Autotools build updates for RHEL 8 #3461

d-torrance · 2024-09-06T21:04:27Z

Otherwise, it may not look in usr-host for dependencies that we've built. For example, the version of gmp in RHEL 8 is too old for flint, so we need to build it and link against that instead.

Here an example of a failed build prior to this patch (https://github.com/d-torrance/M2-workflows/actions/runs/10733556117/job/29767118034), where it was only looking at the RHEL gmp package:

checking for library containing __gmpz_init... -lgmp
configure: error: "`mpn_gcd_11' was not found in the GMP library. It is needed to enable GMP
internals."
checking for library containing __gmpn_gcd_11... no

After this patch (from a local build in an AlmaLinux 8 docker image):

checking for library containing __gmpz_init... -lgmp
checking for library containing __gmpn_gcd_11... none required

This is a followup to #3350 and #3367 in my quest to get the autotools build working with RHEL.

Skipping the GitHub builds since we use the Ubuntu/Homebrew flint packages and this patch won't be tested.

d-torrance · 2024-09-10T03:59:05Z

I ended up simplifying this a bit more. The underlying issue was that flint didn't used to respect LDFLAGS. Instead, it used a variable called LIB_DIRS for a similar purpose, and we needed to change the call to configure pretty significantly to get it to work.

However, now that it does respect LDFLAGS, we can mostly use the defaults from Makefile.library with just a couple tweaks.

d-torrance · 2024-09-26T14:19:39Z

I'm broadening the scope of this PR -- the goal is to get the autotools build to work in RHEL 8.

M2/libraries/flint/Makefile.in

d-torrance · 2024-09-26T19:50:28Z

This now compiles on a Rocky Linux 8 docker image! (With the --disable-tbb configure option.)

It still fails at link time due to FFLAS-FFPACK/Givaro BLAS shenanigans.

TODO:

Add the ability to build TBB
Figure out the BLAS stuff

d-torrance · 2024-09-27T01:53:01Z

Got it working!

Macaulay2, version version-1.14.0.1-9300-gb87f02b5f4 (rhel-8)
with packages: ConwayPolynomials, Elimination, IntegralClosure, InverseSystems, Isomorphism, LLLBases, MinimalPrimes, OnlineLookup, Polyhedra,
               PrimaryDecomposition, ReesAlgebra, Saturation, TangentCone, Truncations, Varieties

i1 : version#"issue"

o1 = rocky-8.9

I think this closes #475. Rather than rolling our own BLAS check, we use AX_BLAS from the autoconf archive and also pass the result to fflas-ffpack.

Still a draft for now -- I'd like to do some more experimenting.

d-torrance · 2024-09-27T21:35:40Z

Assuming the builds all pass, I think this PR is ready to go. I successfully compiled and linked M2 in Rocky Linux 8 using both the openblas-devel package from EPEL and building LAPACK ourselves. I haven't checked building the examples or running the tests yet.

An overview of the changes:

Replace the autotools linear algebra library checks with AX_BLAS and AX_LAPACK from the Autoconf Archive. This simplifies things significantly, and in particular, allows linking with OpenBLAS out of the box, closing #475. We also switch the OpenBLAS for the GitHub Ubuntu builds.
Add a "blas implementation" key/value pair to the version hash table so we can see which BLAS we're linking against.
Simplify the checks for givaro and fflas-ffpack.
Remove the deprecated givaro source files from the e directory.
Add a patch for fflas-ffpack so that it properly detects LAPACK when we build it ourselves. Some of these changes were already submitted upstream, but I also submitted a PR with additional changes.
Improve the TBB check so that it detects when we're using a TBB older than 2020 (the earliest one we support). We also build TBB if needed.
Fix some some bugs with the build when --without-tbb is passed.
Add a version check for MPFR and build it if the detected version is less than 4 (which we need for several functions).
Build eigen if it's not found.
Simplify the flint build, and particular, allow it to link against the gmp we just built.
Build msolve as a static library since we build flint as a static library.

mahrud · 2024-09-27T21:52:11Z

Should the givaro changes be in the same PR? I think @mikestillman wanted to remove them for some time, but I don't know what was the status.

M2/Macaulay2/d/version.dd

M2/Macaulay2/e/m2tbb.hpp

d-torrance · 2024-09-27T22:24:47Z

Should the givaro changes be in the same PR?

Maybe it's outside the scope of the PR, but this was my thought process:

I wanted to simplify the fflas-ffpack and givaro checks since they were muddying up the build flags with a bunch of redundancies with their calls to pkg-config. In particular, I wanted to make sure that any BLAS/LAPACK flags were coming from AX_BLAS and AX_LAPACK and nothing else.
Part of this was investigating whether we needed to still do the isunit/isUnit check. We haven't compiled the file that called this since 1b3c31b, so I figured I'd remove it.

mahrud · 2024-09-27T23:37:22Z

Part of this was investigating whether we needed to still do the isunit/isUnit check.

Maybe remove this from the cmake build as well then.

mahrud · 2024-09-29T21:25:39Z

I think I'm happy with this, but of course I haven't tested it.
@trevorkarn could you test this PR? (I'm requesting a review on github, but you don't need to formally review the changes as long as your build succeeds)

@d-torrance does this close #2507 as well?

mahrud · 2024-09-29T21:27:55Z

Oh also, @d-torrance could you add your RHEL Dockerfile under BUILD/docker?

trevorkarn · 2024-10-01T03:13:14Z

I think I'm happy with this, but of course I haven't tested it. @trevorkarn could you test this PR? (I'm requesting a review on github, but you don't need to formally review the changes as long as your build succeeds)

@d-torrance does this close #2507 as well?

Sad to say the build did not succeed.

d-torrance · 2024-10-01T11:16:54Z

does this close #2507 as well?

No, -lgfortran is in FCLIBS, which we still link against. I think we need it if we link against any Fortran implementation of BLAS, not just one we build, right? And I'm not sure how to detect that, especially if the user can hot-swap with flexiblas or Debian's alternatives system.

d-torrance · 2024-10-02T16:14:32Z

There were several errors building the examples for ForeignFunctions. These were assuming that some of the libraries that M2 links against were linked dynamically and not statically. I pushed a few commits changing these to use things from the C standard library instead. (One of these will close #3374.)

d-torrance · 2024-10-02T18:27:24Z

After the most recent commits, the build finished successfully for me!

gfortan isn't available on the macOS GitHub runners, but gfortran-XX is for various versions XX. Using the latest version (14) for now

Otherwise it won't be able to link the test program it uses to detect BLAS.

[ci skip]

It's not directly related to runProgram. Add a bit more detail about how "get" might be used (by adding a leading "!"), and also link to the docs for the various "open*" functions that also take a string with a leading "!" to interact with a program. The primary reason we're dropping this is example is that it called gfan, which may not be on the PATH, especially if we've just built it.

It will only work if we've dynamically linked against mpfr, which may not be the case.

It will only work if mpfr is available as a dynamic library and we've linked against it, which may not be the case. Replace it with "cos" from the C standard math library

It's in the C standard library, so we know that it will be available. Closes: Macaulay2#3374

M2/autogen.sh

Older versions of autoconf don't install them automatically [ci skip]

Also use a wildcard for the Makefile's in the subdirectories of libraries for simplicity and ease of maintenance. [ci skip]

d-torrance · 2024-10-19T12:29:42Z

I'm going to go ahead and merge this -- building against this branch works for me using docker images for both Rocky Linux and AlmaLinux 8:

(They both fail during make check because the time utility is missing, but that's an easy fix...)

The builds on RHEL 9 are still failing due to #2398.

mahrud · 2024-10-19T13:06:18Z

building against this branch works for me using docker images for both Rocky Linux and AlmaLinux 8

At some point I wanted to create GitHub Actions workflows that build an image based on the contents of the BUILD/docker directory, and setup a workflow_dispatch (like the package review workflow) that could be triggered in a browser. This would make testing on different distributions way easier.

mikestillman · 2024-10-19T13:09:32Z

We should also create docker images (for using M2, and for compiling M2) for each release. We should put the docker images in an "official" location, e.g. under macaulay2 on docker hub, although I'm not sure docker hub is as good as it used to be...

mahrud · 2024-10-19T13:21:15Z

We have one already: https://github.com/orgs/Macaulay2/packages/container/package/testbot
Based on BUILD/testbot/Dockerfile.

trevorkarn · 2024-10-20T20:30:03Z

@d-torrance So sorry for my slow reply! I was able to build a working binary off of commit 4e49717e08fe3166b1115a53778089fb9b40a702 but I never successfully built the docs. Having the binary was enough for the work I wanted to do, so I didn't try, but let me know if you need me to.

d-torrance · 2024-10-20T21:06:44Z

@trevorkarn -- No worries!

I'm pretty sure the docs should build now. This branch has now been merged into the development branch. If you have some time to build the development branch to confirm that the docs build, then that would be much appreciated!

d-torrance requested a review from DanGrayson September 6, 2024 21:05

d-torrance force-pushed the flint-autotools branch from 0c937fc to a631688 Compare September 10, 2024 03:54

d-torrance force-pushed the flint-autotools branch from a631688 to 8477994 Compare September 26, 2024 14:14

d-torrance changed the title ~~Pass LDFLAGS to flint's configure script (autotools)~~ Autotools build updates for RHEL 8 Sep 26, 2024

d-torrance marked this pull request as draft September 26, 2024 14:15

d-torrance mentioned this pull request Sep 26, 2024

Build eigen (autotools) #3487

Closed

mahrud reviewed Sep 26, 2024

View reviewed changes

M2/libraries/flint/Makefile.in Outdated Show resolved Hide resolved

d-torrance force-pushed the flint-autotools branch 3 times, most recently from c45295d to 51b65ec Compare September 27, 2024 21:09

d-torrance marked this pull request as ready for review September 27, 2024 21:35

mahrud reviewed Sep 27, 2024

View reviewed changes

M2/Macaulay2/d/version.dd Outdated Show resolved Hide resolved

mahrud reviewed Sep 27, 2024

View reviewed changes

M2/Macaulay2/e/m2tbb.hpp Show resolved Hide resolved

d-torrance force-pushed the flint-autotools branch from 51b65ec to 8ab49ed Compare September 27, 2024 23:29

mahrud linked an issue Sep 29, 2024 that may be closed by this pull request

allow the use of openblas #475

Closed

mahrud requested a review from trevorkarn September 29, 2024 21:25

d-torrance added 9 commits October 17, 2024 10:32

Install OpenBLAS on Ubuntu GitHub builds

2455903

Set F77 environment variable for GitHub builds

b205705

gfortan isn't available on the macOS GitHub runners, but gfortran-XX is for various versions XX. Using the latest version (14) for now

Add patch to improve fflas-ffpack BLAS detection

e193e30

Pass fortran libs to fflas-ffpack configure script

271e72a

Otherwise it won't be able to link the test program it uses to detect BLAS.

Fetch submodules before trying to copy their contents (autotools)

1288c7f

Add another fflas-ffpack patch; need "static inline"

95298a1

Remove "--without-blas" from flint configure options (autotools)

74f959f

Link against BLAS/LAPACK during engine unit tests

e97cff0

Remove unnecessary givaro isunit v. isUnit check (cmake)

c9af30b

d-torrance force-pushed the flint-autotools branch from 654042e to f3bd8e8 Compare October 17, 2024 14:33

d-torrance added 5 commits October 17, 2024 10:35

Add Dockerfile for Red Hat Enterprise Linux

207bcc7

[ci skip]

Drop mpfr_add example from ForeignFunctions::mpfrT docs

04baaeb

It will only work if we've dynamically linked against mpfr, which may not be the case.

Drop mfpr example from ForeignFunction::foreignFunction docs

46bb7d3

It will only work if mpfr is available as a dynamic library and we've linked against it, which may not be the case. Replace it with "cos" from the C standard math library

Use errno for ForeignFunctions::foreignSymbol example

34c18a5

It's in the C standard library, so we know that it will be available. Closes: Macaulay2#3374

d-torrance force-pushed the flint-autotools branch from f3bd8e8 to 34c18a5 Compare October 17, 2024 14:35

mahrud reviewed Oct 18, 2024

View reviewed changes

M2/autogen.sh Outdated Show resolved Hide resolved

d-torrance added 4 commits October 18, 2024 17:28

Copy config.guess, config.sub, and install-sh if necessary

c5134c2

Older versions of autoconf don't install them automatically [ci skip]

Add some missing Makefile's to .gitignore

77f5796

Also use a wildcard for the Makefile's in the subdirectories of libraries for simplicity and ease of maintenance. [ci skip]

Allow running autogen.sh from build directory

98bf8b0

Update scripts to run autogen.sh from build directory

930f306

d-torrance force-pushed the flint-autotools branch from 8eee18c to 930f306 Compare October 18, 2024 21:45

d-torrance merged commit 64265a4 into Macaulay2:development Oct 19, 2024
5 checks passed

d-torrance mentioned this pull request Oct 19, 2024

allow the use of openblas #475

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Autotools build updates for RHEL 8 #3461

Autotools build updates for RHEL 8 #3461

d-torrance commented Sep 6, 2024

d-torrance commented Sep 10, 2024

d-torrance commented Sep 26, 2024

d-torrance commented Sep 26, 2024 •

edited

Loading

d-torrance commented Sep 27, 2024 •

edited

Loading

d-torrance commented Sep 27, 2024

mahrud commented Sep 27, 2024

d-torrance commented Sep 27, 2024

mahrud commented Sep 27, 2024

mahrud commented Sep 29, 2024

mahrud commented Sep 29, 2024

trevorkarn commented Oct 1, 2024

d-torrance commented Oct 1, 2024

d-torrance commented Oct 2, 2024

d-torrance commented Oct 2, 2024

d-torrance commented Oct 19, 2024

mahrud commented Oct 19, 2024

mikestillman commented Oct 19, 2024

mahrud commented Oct 19, 2024

trevorkarn commented Oct 20, 2024

d-torrance commented Oct 20, 2024

Autotools build updates for RHEL 8 #3461

Autotools build updates for RHEL 8 #3461

Conversation

d-torrance commented Sep 6, 2024

d-torrance commented Sep 10, 2024

d-torrance commented Sep 26, 2024

d-torrance commented Sep 26, 2024 • edited Loading

d-torrance commented Sep 27, 2024 • edited Loading

d-torrance commented Sep 27, 2024

mahrud commented Sep 27, 2024

d-torrance commented Sep 27, 2024

mahrud commented Sep 27, 2024

mahrud commented Sep 29, 2024

mahrud commented Sep 29, 2024

trevorkarn commented Oct 1, 2024

d-torrance commented Oct 1, 2024

d-torrance commented Oct 2, 2024

d-torrance commented Oct 2, 2024

d-torrance commented Oct 19, 2024

mahrud commented Oct 19, 2024

mikestillman commented Oct 19, 2024

mahrud commented Oct 19, 2024

trevorkarn commented Oct 20, 2024

d-torrance commented Oct 20, 2024

d-torrance commented Sep 26, 2024 •

edited

Loading

d-torrance commented Sep 27, 2024 •

edited

Loading