Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bumped ubuntu builds to 24.04 #3288

Merged
merged 15 commits into from
Aug 5, 2024

Conversation

mahrud
Copy link
Member

@mahrud mahrud commented Jun 6, 2024

No description provided.

@mahrud mahrud marked this pull request as ready for review June 6, 2024 18:42
@mahrud mahrud changed the title Added arm64 actions build Added macos build on arm64, bumped ubuntu to 24.04 Jun 6, 2024
@mahrud mahrud force-pushed the quickfix/actions branch 2 times, most recently from 3fc60e2 to b80941d Compare June 6, 2024 22:01
@mahrud
Copy link
Member Author

mahrud commented Jun 6, 2024

Not sure what to do about the /bin/sh: install-info: command not found error in macOS 13. @mikestillman does macOS not include install-info anymore?

@d-torrance
Copy link
Member

I think the following should fix the autotools-ubuntu build:

--- a/.github/workflows/test_build.yml
+++ b/.github/workflows/test_build.yml
@@ -198,7 +198,8 @@ jobs:
       - name: Validate HTML documentation
         if: matrix.build-system == 'autotools' && runner.os == 'Linux'
         run: |
-          python3 -m pip install html5validator
+          sudo apt install pipx
+          pipx install html5validator
           if test ${{ github.event_name }} = pull_request
           then
             git fetch --depth=1 origin development

@mahrud
Copy link
Member Author

mahrud commented Jun 7, 2024

@mikestillman if you have an M1 machine, could you investigate why this is happening?

 -- capturing example results for "Lex"                                     /bin/sh: line 1: 18364 Trace/BPT trap: 5       /Users/runner/work/M2/M2/M2/BUILD/build/usr-dist/arm64-Darwin-macOS-14.5/bin/M2 -q --stop --silent --no-preload --no-threads -e errorDepth=3 -e debugLevel=0 -e "installPackage(\"Macaulay2Doc\", Verbose => false, RerunExamples => false, CheckDocumentation => true, IgnoreExampleErrors => false, RemakeAllDocumentation => false, InstallPrefix => \"/Users/runner/work/M2/M2/M2/BUILD/build/usr-dist/\", UserMode => false, SeparateExec => true, DebuggingMode => true)" -e "\"info-\"|\"Macaulay2Doc\" << Macaulay2Doc#\"test number\" << close" -e "exit 0"

Googling /bin/sh: line 1: 18364 Trace/BPT trap: 5 leads to iOS debugging threads ...

@mahrud
Copy link
Member Author

mahrud commented Jun 18, 2024

Not sure what to do about the /bin/sh: install-info: command not found error in macOS 13. @mikestillman does macOS not include install-info anymore?

Without install-info on macOS Ventura, how have you been using the autotools build to install M2 on macOS until now?!

@mahrud mahrud force-pushed the quickfix/actions branch 2 times, most recently from cc37c56 to db9463d Compare June 18, 2024 17:23
@mahrud
Copy link
Member Author

mahrud commented Jun 18, 2024

@mikestillman could you try running the "Lex" example on arm64 with macos-14? There's no error file saved other than:

 -- capturing example results for "Lex"             /bin/sh: line 1: 18412 Trace/BPT trap: 5

@mahrud
Copy link
Member Author

mahrud commented Jun 19, 2024

On CMake (macos-14 arm64), we're getting a core dump on this line:

i5 : C = QQ[a..d,MonomialOrder=>Lex,MonomialSize=>8];

i6 : try a^(2^15-1) else "failed"

(GRevLex is also causing the same crash)

Meanwhile, autotools (macos-13 x86_64) is failing on several ForeignFunctions examples:

i1 : mpfr = openSharedLibrary "mpfr"
stdio:1:8:(3): error: dlopen(libmpfr.dylib, 0x0001): tried: '/Users/runner/work/M2/M2/M2/BUILD/build/usr-dist/x86_64-Darwin-macOS-13.6.7/bin/../lib/Macaulay2/lib/libmpfr.dylib' (no such file), '/libmpfr.dylib' (no such file), 'libmpfr.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibmpfr.dylib' (no such file), '/Users/runner/work/M2/M2/M2/BUILD/build/usr-dist/x86_64-Darwin-macOS-13.6.7/lib/Macaulay2/lib/libmpfr.dylib' (no such file), '/Users/runner/work/M2/M2/M2/BUILD/build/usr-dist/x86_64-Darwin-macOS-13.6.7/lib/libmpfr.dylib' (no such file), '/usr/local/opt/flint/lib/libmpfr.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/flint/lib/libmpfr.dylib' (no such file), '/usr/local/opt/gmp/lib/libmpfr.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/gmp/lib/libmpfr.dylib' (no such file), '/Users/runner/work/M2/M2/M2/BUILD/build/usr-dist/x86_64-Darwin-macOS-13.6.7/lib/Macaulay2/lib/libmpfr.dylib' (no such file), '/Users/runner/work/M2/M2/M2/BUILD/build/usr-dist/x86_64-Darwin-macOS-13.6.7/lib/libmpfr.dylib' (no such file), '/usr/local/opt/flint/lib/libmpfr.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/flint/lib/libmpfr.dylib' (no such file), '/usr/local/opt/gmp/lib/libmpfr.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/gmp/lib/libmpfr.dylib' (no such file), '/usr/lib/libmpfr.dylib' (no such file, not in dyld cache), 'libmpfr.dylib' (no such file)

@d-torrance
Copy link
Member

Why isn't dlopen looking in /usr/local/lib??? That's where libmpfr.dylib should be.

@d-torrance
Copy link
Member

I ran a little experiment, and in macos-12, dlopen looks in /usr/local/lib, but in macos-13, it doesn't. If we add it to DYLD_LIBRARY_PATH, it does. I'll push a commit adding it, which should hopefully fix that issue.

@d-torrance
Copy link
Member

Oops, that messed up some brew link stuff. I'll remove the commit.

@d-torrance
Copy link
Member

Just pushed a commit with a slightly different approach, adding /usr/local/lib to DYLD_LIBRARY_PATH in the M2 wrapper script. This should have the advantage of working for everyone building on Ventura and not just GitHub.

@d-torrance
Copy link
Member

Wait, there are still some brew link issues... I can't imagine that last commit would have done anything to affect that?

@mahrud
Copy link
Member Author

mahrud commented Jun 19, 2024

Yeah, sorry that's not you. I thought adding --overwrite fixed it, but somehow it came back. I'll look into it.

Also, I'm against hardcoding paths in the wrapper (which is kinda useless these days anyway since we don't distribute other libraries). We could use the value of HOMEBREW_PATH which is /usr/local on intel macs and /opt/homebrew on arm64 macs, but only macos-14 actions use arm64, so I'm surprised that it could find libmpfr on macos-12 but not 13, given that both use intel cpu.

@d-torrance
Copy link
Member

Fair point -- the wrapper that doesn't really do anything on most installations has always bugged me too. I'll go back to the previous solution.

@mahrud
Copy link
Member Author

mahrud commented Jun 19, 2024

To be clear, I'm against hardcoding paths anywhere :) I'm wondering why the section of ForeignFunctions that deals with macOS didn't add the brew path correctly. Can you do an experiment to see if macos-13 has a different value for any of the parameters?

@d-torrance
Copy link
Member

The macOS hack in ForeignFunctions currently only fires on arm machines, since dlopen never checked in /opt/homebrew. I can remove that condition and just have it check in macOS regardless of architecture.

@mahrud
Copy link
Member Author

mahrud commented Jun 19, 2024

There was definitely a build were the linking issues didn't happen: https://github.com/Macaulay2/M2/actions/runs/9569457804/job/26382031099
And although I think that's the commit that added --overwrite, it seems that python wasn't even installed (because macs have it) ... must be a global bug.

@mahrud
Copy link
Member Author

mahrud commented Jun 27, 2024

@mikestillman have you had a chance to build M2 with clang on macos-14? I noticed the linked libraries of the binary built on github are strange:

2024-06-25T06:47:59.3322510Z -- Linked libraries:
2024-06-25T06:47:59.3323230Z /Users/runner/work/M2/M2/M2/BUILD/build/usr-dist/arm64-Darwin-macOS-14.5/bin/M2-binary:
2024-06-25T06:47:59.3324130Z 	/opt/homebrew/opt/libomp/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
2024-06-25T06:47:59.3325160Z 	/opt/homebrew/opt/boost/lib/libboost_stacktrace_addr2line-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
2024-06-25T06:47:59.3326260Z 	/opt/homebrew/opt/boost/lib/libboost_regex-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
2024-06-25T06:47:59.3327110Z 	/opt/homebrew/opt/libffi/lib/libffi.8.dylib (compatibility version 10.0.0, current version 10.4.0)
2024-06-25T06:47:59.3328030Z 	/opt/homebrew/opt/givaro/lib/libgivaro.9.dylib (compatibility version 12.0.0, current version 12.0.0)
2024-06-25T06:47:59.3329040Z 	/opt/homebrew/opt/gmp/lib/libgmpxx.4.dylib (compatibility version 12.0.0, current version 12.0.0)
2024-06-25T06:47:59.3329910Z 	/opt/homebrew/opt/gmp/lib/libgmp.10.dylib (compatibility version 16.0.0, current version 16.0.0)
2024-06-25T06:47:59.3331020Z 	/opt/homebrew/opt/mpsolve/lib/libmps.3.dylib (compatibility version 4.0.0, current version 4.1.0)
2024-06-25T06:47:59.3331950Z 	/opt/homebrew/opt/factory/lib/libfactory-4.3.2.dylib (compatibility version 0.0.0, current version 0.0.0)
2024-06-25T06:47:59.3332840Z 	/opt/homebrew/opt/flint/lib/libflint.19.0.dylib (compatibility version 19.0.0, current version 19.0.0)
2024-06-25T06:47:59.3333840Z 	/opt/homebrew/opt/ntl/lib/libntl.44.dylib (compatibility version 45.0.0, current version 45.1.0)
2024-06-25T06:47:59.3334650Z 	/opt/homebrew/opt/mpfi/lib/libmpfi.0.dylib (compatibility version 1.0.0, current version 1.0.0)
2024-06-25T06:47:59.3335470Z 	/opt/homebrew/opt/mpfr/lib/libmpfr.6.dylib (compatibility version 9.0.0, current version 9.1.0)
2024-06-25T06:47:59.3336430Z 	/opt/homebrew/opt/bdw-gc/lib/libgc.1.dylib (compatibility version 7.0.0, current version 7.3.0)
2024-06-25T06:47:59.3337330Z 	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
2024-06-25T06:47:59.3338200Z 	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)
2024-06-25T06:47:59.3339000Z 	/usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.9.0)
2024-06-25T06:47:59.3339910Z 	/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.12/Python (compatibility version 3.12.0, current version 3.12.0)
2024-06-25T06:47:59.3340860Z 	/opt/homebrew/opt/readline/lib/libreadline.8.dylib (compatibility version 8.2.0, current version 8.2.0)
2024-06-25T06:47:59.3341870Z 	/opt/homebrew/opt/readline/lib/libhistory.8.dylib (compatibility version 8.2.0, current version 8.2.0)
2024-06-25T06:47:59.3343030Z 	/opt/homebrew/opt/gdbm/lib/libgdbm.6.dylib (compatibility version 7.0.0, current version 7.0.0)
2024-06-25T06:47:59.3343860Z 	/opt/homebrew/opt/tbb/lib/libtbb.12.dylib (compatibility version 12.0.0, current version 12.12.0)
2024-06-25T06:47:59.3344680Z 	/opt/homebrew/opt/llvm/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
2024-06-25T06:47:59.3345340Z 	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1700.255.0)
2024-06-25T06:47:59.3346060Z 	/opt/homebrew/opt/llvm/lib/libunwind.1.dylib (compatibility version 1.0.0, current version 1.0.0)

For instance, there are two different libomp.dylib.

@mahrud
Copy link
Member Author

mahrud commented Jul 30, 2024

I'm generally against removing examples/tests because this is pointing to a regression that should be fixed rather than suppressed, but I don't have very strong feelings about it.

@d-torrance
Copy link
Member

I agree that we should definitely avoid skipping the test if at all possible.

One big difference I see is that we're using factory from homebrew instead of building it. From the logs from this PR:

==> Pouring factory--4.4.0.monterey.bottle.tar.gz
🍺  /usr/local/Cellar/factory/4.4.0: 111 files, 7.3MB

But from a recent successful build:

-- Could NOT find Factory (missing: FACTORY_INCLUDE_DIR FACTORY_LIBRARIES FACTORY_VERSION_OK) (Required is at least version "4.2.0")

...

[6/15] Performing download step (download, verify and extract) for 'build-factory'
-- Downloading...
   dst='/Users/runner/work/M2/M2/M2/BUILD/tarfiles/factory-4.4.0.tar.gz'

@mahrud
Copy link
Member Author

mahrud commented Aug 2, 2024

Still failing!!

@mahrud mahrud linked an issue Aug 2, 2024 that may be closed by this pull request
@mahrud
Copy link
Member Author

mahrud commented Aug 2, 2024

Ahh, I just realized: previously we weren't running tests on macos at all! I'll revert to that for now.

mahrud and others added 12 commits August 2, 2024 19:43
Starting macOS 13 (Ventura), install-info is missing from the system, which causes the autotools build to fail.
It was previously only a problem on ARM machines, since dlopen never
checked in /opt/homebrew/lib.  But it's apparently a problem on Intel
Ventura machines as well, since after to switching to macos-13 for the
GitHub builds, dlopen doesn't check in /usr/local/lib either.
@mahrud
Copy link
Member Author

mahrud commented Aug 2, 2024

I also reduced daily tests to weekly, removed the pre-master stuff, and limited concurrent actions to one per PR (so upon new push the old one is cancelled).

@mahrud
Copy link
Member Author

mahrud commented Aug 3, 2024

This PR is legitimately cursed ...

@mahrud
Copy link
Member Author

mahrud commented Aug 4, 2024

I decided to just comment out that test for now, but I think I'll make a list of all these tests and examples disabled throughout the years so someone can look into them.

@d-torrance d-torrance merged commit b4305bc into Macaulay2:development Aug 5, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cancel previous github builds upon new push
3 participants