Windows - Add support for ucrt & Windows-2022 image #224

MSP-Greg · 2021-10-08T20:08:50Z

See commits. On Windows-2022 image, code adds the MSYS2 packages from 'base-devel' group needed for Ruby builds.

Also adds ENV variables by running ridk enable, testing for the file's existence first. Uses code similar to the code for mswin's vcvars script. Parses the output of the set command, so it's not dependent on ridk's output.

Windows 2019 Image - No changes when using with a mingw Ruby. When using a ucrt Ruby, a pre-built ucrt 7z file is installed from https://github.com/MSP-Greg/setup-msys2-gcc/releases/tag/msys2-gcc-pkgs. The 7z files are generated three times a day in that repo.

Windows 2022 Image - Since the image has no MSYS2 build tools installed, both an MSYS2 package and a gcc package (mingw or ucrt) are installed from the MSP-Greg/setup-msys2-gcc repo.

MSP-Greg · 2021-11-15T01:29:32Z

I created a fork of ruby-loco, and built both mingw & ucrt on WIndows 2022 using this PR branch for setup-ruby. Both compiled and passed all tests. This PR runs jobs for mingw & ucrt builds on both Windows 2019 & 2022.

MSP-Greg · 2021-11-24T16:21:44Z

@eregon

Good day.

All the below is Windows specific...

Sorry for all the CI, a few typos, etc. Since windows-2022 does not have MSYS2 build tools installed (there's pro's and con's for that decision), we need to install the tools here, as any CI that installs an extension gem will fail otherwise. With the previous images, we just needed to install the tools for the old Rubies (2.3 and earlier), now that's required for all Rubies.

This is working in ruby/ruby, and I've tested it with nokogiri and puma. nokogiri needs the xml related packages installed, so it's using a branch in setup-ruby-pkgs that pairs with this PR's branch.

I'll check if it works with ruby/openssl, which has an mswin build, and if that's ok, it's ready to go...

bundler.js

eregon · 2022-01-03T12:49:36Z

windows.js

+
+// ssl files cause issues with non RI2 Rubies (<2.4) and ruby/ruby's CI from
+// build folder due to dll resolution
+function renameSystem32Dlls() {


This seems quite risky, is there any other way to solve the issue?
Is it only for Ruby <= 2.3? It seems done for all cases here.
It seems to work fine before for <= 2.3, no?

This is quite involved as to why it's needed.

To be clear, these are not MSFT files. One of the oldest and most popular Windows OpenSSL implementations (built with Visual C) installs the OpenSSL dll's in C:\Windows\System32. Normally, that folder is considered 'off-limits', and should be used only by MSFT.

I've had this in setup-ruby-pkgs for a long time, and it's never caused an issue. It causes all sorts of WTF issues if it's not done...

My concern here is someone might waste a lot of time if for some reason some part of their workflow would use those files and using setup-ruby has the very unexpected side effect to delete or rename files under C:\Windows\System32 (which are not provided by Microsoft but still).
I remember some people were already confused due to the more extensive Path changes on Windows by this action, but this goes another level. If we keep it we should have a clear logged message mentioning what it and why it does it.

I think it's better to not do this. If those files are problematic then filing an issue at https://github.com/actions/virtual-environments seems the best way forward.

@larskanis What do you think? Having a 3rd opinion might be helpful here.

Below is info from a repo I've got that spits out a bunch of data on Actions Images, taken from the windows-2022 image job. All of the openssl.exe files have the dll's in their own directory. The C:\Windows\System32 files are added by the same installer that creates the files at C:\Program Files\OpenSSL.

——————————————————————————————————————————————————————————————————————— libcrypto-1_1-x64.dll C:\mysql\bin\libcrypto-1_1-x64.dll C:\Windows\System32\libcrypto-1_1-x64.dll C:\Program Files\OpenSSL\bin\libcrypto-1_1-x64.dll C:\Program Files\Git\mingw64\bin\libcrypto-1_1-x64.dll c:\tools\php\libcrypto-1_1-x64.dll ——————————————————————————————————————————————————————————————————————— libssl-1_1-x64.dll C:\mysql\bin\libssl-1_1-x64.dll C:\Windows\System32\libssl-1_1-x64.dll C:\Program Files\OpenSSL\bin\libssl-1_1-x64.dll C:\Program Files\Git\mingw64\bin\libssl-1_1-x64.dll c:\tools\php\libssl-1_1-x64.dll ——————————————————————————————————————————————————————————————————————— openssl info OpenSSL 1.1.1 11 Sep 2018 C:\Program Files\OpenSSL\bin\openssl.exe OpenSSL 1.1.1l 24 Aug 2021 C:\Program Files\Git\mingw64\bin\openssl.exe OpenSSL 1.1.1l 24 Aug 2021 C:\Program Files\Git\usr\bin\openssl.exe

My concern here is someone might waste a lot of time

I've worked with quite a few extension gems re Windows, Ruby master/head, and Actions. Never come across a problem. It is a confusing problem if Rubies that use ENV['Path'] for their dll resolution and load them because they're in C:\Windows\System32...

Could you do this only for RI1, so Ruby <= 2.3, and log a message in that case?
I think it's more reasonable to use workarounds like this for old versions, but for recent versions we should avoid such changes.

Windows OpenSSL dll's have different names, depending on the version, the break happens between 1.0.2 and 1.1.0. Updated for Ruby 2.4 and earlier, as some extconf files that work with OpenSSL look for the 1.1.0 names first. Windows Ruby 2.4 uses 1.0.2. See comment

.github/workflows/test.yml

windows.js

MSP-Greg · 2022-01-03T14:50:22Z

Re the windows-2022 JRuby failures:

  C:\Windows\system32\tar.exe -xz -C /d/ -f /d/a/_temp/a1828e52-1472-430c-9f5e-f1202ac6f2b7
  tar.exe: Error opening archive: Failed to open '/d/a/_temp/a1828e52-1472-430c-9f5e-f1202ac6f2b7'

Previous images had the MSYS2 tar installed, windows-2022 does not, so the MSFT tar.exe is used, but it needs a Windows path. Looking at it now...

MSP-Greg · 2022-01-03T15:42:47Z

@eregon re the Windows JRuby extraction issue, I've got that fixed. I just noticed the 'Windows JRuby' step in CI (gem install sassc). It fails on windows-2022 because there are no build tools.

So, should they be installed? If so, I need to see whether they should be mingw or ucrt?

eregon · 2022-01-03T16:51:06Z

So, should they be installed? If so, I need to see whether they should be mingw or ucrt?

I guess it doesn't really matter, so the easier the better.
For sassc, a C++ compiler is needed to create a .dll and that's then loaded via FFI.

MSP-Greg · 2022-01-03T17:58:23Z

Locally, I built the sassc gem with jruby-head, ruby-mingw, & ruby-ucrt.

Using objdump to see what dll's they load, the following is the data. JRuby may be compiled with MSFT Visual Studio. Not sure about enabling that, but if one were to choose between MinGW & UCRT, the MinGW build's dll's line up with JRuby much better. I'll work on it.

———————————————————————————————————————————————— ruby-mingw
ADVAPI32.dll
bcrypt.dll
imagehlp.dll
IPHLPAPI.DLL
KERNEL32.dll
libgmp-10.dll
msvcrt.dll
SHELL32.dll
USER32.dll
DLL Name: WS2_32.dll

———————————————————————————————————————————————— ruby-ucrt
ADVAPI32.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-environment-l1-1-0.dll
api-ms-win-crt-filesystem-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-private-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
api-ms-win-crt-utility-l1-1-0.dll
bcrypt.dll
imagehlp.dll
IPHLPAPI.DLL
KERNEL32.dll
libgmp-10.dll
SHELL32.dll
USER32.dll
WS2_32.dll


———————————————————————————————————————————————— MinGW libsass.so
ADVAPI32.dll
KERNEL32.dll
libgcc_s_seh-1.dll
libstdc++-6.dll
msvcrt.dll

———————————————————————————————————————————————— UCRT libsass.so
ADVAPI32.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-environment-l1-1-0.dll
api-ms-win-crt-filesystem-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-private-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
KERNEL32.dll
libgcc_s_seh-1.dll
libstdc++-6.dll


———————————————————————————————————————————————— Windows JRuby
ADVAPI32.DLL
KERNEL32.dll
msvcrt.dll
USER32.dll
WS2_32.dll

MSP-Greg · 2022-01-03T21:12:12Z

Fixed the issues with JRuby & windows-2022. Appears there are problems with Actions macOS servers. Anyway, I think this is complete, with the issue of RubyGems possibly remaining.

BTW, Happy New Year!

MSP-Greg · 2022-01-04T16:44:09Z

@eregon & @larskanis

This morning I decided to look at the Bundler/RubyGems issue, as this PR currently updates RubyGems instead of updating Bundler for ((bundlerVersionConstraint === '~> 2') && (common.floatVersion(rubyVersion) >= 2.3)).

So, I ran it my fork of this repo, results. The only failing jobs are Windows 3.1.0.

There are a few repos running from the PR branch of my fork, Puma being one. It has been passing using it.

I reverted the RubyGems installation, results. As above, the Windows 3.1.0 jobs failed, but also Ruby 2.5 jobs on every OS failed. The 2.5 failures maybe due to Psych, don't have time to look this morning.

Another 'issue' in both CI runs is that all jobs using Ruby 2.3 thru 2.5 have a warning shown, similar to the below. In the CI in my fork of of this repo, see 'bundle install' of the 'Run /./' step, in the Puma fork, see 'bundle install' of the 'load ruby' step:

Your RubyGems version (2.6.14.4)) has a bug that prevents `required_ruby_version` from working for Bundler

So, not sure what to do about this. Lars, if you have time, maybe see if the issue with the 3.1 bin can be fixed? I'll try to look later... Regardless, we've got warnings and failures in all OS's for Ruby 2.3 thru 2.5 when upgrading to Bundler 2.3.4...

MSP-Greg · 2022-01-04T18:23:07Z

I updated RubyGems/Bundler handling. For Ruby 2.3 thru 2.5 (all EOL), it does gem update --system. For Windows 3.1.0, it installs Bundler with --force. So, it works, no warnings...

eregon · 2022-01-13T17:50:24Z

@MSP-Greg Only 2 remaining then I think it should be ready to be merged:
#224 (comment)
#224 (comment)

eregon

Thank you for this great work!

eregon · 2022-01-13T19:25:16Z

Released: https://github.com/ruby/setup-ruby/releases/tag/v1.92.0

MSP-Greg · 2022-01-13T19:32:23Z

I suspect there are things about TruffleRuby that even without a few nights sleep you would be aware of. Given how long I've used Ruby with Windows, same thing. Hence, sorry about not commenting several things.

One change that's contained in MSP-Greg/setup-msys2-gcc is that I included all the packages needed to build Ruby, which is also true for all the extension stdlib/gems in Ruby repos. I was tired of people using MSP-Greg/setup-ruby-pkgs to install packages that are required by Ruby. Also, I thought matching Ubuntu was a good idea.

But, when removing MSP-Greg/setup-ruby-pkgs from CI here and there, there was a fair bit of code that needed to move to this repo. And that's where I got a little light with the comments... Thanks for reviewing this and keeping the repo up to date. Too many people think it's automatic.

MSP-Greg · 2022-01-13T20:27:59Z

EDIT: I deleted the workflow shown below. Puma uses setup-ruby-pkgs, and there was an odd thing where I pushed a release to it, then immediately used the release in other repos, and the release wasn't updated. Maybe releases take some time to move around GH datacenters, not sure.

Anyway, the weird issue related to Psych (it's in Puma's Gemfile.lock) no longer exists, but the warnings still appear. So, everything is sort of ok. Hence, ignore the below...

———————————————— Ignore below ————————————————

@eregon I just updated Puma's CI in my fork. Due to Psych issues, On all OS's, Ruby 2.5 is failing, and Ruby 2.3 & 2.4 logs warning that I mentioned before in the tests. See https://github.com/MSP-Greg/puma/runs/4808743184.

This was that issue involving:

if ((bundlerVersionConstraint === '~> 2') && (floatVersion >= 2.3) && (floatVersion <= 2.5)) {
  // stop warnings similar to
  // Your RubyGems version (2.6.14.4)) has a bug that prevents `required_ruby_version` from working for Bundler
  await exec.exec(gem, ['update', '--system', '--no-document'])

Not sure what should be done...

Part of the issue is the warnings, and also that new Bundler/RubyGems is compatible with new and old Psych, but old RubyGems is not, so if your bundle/Gemfile update Psych, things explode.

lopopolo · 2022-01-14T02:35:53Z

Hi, is it likely that the MSYS changes mentioned here are interfering with libraries available in the base installed packages on the GHA runner?

It looks like these changes combined with using Ruby 3.1.0 on windows-2019 add a non-default libclang to the path which breaks Rust bindgen.

Example build: https://github.com/artichoke/artichoke/runs/4811419721?check_suite_focus=true

MSP-Greg · 2022-01-14T02:50:25Z

I haven't looked at the build, but Ruby 3.1 is built with the MSYS2 UCRT64 toolset, and Ruby 3.0 (and previous) were built with the MSYS2 MINGW64 toolset.

Not quite sure what you mean by 'non-default libclang'? Nothing has changed in the handling of Path.

lopopolo · 2022-01-14T02:53:10Z

bindgen does a dynamic load of libclang. this is the runtime error:

  thread 'main' panicked at 'Unable to find libclang: "the `libclang` shared library at C:\\msys64\\mingw64\\bin\\libclang.dll could not be opened: LoadLibraryExW failed"', C:\Users\Default\.cargo\registry\src\github.com-1ecc6299db9ec823\bindgen-0.59.2\src/lib.rs:2144:31

MSP-Greg · 2022-01-14T03:08:43Z

When using Ruby 3.1 (or head or ucrt), you'll need to (at a minimum) add the following:

pacman -S mingw-w64-ucrt-x86_64-clang

The windows-latest (windows-2019) image only contains the mingw64 toolset (maybe also mingw32, but we don't use). As mentioned above, Windows Ruby 3.1 uses the ucrt64 toolset, which is not pre-installed on any Actions Windows images.

Note that in the windows-2022 image the mingw64 toolset is not installed, nor are the MSYS2 build tools.

MSP-Greg · 2022-01-14T04:13:32Z

@lopopolo

See artichoke/artichoke#1636

MSP-Greg · 2022-01-14T15:41:16Z

@eregon
Time for a new release? ruby/ruby mingw CI is failing on a few openssl tests due to the dll resolution issue. I'm double checking right now.

eregon · 2022-01-14T16:01:17Z

@MSP-Greg There is already https://github.com/ruby/setup-ruby/releases/tag/v1.92.0 and the two commits after are just cleanups, shouldn't matter. What's the issue?

MSP-Greg · 2022-01-14T16:05:14Z

ruby/ruby mingw CI

  Retrying...
  
    1) Failure:
  OpenSSL::TestConfig#test_s_parse_include [D:/a/ruby/ruby/src/test/openssl/test_config.rb:158]:
  <["default", "sec-a", "sec-b", "sec-main"]> expected but was
  <["default", "sec-main"]>.
  
    2) Error:
  OpenSSL::TestCipher#test_aes_ccm:
  OpenSSL::Cipher::CipherError: 
      D:/a/ruby/ruby/src/test/openssl/test_cipher.rb:374:in `ccm_data_len='
      D:/a/ruby/ruby/src/test/openssl/test_cipher.rb:374:in `block (2 levels) in new_decryptor'
      D:/a/ruby/ruby/src/test/openssl/test_cipher.rb:374:in `each'
      D:/a/ruby/ruby/src/test/openssl/test_cipher.rb:374:in `block in new_decryptor'
      <internal:kernel>:90:in `tap'
      D:/a/ruby/ruby/src/test/openssl/test_cipher.rb:372:in `new_decryptor'
      D:/a/ruby/ruby/src/test/openssl/test_cipher.rb:187:in `test_aes_ccm'

I'm testing right now, I believe the SYSTEM32 dll's need to be renamed for all Rubies.

MSP-Greg · 2022-01-14T16:33:26Z

See #258

MSP-Greg force-pushed the win-ucrt-1 branch 2 times, most recently from 167d4f5 to d1c58dc Compare November 15, 2021 00:35

MSP-Greg marked this pull request as ready for review November 15, 2021 01:29

MSP-Greg force-pushed the win-ucrt-1 branch from 797924f to 2d327b5 Compare November 20, 2021 17:04

MSP-Greg mentioned this pull request Nov 20, 2021

Windows: Add support for MINGW-UCRT based ruby #193

Closed

MSP-Greg force-pushed the win-ucrt-1 branch 2 times, most recently from 293c7b7 to 7079e68 Compare November 21, 2021 20:19

This was referenced Nov 23, 2021

[Actions] use windows-2022 for mingw ruby/ruby#5158

Merged

Actions - use windows-2022, add windows ucrt job sparklemotion/nokogiri#2375

Closed

MSP-Greg force-pushed the win-ucrt-1 branch from 5c4349e to 41bb73e Compare November 24, 2021 19:10

MSP-Greg force-pushed the win-ucrt-1 branch from 41bb73e to 42ff24a Compare December 2, 2021 01:33

MSP-Greg force-pushed the win-ucrt-1 branch from 42ff24a to a6870e8 Compare December 14, 2021 16:26

MSP-Greg force-pushed the win-ucrt-1 branch from a6870e8 to 14de3ca Compare December 26, 2021 19:43

eregon reviewed Jan 3, 2022

View reviewed changes

bundler.js Outdated Show resolved Hide resolved

eregon reviewed Jan 3, 2022

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

eregon reviewed Jan 3, 2022

View reviewed changes

windows.js Outdated Show resolved Hide resolved

eregon reviewed Jan 3, 2022

View reviewed changes

windows.js Outdated Show resolved Hide resolved

eregon mentioned this pull request Jan 3, 2022

Support Ruby 3.1.0 for Windows #253

Merged

MSP-Greg mentioned this pull request Jan 4, 2022

How to bypass RubyGems prompts in CI - bundler's executable "bundle" conflicts with ... rubygems/rubygems#5245

Closed

5 tasks

MSP-Greg force-pushed the win-ucrt-1 branch from 71078e8 to 195822a Compare January 4, 2022 18:14

MSP-Greg added 3 commits January 11, 2022 15:37

windows.js - update addVCVARSEnv to work with all windows images

d12ddae

Only install mingw or ucrt for new Rubies

c5b6e87

Actions - add windows-2022 jobs

87d3145

MSP-Greg force-pushed the win-ucrt-1 branch from 195822a to db14f2d Compare January 11, 2022 22:12

eregon mentioned this pull request Jan 13, 2022

Make the gem a noop on Rubies older than 2.6 ruby/io-wait#9

Merged

MSP-Greg force-pushed the win-ucrt-1 branch from db14f2d to a61e0ce Compare January 13, 2022 17:57

MSP-Greg added 6 commits January 13, 2022 12:08

Windows - Rename 3rd party OpenSSL SYSTEM32 dll's

54611f8

windows.js - suggested changes

5b4cbc3

Windows JRuby - install MinGW tools, typo

85380a5

windows.js - add hasMSYS2PreInstalled constant

d5ff411

package.json & yarn.lock - @vercel/ncc: 0.28.6 -> 0.31.1

415b150

yarn run package

11cc842

MSP-Greg force-pushed the win-ucrt-1 branch from a61e0ce to 11cc842 Compare January 13, 2022 18:09

eregon approved these changes Jan 13, 2022

View reviewed changes

eregon merged commit fb94cfb into ruby:master Jan 13, 2022

MSP-Greg mentioned this pull request Apr 17, 2022

Add Windows MSYS2 ucrt support #194

Closed

eregon mentioned this pull request May 2, 2022

Windows: Use "ridk enable" like variable setup and add UCRT #197

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows - Add support for ucrt & Windows-2022 image #224

Windows - Add support for ucrt & Windows-2022 image #224

MSP-Greg commented Oct 8, 2021 •

edited

Loading

MSP-Greg commented Nov 15, 2021

MSP-Greg commented Nov 24, 2021

eregon Jan 3, 2022

MSP-Greg Jan 3, 2022

eregon Jan 11, 2022

MSP-Greg Jan 11, 2022

eregon Jan 13, 2022 •

edited

Loading

MSP-Greg Jan 13, 2022

MSP-Greg commented Jan 3, 2022

MSP-Greg commented Jan 3, 2022

eregon commented Jan 3, 2022

MSP-Greg commented Jan 3, 2022

MSP-Greg commented Jan 3, 2022

MSP-Greg commented Jan 4, 2022 •

edited

Loading

MSP-Greg commented Jan 4, 2022

eregon commented Jan 13, 2022

eregon left a comment

eregon commented Jan 13, 2022

MSP-Greg commented Jan 13, 2022

MSP-Greg commented Jan 13, 2022 •

edited

Loading

lopopolo commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022

lopopolo commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022

eregon commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022 •

edited

Loading

MSP-Greg commented Jan 14, 2022

Windows - Add support for ucrt & Windows-2022 image #224

Windows - Add support for ucrt & Windows-2022 image #224

Conversation

MSP-Greg commented Oct 8, 2021 • edited Loading

MSP-Greg commented Nov 15, 2021

MSP-Greg commented Nov 24, 2021

eregon Jan 3, 2022

Choose a reason for hiding this comment

MSP-Greg Jan 3, 2022

Choose a reason for hiding this comment

eregon Jan 11, 2022

Choose a reason for hiding this comment

MSP-Greg Jan 11, 2022

Choose a reason for hiding this comment

eregon Jan 13, 2022 • edited Loading

Choose a reason for hiding this comment

MSP-Greg Jan 13, 2022

Choose a reason for hiding this comment

MSP-Greg commented Jan 3, 2022

MSP-Greg commented Jan 3, 2022

eregon commented Jan 3, 2022

MSP-Greg commented Jan 3, 2022

MSP-Greg commented Jan 3, 2022

MSP-Greg commented Jan 4, 2022 • edited Loading

MSP-Greg commented Jan 4, 2022

eregon commented Jan 13, 2022

eregon left a comment

Choose a reason for hiding this comment

eregon commented Jan 13, 2022

MSP-Greg commented Jan 13, 2022

MSP-Greg commented Jan 13, 2022 • edited Loading

lopopolo commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022

lopopolo commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022

eregon commented Jan 14, 2022

MSP-Greg commented Jan 14, 2022 • edited Loading

MSP-Greg commented Jan 14, 2022

MSP-Greg commented Oct 8, 2021 •

edited

Loading

eregon Jan 13, 2022 •

edited

Loading

MSP-Greg commented Jan 4, 2022 •

edited

Loading

MSP-Greg commented Jan 13, 2022 •

edited

Loading

MSP-Greg commented Jan 14, 2022 •

edited

Loading