Add cross-compiling support for the windows toolchain #71584

hjyamauchi · 2024-02-13T18:07:51Z

Extend build.ps1 to enable cross-compile the windows toolchain. For example, building the Windows ARM64 toolchain on a Windows X64 machine.

compnerd · 2024-02-13T18:09:46Z

@swift-ci please build toolchain Windows platform

hjyamauchi · 2024-02-13T18:10:45Z

@swift-ci please build toolchain Windows platform

compnerd · 2024-02-13T18:10:52Z

CC: @shahmishal - as we do not currently have a toolchain that has ARM64 SDKs, we need to use the builds from The Browser Company. If we can roll a new release with the changes, we can adjust that URL.

compnerd · 2024-02-13T18:11:00Z

@swift-ci please build toolchain Windows platform

compnerd · 2024-02-13T18:28:58Z

utils/build.ps1

+      $PinnedSHA256 = "006266d8c2a6a9c70e21b9d161ec35c07bcbb8a452b17e145899d814d07a29e7"
+    }
+    "ARM64" {
+      $PinnedBuild = "https://github.com/thebrowsercompany/swift-build/releases/download/20231117.3/installer-arm64.exe"


@shahmishal - this is the only ARM64 installer currently available that will run on ARM64. Since this script is used by developers and we are using it for ARM64 hosted builds, we do need to encode this.

Removed the hardcoded path to the ARM64 installer

utils/build.ps1

hyp · 2024-02-13T19:04:58Z

utils/build.ps1

        LLDB_PYTHON_EXE_RELATIVE_PATH = "python.exe";
        LLDB_PYTHON_EXT_SUFFIX = ".pyd";
        LLDB_PYTHON_RELATIVE_PATH = "lib/site-packages";
        LLDB_TABLEGEN = (Join-Path -Path $BuildTools -ChildPath "lldb-tblgen.exe");
        LLVM_CONFIG_PATH = (Join-Path -Path $BuildTools -ChildPath "llvm-config.exe");
        LLVM_EXTERNAL_SWIFT_SOURCE_DIR = "$SourceCache\swift";
+        LLVM_INCLUDE_BENCHMARKS = "NO";


why do we need not to include the benchmarks now?

This turned out to be unnecessary. Reverted.

I think that this might be needed still - this avoids unnecessary checks and configuration. I can do the change separately, but the problem was that one of the checks required execution which would not work with cross-compiling.

That may be right. When I rerun without these changes, the build worked successfully.

finagolfin · 2024-02-14T05:28:40Z

utils/build.ps1

+        Python3_EXECUTABLE = "$python";
+        Python3_INCLUDE_DIR = "$BinaryCache\Python$($Arch.CMakeName)-$PythonVersion\tools\include";
+        Python3_LIBRARY = "$BinaryCache\Python$($Arch.CMakeName)-$PythonVersion\tools\libs\python39.lib";
+        Python3_ROOT_DIR = "$BinaryCache\Python$($Arch.CMakeName)-$PythonVersion\tools";
        SWIFT_BUILD_SWIFT_SYNTAX = "YES";


I notice you are cross-compiling the new swift-syntax parser too. Do you have further CMake patches to make that work (similar to my #71552), pass in the extra flags manually, or is this disabled elsewhere?

Or is this just cross-compilation support to build SDKs, not to cross-compile the toolchain itself?

This is cross-compiling the toolchain and we did not find a need to add any additional flags.

This is cross-compiling the toolchain

OK

we did not find a need to add any additional flags.

No, I looked into it now and you were passing in extra SDK flags manually before and are still doing so now, as that's the only way this cross-compile works. You should try my cross-compilation pull out with your cross-compilation workflow, as I set the flags for a freshly-built cross-compilation SDK whereas it appears you manually pass in the extra flags for a prebuilt Windows ARM cross-compilation SDK. I think both should work fine together, as externally supplied flags should override the internal ones, but it would be good to verify that.

@finagolfin Can you elaborate what "try my cross-compilation pull out with your cross-compilation workflow" means? How exactly do we "verify that"? I'd be happy to run or test something.

@finagolfin I reran the cross-compile build with your patch #71552 and it was successful with no issues. The log also didn't say "CROSSCOMPILE". So it seems all good.

Great, what bootstrap mode are you using though?

We don't use any bootstrap mode. We use the host tools.

OK, that is one of the bootstrapping modes, as you can see on the official linux CI, -- Bootstrapping: HOSTTOOLS. Yes, technically that doesn't make sense, but that was the wrong name that was chosen for this internal build variable, instead of something accurate like "Swift compiler source compilation mode."

hjyamauchi · 2024-02-19T23:45:45Z

@swift-ci please build toolchain Windows platform

hjyamauchi · 2024-02-26T22:10:36Z

Rebased

hjyamauchi · 2024-02-26T22:10:49Z

@swift-ci please build toolchain Windows platform

compnerd

I think that this should be mostly fine, outside of the hardcoded Windows bit which interferes with other work. I suppose that I can try to address that, but I'd definitely appreciate it if you could try to proactively avoid some of that pain.

compnerd · 2024-03-05T13:06:11Z

utils/build.ps1

+    TryAdd-KeyValue $Defines CMAKE_SYSTEM_NAME Windows
+    TryAdd-KeyValue $Defines CMAKE_SYSTEM_PROCESSOR $Arch.CMakeName


Hmm, can we hoist this? I think that this might cause some trouble for adding Android support (see #72014).

hjyamauchi · 2024-03-06T23:06:50Z

@compnerd PTAL. The lastest revision is rebased and passes local cross-compiling tests.

compnerd · 2024-03-15T18:28:01Z

@swift-ci please test

etcwilde

LGTM. Eventually it might make sense to have toolchain files and some examples of cache files that can set these variables rather than having a one-size-fits-all script force the values through.

compnerd · 2024-03-15T19:53:12Z

LGTM. Eventually it might make sense to have toolchain files and some examples of cache files that can set these variables rather than having a one-size-fits-all script force the values through.

Which variables are you thinking of? I don't think that we want those files because the majority of the flags being passed are just paths that are dependent on how the build is being executed. The other flags that control what and how should be in cache files. The only exception that we currently have to that are the experimental modules.

etcwilde · 2024-03-15T20:29:45Z

CMAKE__COMPILER_TARGET, the CMAKE_SYSTEM_PROCESSOR, etc. a lot of this is forced through the script when the cache and toolchain would do the job nicely. For the system processor, CMake docs explicitly call out using the toolchain file to set that variable. In this case it's fine, I'm just tired of the prescriptive nature of build-script making it really difficult to build things where you do need the customization, and then making it impossible to upstream things without breaking someone else.

compnerd

I think that with the changes for dependencies, this should be ready to go.

utils/build.ps1

compnerd · 2024-04-17T16:42:42Z

@swift-ci please smoke test

@compnerd

Co-authrored with @compnerd

hjyamauchi · 2024-04-18T16:30:55Z

Rebased and tested now.

hjyamauchi · 2024-04-18T22:45:15Z

@swift-ci please test

shahmishal · 2024-04-19T05:50:20Z

@swift-ci test windows

hjyamauchi force-pushed the crosscompiling-buildps1 branch from 1623490 to 7211d06 Compare February 13, 2024 18:10

hjyamauchi requested a review from compnerd February 13, 2024 18:10

compnerd reviewed Feb 13, 2024

View reviewed changes

hyp reviewed Feb 13, 2024

View reviewed changes

hjyamauchi force-pushed the crosscompiling-buildps1 branch 2 times, most recently from 6906279 to dab2f44 Compare February 13, 2024 21:31

finagolfin reviewed Feb 14, 2024

View reviewed changes

hjyamauchi mentioned this pull request Feb 15, 2024

Build swift-docc and swift-inspect in the cross-compiling case of the windows toolchain build #71655

Open

hjyamauchi force-pushed the crosscompiling-buildps1 branch from dab2f44 to 41545df Compare February 15, 2024 19:16

compnerd mentioned this pull request Feb 19, 2024

Swift Windows ARM + VSCode Issue: SourceKit Language Server client: couldn't create connection to server swiftlang/sourcekit-lsp#1062

Open

hjyamauchi force-pushed the crosscompiling-buildps1 branch 2 times, most recently from f9bcd16 to 6407d4a Compare February 26, 2024 21:57

compnerd approved these changes Mar 5, 2024

View reviewed changes

hjyamauchi force-pushed the crosscompiling-buildps1 branch from 6407d4a to 7fdf14f Compare March 6, 2024 13:53

compnerd approved these changes Mar 15, 2024

View reviewed changes

etcwilde approved these changes Mar 15, 2024

View reviewed changes

compnerd approved these changes Apr 16, 2024

View reviewed changes

utils/build.ps1 Show resolved Hide resolved

hjyamauchi force-pushed the crosscompiling-buildps1 branch from 7fdf14f to 208a066 Compare April 17, 2024 16:34

Add cross-compiling support for the windows toolchain

f6ab729

Co-authrored with @compnerd

hjyamauchi force-pushed the crosscompiling-buildps1 branch from 208a066 to f6ab729 Compare April 18, 2024 16:25

hjyamauchi merged commit 4364612 into swiftlang:main Apr 19, 2024
5 checks passed

shahmishal mentioned this pull request May 27, 2024

[6.0] Add cross-compiling support for the windows toolchain #73920

Merged

		TryAdd-KeyValue $Defines CMAKE_SYSTEM_NAME Windows
		TryAdd-KeyValue $Defines CMAKE_SYSTEM_PROCESSOR $Arch.CMakeName

Add cross-compiling support for the windows toolchain #71584

Add cross-compiling support for the windows toolchain #71584

Conversation

hjyamauchi commented Feb 13, 2024

compnerd commented Feb 13, 2024

hjyamauchi commented Feb 13, 2024

compnerd commented Feb 13, 2024

compnerd commented Feb 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjyamauchi Feb 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjyamauchi commented Feb 19, 2024

hjyamauchi commented Feb 26, 2024

hjyamauchi commented Feb 26, 2024

compnerd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjyamauchi commented Mar 6, 2024

compnerd commented Mar 15, 2024

etcwilde left a comment

Choose a reason for hiding this comment

compnerd commented Mar 15, 2024

etcwilde commented Mar 15, 2024

compnerd left a comment

Choose a reason for hiding this comment

compnerd commented Apr 17, 2024

hjyamauchi commented Apr 18, 2024

hjyamauchi commented Apr 18, 2024

shahmishal commented Apr 19, 2024

hjyamauchi Feb 16, 2024 •

edited

Loading