Custom command wrapping lipo to generate universal binaries in macOS

[czoido]

Motivated by #52
Partially based on: #53 and #58

cc/ @gmeeker

[MartinDelille]

Hi @czoido ! As you may have read here or in the conan slack channel, I'm interested on creating my Macos application universal binary. From what I understand I must currently build 2 application bundles and then run lipo to create the final univeral bundle. Would you document here how I could benefit from your work ?

[czoido]

Hi @czoido ! As you may have read here or in the conan slack channel, I'm interested on creating my Macos application universal binary. From what I understand I must currently build 2 application bundles and then run lipo to create the final univeral bundle. Would you document here how I could benefit from your work ?

Hi @MartinDelille!

What this command does is to generate universal binaries for already existing binaries for multiple architectures by running lipo -create. If you want to test this, you can do conan config install over my branch what would install a new conan bi:lipo custom command. Image that you have a consumer application that uses a conanfile.txt like this:

[requires]
libtiff/4.6.0

And then, if you want to get universal binaries to link with them in your application you can do something like:

conan install . --deployer=full_deploy -s arch=armv8
conan install . --deployer=full_deploy -s arch=x86_64

What will give you all the dependencies in a full_deploy folder in separated folders by architecture. Then running:

conan bin:lipo create full_deploy/host --output-folder=universal

Will iterate through all the files structure, looking for mach-o files and running lipo on them to produce in the output-folder universal an identical file structure of the full_deploy but with universal binaries.

This is really just a convenience wrapper, we don't have a model in Conan currently to manage binaries with multiple architecture while we are starting to investigate it.

[MartinDelille]

Thanks @czoido ! I managed to create the universal folder by fixing two typo 👍

• conan config install => conan config install .
• conan bi:lipo => conan bin:lipo

I'm now wondering how I can use this folder for my application ? The conan_toolchain.cmake file is still pointing to the last generated architecture (x86_64).

[gmeeker]

Looks good to me. Although I'd love to see this available in Conan itself for other extensions: tools.apple.lipo?

@MartinDelille unfortunately that's still an open research topic. One thing that got lost in these PRs was an example Xcode project. See here for older code with an example containing an Xcode project:
https://github.com/gmeeker/conan-deploy-lipo

The conan extension approach only goes as far as using conan to build universal libraries of your projects dependencies. If you're building macOS or iOS only app, this might be sufficient. I don't this solves the universal binary issue for many people, but it's an important first step and having this lipo code somewhere official is good too.

See the longer discussion here to appreciate the difficulties:
#59

I intend to look into this further but haven't had a chance. I have a 1.0 wrapper with a cmake toolchain working (albeit quite ugly) and intend to figure out how to port it. Hopefully show that it's possible to do this with single architecture packages.
https://github.com/gmeeker/BuildConan.git

[czoido]

Thanks @czoido ! I managed to create the universal folder by fixing two typo 👍

• conan config install => conan config install .
• conan bi:lipo => conan bin:lipo

I'm now wondering how I can use this folder for my application ? The conan_toolchain.cmake file is still pointing to the last generated architecture (x86_64).

@MartinDelille, as @gmeeker said, after generating the universal binaries, you have to manually add them to the solution you want to build as Conan does not have built-in support for universal binaries (we are currently investigating how to support them).

One possible workaround is to modify the last generated config file, renaming the file and pointing the folders that point to the last deployed binaries to the folder where you created the universal binaries.

You can find an example for the workaround in this repo. For a simple project that uses fmt:

.
├── CMakeLists.txt
├── conanfile.txt
└── example.cpp

CMakeLists.txt:

cmake_minimum_required(VERSION 3.15)
project(example LANGUAGES CXX)

find_package(fmt REQUIRED CONFIG)

add_executable(example example.cpp)
target_compile_features(example PRIVATE cxx_std_14)
target_link_libraries(example PRIVATE fmt::fmt)

conanfile.txt:

[requires]
fmt/10.2.1
[generators]
CMakeDeps
CMakeToolchain
[layout]
cmake_layout

It could look similar to this

conan install . --deployer=full_deploy -s arch=x86_64 -s build_type=Release --build=missing
conan install . --deployer=full_deploy -s arch=armv8 -s build_type=Release --build=missing

conan bin:lipo create full_deploy/host --output-folder=universal

# just keep the last deployed architecture and rename to the universal architecture armv8.x86_64
rm build/Release/generators/fmt-release-x86_64-data.cmake
mv build/Release/generators/fmt-release-armv8-data.cmake build/Release/generators/fmt-release-armv8.x86_64-data.cmake

# substitute the paths in the files '/full_deploy/host' now should point 
# to '/universal' also point to the correct architecture
# also change the CMAKE_OSX_ARCHITECTURES to include both:
sed -i '' 's|full_deploy/host|universal|g' build/Release/generators/conan_toolchain.cmake
sed -i '' 's|/armv8|/armv8.x86_64|g' build/Release/generators/conan_toolchain.cmake
sed -i '' 's|arm64 CACHE|x86_64;arm64 CACHE|g' build/Release/generators/conan_toolchain.cmake
sed -i '' 's|full_deploy/host|universal|g' build/Release/generators/fmt-release-armv8.x86_64-data.cmake
sed -i '' 's|/armv8|/armv8.x86_64|g' build/Release/generators/fmt-release-armv8.x86_64-data.cmake

cd build

cmake .. -DCMAKE_TOOLCHAIN_FILE=Release/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release

cmake --build .  

lipo example -info

This is just a fast workaround to test, we will be doing some research to do this in Conan natively very soon.

Hope this helps!

[MartinDelille]

Hi @czoido ! Sorry for not answering after your detailed explaination!

This is unfortunately too complicated to integrate sed command to my current build workflow. I will pause universal build for now and either wait for a simpler way to do it.

If I really need to support universal build, I'll build two application bundle and run lipo on it afterward outside of conan.