update build pipeline: simplify ci, move metadata into pyproject, support py312 #89

tlambert03 · 2023-10-11T16:03:08Z

closes #37

I noticed that issue and spent some time updating the build pipeline here. Might have gotten a little carried away, but here's what this PR does now:

removes setup.cfg and places all metadata and setuptools config in pyproject.toml. The only thing setup.py is used for now is the c-extension stuff
uses cibuildwheel to build wheels on github actions: makes the action file significantly simpler. cibuildwheel config is in pyproject.toml
runs pytest on the built wheels
support python 3.12

an example build can be seen here: https://github.com/tlambert03/pymmcore/actions/runs/6484564479
... and in addition to the pytest that gets run by cibuildwheel, I'll download those artifacts and test them on a couple real systems

This lets you build the project with python -m build ... or even with pip install -e . (assuming you have the build deps required)

@marktsuchida, if you're generally supportive of this, I can update the maintainer-notes.md to reflect all the changes made here.

things that could potentially be done as well:

remove smoketest, which is covered by the pytest
remove manylinux/build.sh

For the next deploy, this PR prepares deployment with pypi trusted publishers (rather than tokens) but @marktsuchida, you would need to set it up as described in that link

tlambert03 · 2023-10-11T16:10:45Z

also just noticed that you're currently building universal2 wheels, and this builds both x86_64 and arm64, let me know if you'd prefer me to switch the arm64 to a universal wheel

tlambert03 · 2023-10-11T16:32:09Z

To make it easier to discuss what wheels to keep/change, here's what got shipped for pymmcore v10.4.0.71.0:

cp311-win_amd64
cp311-win32
cp311-manylinux_2_17_x86_64.manylinux2014_x86_64
cp311-macosx_10_9_universal2
cp310-win_amd64
cp310-manylinux_2_17_x86_64.manylinux2014_x86_64
cp310-macosx_10_9_universal2
 cp39-win_amd64
 cp39-win32
 cp39-manylinux_2_17_x86_64.manylinux2014_x86_64
 cp39-manylinux_2_17_i686.manylinux2014_i686
 cp39-macosx_10_9_x86_64
 cp38-win_amd64
 cp38-win32
 cp38-manylinux_2_17_x86_64.manylinux2014_x86_64
 cp38-manylinux_2_17_i686.manylinux2014_i686
 cp38-macosx_10_9_x86_64

and in the artifacts from the last build in this PR, we have:

cp312-win_amd64
cp312-manylinux_2_17_x86_64.manylinux2014_x86_64
cp312-macosx_10_9_x86_64
cp312-macosx_11_0_arm64
cp311-win_amd64
cp311-manylinux_2_17_x86_64.manylinux2014_x86_64
cp311-macosx_10_9_x86_64
cp311-macosx_11_0_arm64
cp310-win_amd64
cp310-manylinux_2_17_x86_64.manylinux2014_x86_64
cp310-macosx_10_9_x86_64
cp310-macosx_11_0_arm64
 cp39-win_amd64
 cp39-manylinux_2_17_x86_64.manylinux2014_x86_64
 cp39-macosx_10_9_x86_64
 cp39-macosx_11_0_arm64
 cp38-win_amd64
 cp38-manylinux_2_17_x86_64.manylinux2014_x86_64
 cp38-macosx_10_9_x86_64
 cp38-macosx_11_0_arm64

let me know what you want changed. (i suspect you might want win32 added back?)

marktsuchida · 2023-10-11T18:16:15Z

Fantastic! Thanks for knocking off a whole bunch of items from my to-do list (especially the use of ci-buildwheel)!

Would be great if you could update the maintainer notes (also feel free to move notes to source comments where things are now in one place, such as NumPy version selection; also feel free to remove any details that ci-buildwheel now takes care of). It is also fine to remove the smoke test and no-longer-needed Linux script.

There is no need to build Windows 32-bit wheels, since we no longer distribute 32-bit device adapters anyway.

I'm also happy with the single-arch wheels for macOS. The universal2 wheels were something of an accident anyway (I didn't explicitly do anything to build them). I think it is good to follow what NumPy (our only dependency) does when there is no reason to do otherwise, and NumPy currently ships 10_9_x86_64 and 11_0_arm64. Also I hope to ship arm64 Micro-Manager soon, and intend to do this as a separate download rather than trying to distribute fat binaries.

I'll review the details and the binary artifacts more carefully later but all the proposed directions look great.

marktsuchida · 2023-10-11T19:34:02Z

Thanks also for getting rid of the distutils imports. Another to-do item I had.

tlambert03 · 2023-10-11T19:40:48Z

ok, updated the notes. Let me know if it all makes sense

tlambert03 · 2023-10-11T19:50:58Z

i was able to test the artifacts on an intel mac and mac silicon (running the pymmcore-plus test suite)... haven't checked windows/linux yet

tlambert03 · 2023-10-12T15:51:35Z

tested wheels on windows as well.

marktsuchida · 2023-10-17T20:57:56Z

Checked cp312 binaries on Win/Mac/Linux; they look good (in particular, weak symbols on macOS look safe).
Also checked that the Linux wheel works (Ubuntu 22.04, Python 3.10).

tlambert03 · 2023-10-17T21:25:23Z

Great!

marktsuchida · 2023-10-17T21:30:12Z

@tlambert03 Thanks for the updated notes! Everything looks good except for just one thing:

The source (.tar.gz) distribution seems to no longer include the MMDevice and MMCore source files (despite the MANIFEST.in -- not immediately sure why this is). I think the old CI built wheels from the sdist so that the sdist would be verified; I don't think we necessarily need to do that but it would be nice if the build from sdist worked out of the box (provided that SWIG etc. are available).

tlambert03 · 2023-10-17T21:40:51Z

Will check again, surprised to hear that though! Certainly looked at one point. Will figure out... we can also add check-manifest to be sure https://pypi.org/project/check-manifest/

tlambert03 · 2023-10-17T21:59:05Z

The source (.tar.gz) distribution seems to no longer include the MMDevice and MMCore source files (despite the MANIFEST.in -- not immediately sure why this is)

ah, if you mean the artifact from ci (which indeed lacks it) ... it's because git checkout there doesn't recurse submodules. locally, it seems to work fine. lemme update. locally, here's what the sdist looks like:

dist/pymmcore-10.4.0.71.1.dev0/
├── LICENSE.txt
├── MANIFEST.in
├── PKG-INFO
├── README.md
├── mmCoreAndDevices
│   ├── MMCore
│   │   ├── AppleHost.h
│   │   ├── CircularBuffer.cpp
│   │   ├── CircularBuffer.h
│   │   ├── ConfigGroup.h
│   │   ├── Configuration.cpp
│   │   ├── Configuration.h
│   │   ├── CoreCallback.cpp
│   │   ├── CoreCallback.h
│   │   ├── CoreProperty.cpp
│   │   ├── CoreProperty.h
│   │   ├── CoreUtils.h
│   │   ├── DeviceManager.cpp
│   │   ├── DeviceManager.h
│   │   ├── Devices
│   │   │   ├── AutoFocusInstance.cpp
│   │   │   ├── AutoFocusInstance.h
│   │   │   ├── CameraInstance.cpp
│   │   │   ├── CameraInstance.h
│   │   │   ├── DeviceInstance.cpp
│   │   │   ├── DeviceInstance.h
│   │   │   ├── DeviceInstanceBase.h
│   │   │   ├── DeviceInstances.h
│   │   │   ├── GalvoInstance.cpp
│   │   │   ├── GalvoInstance.h
│   │   │   ├── GenericInstance.h
│   │   │   ├── HubInstance.cpp
│   │   │   ├── HubInstance.h
│   │   │   ├── ImageProcessorInstance.cpp
│   │   │   ├── ImageProcessorInstance.h
│   │   │   ├── MagnifierInstance.cpp
│   │   │   ├── MagnifierInstance.h
│   │   │   ├── SLMInstance.cpp
│   │   │   ├── SLMInstance.h
│   │   │   ├── SerialInstance.cpp
│   │   │   ├── SerialInstance.h
│   │   │   ├── ShutterInstance.cpp
│   │   │   ├── ShutterInstance.h
│   │   │   ├── SignalIOInstance.cpp
│   │   │   ├── SignalIOInstance.h
│   │   │   ├── StageInstance.cpp
│   │   │   ├── StageInstance.h
│   │   │   ├── StateInstance.cpp
│   │   │   ├── StateInstance.h
│   │   │   ├── XYStageInstance.cpp
│   │   │   └── XYStageInstance.h
│   │   ├── Error.cpp
│   │   ├── Error.h
│   │   ├── ErrorCodes.h
│   │   ├── FrameBuffer.cpp
│   │   ├── FrameBuffer.h
│   │   ├── Host.cpp
│   │   ├── Host.h
│   │   ├── LibraryInfo
│   │   │   ├── LibraryPaths.h
│   │   │   ├── LibraryPathsUnix.cpp
│   │   │   └── LibraryPathsWindows.cpp
│   │   ├── LoadableModules
│   │   │   ├── LoadedDeviceAdapter.cpp
│   │   │   ├── LoadedDeviceAdapter.h
│   │   │   ├── LoadedModule.cpp
│   │   │   ├── LoadedModule.h
│   │   │   ├── LoadedModuleImpl.cpp
│   │   │   ├── LoadedModuleImpl.h
│   │   │   ├── LoadedModuleImplUnix.cpp
│   │   │   ├── LoadedModuleImplUnix.h
│   │   │   ├── LoadedModuleImplWindows.cpp
│   │   │   └── LoadedModuleImplWindows.h
│   │   ├── LogManager.cpp
│   │   ├── LogManager.h
│   │   ├── Logging
│   │   │   ├── GenericEntryFilter.h
│   │   │   ├── GenericLinePacket.h
│   │   │   ├── GenericLogger.h
│   │   │   ├── GenericLoggingCore.h
│   │   │   ├── GenericMetadata.h
│   │   │   ├── GenericPacketArray.h
│   │   │   ├── GenericPacketQueue.h
│   │   │   ├── GenericSink.h
│   │   │   ├── GenericStreamSink.h
│   │   │   ├── Logger.h
│   │   │   ├── Logging.h
│   │   │   ├── Metadata.cpp
│   │   │   ├── Metadata.h
│   │   │   └── MetadataFormatter.h
│   │   ├── MMCore.cpp
│   │   ├── MMCore.h
│   │   ├── MMEventCallback.h
│   │   ├── PluginManager.cpp
│   │   ├── PluginManager.h
│   │   ├── Semaphore.cpp
│   │   ├── Semaphore.h
│   │   ├── Task.cpp
│   │   ├── Task.h
│   │   ├── TaskSet.cpp
│   │   ├── TaskSet.h
│   │   ├── TaskSet_CopyMemory.cpp
│   │   ├── TaskSet_CopyMemory.h
│   │   ├── ThreadPool.cpp
│   │   └── ThreadPool.h
│   └── MMDevice
│       ├── Debayer.cpp
│       ├── Debayer.h
│       ├── DeviceBase.h
│       ├── DeviceThreads.h
│       ├── DeviceUtils.cpp
│       ├── DeviceUtils.h
│       ├── FixSnprintf.h
│       ├── ImageMetadata.h
│       ├── ImgBuffer.cpp
│       ├── ImgBuffer.h
│       ├── MMDevice.cpp
│       ├── MMDevice.h
│       ├── MMDeviceConstants.h
│       ├── ModuleInterface.cpp
│       ├── ModuleInterface.h
│       ├── Property.cpp
│       └── Property.h
├── pymmcore
│   ├── __init__.py
│   ├── __init__.pyi
│   ├── _version.py
│   ├── py.typed
│   └── pymmcore_swig.i
├── pymmcore.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   ├── requires.txt
│   └── top_level.txt
├── pyproject.toml
├── setup.cfg
├── setup.py
└── tests
    └── test_mmcore.py

tlambert03 · 2023-10-17T22:17:15Z

ok, I added two things

when building the sdist:

      - name: Build sdist
        run: |
          pip install -U pip build check-manifest
          check-manifest 
          python -m build --sdist

check-manifest ensures that everything that is checked into source (including submodules) is either contained in the sdist, or explicitly excluded either in Manifest.in or in pyproject.toml [tool.check-manifest]. So, provided you like the list above, this will ensure that's always the case.

I also added a check-manifest job as well that should give an early warning that the sdist will fail

marktsuchida · 2023-10-18T19:36:40Z

Perfect, thanks! Will probably do a release soon after deciding what to do with the few other outstanding PRs/issues.

tlambert03 added 14 commits October 11, 2023 08:10

wip

d6db8d6

wip

37e6f4d

add new deploy

b9424a3

use submodules

28189d8

add tests

015baf8

simplify

c56e7b6

simplify

93b0766

one file

df2dfaf

ignore deprecated

59843f8

remove musllinux

54aae07

remove deprecat ignore

d2dce8e

drop py37

cb6502c

split up macos

1ef5263

silence test warnings

4b7267e

tlambert03 changed the title ~~update build pipeline: simplify ci, move metadata into pyproject, support py312 and mac silicon~~ update build pipeline: simplify ci, move metadata into pyproject, support py312 Oct 11, 2023

tlambert03 added 2 commits October 11, 2023 12:05

fix imports

6588318

stupid, fix again

e982d0e

tlambert03 added 2 commits October 11, 2023 14:59

update readme

9ce366f

remove files

9a1e6a2

tlambert03 added 2 commits October 11, 2023 15:38

update maintainer notes

c927ee2

trusted pub

e549f86

tlambert03 added 2 commits October 17, 2023 18:00

recurse checkout

8bdfe78

add check manifest

343b903

fix depth

cc59d56

marktsuchida merged commit f08fcd7 into micro-manager:main Oct 18, 2023
7 checks passed

tlambert03 deleted the pyproject branch October 18, 2023 19:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update build pipeline: simplify ci, move metadata into pyproject, support py312 #89

update build pipeline: simplify ci, move metadata into pyproject, support py312 #89

tlambert03 commented Oct 11, 2023 •

edited

Loading

tlambert03 commented Oct 11, 2023

tlambert03 commented Oct 11, 2023 •

edited

Loading

marktsuchida commented Oct 11, 2023

marktsuchida commented Oct 11, 2023

tlambert03 commented Oct 11, 2023

tlambert03 commented Oct 11, 2023

tlambert03 commented Oct 12, 2023

marktsuchida commented Oct 17, 2023

tlambert03 commented Oct 17, 2023

marktsuchida commented Oct 17, 2023

tlambert03 commented Oct 17, 2023

tlambert03 commented Oct 17, 2023

tlambert03 commented Oct 17, 2023

marktsuchida commented Oct 18, 2023

update build pipeline: simplify ci, move metadata into pyproject, support py312 #89

update build pipeline: simplify ci, move metadata into pyproject, support py312 #89

Conversation

tlambert03 commented Oct 11, 2023 • edited Loading

tlambert03 commented Oct 11, 2023

tlambert03 commented Oct 11, 2023 • edited Loading

marktsuchida commented Oct 11, 2023

marktsuchida commented Oct 11, 2023

tlambert03 commented Oct 11, 2023

tlambert03 commented Oct 11, 2023

tlambert03 commented Oct 12, 2023

marktsuchida commented Oct 17, 2023

tlambert03 commented Oct 17, 2023

marktsuchida commented Oct 17, 2023

tlambert03 commented Oct 17, 2023

tlambert03 commented Oct 17, 2023

tlambert03 commented Oct 17, 2023

marktsuchida commented Oct 18, 2023

tlambert03 commented Oct 11, 2023 •

edited

Loading

tlambert03 commented Oct 11, 2023 •

edited

Loading