[all] Support for M1 builds #808

mantissaman · 2021-03-27T19:01:01Z

While trying to run OSO Python sample app or trying to install oso (pip install oso) on Apple M1 architecture I am getting following error:
(venv) ➜ oso-python-quickstart git:(main) pip install -r requirements.txt
ERROR: Could not find a version that satisfies the requirement oso
ERROR: No matching distribution found for oso

saolsen · 2021-03-29T13:37:15Z

Yeah sorry we don't have M1 builds yet. The easiest thing to do is use x64 python via rosetta for development. We hope to have M1 builds available soon.

delicb · 2021-04-04T12:50:41Z

@mantissaman
I did manage to make it work locally with Python 3.9 on M1 chip, with the following steps.

install Rust - brew install rust - needed for compile
create virtual env - python -m venv <path-to-your-env>
activate virtual env - source <path-to-your-env>/bin/activate
install wheel (not sure if required, but some errors have gone after I did this) - pip install wheel
clone oso repo - git clone https://github.com/osohq/oso.git
build - cd oso && make python-build

The last step builds OSO locally and installs it in the active python environment by creating <path-to-your-env>/site-packages/oso.egg-link.

After that, I was able to import and use OSO. Note that I needed OSO only, not flask or sqlalchemy libs, but you should be able to install those manually once you have OSO core installed.

oso ❯ python
Python 3.9.2 (default, Mar  5 2021, 18:57:15)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.processor()
'arm'
>>> import oso
>>> o = oso.Oso()
>>> o.load_str('allow("foo", "bar", "baz");')
>>> o.is_allowed("foo", "bar", "baz")
True
>>>

I hope this helps.

mantissaman · 2021-04-12T04:18:23Z

@delicb - Many thanks. It worked like a charm.

rcrowe · 2021-07-26T20:00:03Z

Unable to develop with Go; either the linker fails or if I try forcing the architecture I see...

> env GOOS=darwin GOARCH=amd64 go build .
package oso-test
	imports github.com/osohq/go-oso
	imports github.com/osohq/go-oso/internal/ffi: build constraints exclude all Go files in $GOPATH/pkg/mod/github.com/osohq/[email protected]/internal/ffi

saolsen · 2021-07-26T20:16:58Z

Hey @rcrowe. We aren't set up to build arm64 yet but you can build it yourself with a couple extra steps.

Clone the oso repo and build the library for arm64. cargo build --release
Clone the go-oso repo and replace the library in internal/ffi/native/macos/libpolar.a With the one you just built (found in target/release/libpolar.a)
Replace the line in the go-oso library in internal/ffi/ffi.go that says

// #cgo darwin,amd64 LDFLAGS: ${SRCDIR}/native/macos/libpolar.a -ldl -lm

With

// #cgo darwin,arm64 LDFLAGS: ${SRCDIR}/native/macos/libpolar.a -ldl -lm

rcrowe · 2021-08-05T14:43:51Z

Thanks @saolsen that worked locally 🙇🏻

What would need to happen in order to support this within the various languages?

Under macos_libs run rustup target add x86_64-apple-darwin & cargo build --target aarch64-apple-darwin --release
Copy libpolar.a with the target in the filename, into each language
For go, add a new cgo directive
??

saolsen · 2021-08-05T16:00:58Z

We are planning to include builds for arm in the official release, we just haven't gotten to it yet. The way you would have to build it yourself is a little bit different for every language.

asyncee · 2021-08-27T06:30:26Z

Leaving a comment to indicate demand on this feature. Currently we have to support two separate paths to deploy a project :)

gj · 2021-09-24T23:52:32Z

We might be able to take some inspiration from rust-lang/rust#75991.

taybenlor · 2021-10-13T23:53:46Z

Is there a planned release date for a version with ARM wheels?

saolsen · 2021-10-14T14:22:55Z

We're planning to include a macOS ARM wheel for our python library in our release next week. Other languages will be a bit later but hopefully not too much longer.

taybenlor · 2021-10-14T21:42:09Z

Awesome! Thanks. Will that include Linux ARM as well? I’m using the Python library inside docker on an M1.

tedivm · 2021-10-22T15:15:25Z

A linux ARM would be great- basically something that can run in the official Python ARM64 containers.

tedivm · 2021-10-28T01:20:20Z

For anyone else who needs it I'm building docker containers with ARM support here.

tedivm · 2021-11-01T22:36:33Z

And in further good news, the automation I set up to keep the containers up to date picked up the new release and automatically built new containers.

dhatch · 2021-11-09T17:52:12Z

An update here! oso in Python now supports M1 processors. We don't yet have support for ARM linux. Support for M1 in other languages & ARM linux is coming! Our next priority is Go!

jlk · 2021-12-31T21:48:07Z

Just a note on this: The lack of linux,arm64 support (in go) means I can't deploy (using CDK) to arm64 lambda from my m1 mbp.build.

So, for others - if you see the error below, that's why:

imports github.com/osohq/go-oso/internal/ffi: build constraints exclude all Go files in /Users/home/go/src/github.com/org/project/vendor/github.com/osohq/go-oso/internal/ffi

tedivm · 2022-01-03T14:16:44Z

@jlk- AWS Lambda supports using images instead of zip files to launch. As a work around you can use the multi-py oso image as the base for your lambda.

hobofan · 2022-01-27T17:58:48Z

The Java version of oso also seems to have an M1 related issue.

When running with a JRA built for aarch64 I'm getting the following error:

Caused by: java.lang.UnsatisfiedLinkError: could not get native definition for type `POINTER`, original error message follows: java.lang.UnsatisfiedLinkError: Unable to execute or load jffi binary stub from `/var/folders/2d/lkf35y614bx09d6cdpvz5pt40000gn/T/`. Set `TMPDIR` or Java property `java.io.tmpdir` to a read/write path that is not mounted "noexec".
/Users/fld33/fld33/field33/.ijwb/jffi12706078071003416507.dylib: dlopen(/Users/fld33/fld33/field33/.ijwb/jffi12706078071003416507.dylib, 0x0001): tried: '/Users/fld33/fld33/field33/.ijwb/jffi12706078071003416507.dylib' (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64e')), '/usr/lib/jffi12706078071003416507.dylib' (no such file)
	at com.kenai.jffi.internal.StubLoader.tempLoadError(StubLoader.java:424)
	at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:409)
	at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:278)
	at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:487)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at com.kenai.jffi.Init.load(Init.java:68)
	at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49)
	at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45)
	at com.kenai.jffi.Foreign.getInstance(Foreign.java:103)
	at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242)
	at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
	at com.kenai.jffi.Type.resolveSize(Type.java:155)
	at com.kenai.jffi.Type.size(Type.java:138)
	at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:178)
	at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:57)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:41)
	at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:53)
	at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:49)
	at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/java.lang.Class.newInstance(Class.java:584)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
	at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
	at jnr.ffi.LibraryLoader.create(LibraryLoader.java:74)
	at com.osohq.oso.Ffi.<init>(Ffi.java:314)
	at com.osohq.oso.Ffi.get(Ffi.java:322)
	at com.osohq.oso.Polar.<init>(Polar.java:24)
	at com.osohq.oso.Oso.<init>(Oso.java:11)

Running the same built .jar with a non-aarch64 JRE works, though with a big performance penalty.

Apart from that, when looking into the issue I also stumbled upon jnr/jnr-ffi#257, which might be a hurdle for M1 support (but should by now be easily fixable by updating the dependency).

EDIT: Can confirm that bumping the jnr dependency fixes the problem

chrichts · 2022-02-13T03:49:49Z

@saolsen we've built linux/arm64 for go. Any way we can add it to https://github.com/osohq/go-oso?

gj · 2022-02-25T16:48:49Z

Hey @chrichts, do you mean you were able to do a manual local build for Go or that you've set up a GitHub Actions workflow to build it (the same way the rest of our libs are built)?

chrichts · 2022-02-26T07:43:28Z

We're doing a manual local build. Its fine and it works, but everytime we update oso we need to rerun it so would be much more helpful having it included.

But I see now it looks like you're building it here: https://github.com/osohq/oso/blob/main/.github/workflows/release.yml. Can I open a PR to add support for linux/arm64?

adilnaimi · 2024-01-19T13:30:34Z

I manage to fix our local build using this:

Dockerfile:

# Use manylinux_2_28 image
FROM quay.io/pypa/manylinux_2_28_aarch64 as build

RUN yum -y install gcc make patch zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel tk-devel libffi-devel xz-devel jq
RUN curl https://pyenv.run | bash
ENV PATH="/root/.pyenv/bin:/root/.pyenv/shims:${PATH}"
RUN pyenv install 3.12
RUN pyenv global 3.12
RUN python --version

RUN pip install --upgrade pip setuptools wheel

RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"

ARG OSO_VERSION
WORKDIR /opt
RUN git clone --depth 1 --branch "v$OSO_VERSION" https://github.com/osohq/oso.git /opt/oso
WORKDIR /opt/oso
RUN rm -Rf .git
RUN make python-build
RUN make python-test
WORKDIR /opt/oso/languages/python/oso
RUN python setup.py bdist_wheel
RUN auditwheel repair dist/*.whl -w wheelhouse/

WORKDIR /opt/oso
RUN make python-django-build
RUN make python-django-test
WORKDIR /opt/oso/languages/python/django-oso
RUN python setup.py bdist_wheel
# RUN auditwheel repair dist/*.whl -w wheelhouse/

FROM scratch as runtime
ARG OSO_VERSION
COPY --from=build /opt/oso/languages/python/oso/wheelhouse/oso-"$OSO_VERSION"-cp312-cp312-manylinux_2_28_aarch64.whl /
COPY --from=build /opt/oso/languages/python/django-oso/dist/django_oso-0.27.1-py3-none-any.whl /

Run Command:

DOCKER_BUILDKIT=1 docker build \
    --output . \
    --build-arg="OSO_VERSION=0.27.3" \
    --target runtime \
    .

samscott89 changed the title ~~Python - Could not find a version that satisfies~~ [all] Support for M1 builds May 26, 2021

samscott89 pinned this issue Oct 13, 2021

leina05 mentioned this issue Oct 13, 2021

pip installation error #1278

Closed

hobofan mentioned this issue Jan 27, 2022

(Java): Support M1 #1518

Merged

connec mentioned this issue Feb 26, 2022

Use a matrix for library build in release workflow #1535

Closed

robertkrossa mentioned this issue Sep 13, 2022

No Python wheels for ARM64 Linux #1580

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[all] Support for M1 builds #808

[all] Support for M1 builds #808

mantissaman commented Mar 27, 2021

saolsen commented Mar 29, 2021

delicb commented Apr 4, 2021 •

edited

Loading

mantissaman commented Apr 12, 2021

rcrowe commented Jul 26, 2021

saolsen commented Jul 26, 2021

rcrowe commented Aug 5, 2021

saolsen commented Aug 5, 2021

asyncee commented Aug 27, 2021

gj commented Sep 24, 2021

taybenlor commented Oct 13, 2021

saolsen commented Oct 14, 2021 •

edited by gj

Loading

taybenlor commented Oct 14, 2021 •

edited

Loading

tedivm commented Oct 22, 2021

tedivm commented Oct 28, 2021

tedivm commented Nov 1, 2021

dhatch commented Nov 9, 2021 •

edited by gj

Loading

jlk commented Dec 31, 2021 •

edited

Loading

tedivm commented Jan 3, 2022

hobofan commented Jan 27, 2022 •

edited

Loading

chrichts commented Feb 13, 2022

gj commented Feb 25, 2022

chrichts commented Feb 26, 2022

adilnaimi commented Jan 19, 2024

[all] Support for M1 builds #808

[all] Support for M1 builds #808

Comments

mantissaman commented Mar 27, 2021

saolsen commented Mar 29, 2021

delicb commented Apr 4, 2021 • edited Loading

mantissaman commented Apr 12, 2021

rcrowe commented Jul 26, 2021

saolsen commented Jul 26, 2021

rcrowe commented Aug 5, 2021

saolsen commented Aug 5, 2021

asyncee commented Aug 27, 2021

gj commented Sep 24, 2021

taybenlor commented Oct 13, 2021

saolsen commented Oct 14, 2021 • edited by gj Loading

taybenlor commented Oct 14, 2021 • edited Loading

tedivm commented Oct 22, 2021

tedivm commented Oct 28, 2021

tedivm commented Nov 1, 2021

dhatch commented Nov 9, 2021 • edited by gj Loading

jlk commented Dec 31, 2021 • edited Loading

tedivm commented Jan 3, 2022

hobofan commented Jan 27, 2022 • edited Loading

chrichts commented Feb 13, 2022

gj commented Feb 25, 2022

chrichts commented Feb 26, 2022

adilnaimi commented Jan 19, 2024

delicb commented Apr 4, 2021 •

edited

Loading

saolsen commented Oct 14, 2021 •

edited by gj

Loading

taybenlor commented Oct 14, 2021 •

edited

Loading

dhatch commented Nov 9, 2021 •

edited by gj

Loading

jlk commented Dec 31, 2021 •

edited

Loading

hobofan commented Jan 27, 2022 •

edited

Loading