Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add git-lfs Large File Storage support #40

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

moto-timo
Copy link
Member

"Git Large File Storage (LFS) replaces large files such as audio samples,
videos, datasets, and graphics with text pointers inside Git, while storing
the file contents on a remote server."

https://git-lfs.github.com/

This is useful when including proprietary pre-built binaries and also
when adding virtualization guests, such as Windows as a Guest or
Android as a Guest.

Since http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a2d6792f3a531b557bffaec4387d8e0e4eb3dab0
git-lfs is a non-fatal HOST_TOOLS addition, meaning it will be linked if present.

debian-9 only supports it with stretch-backports, so skip
opensuse-15.0,-15.1 do not have git-lfs except in community packages, so skip
ubuntu-16.04 does not have git-lfs except in package cloud, so skip

If someone really wants support in those distro versions, patches are welcome.

Signed-off-by: Tim Orling [email protected]

@rewitt1
Copy link
Contributor

rewitt1 commented Feb 18, 2020

Can this cause any build reproducibility issues between different distros? i.e. Would the behavior be the same between Fedora 29 and Ubuntu 18.04, but not between Fedora 29 and Ubuntu 16.04.

I ask because that was one useful aspect, being able to check for common behavior across distros.

@moto-timo moto-timo force-pushed the git-lfs branch 2 times, most recently from dee3586 to 9a535ed Compare March 10, 2020 00:37
@moto-timo
Copy link
Member Author

@rewitt1 I completely agree about your concern, so the most recent updates to the pull request address debian-9, ubuntu-16.04 and opensuse-15.0,15.1

# wget -O - https://packagecloud.io/github/git-lfs/gpgkey | gpg --with-fingerprint
RUN apt-get update && apt-get install -y apt-transport-https && \
echo "deb https://packagecloud.io/github/git-lfs/ubuntu/ xenial main" >> /etc/apt/sources.list && \
apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 6D398DBD30DD78941E2C4797FE2A5F8BDC282033 && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a fingerprint for the key you can use and verify, since the key will be downloaded every time the image is built?

The manpage for apt-key even says

With adv --recv-key you can e.g. download key from
keyservers directly into the trusted set of keys. Note that there are no checks
performed, so it is easy to completely undermine the apt-secure(8) infrastructure if
used without care.

Does the same question apply in the zypper case?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rewitt1 --recv-keys 6D398DBD30DD78941E2C4797FE2A5F8BDC282033 is the PGP fingerprint. It seemed better to hard code a verifiable fingerprint than to trust running a shell script or carrying the gpg key as an artifact. This also applies for zypper because OpenSUSE has no packaging for git-lfs, at least not in the 15.1 release. As soon as we feel comfortable dropping Ubuntu 16.04 and OpenSUSE 15.2 is released we can get rid of this packagecloud workaround.

Tim Orling added 4 commits May 5, 2020 09:54
"Git Large File Storage (LFS) replaces large files such as audio samples,
videos, datasets, and graphics with text pointers inside Git, while storing
the file contents on a remote server."

https://git-lfs.github.com/

This is useful when including proprietary pre-built binaries and also
when adding virtualization guests, such as Windows as a Guest or
Android as a Guest.

Since http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a2d6792f3a531b557bffaec4387d8e0e4eb3dab0
git-lfs is a non-fatal HOST_TOOLS addition, meaning it will be linked if present.

debian-9 only supports it with stretch-backports, patch to follow
opensuse-15.0,-15.1 do not have git-lfs except in package cloud, patch to follow
ubuntu-16.04 does not have git-lfs except in package cloud, patch to follow

Signed-off-by: Tim Orling <[email protected]>
Add stretch-backports to /etc/apt/sources.list and
use it to install git-lfs.

Signed-off-by: Tim Orling <[email protected]>
Add "github_git-lfs" rpm repo excerpted from install script:
https://packagecloud.io/github/git-lfs/install#bash-rpm
(e.g. https://packagecloud.io/install/repositories/github/git-lfs/config_file.repo?os=opensuse&dist=15.0&source=script)

Automatically add gpg keys with:
zypper --gpg-auto-import-keys refresh github_git-lfs

Signed-off-by: Tim Orling <[email protected]>
@dkruces
Copy link

dkruces commented Apr 21, 2021

Hi guys,

any chance to get this merged? Don't know how busy you are and if I can take over.

@rewitt1
Copy link
Contributor

rewitt1 commented Apr 23, 2021

Hi guys,

any chance to get this merged? Don't know how busy you are and if I can take over.

I can merge it if it gets updated. But right now, I don't have time to update it myself. I'll circle back in a few weeks when I have more time, and if it hasn't been updated, then I'll try to help out with updating the pull request.

@moto-timo
Copy link
Member Author

moto-timo commented May 6, 2021

Internally we went a different route to add "repo" and "git-lfs" via a derivative container model (which is in the process of being open sourced). The issue with "git-lfs" specifically was that it wasn't directly installable as a package for all the items in the supported Distro/Version matrix. Ergo, I have no motivation to rebase the PR at this time.

Happy to have you take it over @dagmcr if you have a driving need and can bring it up to date. It is possible the Distro/Version package manager issue is no longer a problem (for "git-lfs" specifically).

moto-timo added a commit to moto-timo/yocto-dockerfiles that referenced this pull request Apr 23, 2022
Although ubuntu-16.04 is in Extended Security Maintenance until 2026, it
is holding us back from adding git-lfs (see
crops#40)

Signed-off-by: Tim Orling <[email protected]>
moto-timo added a commit to moto-timo/yocto-dockerfiles that referenced this pull request Apr 23, 2022
Although ubuntu-16.04 is in Extended Security Maintenance until 2026, it
is holding us back from adding git-lfs (see
crops#40)

Signed-off-by: Tim Orling <[email protected]>
moto-timo added a commit to moto-timo/yocto-dockerfiles that referenced this pull request Apr 23, 2022
Although ubuntu-16.04 is in Extended Security Maintenance until 2026, it
is holding us back from adding git-lfs (see
crops#40)

Signed-off-by: Tim Orling <[email protected]>
moto-timo added a commit to moto-timo/yocto-dockerfiles that referenced this pull request Apr 23, 2022
Although ubuntu-16.04 is in Extended Security Maintenance until 2026, it
is holding us back from adding git-lfs (see
crops#40)

Signed-off-by: Tim Orling <[email protected]>
moto-timo added a commit to moto-timo/yocto-dockerfiles that referenced this pull request Apr 23, 2022
Although ubuntu-16.04 is in Extended Security Maintenance until 2026, it
is holding us back from adding git-lfs (see
crops#40)

Signed-off-by: Tim Orling <[email protected]>
moto-timo added a commit that referenced this pull request Apr 23, 2022
Although ubuntu-16.04 is in Extended Security Maintenance until 2026, it
is holding us back from adding git-lfs (see
#40)

Signed-off-by: Tim Orling <[email protected]>
moto-timo added a commit to moto-timo/poky-container that referenced this pull request Apr 23, 2022
Ubuntu "Jammy Jellyfish" 22.04 has been released, with long-term support
until at least 2027 (maintenance) and 2032 (extended security
maintenance).

https://ubuntu.com/about/release-cycle
https://ubuntu.com/blog/ubuntu-22-04-lts-released

Although ubuntu-16.04 is in Extended Security Maintenance until 2026, it
is holding us back from adding git-lfs (see
crops/yocto-dockerfiles#40).

Signed-off-by: Tim Orling <[email protected]>
moto-timo added a commit to moto-timo/poky-container that referenced this pull request Apr 23, 2022
Ubuntu "Jammy Jellyfish" 22.04 has been released, with long-term support
until at least 2027 (maintenance) and 2032 (extended security
maintenance).

https://ubuntu.com/about/release-cycle
https://ubuntu.com/blog/ubuntu-22-04-lts-released

Although ubuntu-16.04 is in Extended Security Maintenance until 2026, it
is holding us back from adding git-lfs (see
crops/yocto-dockerfiles#40).

Signed-off-by: Tim Orling <[email protected]>
moto-timo added a commit to crops/poky-container that referenced this pull request Apr 23, 2022
Ubuntu "Jammy Jellyfish" 22.04 has been released, with long-term support
until at least 2027 (maintenance) and 2032 (extended security
maintenance).

https://ubuntu.com/about/release-cycle
https://ubuntu.com/blog/ubuntu-22-04-lts-released

Although ubuntu-16.04 is in Extended Security Maintenance until 2026, it
is holding us back from adding git-lfs (see
crops/yocto-dockerfiles#40).

Signed-off-by: Tim Orling <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants