Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add import blocks supporting git #22

Merged
merged 4 commits into from
May 22, 2024
Merged

Add import blocks supporting git #22

merged 4 commits into from
May 22, 2024

Conversation

patricoferris
Copy link
Collaborator

In step with the "how far can we get with paths starting with /data" this PR adds very basic Import blocks to Shark. Import blocks at the moment are essentially run blocks using git to pull a repository into /data using a basic Alpine image to do so. For now making them a distinct separate block that is translated into a build script is a little overkill, but I think in the long run it will make it easier for us to support other sources of data and make it nicer for users to simple write:

```shark-import
https://foo.git /data/foo
ftp://bar /data/bar
file:///home/alice/project /data/project
```

Or something like that rather than manually doing an Obuilder script. For now the format is simple, (URL, PATH) pairs with everything being assumed to be git. There could some role for Vurl here for extra metadata, but I could imagine coming up with am Import DSL for doing more interesting imports of data e.g. importing branches from a git repo and embedding them in the file system.

```shark-import
https://foo.git |> branches test* |> /data/foo
```

The tests now run a simple ocaml-test.md script that uses an import statement and then opam and dune to build things. The invocations of opam and dune use there less common "provide the file path" variants in order to activate the automatic dependency tracking. A small price to pay for what is otherwise a very nice setup I think!

@patricoferris
Copy link
Collaborator Author

The test here uses ocaml/opam so we need to implement support in OBuilder for finding the image's user and actually uid and gid

@patricoferris
Copy link
Collaborator Author

Right -- opam install mutates /home/opam/.opam/default which is implicitly needed by dune build because that is where dune is installed, but that is all implicit! Hmmm :S

@mdales mdales merged commit bdfea5b into main May 22, 2024
2 checks passed
@mdales mdales deleted the pf341-import branch May 22, 2024 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants