Package Manager MVP #14265
Package Manager MVP #14265
Conversation
|
Congratulations with the start! Future proposal: would you consider an optional vendoring of dependencies in an adjacent git ref (background) ? In this example: When zig finds a "git-ref"-vendored dependency, instead of downloading from github, it checks it out from the project's git tree, similarly how Main benefits:
I would be willing to work on this assuming there is no outright rejection. Let me know! |
|
That looks like a neat approach. If I understand correctly, that approach can already be done even with this minimal MVP. In fact it's exactly what is being done in my example above. Perhaps what is missing would be achieving this, but for a URL that only supports the git protocol and does not have that convenient https download that GitHub has made available. Or perhaps what is missing, more importantly, would be some kind of detection that the URL is in fact a git repository that is already cloned and corresponds to the same one that the Let me know what you think. |
|
I get this is a PoC / prototype, but please don't leave crypto signing / verification as an afterthought when you ship something (just as rust, python, npm (in fact all of them) have done), as it's always tricky to retrofit. happy to work with you from the sigstore community |
|
The current plan, as already implemented in this branch, is Trust On First Use. If you would like to make a case for doing something more complicated than this, you are welcome to send some learning materials our way. |
How's that work @andrewrk , typically ToFu means you cache an identifier for subsequent use (like ssh, I select yes and tofu is set up for later connections)? I will look to gather some materials for you. |
I think we misunderstood each other. The repository would look like this (feel free to poke at it): Note that f0e53cc2391741034b144a2c2076ed8a9937b29b comes from libz. And the
If zig recognizes the repository as "git-vendored", zig should, instead of downloading it, take the git tree from the same repository. However, there are two caveats:
|
|
If you'd be interested in a new config format to adopt, I've been very impressed with the design of the "gura" format (https://github.com/gura-conf/gura). It seems to combine a lot of the best of JSON, YAML and TOML while being relatively simple. I don't think it's seen widespread adoption anywhere though, so tooling support may be limited. |
|
A couple of concerns that I have:
|
|
Using this with package managers like Nix or Guix requires support for separating the build into two steps:
Step 1. could be implemented by zig (like |
|
Zig dependency file naming proposal: Ziggyfile |
|
I don’t think the file format is valid ini. All the ini parsers I know of either outright reject the duplicated Some other thoughts coming from another packaging ecosystem (I maintain pip and contribute to many specs and tools in Python packaging): If you are to support installing from VCS, strictly only support commit IDs (or equivalent), at least for now. Anything that can potentially change (especially branches, but also others) would introduce endless cachability and reproducibility issues. You could leave room and figure them out later, but don’t commit (not a pun) to supporting them too early. Also big +1 on not relying on arbitrary code execution, it’s definitely one of the worst things you could do. Another thing worth leaving room for is to allow multiple sources to satisfy one dependency (that the resolver can choose on build-time based on various criteria, such as OS, CPU, GPU, etc.). Eventually you’ll reach a point where people wants to distribute pre-built binaries, and it’ll cause much growing pain to change a one-to-one assumption to many-to-one unless you plan ahead. |
|
I'm interested in how the args are planned to work. From what I'm reading, will it be command-line-style arguments, and they will all be applied to the child builder just as if those arguments were passed? I think that'll work okay for my use cases, even if the usage seems a bit odd in my head. Is there really any need to pass options other than |
|
So exciting to see progress towards a package manager! (For context, here is my only comment on the other PR, where I advocate for a package manager in the style of Elixir.) I'm curious how this will extend to transitive dependencies. The example here shows two dependencies that don't themselves have any dependencies (which is itself kind of neat that the deps don't need to be a part of this system). But I'm wondering how it will work when the library you want to use depends on other libraries. I can think of three general approaches:
Per my comment in the other thread, I'm personally in favor of an approach like in (3), but given the MVP here, it seems to be missing foundational pieces for that. So am I right in assuming that the eventual vision will be either (1) or (2)? Edit: Hm, I'm realizing that the |
|
@lukehinds It's a new package manager, and an opportunity to get things right from the beginning. And yep, Sigstore is something we should absolutely support. Happy to work with you on that. Maybe we can even reuse some of the work we did on wasm signatures (multiple signers, policies, etc. would apply very well, and would be a significant improvement over other package managers). |
| return std.fmt.parseInt(u64, rtrimmed, 8); | ||
| } | ||
|
|
||
| pub fn is_ustar(header: Header) bool { |
There was a problem hiding this comment.
This doesn't follow the style guide.
| if (file_size == 0 and unstripped_file_name.len == 0) return; | ||
| const file_name = try stripComponents(unstripped_file_name, options.strip_components); | ||
|
|
||
| var file = try dir.createFile(file_name, .{}); |
There was a problem hiding this comment.
Files are not guaranteed to be laid out in an order that will create the directory the file will live in before creating the file itself. This may lead to errors about the directory not existing yet.
| @@ -17,6 +17,9 @@ const FCOMMENT = 1 << 4; | |||
|
|
|||
| const max_string_len = 1024; | |||
|
|
|||
| /// TODO: the fully qualified namespace to this declaration is | |||
| /// std.compress.gzip.GzipStream which has a redundant "gzip" in the name. | |||
| /// Instead, it should be `std.compress.gzip.Stream`. | |||
There was a problem hiding this comment.
It would be better if all of the algorithms under std.compress could use the same naming conventions. Like std.compress.deflate uses Decompressor() and Compressor()
Pre-built artifacts aside, multiple URIs for source code is also forward-looking. Example: Multiple HTTP mirrors could be useful now and other protocols can be ignored. Down the line, Zig could integrate with operating systems' registered protocol handlers or allow users to configure an arbitrary external command for a protocol. My other suggestion is to specify hashing algorithm(s) in-band (e.g. replace |
|
@uranusjr makes an important point. It's good that most editors will highlight Since a subset of a common syntax is considered an advantage, both from a perspective of simplicity and interoperability with existing tools, depth-1 s-expressions might be an interesting choice. No parenthesis hell if you only get a couple =) Doesn't really need the By limiting to depth 2, we could achieve simple lists and maps: There is no obvious way to add comments, there's no real "extension" for it, |
[dep:libz]
url=https://github.com/andrewrk/libz/archive/f0e53cc2391741034b144a2c2076ed8a9937b29b.tar.gz
sha1=c9b30cffc40999d2c078ff350cbcee642970a224fe123c756d0892f876cf1aae🤔
JSON is not about JS only for a long time already. But is not suitable for this task definitely with its mandatory quotes. |
This allows setting a custom buffer size. In this case I wanted it because using a buffer size large enough to fit a TLS ciphertext record elides a memcpy(). This commit also adds `readAtLeast` to the Reader interface.
This makes building from source go faster and avoids tripping over unimplemented things in the C backend.
The `zig build` command now makes `@import("@Dependencies")` available
to the build runner package. It contains all the dependencies in a
generated file that looks something like this:
```zig
pub const imports = struct {
pub const foo = @import("foo");
pub const @"bar.baz" = @import("bar.baz");
};
pub const build_root = struct {
pub const foo = "<path>";
pub const @"bar.baz" = "<path>";
};
```
The build runner exports this import so that `std.build.Builder` can
access it. `std.build.Builder` uses it to implement the new `dependency`
function which can be used like so:
```zig
const libz_dep = b.dependency("libz", .{});
const libmp3lame_dep = b.dependency("libmp3lame", .{});
// ...
lib.linkLibrary(libz_dep.artifact("z"));
lib.linkLibrary(libmp3lame_dep.artifact("mp3lame"));
```
The `dependency` function calls the build.zig file of the dependency as
a child Builder, and then can be ransacked for its build steps via the
`artifact` function.
This commit also renames `dependency.id` to `dependency.name` in the
`build.zig.ini` file.
|
Again coming from another packaging ecosystem, I have to also put my vote against using code for metadata configuration, even a subset of it. Most of the hurdle users have to effective write a package metadata file is not the language, but what keys exactly are needed for what, and most people simply don’t do packaging enough to remember and have to resort to copy-pasting anyway. It is therefore not particularly useful to optimise how easy users can memorise and write the declaration—they can’t, no matter what you use. It would be better to optimise for
The originally proposed section-key-value non-ini format actually satisfies these quite well. The main problem of it would be tooling and extendability in the future, but as long as you spec it well enough (big if!) this is at least solvable. |
|
For those who want to influence Zig's package manager as the project moves forward, here is how to do it:
|
Now I can see why my suggested enhancement is a subset of your proposal. Thank you for your thoughtful reply!
This is an excellent point, and a way to resolve it, which I did not originally consider. |
|
I updated https://github.com/ziglang/zig/projects/4 to help organize efforts related to package management. |
|
I couldn't find a reference in any of the earlier comments/issues and didn't want (yet) to create a new issue, but is there any planned support for monorepos, i.e. git repos hosting multiple projects/packages which are being developed in parallel and which might have some inter-dependencies with other packages in that repo? An extreme case is Google's monorepo, but I'm interested in growing my own super-early stage Zig monorepo, which currently is using gyro, but I'm also considering adding more hybrid Zig/TypeScript libs (for WASM apps) to my main monorepo (see below)... What I think is needed for this general use case (and couldn't find info on): When specifying a dependency on a package from such a monorepo, we'd also be required to provide a subdirectory or direct path to that package's own Ps. I perfectly understand there's no major interest in these setups, but it'd be nice if they aren't categorically excluded from the outset (and I'm not saying that they are!) 😉 FWIW I've been using a monorepo for the past 5 years for all my TypeScript work and wouldn't go back. It's IMHO the only feasible/sensible way to manage & automate scaffolding, maintenance, version bumps, change logs, CI/releases of all the 175+ packages in there (rather than requiring the same number of individual git repos and duplicating a ton of infrastructure setup)... |
Demo
You can play with this yourself:
Keep in mind, some enhancements to TLS need to be made before downloading https will work for non-Linux users:
Why .ini format for the declarative file?
Here are options that I considered and reasons for why I ultimately rejected them:
So here's what I came up with: we will specify a very restricted subset of .ini. The restricted subset of .ini that zig supports for this file is trivial to parse. It's also technically a subset of TOML, and intended to be a subset of most .ini parsers in the wild, so existing libraries & tools can be reused, probably.
The filename,
build.zig.iniis intended to imply that it is an appendage tobuild.zigbecause that is exactly what it is. The real, actual file that signifies a zig package is a build.zig, and the existence of this extra file is bonus - it is for the case of declarative information that we want to expose without requiring execution of zig code.In fact, if you look at the example above - libz and libmp3lame - these packages do not actually have a
build.zig.ini. They do not have any declarative information to declare. This is perfectly legal, and has the same repercussions as if you copy-pasted the files directly into VCS.It's very easy to bikeshed this topic (the topic of chosen file format). Feel free to discuss, but please make sure new comments add something to the discussion, not repeat something already said.
Follow-up Issues
@importdependencies directly in build.zig scripts #14279zig buildexecutions are guaranteed to not attempt network access #14280competing proposal: use s-expressions and the file can be namednahbuild.zig.sex.tar.zstextension #14299.tar.xzextension #14300closes #353
closes #368
closes #943