Swappable Backends

[drkibitz]

Did @mbrevoort or anyone else discuss this with the core node team? It makes sense, and IMO the appropriate step would be to have a frontend server api, and a backend store api. This way the backend store API could be swapped out for filesystem, couchdb, etc..

[drkibitz]

This approach could actually be in favor over, yet integrated with https://github.com/isaacs/npmjs.org

[drkibitz]

Could https://github.com/mbrevoort/node-reggie/blob/master/lib/data.js be considered the backend store? Sorry I haven't had the chance to see how all this is coupled together.

[bajtos]

Pluggable backends require more work that just replacing CouchDB with our data.js, becase the protocol used between the npm client and the registry is heavily based on the way how CouchDB operates. In fact, the public registry is implemented as a bunch of stored procedures and rewrite rules in CouchDB, plus a part of the work is left for the npm client.

There was another project started on github recently that aimed to provide different backends (filesystem, MongoDB), but I can't find it now.

In the ideal world, you would have a nice API that is agnostic of backend-store and that would be easy to implement for both clients and servers. To get there, you will have to rewrite both the npm client and the registry server, which is a lot of work.

Anyway, I haven't discussed this matter with @isaacs (the maintainer of npm registry implementation), feel free to take the initiative here.

[isaacs]

In the ideal world, you would have a nice API that is agnostic of backend-store and that would be easy to implement for both clients and servers.

I agree with the intent, but disagree somewhat with the specific sentiment here. In an ideal world, I would have known from the start that it would basically always be a CouchDB-like interface, and played more to CouchDB's strengths from the outset.

The benefit of "different backends" is somewhat lost on me. At best, it is equivalent. At worst, it is not. This seems like added complexity for (hopefully) no discernible change.

A better approach would be for a database API that mimics the CouchDB API. There are some interesting developments in the LevelDB community around this. I'm told that @mikeal is working on one with the explicit intent of supporting the npm registry.

[mikeal]

yup, i've got a data model close enough to CouchDB that I can replicate all the data but i'm taking a very different approach to indexing and a lot of other stuff.

the real "npm support" would be done in re-creating the REST interface without _list and _show functions, which would be much faster and cleaner and probably easier than _list and _show functions. very little of the registry uses raw CouchDB REST interfaces, most of it is a custom API written with couchdb's url rewriter and _list and _show.

[mbrevoort]

I think the most needed feature is for the npm client to support resolving packages from multiple registries. If it can, then registries don't necessarily need to synchronize beyond durability, and the backend-store API and details can be encapsulated away. The registries would then just need to implement a known standard, preferable REST like API.

For example, I would love to back a private registry with something like Amazon S3 that in and of itself is very reliable and durable. I could then stand up a basically stateless node process (or whatever) in front of it that implements the registry API for publishing, searching, returning packages, etc. This could be very turnkey and very durable. And with the npm client able to support multiple registries, I can resolve, search and publish public packages to the public registry, while maintaining my private registry that just contains my private modules.

If some registry implementation wants to implement synchronization in standard way or between instances of itself then great but that misses the crux of the problem IMHO. On our current trajectory, it's not feasible to assume that one registry or any copy of the registry MUST contain all of the things.

[mikeal]

the reality is that considerably less than 1% of people need to resolve from multiple registries and the ones that do are usually in a position to run their own registry if it were just a simple node process. if the registry were written in node, using leveldb for storage, it would be trivial to write a registry that cross-resolved from other registries.

[mbrevoort]

@mikeal "considerably less than 1%" is short-sighted and a gross underestimation.

I'm in a position to run a private + full blow copy of the public registry, we do and it is costly and a PITA.

@isaac (you are mentioned)[http://blog.nodejitsu.com/npm-innovation-through-modularity] in the context of the npm roadmap and "Multiple registries". Do you see the npm client having the ability to support multiple registries or requiring a proxy?

[mikeal]

caveat: running your own registry should not require you to host all the binaries. if it didn't it would be rather simple to run your own.

[mikeal]

@mbrevoort running today's registry is only accessible for a few people. i'm saying that it should be more accessible, and simple, and it should be easy to add these kinds of features in the registry rather than adding them to the client.

[mbrevoort]

@mikeal I can see your point and changing the behavior of the client is more intrusive and impactful. But I think that allowing the client to directly resolve from different registries is a better long term solution that would foster a whole other round of interesting innovation.