June 10, 2019 via IRC (#datprotocol)
- pfrazee
- rangermauve
- andrewosh
- mafintosh
- okdistribute
- What's everyone up to
- hyperdrive-daemon
- FUSE dependency, can we make it optional? (Windows support)
- Is the RPC ready for Beaker integration?
- Is the RPC ready for SDK integration?
- UI for debugging networking
- What's everyone up to
- rangermauve: Got PRs into hypercore and hyperdrive for surfacing extensions. Finishing up initial release of Dat SDK. Looked into daemon
- pfrazee: Mostly working n beaker-specific dev while waiting for daemon. Improvements to the site editor, comments sidebar, user management, unwalled garden stuff.
- andrewosh: Playing around with hyperdrive-daemon. CLI fixes, installation issues, building up set of issues to work on. Core functionality is stable once installed. Lots of nice extension points for people to jump in.
- hyperdrive-daemon - FUSE dependency
- rangermauve: Is there a way to decouple FUSE from the daemon, or make it optional for installation
- andrewosh: Going to be necessary for things like Beaker, fuse-native installation should be do inside
hyperdrive setupalong side rest of fuse configuration. the problem now is just that fuse-native is compiled duringnpm i, all fuse-native references are already try-catched - rangermauve: I'll open an issue to discuss. Could take a shot to get it working.
- hyperdrive-daemon - Beaker integration
- pfrazee: Discussed with andrewosh over a call, there are a few APIs that I still need, but we're close to being ready to start implementing against it.
- andrewosh: Daemon only supports 5 most basic hyperdrive methods at the moment
- hyperdrive-daemon - debugging UI
- pfrazee: Also talked about the debugging interface. Beaker has one implemented that might be replaced with one that's native to the daemon. An html-based UI.
- andrewosh: A nice swarm debugging UI is becoming crucial, mathias wants to use it to debvug intermittent utp issues
- pfrazee: Y'all should look at copying my code and building off of it. It's got decent features like filtering logs
- pfrazee: if you want to pull from that UI I built, I can help with that
- mafintosh: but the really nice thing is that 10 has been incredible solid since we started testing it for reals these last two days. All the bugs was in confs etc. All stuff we would have spotted in 10s given the stats ui. now it took us 1 days instead... :)
- okdistribute: yes I agree mafintosh and pfrazee, the biggest complaint I've heard from devs over the past year has been debugging. lso relatedly, 'who has replicated the data?' -- there's been prototype uis for this in many places but still haven't had one readily accessible to users
- rangermauve: Hopefully the better NAT hole punching and having a single process managing the dats will help a bunch with weird errors
- pfrazee: this is the interface that's already in beaker https://usercontent.irccloud-cdn.com/file/h1XXFdiC/Screen%20Recording%202019-07-10%20at%2011.46.49%20AM.mov
- rangermauve: So, pfrazee, are you thinking of doing a pr to the daemon with your slick UI? 😻
- pfrazee: I'll need to work with maf to update it and etc. The interface I have there is only a 5/10 in usefulness, but that's just because the information it's rendering is sometimes too limited
- hyperdrive-daemon - Corestores / SDK
- rangermauve: I thought corestores would be created per data structure and managed with megastore, but the daemon is using one corestore for all the drives, and they add themselves to be seeded.
- andrewosh: originally designed for replication optimization, but it turned out to be more of a pain that it was worth. Relying on a designed but unimplemented update to hypercore-protocol that adds a better capability system. The gist is that peers will exchange proofs that they have certain discovery keys instead of the keys themselves, so key information isn't leaked. that way we can support one peer with N hypercores connecting to another with M, where N is a subset of M, without learning anything about the complement
- rangermauve: how do you identify which discovery key should be used for seeding from within the structure? multifeed has the
defaultfeed which it uses to replicate, any tips on how it can tell the corestore to start seeding that? do you have any other thoughts on how we might standardize around that? - andrewosh: decided to make seeding orthogonal. if you want to seed a key you need to explicitly tell the swarm networker to seed it. with the new model, the "default" feed isn't relevant for replication anymore (only relevant for storage bootstrapping, so you can restart a hyperdrive from the same directory and the default feed will be loaded). Using the default feed to encrypt the whole channel was never ideal for corestore. cap system will be a major improvement there. the cap system, along with a major update to corestore that preserves parent relationships between hypercores (so corestore can do the replication optimization that megastore intended to do, but couldn't without em), are the main networking todos here
- NOISE encryption
- pfrazee: are we switching to the NOISE encryption with the new daemon?
- mafintosh: once we get to updating the capability system, yes. It's currently running without encryption
- rangermauve: I guess NOISE and the capability system will be blocking widespread use of the daemon in the meantime?
- pfrazee: yeah I think realistically the timeline on all of this is, what, 2 months?
- mafintosh: Prob yea, but it’s important that people start testing / building on it now
- pfrazee: yeah right - my read is that the PoC is the point that we can start helping move forward, but there's still going to be a lot of work
- hyperdrive-daemon - hypercore support
- rangermauve: Regarding the SDK I think I'm going to do one release with v10 and hyper swarm before I get to integrating the Daemon and I think I'll help out with getting to work in the meantime. Are y'all okay with hypercore being added to the RPC? I'm cool with pushing that forward
- mafintosh: yea, but it’s low on my list compared to other stuff so we can’t help much with the effort. Other than encouraging support haha
- rangermauve: That's okay. As long as I can get a PR with passing tests merged that's all I need. Having hypercore support is super important for applications building on dat. Not sure of the core store API would be as necessary given how it works. 🤔
- Pfrazee will help with the daemon's debugger
- We'll start integrating and testing the daemon while it gets polished
- rangermauve will help with hypercore integration and getting it to run on Windows