You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Dagger which aims to implement a p2p storage network that relies on proofs of storage and scales across a wide variety of devices, requires a focused research phase to answer outstanding questions about the viability and real life requirements of such a solution.
This document attempts to give a high level overview of the research directions and provide a very rough outline of the topics and timelines. We expect this roadmap to evolve as the research progresses and we gain more clarity on the overall approach.
Components
Dagger is currently outlined as a two step mechanism - a non-interactive "Proof of Storage-Time" (PoSt) scheme, potentially based around either a PDP or PoR scheme; and an interactive scheme that has the double purpose of preventing withholding and blackmailing attacks as well as complementing the PoSt scheme.
The current outline is preliminary and rests on circular assumptions around the capabilities and security of the proof of storage schemes as well as networks limitations/capabilities, economic incentives, security and others.
It is the goal of this research to validate and (potentially drastically) improve on the approach outlined in Dagger proposal and Robust Proofs of Data Possession and Retrievability documents. Our research is focused on finding the best suitable solution if it already exists, but also to improve on the state of art where possible and/or no other viable solutions is identified.
Proofs of Storage
Proofs of storage are an integral part of the current Dagger design. They are crucial at providing both security of data as well as an environment capable of sustaining economic incentives.
The topic of proofs of storage has been well researched in academia and in the context of centralized storage providers (Microsoft, Amazon, Google, etc...)
Our objective is to identify a solution that will provide strong data security, persistence and availability guarantees. So far the two most well known approaches are "Provable Data Possession" and "Proofs of Retrievability"; a third solution known as "Proofs of Replication" is also relevant, but so far seems the least suitable for our setup. Nevertheless, we're considering any approach that will yield the expected properties.
Desired Outcome:
Determine the best possible schemes in terms of
Security - how secure the overall schemes is in a trustless p2p network
Performance - how fast it performs in a real setting
when applied to large files/data sets (GB/TB sizes)
when used on fast changing data
and under what assumptions
in a p2p network conformed by many disparate devices
Compactness - how large are the proofs both at the storage origin as well as the "public verifier" perspective
we need compact proofs for optimal on-the-wire performance
we need compact proofs due to potential on chain storage limitations
can we aggregate the proofs?
we need compact proofs due to cost-of-storage for clients, verifiers, etc...
preferably stateless to prevent state management complexity
The Dagger p2p network is expected to be comprised of many disparate device types. Of particular concern are devices with reduced capacity - be it processing, power, network, storage or all of the above. This are phones, IoT or wearable devices that might want to interact with the network directly or indirectly (by means of a third party app, such as a chat app, for example).
We expect Dagger (and most p2p networks in the mobile era) to be comprised of this reduced capacity devices and as such, we want to build our network from the ground up with this devices in mind.
A potential direction we want to explore in this context is the idea of "Adaptive Devices". The main theme of adaptive devices is to scale up or down depending on the current conditions that the device is operating under. For example, a phone might not be able to operate as a full fledged node while on the go - due to bandwidth and power constraints, but as soon as it's "plugged in the wall", it can switch back to a full node, potentially serving other nodes in the network and thus increasing the networks overall capacity.
There are many more aspects that we expect to revice as part of our research - for example, UDP based protocols and their viability for mobile and IoT devices; discovery mechanism such as DHTs; gossip and mesh networks; anonymity and metadata leakage; security and overall network resilience, etc...
Another crucial area of interest is node anonymity. For example, the Robust Proofs of Data Possession and Retrievability approach, assumes that verifiers are anonymous and/or at the very least aren't identifiable as verifiers. In this context, anonymity, privacy and information/metadata leakage is crucial to realize the scheme (however, as a backup and/or complement to this, there are other approaches in the context of PDPs and/or PoRs that might bypass this requirement).
Desired Outcome:
Determine the best network topology and configuration to support
Fast and frequent message gossiping (i.e. to gossip proofs of storage)
Efficient, secure and lightweight discovery mechanisms
Capable of supporting a wide range of device types and
Network topologies (NAT, firewalls, etc) and
Changing connectivity (mobile devices)
Understand what levels of churn we should expect and what can be tolerated by this types of networks - i.e. networks with potentially large amounts of mobile devices
Explore and develop further the ideas behind "Adaptive Devices"
From the standpoint of the Dagger network, consensus is maximally isolated and where possible treated as a blackbox.
This is done for several reasons:
Complexity - consensus entails a high level of complexity that can significantly increase the project's scope, this is something we want to avoid.
Specificity - Dagger's consensus requirements are modest and generic. They mostly revolve around registration, coordination of validator and storage nodes and adjudication of penalties and rewards. Of particular concern are micropayments used in bandwidth payments and potentially the most tricky requirement as it relies on fast and lightweight consensus interactions.
Plugability - isolating consensus gives us the possibility of plugin in many different consensus engines without being tied to any specific implementation. This has advantages in terms of developing our solution as it allows us to "pick what works" during our R&D phase. It also seems like developing all or most of our consensus related logic as EVM based contracts gives us both the best level of interoperability with many consensus engines as well as the lowest level of implementation complexity, since contracts can be implemented in a variety of high level languages.
It's worth understanding that the requirements of fast and lightweight consensus are crucial for the success of the project as this is potentially the biggest bottleneck that we will deal with. Our definition of fast and lightweight is concerned with latency - how long does it take to process transactions and runtime requirements - what does it take to run a secure node. The economic cost of execution is also important as it should be affordable to be viable.
Thus, our research is centered around identifying fast and lightweight consensus and scalability mechanisms from a latency and cost perspective.
Desired Outcome:
Identify a practical way forward to satisfy our immediate requirements such as
Fast, lightweight and cheap consensus interactions
Where no other alternatives exist, identify potential scaling solutions such as L2 (rollups or state channels)
Devise a suitable architecture to interact with the consensus engine
For example, do all required nodes (storage and verifiers) write to the chain or are "aggregator" nodes required due to cost and throughput?
Identify what can be offloaded from consensus interactions and what is consensus bound
Topics of Interest:
Mechanism design as it concerns the verification and incentives schemes
Smart contract architecture of the Dagger two step verification model
Consensus mechanisms and how they fit the Dagger model
Scaling solutions such as rollups (optimistic and ZK), state/payment channels
On-chain vs off-chain execution and security implications
For example data availability and subsequent funds lockup
PoC Opportunities:
Design and implement the registration, coordination and adjudication logic
Design and implement state/payment channels for bandwidth micropayments
Experiment with different scaling solutions and how they would perform in a real life setting
Incentives
Incentives are a crucial part of the system and play a big role in shaping interactions in the network. A core intuition that permeates our design is that without properly aligned incentives a network such as Dagger cannot exist without specializing around some specific interaction. We've outline some of this intuitions in our Incentives Rationale document.
Our current assumption around incentives is that we can scope all interactions to be 1 to 1 and thus reduce the complexity of the incentives mechanism. It is important that we validate this assumptions both through literature review and by simulating some of the expected emergent properties.
Desired Outcome:
Validate our core assumptions around simple 1 to 1 interactions
Can demand and supply steer the network to optimally utilize its resources?
Will the expected emergent properties materialize?
Organic CDNs through ephemeral and caching nodes
Opportunistic providing and tit-for-tat settlements
Are there any others that we can anticipate?
Identify and validate workarounds for the zero-entry problem
Topics of Interest:
Incentives mechanism design
Emergent properties in p2p networks
Cryptoeconomic models
Spam resistance through economic incentives and penalties
PoC Opportunities:
There are a lot of opportunities for simulations here
Simulate the expected emergent properties
Simulate different adversarial conditions that will break our current assumptions
Deliverables
Our goal with the research phase is to
Increase the team's understanding of the problem space
Identify the best overall solutions for each domain
Produce usable PoCs and actionable simulation results
Improve on the state of the art where possible or required
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Codex Storage
-
Dagger Research Scope
Dagger which aims to implement a p2p storage network that relies on proofs of storage and scales across a wide variety of devices, requires a focused research phase to answer outstanding questions about the viability and real life requirements of such a solution.
This document attempts to give a high level overview of the research directions and provide a very rough outline of the topics and timelines. We expect this roadmap to evolve as the research progresses and we gain more clarity on the overall approach.
Components
Dagger is currently outlined as a two step mechanism - a non-interactive "Proof of Storage-Time" (PoSt) scheme, potentially based around either a PDP or PoR scheme; and an interactive scheme that has the double purpose of preventing withholding and blackmailing attacks as well as complementing the PoSt scheme.
The current outline is preliminary and rests on circular assumptions around the capabilities and security of the proof of storage schemes as well as networks limitations/capabilities, economic incentives, security and others.
It is the goal of this research to validate and (potentially drastically) improve on the approach outlined in Dagger proposal and Robust Proofs of Data Possession and Retrievability documents. Our research is focused on finding the best suitable solution if it already exists, but also to improve on the state of art where possible and/or no other viable solutions is identified.
Proofs of Storage
Proofs of storage are an integral part of the current Dagger design. They are crucial at providing both security of data as well as an environment capable of sustaining economic incentives.
The topic of proofs of storage has been well researched in academia and in the context of centralized storage providers (Microsoft, Amazon, Google, etc...)
Our objective is to identify a solution that will provide strong data security, persistence and availability guarantees. So far the two most well known approaches are "Provable Data Possession" and "Proofs of Retrievability"; a third solution known as "Proofs of Replication" is also relevant, but so far seems the least suitable for our setup. Nevertheless, we're considering any approach that will yield the expected properties.
Desired Outcome:
Determine the best possible schemes in terms of
Topics of Interest:
PoC Opportunities:
Network
The Dagger p2p network is expected to be comprised of many disparate device types. Of particular concern are devices with reduced capacity - be it processing, power, network, storage or all of the above. This are phones, IoT or wearable devices that might want to interact with the network directly or indirectly (by means of a third party app, such as a chat app, for example).
We expect Dagger (and most p2p networks in the mobile era) to be comprised of this reduced capacity devices and as such, we want to build our network from the ground up with this devices in mind.
A potential direction we want to explore in this context is the idea of "Adaptive Devices". The main theme of adaptive devices is to scale up or down depending on the current conditions that the device is operating under. For example, a phone might not be able to operate as a full fledged node while on the go - due to bandwidth and power constraints, but as soon as it's "plugged in the wall", it can switch back to a full node, potentially serving other nodes in the network and thus increasing the networks overall capacity.
There are many more aspects that we expect to revice as part of our research - for example, UDP based protocols and their viability for mobile and IoT devices; discovery mechanism such as DHTs; gossip and mesh networks; anonymity and metadata leakage; security and overall network resilience, etc...
Another crucial area of interest is node anonymity. For example, the Robust Proofs of Data Possession and Retrievability approach, assumes that verifiers are anonymous and/or at the very least aren't identifiable as verifiers. In this context, anonymity, privacy and information/metadata leakage is crucial to realize the scheme (however, as a backup and/or complement to this, there are other approaches in the context of PDPs and/or PoRs that might bypass this requirement).
Desired Outcome:
Determine the best network topology and configuration to support
Topics of Interest:
PoC Opportunities:
Consensus Interaction
From the standpoint of the Dagger network, consensus is maximally isolated and where possible treated as a blackbox.
This is done for several reasons:
Complexity - consensus entails a high level of complexity that can significantly increase the project's scope, this is something we want to avoid.
Specificity - Dagger's consensus requirements are modest and generic. They mostly revolve around registration, coordination of validator and storage nodes and adjudication of penalties and rewards. Of particular concern are micropayments used in bandwidth payments and potentially the most tricky requirement as it relies on fast and lightweight consensus interactions.
Plugability - isolating consensus gives us the possibility of plugin in many different consensus engines without being tied to any specific implementation. This has advantages in terms of developing our solution as it allows us to "pick what works" during our R&D phase. It also seems like developing all or most of our consensus related logic as EVM based contracts gives us both the best level of interoperability with many consensus engines as well as the lowest level of implementation complexity, since contracts can be implemented in a variety of high level languages.
It's worth understanding that the requirements of fast and lightweight consensus are crucial for the success of the project as this is potentially the biggest bottleneck that we will deal with. Our definition of fast and lightweight is concerned with latency - how long does it take to process transactions and runtime requirements - what does it take to run a secure node. The economic cost of execution is also important as it should be affordable to be viable.
Thus, our research is centered around identifying fast and lightweight consensus and scalability mechanisms from a latency and cost perspective.
Desired Outcome:
Identify a practical way forward to satisfy our immediate requirements such as
Topics of Interest:
PoC Opportunities:
Incentives
Incentives are a crucial part of the system and play a big role in shaping interactions in the network. A core intuition that permeates our design is that without properly aligned incentives a network such as Dagger cannot exist without specializing around some specific interaction. We've outline some of this intuitions in our Incentives Rationale document.
Our current assumption around incentives is that we can scope all interactions to be 1 to 1 and thus reduce the complexity of the incentives mechanism. It is important that we validate this assumptions both through literature review and by simulating some of the expected emergent properties.
Desired Outcome:
Validate our core assumptions around simple 1 to 1 interactions
Topics of Interest:
PoC Opportunities:
There are a lot of opportunities for simulations here
Deliverables
Our goal with the research phase is to
Beta Was this translation helpful? Give feedback.
All reactions