Conceptual topic on the DAL (#270)

* WIP Conceptual topic on the DAL

* import lucid component

* Getting DAL parameters

* repeated info

* tweaks

* Add to TOC

* User tasks for the DAL

* redundancy factor

* clarify data storage and use

* diagram and some tweaks

* Make clear that this is Tezos layer 1 for the first mention of L1

* DAL splits the data into shards

* Capitalize DAL

* DAL nodes share the data through the peer-to-peer network.

* Rework steps based on @iguerNL's comment

* Hard-code these diagrams instead of linking to lucidchart

* More work on this workflow

* When clients publish data

* The maximum number of slots in each block

* Clarify attestation log

* Update steps in DAL workflow and use the same diagram

* Data available for only a short time

* Use the same diagrams

* Clarify pages with ellipses to avoid connection to "n"

* This is an internal detail

* Link to new DAL topic

* Use steps from tutorial

* Link to new env setup page

* Remove because the References section is there

* typo

docs/architecture/data-availability-layer.md

@@ -0,0 +1,157 @@
+---
+title: The Data Availability Layer
+authors: "Tim McMackin"
+last_update:
+  date: 7 February 2024
+---
+
+:::note Experimental
+The Data Availability Layer is an experimental feature that is not yet available on Tezos Mainnet.
+The way the DAL works may change significantly before it is generally available.
+:::
+
+The Data Availability Layer (DAL) is a companion peer-to-peer network for the Tezos blockchain, designed to provide additional data bandwidth to Smart Rollups.
+It allows users to share large amounts of data in a way that is decentralized and permissionless, because anyone can join the network and post and read data on it.
+
+For a tutorial on how to use the DAL, see [Implement a file archive with the DAL and a Smart Rollup](../tutorials/build-files-archive-with-dal).
+
+## How the DAL works
+
+The DAL relies on a network of DAL nodes that distribute data via a peer-to-peer network.
+Layer 1 bakers verify that the data is available.
+After the bakers attest that the data is available, the DAL nodes provide the data to Smart Rollups.
+Smart Rollups that need the data must use it or store it promptly, because it is available only temporarily on the DAL.
+
+The DAL works like this:
+
+1. Users post data to a DAL node.
+1. The DAL node returns a certificate, which includes two parts:
+
+   - The _commitment_ is like a hash of the data but has the additional ability to identify individual shards of the data and reconstruct the original data from a certain percentage of the shards.
+   The number of shards needed depends on how the data is spread across shards, which is controlled by a parameter called the _redundancy factor_.
+   - The _proof_ certifies the length of the data to prevent malicious users from overloading the layer with data.
+
+1. Users post the certificate to Tezos layer 1 via the Octez client.
+1. When the certificate is confirmed in a block, the DAL splits the data into shards and shares it through the peer-to-peer network.
+1. Layer 1 assigns the shards to bakers.
+1. Bakers verify that they are able to download the shards that they are assigned to.
+1. Bakers attest that the data is available in their usual block attestations to layer 1.
+
+   Each Tezos network has a delay of a certain number of blocks known as the _attestation lag_.
+   This number of blocks determines when bakers attest that the data is available and when the data becomes available to Smart Rollups.
+   For example, if a certificate is included in level 100 and the attestation lag is 4, bakers must attest that the data is available in level 104, along with their usual attestations that build on level 103.
+
+   If enough shards are attested in that level, the data becomes available to Smart Rollups at the end of layer 104.
+   If not enough shards are attested in that level, the certificate is considered bogus and the related data is dropped.
+
+1. The Smart Rollup node monitors the blocks and when it sees attested DAL data, it connects to a DAL node to request the data.
+Smart Rollups must store the data if they need it because it is available on the DAL for only a short time.
+
+The overall workflow is summarized in the following figure:
+
+![Overall diagram of the workflow of the Data Availability Layer](/img/architecture/dal-workflow.png)
+<!-- https://lucid.app/lucidchart/cc422278-7319-4a2f-858a-a7b72e1ea3a6/edit -->
+
+## Data structure
+
+Internally, the Data Availability Layer stores information about the available data in layer 1 blocks.
+Each block has several byte-vectors called _slots_, each with a maximum size.
+DAL users can add information about the available data as _pages_ in these slots, as shown in this figure:
+
+![Two example blocks with different DAL slots in use in each](/img/architecture/dal-slots-in-blocks.png)
+<!-- https://lucid.app/lucidchart/46fa8412-8443-4491-82f6-305aafaf85f2/edit -->
+
+The data in a slot is broken into pages to ensure that each piece of data can fit in a single Tezos operation.
+This data must fit in a single operation to allow the Smart Rollup refutation game to work, in which every execution step of the Smart Rollup must be provable to layer 1.
+
+When clients publish data, they must specify which slot to add it to.
+Note that because the DAL is permissionless, clients may try to add data to the same slot in the same block.
+In this case, the first operation in the block takes precedence, which leaves the baker that creates the block in control of which data makes it into the block.
+Other operations that try to add data to the same slot fail.
+
+The number and size of these slots can change.
+Different networks can have different DAL parameters.
+Future changes to the protocol may allow the DAL to resize dynamically based on usage.
+
+## Getting the DAL parameters
+
+Clients can get information about the current DAL parameters from the RPC endpoint `GET /chains/main/blocks/head/context/constants` or the Smart Rollup kernel SDK function `reveal_dal_parameters`.
+These parameters include:
+
+- `number_of_slots`: The maximum number of slots in each block
+- `slot_size`: The size of each slot in bytes
+- `page_size`: The size of each page in bytes
+- `attestation_lag`: The number of blocks after a certificate is published that bakers attest that the data is available; if enough attestations are available in this block, the data becomes available to Smart Rollups
+- `redundancy_factor`: How much redundancy is used to split the data into shards; for example, a redundancy factor of 2 means that half of all shards are enough to reconstruct the original data and a redundancy factor of 4 means that 25% of all shards are required
+
+## Sending data to the DAL
+
+Sending data to the DAL is a two-step process:
+
+1. Send the data to a DAL node by passing it to its `POST /slot` endpoint, as in this example:
+
+   ```bash
+   curl -X POST http://dal-node.example.com:10732/slot --data '"Hello, world!"' -H 'Content-Type: application/json'
+   ```
+
+   The DAL node returns the commitment and proof of the data, as in this abbreviated example:
+
+   ```json
+   {
+     "commitment": "sh1u3tr3YKPDY",
+     "commitment_proof": "8229c63b8e858d9a9"
+   }
+   ```
+
+1. Send an operation to include the commitment and proof in a block by running this Octez client command, using an RPC endpoint for `$ENDPOINT` and an account alias or address for `$MY_ACCOUNT`:
+
+   ```bash
+   commitment="sh1u3tr3YKPDY"
+   proof="8229c63b8e858d9a9"
+   octez-client --endpoint ${ENDPOINT} \
+     publish dal commitment "${commitment}" from ${MY_ACCOUNT} for slot 10 \
+     with proof "${proof}"
+   ```
+
+For an example of sending larger amounts of data, see [Implement a file archive with the DAL and a Smart Rollup](../tutorials/build-files-archive-with-dal).
+
+## Getting data from the DAL
+
+Smart Rollups can use data from the DAL **only after it has been attested by the bakers**.
+Therefore, they cannot access DAL data in the current level, because not enough blocks have elapsed to allow bakers to attest the data.
+
+The latest level that Smart Rollups can access is the current level minus the attestation lag.
+They can access the data in that level with the Smart Rollup kernel SDK function `reveal_dal_page`, which accepts the level, slot, and page to receive, as in this example:
+
+```rust
+let param = host.reveal_dal_parameters();
+
+let sol = host.read_input()?.unwrap();
+
+let target_level = sol.level as usize - param.attestation_lag as usize;
+
+let mut buffer = vec![0u8; param.page_size as usize];
+
+let bytes_read = host.reveal_dal_page(target_level as i32, slot_index, 0, &mut buffer)?;
+
+if 0 < bytes_read {
+    debug_msg!(
+        host,
+        "Attested slot at index {} for level {}: {:?}\n",
+        slot_index,
+        target_level,
+        &buffer.as_slice()[0..10]
+    );
+} else {
+    debug_msg!(
+        host,
+        "No attested slot at index {} for level {}\n",
+        slot_index,
+        target_level
+    );
+}
+```
+
+## Reference
+
+For more information about the DAL, see [DAL overview](https://tezos.gitlab.io/shell/dal_overview.html) in the Octez documentation.

docs/tutorials/build-files-archive-with-dal.md

@@ -77,17 +77,27 @@ You also create a Smart Rollup that listens to the DAL and responds to that data
 The DAL works like this:
 
 1. Users post data to a DAL node.
-1. The DAL node returns a certificate.
-This certificate includes a commitment that the data is available and a proof of the data.
-1. Users post the certificate to layer 1 via the Octez client, which is much cheaper than posting the complete data.
-1. When the certificate is confirmed in a block, layer 1 splits the data into shards and assigns those shards to bakers, who verify that the data is available.
-1. Bakers verify that the data is available and attest that the data is available in their usual block attestations to layer 1.
-They have a certain number of blocks to do so, known as the _attestation lag_, and if they don't by the end of this period, the certificate is considered bogus and the related data is dropped.
-1. Other DAL nodes get the data from the initial DAL node through the peer-to-peer network.
+1. The DAL node returns a certificate, which includes two parts:
+
+   - The _commitment_ is like a hash of the data but has the additional ability to identify individual shards of the data and reconstruct the original data from a certain percentage of the shards.
+   The number of shards needed depends on how the data is spread across shards, which is controlled by a parameter called the _redundancy factor_.
+   - The _proof_ certifies the length of the data to prevent malicious users from overloading the layer with data.
+
+1. Users post the certificate to Tezos layer 1 via the Octez client.
+1. When the certificate is confirmed in a block, the DAL splits the data into shards and shares it through the peer-to-peer network.
+1. Layer 1 assigns the shards to bakers.
+1. Bakers verify that they are able to download the shards that they are assigned to.
+1. Bakers attest that the data is available in their usual block attestations to layer 1.
+
+   Each Tezos network has a delay of a certain number of blocks known as the _attestation lag_.
+   This number of blocks determines when bakers attest that the data is available and when the data becomes available to Smart Rollups.
+   For example, if a certificate is included in level 100 and the attestation lag is 4, bakers must attest that the data is available in level 104, along with their usual attestations that build on level 103.
+
+   If enough shards are attested in that level, the data becomes available to Smart Rollups at the end of layer 104.
+   If not enough shards are attested in that level, the certificate is considered bogus and the related data is dropped.
+
 1. The Smart Rollup node monitors the blocks and when it sees attested DAL data, it connects to a DAL node to request the data.
-1. The Smart Rollup node stores the data in its durable storage, addressed by its hash.
-Smart Rollups must store the data because it is available on the DAL for only a short time.
-1. Users who know the hash of the data can download it from the Smart Rollup node.
+Smart Rollups must store the data if they need it because it is available on the DAL for only a short time.
 
 The overall workflow is summarized in the following figure:
 

docs/tutorials/join-dal-baker.md

@@ -2,7 +2,7 @@
 title: Join the DAL as a baker, in 5 steps
 authors: Tezos core developers
 last_update:
-  date: 24 January 2024
+  date: 7 February 2024
 ---
 
 The Tezos data availability layer (DAL) is a key component for the scalability of Tezos.
@@ -22,8 +22,6 @@ For now, bakers can join the DAL without risking any reward loss, ensuring a smo
 This incentive-free version of the DAL is currently available on the Weeklynet test network.
 In this tutorial you learn how to join Weeklynet as a baker and attest the publication of data on the DAL network.
 
-For an introduction to how the DAL works, see the tutorial [Implement a file archive with the DAL and a Smart Rollup](./build-files-archive-with-dal).
-
 ## Tutorial diagram
 
 In this tutorial, you set up the Octez client and several Octez daemons, including a layer 1 node, a baker, and a DAL baking node.
@@ -32,6 +30,12 @@ The following diagram shows these daemons with a blue background:
 ![A diagram of the DAL architecture, with the daemons that you create in this tutorial highlighted](/img/tutorials/join-dal-baker-overview.png)
 <!-- https://lucid.app/lucidchart/b6b076ec-194c-4011-8e20-fa348bb983f3/edit?page=0_0# -->
 
+## References
+
+- For an overview of the DAL, see [Data Availability Layer](../architecture/data-availability-layer).
+- For an introduction to how the DAL works, see the tutorial [Implement a file archive with the DAL and a Smart Rollup](./build-files-archive-with-dal).
+- For technical information about the DAL, see [Data-Availability Layer](https://tezos.gitlab.io/shell/dal.html) in the Octez documentation.
+
 :::warning
 This tutorial uses a very simple setup running all required daemons on the same machine. In a production environment, we advise against running a DAL attester node under the same IP address than a baker's node because the DAL node may leak the IP address and ease DOS attacks on the baker. See also [the DAL documentation page on baking](https://tezos.gitlab.io/shell/dal_bakers.html).
 :::
@@ -40,9 +44,6 @@ This tutorial uses a very simple setup running all required daemons on the same
 The UX of the DAL components will be subject to changes with the feedback from the testers following this tutorial, so this tutorial will be updated accordingly. Feel free to file issues if it's not up-to-date.
 :::
 
-For more information about the DAL, see [Data-Availability Layer](https://tezos.gitlab.io/shell/dal.html) in the Octez documentation.
-<!-- TODO link to page on DAL when it's available: https://github.com/trilitech/tezos-developer-docs/pull/270 -->
-
 - [Step 1: Get a Weeklynet-compatible Octez version](./join-dal-baker/get-octez)
 - [Step 2: Run an Octez node on Weeklynet](./join-dal-baker/run-node)
 - [Step 3: Set up a baker account on Weeklynet](./join-dal-baker/prepare-account)

sidebars.js

@@ -48,7 +48,7 @@ const sidebars = {
         // },
         'architecture/rpc',
         'architecture/smart-rollups',
-        // 'architecture/data-availability', // TODO
+        'architecture/data-availability-layer',
         {
           type: 'category',
           label: 'Governance',