Data availability layer tutorial (#264)

* WIP: Kernel developers tutorial for DAL * ~400 bakers Co-authored-by: Raphaël Cauderlier <[email protected]> * This number is `attestation_lag`. Co-authored-by: Raphaël Cauderlier <[email protected]> * with enough tez. Co-authored-by: Raphaël Cauderlier <[email protected]> * getting tez for free on Co-authored-by: Raphaël Cauderlier <[email protected]> * request new tez Co-authored-by: Raphaël Cauderlier <[email protected]> * typically the ones at index 0, 30, and 31 Co-authored-by: Raphaël Cauderlier <[email protected]> * for this experiment. Co-authored-by: Raphaël Cauderlier <[email protected]> * deposit tez to the Smart Rollup Co-authored-by: Raphaël Cauderlier <[email protected]> * file archive * octez-dal-node run * octez-dal-node run * need comma here * fixes * Intro and learning objectives * Why the DAL? section * typo * Rust prereq * tutorial application * wasm compilation target * Move to MDX file for diagrams * lucid diagram * Enable syntez highlighting for toml * Setting up weeklynet * Move part 1 to a new page * Move parts 2 and 3 to separate pages * jq and xxd required * Clarify objective * Add command example * error message for dual write * Screencap from explorus * Lower case DAL * Typo * Clarify conflicts * Move part 4 to separate page * teztnets.com * Use data instead of file * Correct location for this error * How to see unattested slots * drop beta from explorus * rename index to slot_index * rename index to slot_index * next steps * additional data bandwidth * links for more info * endpoint is /slot * correct slot size * why use observer mode * it is available on the DAL for only a short time. * Add note about experimental DAL * title * The Smart Rollup Installer does not support the DAL Co-authored-by: Thomas Letan <[email protected]> * typo * How to see the hex dump as text --------- Co-authored-by: Thomas Letan <[email protected]> Co-authored-by: Raphaël Cauderlier <[email protected]> Co-authored-by: Thomas Letan <[email protected]>
trilitech · Jan 22, 2024 · c1b310f · c1b310f
1 parent 64b8ae1
commit c1b310f
Show file tree

Hide file tree

Showing 9 changed files with 774 additions and 1 deletion.
diff --git a/docs/tutorials.mdx b/docs/tutorials.mdx
@@ -117,4 +117,12 @@ These tutorials are intended for developers who are familiar with Tezos and want
   link="Start tutorial"
 />
 
+<TutorialCard
+  title="Implement a file archive with the DAL"
+  emoji="🗃️"
+  href="/tutorials/build-files-archive-with-dal"
+  description="Learn how to build a file archive with a Smart Rollup and the data availability layer"
+  link="Start tutorial"
+/>
+
 </TutorialCardContainer>
diff --git a/docs/tutorials/build-files-archive-with-dal.mdx b/docs/tutorials/build-files-archive-with-dal.mdx
@@ -0,0 +1,153 @@
+---
+title: Implementing a file archive with the DAL and a Smart Rollup
+authors: 'Tezos Core Developers'
+last_update:
+  date: 17 January 2024
+---
+
+import LucidDiagram from '@site/src/components/LucidDiagram';
+
+:::note Experimental
+The data availability layer is an experimental feature that is not yet available on Tezos Mainnet.
+The way the DAL works may change significantly before it is generally available.
+:::
+
+The data availability layer (DAL) is a companion peer-to-peer network for the Tezos blockchain, designed to provide additional data bandwidth to Smart Rollups.
+It allows users to share large amounts of data in a way that is decentralized and permissionless, because anyone can join the network and post and read data on it.
+
+In this tutorial, you will set up a file archive that stores and retrieves files with the DAL.
+You will learn:
+
+- How data is organized and shared with the DAL and the reveal data channel
+- How to read data from the DAL in a Smart Rollup
+- How to host a DAL node
+- How to publish data and files with the DAL
+
+Because the DAL is not yet available on Tezos Mainnet, this tutorial uses the [Weeklynet test network](https://teztnets.com/weeklynet-about), which runs on a newer version of the protocol that includes the DAL.
+
+See these links for more information about the DAL:
+
+- For technical information about how the DAL works, see [Data Availability Layer](https://tezos.gitlab.io/shell/dal.html) in the Octez documentation.
+- For more information about the approach for the DAL, see [The Rollup Booster: A Data-Availability Layer for Tezos](https://research-development.nomadic-labs.com/data-availability-layer-tezos.html).
+
+## Prerequisites
+
+This article assumes some familiarity with Smart Rollups.
+If you are new to Smart Rollups, see the tutorial [Deploy a Smart Rollup](./smart-rollup).
+
+### Set up a Weeklynet environment and account
+
+Because Weeklynet requires a specific version of the Octez suite, you can't use most wallet applications and installations of the Octez suite with it.
+Instead, you must set up an environment with a specific version of the Octez suite and use it to create and fund an account.
+Note that Weeklynet is reset every Wednesday, so you must recreate your environment and account after the network resets.
+
+The easiest way to do this is to use the Docker image that is generated each time Weeklynet is reset and recreated.
+As another option, you can build the specific version of the Octez suite locally.
+For instructions, see the Weeklynet page at https://teztnets.com/weeklynet-about.
+
+To set up an environment and account in a Docker container, follow these steps:
+
+1. From the [Weeklynet](https://teztnets.com/weeklynet-about) page, find the Docker command to create a container from the correct Docker image, as in this example:
+
+   ```bash
+   docker run -it --entrypoint=/bin/sh tezos/tezos:master_7f3bfc90_20240116181914
+   ```
+
+   The image tag in this command changes each time the network is reset.
+
+1. Copy the URL of the public RPC endpoint for Weeklynet, such as `https://rpc.weeklynet-2024-01-17.teztnets.com`.
+This endpoint also changes each time the network is reset.
+
+1. For convenience, you may want to set this endpoint as the value of the `ENDPOINT` environment variable.
+
+1. In the container, initialize the Octez client with that endpoint, such as this example:
+
+   ```bash
+   octez-client -E https://rpc.weeklynet-2024-01-17.teztnets.com config init
+   ```
+
+1. Create an account with the command `octez-client gen keys $MY_ACCOUNT`, where `$MY_ACCOUNT` is an alias for your account.
+
+1. Get the public key hash of the new account by running the command `octez-client show address $MY_ACCOUNT`.
+
+1. From the [Weeklynet](https://teztnets.com/weeklynet-about) page, open the Weeklynet faucet and send some tez to the account.
+
+Now you can use this account to deploy Smart Rollups.
+
+### Install Rust
+
+To run this tutorial, install Rust by running the following command.
+The application in this tutorial uses Rust because of its support for WebAssembly (WASM), the language that Smart Rollups use to communicate.
+Rollups can use any language that has WASM compilation support.
+
+```bash
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+```
+
+Then, add WASM as a compilation target for Rust by running this command:
+
+```bash
+rustup target add wasm32-unknown-unknown
+```
+
+You can see other ways of installing Rust at https://www.rust-lang.org.
+
+## Why the DAL?
+
+The DAL has earned the nickname "Rollup Booster" from its ability to address
+the last bottleneck Smart Rollups developers could not overcome without
+sacrificing decentralization: block space. Smart Rollups offload
+*computation* from layer 1, but the transactions that they process still need to
+originate from somewhere.
+
+By default, that "somewhere" is the layer 1 blocks, yet the size of a Tezos
+block is limited to around 500KBytes. In this model, while Smart Rollups do not
+compete for layer 1 gas anymore, they still compete for block space.
+
+{/* Is this info about the reveal data channel needed here? */}
+Additionally, a Smart Rollup can fetch data from an additional source called the
+reveal data channel, which allows them to retrieve arbitrary data.
+The reveal channel is a powerful way to share data, because it allows a Smart Rollup
+operator to post hashes instead of full data files on layer 1. But it is a
+double-edged sword, because nothing enforces the availability of the data in the
+first place. [Solutions exist to address this
+challenge](https://research-development.nomadic-labs.com/introducing-data-availability-committees.html),
+but they are purely off-chain ones, coming with no guarantee from layer 1.
+
+The DAL allows third parties to publish data and have bakers attest that the data is available.
+When enough bakers have attested that the data is available, Smart Rollups can retrieve the data without the need for additional trusted third-parties.
+
+## How the DAL works
+
+In this tutorial, you create a file archive application that allows clients to upload data to the DAL.
+You also create a Smart Rollup that listens to the DAL and responds to that data.
+
+The DAL works like this:
+
+1. Users post data to a DAL node.
+1. The DAL node returns a certificate.
+This certificate includes a commitment that the data is available and a proof of the data.
+1. Users post the certificate to layer 1 via the Octez client, which is much cheaper than posting the complete data.
+1. When the certificate is confirmed in a block, layer 1 splits the data into shards and assigns those shards to bakers, who verify that the data is available.
+1. Bakers verify that the data is available and attest that the data is available in their usual block attestations to layer 1.
+They have a certain number of blocks to do so, known as the _attestation lag_, and if they don't by the end of this period, the certificate is considered bogus and the related data is dropped.
+1. Other DAL nodes get the data from the initial DAL node through the peer-to-peer network.
+1. The Smart Rollup node monitors the blocks and when it sees attested DAL data, it connects to a DAL node to request the data.
+1. The Smart Rollup node stores the data in its durable storage, addressed by its hash.
+Smart Rollups must store the data because it is available on the DAL for only a short time.
+1. Users who know the hash of the data can download it from the Smart Rollup node.
+
+The overall workflow is summarized in the following figure:
+
+<LucidDiagram width="640px" height="480px" src="https://lucid.app/documents/embedded/cc422278-7319-4a2f-858a-a7b72e1ea3a6" id="ljs6hoejIr1H" />
+
+There are many steps in the DAL process, but the most complicated parts (storing and sharing data) are handled automatically by the various daemons in the Octez suite.
+
+:::note The Smart Rollup installer does not support the DAL
+As of today, the Smart Rollup installer does not support the DAL as a
+data availability solution. This means we will need to rely on the reveal
+channel to initialize our Smart Rollup correctly (which is not ideal for a
+decentralized file archive).
+:::
+
+When your environment is ready, get started by going to [Part 1: Getting the DAL parameters](./build-files-archive-with-dal/get-dal-params).
diff --git a/docs/tutorials/build-files-archive-with-dal/get-dal-params.mdx b/docs/tutorials/build-files-archive-with-dal/get-dal-params.mdx
@@ -0,0 +1,173 @@
+---
+title: "Part 1: Getting the DAL parameters"
+authors: 'Tezos Core Developers'
+last_update:
+  date: 17 January 2024
+---
+
+import LucidDiagram from '@site/src/components/LucidDiagram';
+
+The data availability layer stores information about the available data in layer 1 blocks.
+Each block has several byte-vectors called _slots_, each with a maximum size.
+DAL users can add information about the available data as _pages_ in these slots, as shown in this figure:
+
+<LucidDiagram width="640px" height="480px" src="https://lucid.app/documents/embedded/46fa8412-8443-4491-82f6-305aafaf85f2" id="Hxs62lrO0C4d" />
+
+The data in a slot is broken into pages to ensure that each piece of data can fit in a single Tezos operation.
+This data must fit in a single operation to allow the Smart Rollup refutation game to work, in which every execution step of the Smart Rollup must be provable to layer 1.
+{/* TODO link to Smart Rollup topic for more info on the refutation game */}
+
+When clients add data, they must specify which slot to add it to.
+Note that because the DAL is permissionless, clients may try to add data to the same slot in the same block.
+In this case, the first operation in the block takes precedence, which leaves the baker that creates the block in control of which data makes it into the block.
+Other operations that try to add data to the same slot fail.
+
+The number and size of these slots can change.
+Different networks can have different DAL parameters.
+Future changes to the protocol may allow the DAL to resize dynamically based on usage.
+
+Therefore, clients must get information about the DAL before sending data to it.
+In these steps, you set up a simple Smart Rollup to get the current DAL parameters and print them to the log.
+
+## Prerequisites
+
+Before you begin, make sure that you have installed the prerequisites and set up an environment and an account as described in [Implementing a File Archive with the DAL and a Smart Rollup](../build-files-archive-with-dal).
+
+## Fetching the DAL parameters in a kernel
+
+To get the DAL parameters, you can use built-in functions in the Tezos [Rust SDK](https://crates.io/crates/tezos-smart-rollup).
+
+1. In a folder for your project, create a file named `Cargo.toml` with this code:
+
+   ```toml
+   [package]
+   name = "files_archive"
+   version = "0.1.0"
+   edition = "2021"
+
+   [lib]
+   crate-type = ["cdylib", "lib"]
+
+   [dependencies]
+   tezos-smart-rollup = { version = "0.2.2", features = [ "proto-alpha" ] }
+   ```
+
+   As a reminder, the kernel of a Smart Rollup is a WASM program.
+   The `proto-alpha` feature is necessary to get access to the functions specific to the DAL.
+
+1. Create a file named `src/lib.rs` to be the kernel.
+
+1. In the `src/lib.rs` file, add this code:
+
+   ```rust
+   use tezos_smart_rollup::{kernel_entry, prelude::*};
+
+   pub fn entry<R: Runtime>(host: &mut R) {
+       let param = host.reveal_dal_parameters();
+       debug_msg!(host, "{:?}\n", param);
+   }
+
+   kernel_entry!(entry);
+   ```
+
+   This function gets the DAL parameters of the currently connected network and prints them to the log.
+
+1. Build the kernel:
+
+   ```bash
+   cargo build --release --target wasm32-unknown-unknown
+   cp target/wasm32-unknown-unknown/release/files_archive.wasm .
+   ```
+
+1. Get the installer kernel:
+
+   ```bash
+   cargo install tezos-smart-rollup-installer
+   export PATH="${HOME}/.local/bin:${PATH}"
+   smart-rollup-installer get-reveal-installer \
+       -P _rollup_node/wasm_2_0_0 \
+       -u files_archive.wasm \
+       -o installer.hex
+   ```
+
+Now the Smart Rollup is ready to deploy.
+
+## Deploying the Smart Rollup and starting a node
+
+Follow these steps to deploy the Smart Rollup to Weeklynet and start a node:
+
+1. Run this command to deploy the Smart Rollup, replacing `$MY_ACCOUNT` with your account alias and `$ENDPOINT` with the RPC endpoint:
+
+   ```bash
+   octez-client --endpoint ${ENDPOINT} \
+       originate smart rollup files_archive from ${MY_ACCOUNT} \
+       of kind wasm_2_0_0 of type unit with kernel "$(cat installer.hex)" \
+       --burn-cap 2.0 --force
+   ```
+
+1. Start the node with this command:
+
+   ```bash
+   octez-smart-rollup-node --endpoint ${ENDPOINT} \
+       run observer for files_archive with operators \
+       --data-dir ./_rollup_node --log-kernel-debug
+   ```
+
+   For simplicity, this command runs the Smart Rollup in observer mode, which does not require a stake of 10,000 tez to publish commitments.
+
+1. Open a new terminal window and run this command to watch the node's log:
+
+   ```bash
+   tail -F _rollup_node/kernel.log
+   ```
+
+The log prints the current DAL parameters, as in this example:
+
+```
+RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 }
+RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 }
+RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 }
+RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 }
+RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 }
+```
+
+These parameters are:
+
+- `number_of_slots`: The number of slots in each block
+- `slot_size`: The size of each slot in bytes
+- `page_size`: The size of each page in bytes
+- `attestation_lag`: The number of subsequent blocks in which bakers can attest that the data is available; if enough attestations are available by the time this number of blocks have been created, the data becomes available to Smart Rollups
+
+## Setting up a deployment script
+
+In later parts of this tutorial, you will update and redeploy the Smart Rollup multiple times.
+To simplify the process, you can use this script.
+To use it, pass the alias of your account in the Octez client:
+
+```bash
+#!/usr/bin/bash
+
+alias="${1}"
+
+set -e
+
+cargo build --release --target wasm32-unknown-unknown
+
+rm -rf _rollup_node
+
+cp target/wasm32-unknown-unknown/release/files_archive.wasm .
+
+smart-rollup-installer get-reveal-installer -P _rollup_node/wasm_2_0_0 \
+  -u files_archive.wasm -o installer.hex
+
+octez-client --endpoint ${ENDPOINT} \
+  originate smart rollup files_archive from "${alias}" of kind wasm_2_0_0 \
+  of type unit with kernel "$(cat installer.hex)" --burn-cap 2.0 --force
+
+octez-smart-rollup-node --endpoint ${ENDPOINT} \
+  run observer for files_archive with operators --data-dir _rollup_node \
+  --dal-node http://localhost:10732 --log-kernel-debug
+```
+
+In the next section, you will get information about the state of slots in the DAL.
+See [Part 2: Getting slot information](./get-slot-info).