diff --git a/docs/tutorials.mdx b/docs/tutorials.mdx index 8149c111a..a87efc5a5 100644 --- a/docs/tutorials.mdx +++ b/docs/tutorials.mdx @@ -117,4 +117,12 @@ These tutorials are intended for developers who are familiar with Tezos and want link="Start tutorial" /> + + diff --git a/docs/tutorials/build-files-archive-with-dal.mdx b/docs/tutorials/build-files-archive-with-dal.mdx new file mode 100644 index 000000000..121a98a0f --- /dev/null +++ b/docs/tutorials/build-files-archive-with-dal.mdx @@ -0,0 +1,153 @@ +--- +title: Implementing a file archive with the DAL and a Smart Rollup +authors: 'Tezos Core Developers' +last_update: + date: 17 January 2024 +--- + +import LucidDiagram from '@site/src/components/LucidDiagram'; + +:::note Experimental +The data availability layer is an experimental feature that is not yet available on Tezos Mainnet. +The way the DAL works may change significantly before it is generally available. +::: + +The data availability layer (DAL) is a companion peer-to-peer network for the Tezos blockchain, designed to provide additional data bandwidth to Smart Rollups. +It allows users to share large amounts of data in a way that is decentralized and permissionless, because anyone can join the network and post and read data on it. + +In this tutorial, you will set up a file archive that stores and retrieves files with the DAL. +You will learn: + +- How data is organized and shared with the DAL and the reveal data channel +- How to read data from the DAL in a Smart Rollup +- How to host a DAL node +- How to publish data and files with the DAL + +Because the DAL is not yet available on Tezos Mainnet, this tutorial uses the [Weeklynet test network](https://teztnets.com/weeklynet-about), which runs on a newer version of the protocol that includes the DAL. + +See these links for more information about the DAL: + +- For technical information about how the DAL works, see [Data Availability Layer](https://tezos.gitlab.io/shell/dal.html) in the Octez documentation. +- For more information about the approach for the DAL, see [The Rollup Booster: A Data-Availability Layer for Tezos](https://research-development.nomadic-labs.com/data-availability-layer-tezos.html). + +## Prerequisites + +This article assumes some familiarity with Smart Rollups. +If you are new to Smart Rollups, see the tutorial [Deploy a Smart Rollup](./smart-rollup). + +### Set up a Weeklynet environment and account + +Because Weeklynet requires a specific version of the Octez suite, you can't use most wallet applications and installations of the Octez suite with it. +Instead, you must set up an environment with a specific version of the Octez suite and use it to create and fund an account. +Note that Weeklynet is reset every Wednesday, so you must recreate your environment and account after the network resets. + +The easiest way to do this is to use the Docker image that is generated each time Weeklynet is reset and recreated. +As another option, you can build the specific version of the Octez suite locally. +For instructions, see the Weeklynet page at https://teztnets.com/weeklynet-about. + +To set up an environment and account in a Docker container, follow these steps: + +1. From the [Weeklynet](https://teztnets.com/weeklynet-about) page, find the Docker command to create a container from the correct Docker image, as in this example: + + ```bash + docker run -it --entrypoint=/bin/sh tezos/tezos:master_7f3bfc90_20240116181914 + ``` + + The image tag in this command changes each time the network is reset. + +1. Copy the URL of the public RPC endpoint for Weeklynet, such as `https://rpc.weeklynet-2024-01-17.teztnets.com`. +This endpoint also changes each time the network is reset. + +1. For convenience, you may want to set this endpoint as the value of the `ENDPOINT` environment variable. + +1. In the container, initialize the Octez client with that endpoint, such as this example: + + ```bash + octez-client -E https://rpc.weeklynet-2024-01-17.teztnets.com config init + ``` + +1. Create an account with the command `octez-client gen keys $MY_ACCOUNT`, where `$MY_ACCOUNT` is an alias for your account. + +1. Get the public key hash of the new account by running the command `octez-client show address $MY_ACCOUNT`. + +1. From the [Weeklynet](https://teztnets.com/weeklynet-about) page, open the Weeklynet faucet and send some tez to the account. + +Now you can use this account to deploy Smart Rollups. + +### Install Rust + +To run this tutorial, install Rust by running the following command. +The application in this tutorial uses Rust because of its support for WebAssembly (WASM), the language that Smart Rollups use to communicate. +Rollups can use any language that has WASM compilation support. + +```bash +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +``` + +Then, add WASM as a compilation target for Rust by running this command: + +```bash +rustup target add wasm32-unknown-unknown +``` + +You can see other ways of installing Rust at https://www.rust-lang.org. + +## Why the DAL? + +The DAL has earned the nickname "Rollup Booster" from its ability to address +the last bottleneck Smart Rollups developers could not overcome without +sacrificing decentralization: block space. Smart Rollups offload +*computation* from layer 1, but the transactions that they process still need to +originate from somewhere. + +By default, that "somewhere" is the layer 1 blocks, yet the size of a Tezos +block is limited to around 500KBytes. In this model, while Smart Rollups do not +compete for layer 1 gas anymore, they still compete for block space. + +{/* Is this info about the reveal data channel needed here? */} +Additionally, a Smart Rollup can fetch data from an additional source called the +reveal data channel, which allows them to retrieve arbitrary data. +The reveal channel is a powerful way to share data, because it allows a Smart Rollup +operator to post hashes instead of full data files on layer 1. But it is a +double-edged sword, because nothing enforces the availability of the data in the +first place. [Solutions exist to address this +challenge](https://research-development.nomadic-labs.com/introducing-data-availability-committees.html), +but they are purely off-chain ones, coming with no guarantee from layer 1. + +The DAL allows third parties to publish data and have bakers attest that the data is available. +When enough bakers have attested that the data is available, Smart Rollups can retrieve the data without the need for additional trusted third-parties. + +## How the DAL works + +In this tutorial, you create a file archive application that allows clients to upload data to the DAL. +You also create a Smart Rollup that listens to the DAL and responds to that data. + +The DAL works like this: + +1. Users post data to a DAL node. +1. The DAL node returns a certificate. +This certificate includes a commitment that the data is available and a proof of the data. +1. Users post the certificate to layer 1 via the Octez client, which is much cheaper than posting the complete data. +1. When the certificate is confirmed in a block, layer 1 splits the data into shards and assigns those shards to bakers, who verify that the data is available. +1. Bakers verify that the data is available and attest that the data is available in their usual block attestations to layer 1. +They have a certain number of blocks to do so, known as the _attestation lag_, and if they don't by the end of this period, the certificate is considered bogus and the related data is dropped. +1. Other DAL nodes get the data from the initial DAL node through the peer-to-peer network. +1. The Smart Rollup node monitors the blocks and when it sees attested DAL data, it connects to a DAL node to request the data. +1. The Smart Rollup node stores the data in its durable storage, addressed by its hash. +Smart Rollups must store the data because it is available on the DAL for only a short time. +1. Users who know the hash of the data can download it from the Smart Rollup node. + +The overall workflow is summarized in the following figure: + + + +There are many steps in the DAL process, but the most complicated parts (storing and sharing data) are handled automatically by the various daemons in the Octez suite. + +:::note The Smart Rollup installer does not support the DAL +As of today, the Smart Rollup installer does not support the DAL as a +data availability solution. This means we will need to rely on the reveal +channel to initialize our Smart Rollup correctly (which is not ideal for a +decentralized file archive). +::: + +When your environment is ready, get started by going to [Part 1: Getting the DAL parameters](./build-files-archive-with-dal/get-dal-params). diff --git a/docs/tutorials/build-files-archive-with-dal/get-dal-params.mdx b/docs/tutorials/build-files-archive-with-dal/get-dal-params.mdx new file mode 100644 index 000000000..03ea7292f --- /dev/null +++ b/docs/tutorials/build-files-archive-with-dal/get-dal-params.mdx @@ -0,0 +1,173 @@ +--- +title: "Part 1: Getting the DAL parameters" +authors: 'Tezos Core Developers' +last_update: + date: 17 January 2024 +--- + +import LucidDiagram from '@site/src/components/LucidDiagram'; + +The data availability layer stores information about the available data in layer 1 blocks. +Each block has several byte-vectors called _slots_, each with a maximum size. +DAL users can add information about the available data as _pages_ in these slots, as shown in this figure: + + + +The data in a slot is broken into pages to ensure that each piece of data can fit in a single Tezos operation. +This data must fit in a single operation to allow the Smart Rollup refutation game to work, in which every execution step of the Smart Rollup must be provable to layer 1. +{/* TODO link to Smart Rollup topic for more info on the refutation game */} + +When clients add data, they must specify which slot to add it to. +Note that because the DAL is permissionless, clients may try to add data to the same slot in the same block. +In this case, the first operation in the block takes precedence, which leaves the baker that creates the block in control of which data makes it into the block. +Other operations that try to add data to the same slot fail. + +The number and size of these slots can change. +Different networks can have different DAL parameters. +Future changes to the protocol may allow the DAL to resize dynamically based on usage. + +Therefore, clients must get information about the DAL before sending data to it. +In these steps, you set up a simple Smart Rollup to get the current DAL parameters and print them to the log. + +## Prerequisites + +Before you begin, make sure that you have installed the prerequisites and set up an environment and an account as described in [Implementing a File Archive with the DAL and a Smart Rollup](../build-files-archive-with-dal). + +## Fetching the DAL parameters in a kernel + +To get the DAL parameters, you can use built-in functions in the Tezos [Rust SDK](https://crates.io/crates/tezos-smart-rollup). + +1. In a folder for your project, create a file named `Cargo.toml` with this code: + + ```toml + [package] + name = "files_archive" + version = "0.1.0" + edition = "2021" + + [lib] + crate-type = ["cdylib", "lib"] + + [dependencies] + tezos-smart-rollup = { version = "0.2.2", features = [ "proto-alpha" ] } + ``` + + As a reminder, the kernel of a Smart Rollup is a WASM program. + The `proto-alpha` feature is necessary to get access to the functions specific to the DAL. + +1. Create a file named `src/lib.rs` to be the kernel. + +1. In the `src/lib.rs` file, add this code: + + ```rust + use tezos_smart_rollup::{kernel_entry, prelude::*}; + + pub fn entry(host: &mut R) { + let param = host.reveal_dal_parameters(); + debug_msg!(host, "{:?}\n", param); + } + + kernel_entry!(entry); + ``` + + This function gets the DAL parameters of the currently connected network and prints them to the log. + +1. Build the kernel: + + ```bash + cargo build --release --target wasm32-unknown-unknown + cp target/wasm32-unknown-unknown/release/files_archive.wasm . + ``` + +1. Get the installer kernel: + + ```bash + cargo install tezos-smart-rollup-installer + export PATH="${HOME}/.local/bin:${PATH}" + smart-rollup-installer get-reveal-installer \ + -P _rollup_node/wasm_2_0_0 \ + -u files_archive.wasm \ + -o installer.hex + ``` + +Now the Smart Rollup is ready to deploy. + +## Deploying the Smart Rollup and starting a node + +Follow these steps to deploy the Smart Rollup to Weeklynet and start a node: + +1. Run this command to deploy the Smart Rollup, replacing `$MY_ACCOUNT` with your account alias and `$ENDPOINT` with the RPC endpoint: + + ```bash + octez-client --endpoint ${ENDPOINT} \ + originate smart rollup files_archive from ${MY_ACCOUNT} \ + of kind wasm_2_0_0 of type unit with kernel "$(cat installer.hex)" \ + --burn-cap 2.0 --force + ``` + +1. Start the node with this command: + + ```bash + octez-smart-rollup-node --endpoint ${ENDPOINT} \ + run observer for files_archive with operators \ + --data-dir ./_rollup_node --log-kernel-debug + ``` + + For simplicity, this command runs the Smart Rollup in observer mode, which does not require a stake of 10,000 tez to publish commitments. + +1. Open a new terminal window and run this command to watch the node's log: + + ```bash + tail -F _rollup_node/kernel.log + ``` + +The log prints the current DAL parameters, as in this example: + +``` +RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } +RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } +RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } +RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } +RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } +``` + +These parameters are: + +- `number_of_slots`: The number of slots in each block +- `slot_size`: The size of each slot in bytes +- `page_size`: The size of each page in bytes +- `attestation_lag`: The number of subsequent blocks in which bakers can attest that the data is available; if enough attestations are available by the time this number of blocks have been created, the data becomes available to Smart Rollups + +## Setting up a deployment script + +In later parts of this tutorial, you will update and redeploy the Smart Rollup multiple times. +To simplify the process, you can use this script. +To use it, pass the alias of your account in the Octez client: + +```bash +#!/usr/bin/bash + +alias="${1}" + +set -e + +cargo build --release --target wasm32-unknown-unknown + +rm -rf _rollup_node + +cp target/wasm32-unknown-unknown/release/files_archive.wasm . + +smart-rollup-installer get-reveal-installer -P _rollup_node/wasm_2_0_0 \ + -u files_archive.wasm -o installer.hex + +octez-client --endpoint ${ENDPOINT} \ + originate smart rollup files_archive from "${alias}" of kind wasm_2_0_0 \ + of type unit with kernel "$(cat installer.hex)" --burn-cap 2.0 --force + +octez-smart-rollup-node --endpoint ${ENDPOINT} \ + run observer for files_archive with operators --data-dir _rollup_node \ + --dal-node http://localhost:10732 --log-kernel-debug +``` + +In the next section, you will get information about the state of slots in the DAL. +See [Part 2: Getting slot information](./get-slot-info). diff --git a/docs/tutorials/build-files-archive-with-dal/get-slot-info.mdx b/docs/tutorials/build-files-archive-with-dal/get-slot-info.mdx new file mode 100644 index 000000000..f8c893c68 --- /dev/null +++ b/docs/tutorials/build-files-archive-with-dal/get-slot-info.mdx @@ -0,0 +1,138 @@ +--- +title: "Part 2: Getting slot information" +authors: 'Tezos Core Developers' +last_update: + date: 17 January 2024 +--- + +When clients send data to the DAL, they must choose which slot to put it in. +This can cause conflicts, because only one client can write data to a given slot in a single block. +If more than one client tries to write to the same slot and a baker includes those operations in the same block, only the first operation in the block succeeds in writing data to the slot. +The other operations fail and the clients must re-submit the data to be included in a future block. + +For this reason, clients should check the status of slots to avoid conflicts. +For example, slots 0, 30, and 31 are often used for regression tests. + +To see which slots are in use, you can use the Explorus indexer at https://explorus.io/dal. +For example, this screenshot shows that slots 10 and 25 are in use: + +![The Explorus indexer, showing the slots that are in use in each block](/img/tutorials/dal-explorus-slots.png) + +You can also see the state of the DAL slots by running a DAL node. +To reduce the amount of data that they have to manage, DAL nodes can subscribe to certain slots and ignore the data in others. +Similarly, the protocol assigns bakers to monitor certain slots. + +## Starting a DAL node + +To run a DAL node, use the Octez `octez-dal-node` command and pass the slots to monitor in the `--producer-profiles` argument. + +Run this command to start a DAL node and monitor slot 0: + +```bash +octez-dal-node run --endpoint ${ENDPOINT} \ + --producer-profiles=0 --data-dir _dal_node +``` + +## Accessing the slot data from a Smart Rollup + +Follow these steps to update the Smart Rollup to access information about slot 0: + +1. Update the `src/lib.rs` file to have this code: + + ```rust + use tezos_smart_rollup::{host::RuntimeError, kernel_entry, prelude::*}; + use tezos_smart_rollup_host::dal_parameters::RollupDalParameters; + + pub fn run( + host: &mut R, + param: &RollupDalParameters, + slot_index: u8, + ) -> Result<(), RuntimeError> { + let sol = host.read_input()?.unwrap(); + + let target_level = sol.level as usize - param.attestation_lag as usize; + + let mut buffer = vec![0u8; param.page_size as usize]; + + let bytes_read = host.reveal_dal_page(target_level as i32, slot_index, 0, &mut buffer)?; + + if 0 < bytes_read { + debug_msg!( + host, + "Attested slot at index {} for level {}: {:?}\n", + slot_index, + target_level, + &buffer.as_slice()[0..10] + ); + } else { + debug_msg!( + host, + "No attested slot at index {} for level {}\n", + slot_index, + target_level + ); + } + + Ok(()) + } + + pub fn entry(host: &mut R) { + let param = host.reveal_dal_parameters(); + debug_msg!(host, "{:?}\n", param); + + match run(host, ¶m, 0) { + Ok(()) => debug_msg!(host, "See you in the next level\n"), + Err(_) => debug_msg!(host, "Something went wrong for some reasons"), + } + } + + kernel_entry!(entry); + ``` + + The key change is the addition of the function `run`. + Using this function allows the code to use the `?` operator of Rust by using a function that returns a `Result` type. + + The `run` function proceeds as follows: + + 1. First, it uses the DAL parameters to know the first level where a slot might be used. + It subtracts the attestation lag from the current level, which it gets from the Smart Rollup inbox; the result is the most recent block that may have attested data in it. + 1. It allocates `Vec` buffer of the current page size. + 1. It attempts to fill the buffer with the `read_dal_page` function provided + by the SDK. + 1. It checks the value returned by the function, which is the number of bytes + read. + Zero bytes mean that the slot has no attested data in it. + Otherwise, it is necessarily the size of the page, because that's the size of the buffer. + +1. Update the `Cargo.toml` file to add this dependency at the end: + + ```toml + tezos-smart-rollup-host = { version = "0.2.2", features = [ "proto-alpha" ] } + ``` + +1. Run the commands to build and deploy the Smart Rollup and start the node. +You can use the script in [Part 1: Getting the DAL parameters](./get-dal-params) to simplify the process. + +1. In another terminal window, view the log with the command `tail -F _rollup_node/kernel.log`. + +The log shows information about slot 0, as in this example: + +``` +RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } +No attested slot at index 0 for level 56875 +See you in the next level +RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } +Attested slot at index 0 for level 56876: [16, 0, 0, 2, 89, 87, 0, 0, 0, 0] +See you in the next level +RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } +No attested slot at index 0 for level 56877 +See you in the next level +``` + +For the first 4 Tezos blocks produced after the origination of the Smart Rollup, the kernel will report that no slot has been attested for the targeted level, _even if Explorus states the opposite_. +This is because, as of January, 2024, a Smart Rollup cannot fetch the content of a slot published before it is originated. +This is why you must wait for 4 blocks before seeing slot page contents being +logged. + +Now that you can see the state of the slots, you can find an unused slot and publish data to it. +When you are ready, continue to [Part 3: Publishing on the DAL](./publishing-on-the-dal). diff --git a/docs/tutorials/build-files-archive-with-dal/publishing-on-the-dal.mdx b/docs/tutorials/build-files-archive-with-dal/publishing-on-the-dal.mdx new file mode 100644 index 000000000..0eca70dd1 --- /dev/null +++ b/docs/tutorials/build-files-archive-with-dal/publishing-on-the-dal.mdx @@ -0,0 +1,141 @@ +--- +title: "Part 3: Publishing on the DAL" +authors: 'Tezos Core Developers' +last_update: + date: 17 January 2024 +--- + +Now that you can get information about the DAL, the next step is to publish data to it and verify that the kernel can access it. + +:::note Planning ahead +Before trying to run the code yourself, look at [Explorus](https://explorus.io/dal) and choose a slot that is not currently being used. +::: + +The examples in this tutorial use slot 10. + +## Switching slots + +When you have selected a slot that does not appear to be in use, follow these steps to restart the Smart Rollup and DAL node: + +1. Stop the DAL node and restart it with a new `--producer-profiles` argument. +For example, this command uses slot 10: + + ```bash + octez-dal-node run --endpoint ${ENDPOINT} \ + --producer-profiles=10 --data-dir _dal_node + ``` + +1. Update the kernel to monitor that slot by updating this line: + + ```rust + match run(host, ¶m, 0) { + ``` + + For example, to monitor slot 10, change the 0 to a 10, as in this code: + + ```rust + match run(host, ¶m, 10) { + ``` + +1. Run the commands to build and deploy the Smart Rollup and start the node. +You can use the script in [Part 1: Getting the DAL parameters](./get-dal-params) to simplify the process. + +## Publishing messages + +The DAL node provides an RPC endpoint for clients to send data to be added to a slot: `POST /slot`, whose body is the contents of the slot. + +1. Run this command to publish a message to the DAL: + + ```bash + curl localhost:10732/slot --data '"Hello, world!"' -H 'Content-Type: application/json' + ``` + + This command assumes that you have not changed the default RPC server address. + + The command returns the certificate from the DAL node, which looks like this example: + + ```json + { + "commitment": "sh1u3tr3YKPDYUp2wWKCfmV5KZb82FREhv8GtDeR3EJccsBerWGwJYKufsDNH8rk4XqGrXdooZ", + "commitment_proof":"8229c63b8e858d9a96321c80a204756020dd13243621c11bec61f182a23714cf6e0985675fff45f1164657ad0c7b9418" + } + ``` + + Note that the value of the message is in double quotes because it must be a valid JSON string, as hinted by the `Content-Type` header. + +1. Using the values of the commitment and proof from the previous command, post the certificate to layer 1 with this command: + + ```bash + commitment="sh1u3tr3YKPDYUp2wWKCfmV5KZb82FREhv8GtDeR3EJccsBerWGwJYKufsDNH8rk4XqGrXdooZ" + proof="8229c63b8e858d9a96321c80a204756020dd13243621c11bec61f182a23714cf6e0985675fff45f1164657ad0c7b9418" + octez-client --endpoint ${ENDPOINT} \ + publish dal commitment "${commitment}" from ${MY_ACCOUNT} for slot 10 \ + with proof "${proof}" + ``` + + After 4 blocks, you should see a message in the kernel log that looks like this: + + ``` + RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } + Attested slot at index 10 for level 57293: [72, 101, 108, 108, 111, 44, 32, 119, 111, 114] + See you in the next level + ``` + + You can verify your message by converting the bytes in the message back to the first 10 characters of the string "Hello, World!" + + If you see a message that says "A slot header for this slot was already proposed," another transaction tried to write to that slot in the same block, so you must try again. + + If you don't see information about the attested slot, check the page at https://explorus.io/dal. + If that page shows red (unattested) slots, it's possible that the attesters for the network are offline. + +## Publishing files + +You can also send raw bytes to the DAL node with the header `Content-Type: application/octet-stream`. +In this case, you must prefix the data with its size due to limitations of the DAL. + +1. Install the `jq` and `xxd` programs. + +1. Create a file named `upload_file.sh` and add this code: + + ```bash + #!/usr/bin/bash + + path="${1}" + alias="${2}" + index="${3}" + + target="$(mktemp)" + echo "storing temporary file at ${target}" + file_size="$(cat "${path}" | wc -c)" + slot_size_bin="$(printf "%08x" "${file_size}")" + slot_contents="$(cat ${path} | xxd -p)" + + echo -n "${slot_size_bin}${slot_contents}" | xxd -p -r > "${target}" + + certificate="$(curl localhost:10732/slot --data-binary "@${target}" -H 'Content-Type: application/octet-stream')" + + echo "${certificate}" + + commitment="$(echo -n ${certificate} | jq '.commitment' -r)" + proof="$(echo -n ${certificate} | jq '.commitment_proof' -r)" + + octez-client --endpoint ${ENDPOINT} \ + publish dal commitment "${commitment}" from "${alias}" \ + for slot "${index}" with proof "${proof}" + + rm "${target}" + ``` + + The script accepts three arguments: the file to send, the account alias to use and the slot index to use. + This script also assumes that the `PATH` and `ENDPOINT` environment variables are correctly set. + For example: + + ```bash + ./upload_file.sh myFile.txt $MY_ACCOUNT 10 + ``` + + Again, by inspecting the kernel logs, you should be able to see that the file that you wanted to publish is indeed the one fetched by the Smart Rollup. + +Now you can publish data to the DAL and use it in a Smart Rollup. +In the next section, you write to and retrieve the entire slot. +When you are ready, go to [Part 4: Using the entire slot](./using-full-slot). diff --git a/docs/tutorials/build-files-archive-with-dal/using-full-slot.mdx b/docs/tutorials/build-files-archive-with-dal/using-full-slot.mdx new file mode 100644 index 000000000..6ee53c95b --- /dev/null +++ b/docs/tutorials/build-files-archive-with-dal/using-full-slot.mdx @@ -0,0 +1,146 @@ +--- +title: "Part 4: Using the entire slot" +authors: 'Tezos Core Developers' +last_update: + date: 18 January 2024 +--- + +In some cases, you may want to retrieve the entire contents of a slot. +For example, it can be convenient to get the entire slot because it has a fixed size, while the data in the slot may be smaller and padded to fit the slot. + +## Fetching and storing the full slot + +Retrieving the full slot is similar to retrieving any data from the slot. +In this case, you change the kernel to retrieve data of the exact size of the slot. + +1. Update the `run` function in the `lib/rs` file to this code: + + ```rust + pub fn run( + host: &mut R, + param: &RollupDalParameters, + slot_index: u8, + ) -> Result<(), RuntimeError> { + // Reading one message from the shared inbox is always safe, + // because the shared inbox contains at least 3 messages per + // Tezos block. + let sol = host.read_input()?.unwrap(); + + let target_level = sol.level as usize - param.attestation_lag as usize; + + let mut buffer = vec![0u8; param.slot_size as usize]; + + let bytes_read = host.reveal_dal_page(target_level as i32, slot_index, 0, &mut buffer)?; + + if bytes_read == 0 { + debug_msg!( + host, + "No attested slot at index {} for level {}\n", + slot_index, + target_level + ); + + return Ok(()); + } + + debug_msg!( + host, + "Attested slot at index {} for level {}\n", + slot_index, + target_level + ); + + let num_pages = param.slot_size / param.page_size; + + for page_index in 1..num_pages { + let _result = host.reveal_dal_page( + target_level as i32, + slot_index, + page_index.try_into().unwrap(), + &mut buffer[page_index as usize * (param.page_size as usize) + ..(page_index as usize + 1) * (param.page_size as usize)], + ); + } + + let hash = blake2b::digest(&buffer, 32).unwrap(); + let key = hex::encode(hash); + let path = OwnedPath::try_from(format!("/{}", key)).unwrap(); + + debug_msg!(host, "Saving slot under `{}'\n", path); + + let () = host.store_write_all(&path, &buffer)?; + + Ok(()) + } + ``` + + Now the `run` function works like this: + + 1. It allocates a buffer of the size of a slot, not a size of a page. + 1. It tries to fetch the contents of the first page. + If 0 bytes are written by `reveal_dal_page`, the targeted slot has not been + attested for this block. + 1. If the targeted slot has been attested, the function reads as many pages as necessary to get the full slot data. + 1. It stores the data in the durable storage, using the Blake2B hash (encoded in hexadecimal) as its key. + +1. Add these `use` statements to the beginning of the file: + + ```rust + use tezos_crypto_rs::blake2b; + use tezos_smart_rollup::storage::path::OwnedPath; + ``` + + These dependencies use `tezos_crypto_rs` for hashing, and `hex` for encoding. + +1. Add the matching dependencies to the `Cargo.toml` file: + + ```toml + tezos_crypto_rs = { version = "0.5.2", default-features = false } + hex = "0.4.3" + ``` + + Adding `default-features = false` for `tezos_crypto_rs` is necessary for the crate to be compatible with Smart Rollups. + +1. Deploy the Smart Rollup again, publish a file as you did in the previous section, and wait for enough levels to pass. +The Smart Rollup log shows the hash of the data, as in this example: + + ``` + RollupDalParameters { number_of_slots: 32, attestation_lag: 4, slot_size: 65536, page_size: 4096 } + Attested slot at index 10 for level 15482 + Saving slot under `/6a578d1e6746d29243ff81923bcea6375e9344d719ca118e14cd9f3d3b00cd96' + See you in the next level + ``` + +1. Get the data from the slot by passing the hash, as in this example: + + ```bash + hash=6a578d1e6746d29243ff81923bcea6375e9344d719ca118e14cd9f3d3b00cd96 + curl "http://localhost:8932/global/block/head/durable/wasm_2_0_0/value?key=/${hash}" \ + -H 'Content-Type: application/octet-stream' \ + -o slot.bin + ``` + +1. Convert the contents of the slot to text by running this command: + + ```bash + xxd -r -p slot.bin + ``` + + The console shows your message in text, such as "Hi! This is a message to go on the DAL." + +:::note Why `diff` won't work +You cannot use `diff` to ensure that the file you originally published and the one that you downloaded from the rollup node are equal. +Indeed, they are not: because the size of a slot is fixed, the DAL node pads the value it receives from `POST /slot` in order to ensure that it has the correct slot size. +::: + +## Next steps + +Now you know how to send files to the DAL and use a Smart Rollup to store the data. + +From there, the sky's the limit. +You can implement many other features, such as: + +- Handling more than one file per level +- Having file publishers pay for the storage that they are using in layer 2 by allowing them to deposit tez to the Smart Rollup and sign the files they publish +- Building a frontend to visualize the files in the archive +- Providing the original size of the file by modifying the script to prefix the file with its size diff --git a/docusaurus.config.js b/docusaurus.config.js index 26b5ba1fc..f2ba4c536 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -121,7 +121,7 @@ const config = { }, prism: { theme: require('prism-react-renderer/themes/github'), - additionalLanguages: ['csharp'], + additionalLanguages: ['csharp', 'toml'], }, // https://github.com/flexanalytics/plugin-image-zoom // Enable click to zoom in to large images diff --git a/sidebars.js b/sidebars.js index dae502ddc..d0ac6504c 100644 --- a/sidebars.js +++ b/sidebars.js @@ -341,6 +341,20 @@ const sidebars = { 'tutorials/smart-rollup/run', ], }, + { + type: 'category', + label: 'Implement a file archive with the DAL', + link: { + type: 'doc', + id: 'tutorials/build-files-archive-with-dal', + }, + items: [ + 'tutorials/build-files-archive-with-dal/get-dal-params', + 'tutorials/build-files-archive-with-dal/get-slot-info', + 'tutorials/build-files-archive-with-dal/publishing-on-the-dal', + 'tutorials/build-files-archive-with-dal/using-full-slot', + ], + }, { type: 'category', label: 'Build an NFT marketplace', diff --git a/static/img/tutorials/dal-explorus-slots.png b/static/img/tutorials/dal-explorus-slots.png new file mode 100644 index 000000000..01978f187 Binary files /dev/null and b/static/img/tutorials/dal-explorus-slots.png differ