feat(docs): add some documentation on starting a chain from mainnet s…

…tate (near#8753)

* feat(docs): add some documentation on starting a chain from mainnet state

* nits

* move to new section

* add a section on running the network

docs/SUMMARY.md

@@ -52,6 +52,11 @@
 
 - [Networking](./advanced_configuration/networking.md)
 
+# Custom test networks
+
+- [Starting a network from mainnet state](./test_networks/mainnet_spoon.md)
+
 # Misc
 
-- [Misc](./misc/README.md)
+- [Overview](./misc/README.md)
+- [State Sync Dump](./misc/state_sync_dump.md)

docs/test_networks/mainnet_spoon.md

@@ -0,0 +1,223 @@
+# Starting a test chain from state taken from mainnet or testnet
+
+## Purpose
+
+For testing purposes, it is often desirable to start a test chain with
+a starting state that looks like mainnet or testnet. This is usually
+done for the purpose of testing changes to neard itself, but it's also
+possible to do this if you're a contract developer and want to see
+what a change to your contract would look like on top of the current
+mainnet state. At the end of the process described here, you'll have
+a set of genesis records that can be used to start your own test chain,
+that'll be like any other test chain like the ones generated by the
+`neard localnet` command, except with account balances and data taken
+from mainnet
+
+## How-to
+
+The first step is to obtain an RPC node home directory for the chain
+you'd like to spoon. So if you want to use mainnet state, you can
+follow the instructions
+[here](https://near-nodes.io/rpc/run-rpc-node-without-nearup#5-get-data-backup-1)
+to obtain a recent snapshot of a mainnet node's home directory. Once
+you have your node's home directory set up, run the following
+`state-viewer` command to generate a dump of the chain's state:
+
+```shell
+$ neard --home $NEAR_HOME_DIRECTORY view-state dump-state --stream
+```
+
+This command will take a while (possibly many hours) to run. But at the
+end you should have `genesis.json` and `records.json` files in
+`$NEAR_HOME_DIRECTORY/output`. This records file represents all of the
+chain's current state, and is what we'll use to start our chain.
+
+From here, we need to make some changes to the `genesis.json` that was
+generated in `$NEAR_HOME_DIRECTORY/output`. To see why, note that the
+validators field of this genesis file lists all the current mainnet
+validators and their keys. So that means if we were to try and start a
+test chain from the generated genesis and records files as-is, it
+would work, but our node would expect the current mainnet validators
+to be producing blocks and chunks (which they definitely won't be!
+Because we're the only ones who know or care about this new test
+chain).
+
+So we need to select a new list of validators to start off our
+chain. Suppose that we want our chain to have two validators,
+`validator0.near` and `validator1.near`. Let's make a new directory
+where we'll be storing intermediate files during this process:
+
+```shell
+$ mkdir ~/test-chain-scratch
+```
+
+then using your favorite editor, lay out the validators you want in
+the test chain as a JSON list in the same format as the `validators`
+field in `genesis.json`, maybe in the file
+`~/test-chain-scratch/validators.json`
+
+```json
+[
+  {
+    "account_id": "validator0.near",
+    "public_key": "ed25519:GRAFkrqEkJAbdbWUgc6fDnNpCTE83C3pzdJpjAHkMEhq",
+    "amount": "100000000000000000000000000000000"
+  },
+  {
+    "account_id": "validator1.near",
+    "public_key": "ed25519:5FxQQTC9mk5kLAhTF9ffDMTXiyYrDXyGYskgz46kHMdd",
+    "amount": "100000000000000000000000000000000"
+  }
+]
+```
+
+These validator keys should be keys you've already generated. So for
+the rest of this document, we'll assume you've run:
+
+```shell
+$ neard --home ~/near-test-chain/validator0 init --account-id validator0.near
+$ neard --home ~/near-test-chain/validator1 init --account-id validator1.near
+```
+
+This is also a good time to think about what extra accounts you might
+want in your test chain. Since all accounts in the test chain will
+have the same keys as they do on mainnet, you'll only have access to
+the accounts that you have access to on mainnet. If you want to add an
+account with a large balance to properly test things out, you can
+write them out in a file as a JSON list of state records (in the same
+format as they appear in `records.json`). For example, you could put
+the following in `~/test-chain-scratch/extra-records.json`:
+
+```json
+[
+  {
+    "Account": {
+      "account_id": "my-test-account.near",
+      "account": {
+        "amount": "10000000000000000000000000000000000",
+        "locked": "0",
+        "code_hash": "11111111111111111111111111111111",
+        "storage_usage": 182,
+        "version": "V1"
+      }
+    }
+  },
+  {
+    "AccessKey": {
+      "account_id": "my-test-account.near",
+      "public_key": "ed25519:Eo9W44tRMwcYcoua11yM7Xfr1DjgR4EWQFM3RU27MEX8",
+      "access_key": {
+        "nonce": 0,
+        "permission": "FullAccess"
+      }
+    }
+  }
+]
+```
+
+You'll want to include an access key here, otherwise you won't be able
+to do anything with the account. Note that here you can also add
+access keys for any mainnet account you want, so you'll be able to
+control it in the test chain.
+
+Now to make these changes to the genesis and records files, you can
+use the `neard amend-genesis` command like so:
+
+```shell
+# mkdir ~/near-test-chain/
+$ neard amend-genesis --genesis-file-in $NEAR_HOME_DIRECTORY/output/genesis.json --records-file-in $NEAR_HOME_DIRECTORY/output/records.json --validators ~/test-chain-scratch/validators.json --extra-records ~/test-chain-scratch/extra-records.json --chain-id $TEST_CHAIN_ID --records-file-out ~/near-test-chain/records.json --genesis-file-out ~/near-test-chain/genesis.json
+```
+
+## Starting the network
+
+After running the previous steps you should have the files
+`genesis.json` and `records.json` in `~/near-test-chain/`. Assuming
+you've started it with the two validators `validator0.near` and
+`validator1.near` as described above, you'll want to run at least two
+nodes, one for each of these validator accounts. If you're working
+with multiple computers or VMs that can connect to each other over the
+internet, you'll be able to run your test network over the internet as
+is done with the "real" networks (mainnet, testnet, etc.). But for now
+let's assume that you want to run this on only one machine.
+
+So assuming you've initialized home directories for each of the
+validators with the `init` command described above, you'll want to
+copy the records and genesis files generated in the previous step to
+each of these:
+
+```shell
+$ cp ~/near-test-chain/records.json ~/near-test-chain/validator0
+$ cp ~/near-test-chain/genesis.json ~/near-test-chain/validator0
+$ cp ~/near-test-chain/records.json ~/near-test-chain/validator1
+$ cp ~/near-test-chain/genesis.json ~/near-test-chain/validator1
+```
+
+Now we'll need to make a few config changes to each of
+`~/near-test-chain/validator0/config.json` and
+`~/near-test-chain/validator1/config.json`:
+
+changes to `~/near-test-chain/validator0/config.json`:
+
+```json
+{
+  "genesis_records_file": "records.json",
+  "rpc": {
+    "addr": "0.0.0.0:3030"
+  },
+  "network": {
+    "addr": "0.0.0.0:24567",
+    "boot_nodes": "ed25519:[email protected]:24568",
+    "skip_sync_wait": false,
+  },
+  "consensus": {
+    "min_num_peers": 1
+  },
+  "tracked_shards": [0],
+}
+```
+
+changes to `~/near-test-chain/validator1/config.json`:
+
+```json
+{
+  "genesis_records_file": "records.json",
+  "rpc": {
+    "addr": "0.0.0.0:3031"
+  },
+  "network": {
+    "addr": "0.0.0.0:24568",
+    "boot_nodes": "ed25519:[email protected]:24567",
+    "skip_sync_wait": false,
+  },
+  "consensus": {
+    "min_num_peers": 1
+  },
+  "tracked_shards": [0],
+}
+```
+
+Here we make sure to have each node listen on different ports, while
+telling each about the other via `network.boot_nodes`. In this
+`boot_nodes` string, we set the public key not to the validator key,
+but to whatever key is present in the `node_key.json` file you got
+when you initialized the home directory. So for `validator0`'s config,
+we set its boot node to `validator1`'s node key, followed by the
+address of the socket it should be listening on. We also want to drop
+the minimum required number of peers, since we're just running a small
+test network locally. We set `skip_sync_wait` to `false`, because
+otherwise we get strange behavior that will often make your network
+stall.
+
+After making these changes, you can try running one neard process for
+each of your validators:
+
+```shell
+$ neard --home ~/near-test-chain/validator0 run
+$ neard --home ~/near-test-chain/validator1 run
+```
+
+Now these nodes will begin by taking the records laid out in
+`records.json` and turning them into a genesis state. At the time of
+this writing, using the latest nearcore version from the master
+branch, this will take a couple hours. But your validators should
+begin producing blocks after that's done.