cardano-testnet: allow to take node config file as input #6103

smelc · 2025-02-05T14:45:53Z

Description

With @CarlosLopezDeLara's input, I have chosen an all or nothing behavior: either you pass the configuration file and all genesis files or you pass none of them. If we choose an à la carte behavior, it's cumbersome to implement, and anyway since the node configuration file contains the hashes of genesis files, there is at least some degree of binding between all of these files.

After feedback from Jordan, we finally only ask the user to provide the node configuration file. Then we read the eras' genesis files by looking at the paths specified in the node configuration file. This way, no additional flag needed, nor checking consistency between the flags and the node configuration file.

How to trust this PR

TBD

Checklist

Commit sequence broadly makes sense and commits have useful messages
CI passes.
Self-reviewed the diff

Jimbo4350

How do you plan to deal with conflicts in the node configuration yaml file? I think this is the wrong approach. We don't want to be forced to reconcile configuration file differences. All we should need the is the node configuration yaml file.

Jimbo4350 · 2025-02-07T14:22:59Z

cardano-testnet/src/Parsers/Run.hs

+          decodeFileThrow $ icfAlonzoGenesisConfigFile inputConfigFiles
+        conwayGenesisFile ::  ConwayGenesis StandardCrypto <-
+          decodeFileThrow $ icfConwayGenesisConfigFile inputConfigFiles
+        runTestnet testnetOptions $ cardanoTestnet


why does runTestnet result in a Property??

runTestnet doesn't return a property but it does take a Conf -> H.Integration a callback, as visible here:

cardano-node/cardano-testnet/src/Testnet/Property/Run.hs

Line 41 in dad33d9

runTestnet :: CardanoTestnetOptions -> (Conf -> H.Integration a) -> IO ()

This is a bit meh and we can probably improve that: in the case where the user provides a configuration file, we would like not be tied to the hedgehog-extras/hedgehog APIs like Integration and Property. But this is something we would better do in the future, not in this PR, because it's not required for reaching a first release of cardano-testnet IMHO. And also, it will be some work, because our existing "backend", e.g. cardanoTestnet here is in the Integration monad. So it's not an easy change.

We can return to this. Could you create an issue to track it? The issue just needs to point out its an issue 😆

@Jimbo4350> created follow-up issue: #6122

smelc · 2025-02-12T10:13:51Z

For future reference, if we leave the call to cardano-cli debug check-node-configuration, we have the following failures with the following genesis files:

> cabal run cardano-testnet -- cardano --output-dir cl-output-dir --node-config cl-example/configuration.json --testnet-magic 44
  ✗ <interactive> failed at src/Testnet/Start/Cardano.hs:283:3
    after 1 test.
    shrink path: 1:
  
    forAll0 =
      Process exited with non-zero exit-code: 1
      ━━━━ stdout ━━━━
      Checking byron genesis file: cl-example/byron.genesis.spec.json
      
      ━━━━ stderr ━━━━
      Error reading node configuration at: cl-example/byron.genesis.spec.json: Incorrect schema for GenesisData.
       Error: expected field protocolConsts

It seems the code is requiring too much data here when deserializing the Byron genesis. This data is actually not required to start the node ~~(i.e. if we remove the check, the testnet starts fine). So it seems overkill to require this data when starting a testnet.~~

[edit] Oh well, it seems the node doesn't like it either actually:

✗ <interactive> failed at src/Testnet/Start/Cardano.hs:342:5
    after 1 test.
    shrink path: 1:
  
    forAll0 =
      Cardano node process did not start: CardanoProtocolInstantiationError (CardanoProtocolInstantiationErrorByron (GenesisReadError "/home/churlin/dev/cardano-node/cl-example/byron.genesis.spec.json" (GenesisDataSchemaError (SchemaError {seExpected = "field protocolConsts", seActual = Nothing}))))
      
      cardano-node: There was an error parsing the genesis file: "/home/churlin/dev/cardano-node/cl-example/byron.genesis.spec.json" Error: GenesisDataSchemaError (SchemaError {seExpected = "field protocolConsts", seActual = Nothing})

The node itself, when it starts, doesn't require hashes of genesis files. So it seems overkill to require it to start a testnet. See also #6103 (comment)

smelc · 2025-02-12T16:18:55Z

Starting with the configuration.yaml file created by running the queries (e.g. the sandbox created by DISABLE_RETRIES=1 cabal test cardano-testnet-test --test-options '-p "/CliQueries/"') test fails as follows:

> cabal run cardano-testnet -- cardano --output-dir cl-output-dir --node-config nix-shell.Ai0S6G/cli-queries-test-6184168d6b9797eb/configuration.yaml --testnet-magic 44

✗ <interactive> failed at src/Testnet/Start/Cardano.hs:342:5
    after 1 test.
    shrink path: 1:
  
    forAll0 =
      Cardano node process did not start: CardanoProtocolInstantiationError (CardanoProtocolInstantiationErrorByron (CredentialsError DelegationCertificateNotFromGenesisKey))
      
      cardano-node: Byron leader credentials error: DelegationCertificateNotFromGenesisKey

~~I'm a bit puzzled.~~

[edit] Solved later on, it was related to not honoring the custom configuration file everywhere

The node itself, when it starts, doesn't require hashes of genesis files. So it seems overkill to require it to start a testnet. See also #6103 (comment)

smelc · 2025-02-18T12:56:14Z

All we should need the is the node configuration yaml file.

@Jimbo4350> Updated version of this PR now does that 👍

… node configuration file (it was unused) This will be done differently in #6103

Jimbo4350

This is an improvement 👍 . Have a look at my comments.

Jimbo4350 · 2025-02-18T13:41:21Z

cardano-testnet/src/Parsers/Cardano.hs

+        <>  OA.showDefault
+        <>  OA.value 1)
+    , UserNodeOptions
+        <$> strOption ( long "node-config"


So this is an improvement however the resultant type should be [TestnetNodeOptions]. Users (or QA) may want to for example create forking conditions whereby nodes have different configurations. This will have to be done in another PR at some point.

Jimbo4350 · 2025-02-18T13:49:46Z

cardano-testnet/src/Parsers/Run.hs

+                Nothing -> error $ "Genesis files not specified in node configuration file: " <> nodeInputConfigFile
+                Just x -> x
+            adjustedProtocolConfig =
+              -- Make all the files be relative to the location of the config file.


Jimbo4350 · 2025-02-18T13:52:06Z

cardano-testnet/src/Parsers/Run.hs

+          alonzoGenesisFile
+          conwayGenesisFile
+  where
+    getShelleyGenesises (NodeProtocolConfigurationCardano _byron shelley alonzo conway _hardForkConfig) =


We also want to confirm the validity of the byron genesis but that can be a separate PR in the future.

Jimbo4350 · 2025-02-18T13:55:50Z

cardano-testnet/src/Testnet/Start/Cardano.hs

+  where
+    nSpoNodes =
+      case cardanoNodes of
+        UserNodeOptions _ -> 1


Hardcoding this isn't great. This is related to my comment above regarding [TestnetNodeOptions]

Jimbo4350 · 2025-02-18T13:58:42Z

cardano-testnet/src/Testnet/Start/Cardano.hs

-    -- to run a testnet. "create-staked" is not a good way to do this especially because it
-    -- makes assumptions about where things should go and where genesis template files should be.
-    -- See all of the ad hoc file creation/renaming/dir creation etc below.
-    H.failMessage GHC.callStack "Specifying node configuration files per node not supported yet."


We need to note that we want to support configuration files per node. However it's not urgent to do this now.

Jimbo4350 · 2025-02-18T14:51:23Z

cardano-testnet/src/Testnet/Start/Types.hs

@@ -67,7 +66,7 @@ instance Default CardanoTestnetCliOptions where
 data CardanoTestnetOptions = CardanoTestnetOptions
  { -- | List of node options. Each option will result in a single node being
    -- created.
-    cardanoNodes :: [TestnetNodeOptions]
+    cardanoNodes :: TestnetNodeOptions


The haddock should be updated

Jimbo4350 · 2025-02-18T14:51:37Z

cardano-testnet/src/Testnet/Start/Types.hs

-cardanoNumRelays CardanoTestnetOptions{cardanoNodes} =
-  NumRelays . length $ filter isRelayNodeOptions cardanoNodes
+cardanoNumRelays CardanoTestnetOptions{cardanoNodes=_} =
+  undefined


Jimbo4350 · 2025-02-18T14:59:16Z

cardano-testnet/src/Testnet/Start/Cardano.hs

@@ -193,16 +197,15 @@ getDefaultShelleyGenesis asbe maxSupply opts = do
 cardanoTestnet :: ()
  => HasCallStack
  => CardanoTestnetOptions -- ^ The options to use
-  -> Conf
-  -> UserNodeConfig -- ^ The node configuration file to use. If omitted it's generated.


This is an indication that TestnetNodeOptions ought to be renamed to TestnetNodeConfiguration.
A type like the following would be more indicative:

data TestnetNodeConfiguration = UserSpecifiedConfiguration FilePath | DefaultConfiguration [AutomaticNodeOption]

However what we really want is:

data TestnetNodeConfiguration = UserSpecifiedConfiguration FilePath | DefaultSpoConfiguration [String] | DefaultRelayConfiguration [String]

This gives users the most flexibility in that they can spin up a testnet with a mixture of nodes. Haddocks are desperately needed to indicate that [String] types are command line executable modifiers.

… node configuration file (it was unused) This will be done differently in #6103

…erve existing behavior too

smelc added the cardano-testnet label Feb 5, 2025

smelc changed the title ~~cardano-testnet: allow to take config files as input~~ cardano-testnet: allow to take node config and genesis files as input Feb 5, 2025

smelc linked an issue Feb 5, 2025 that may be closed by this pull request

[FEATURE] - cardano-testnet: Use flags to pass genesis and configuration files #6069

Open

17 tasks

Base automatically changed from smelc/cardano-testnet-support-custom-node-config-file to master February 5, 2025 16:14

smelc force-pushed the smelc/testnet-pass-config-files branch from 3894926 to 8e37423 Compare February 7, 2025 13:53

smelc marked this pull request as ready for review February 7, 2025 13:53

smelc requested a review from a team as a code owner February 7, 2025 13:53

Jimbo4350 requested changes Feb 7, 2025

View reviewed changes

smelc force-pushed the smelc/testnet-pass-config-files branch 2 times, most recently from 63c60a5 to 6ca79c5 Compare February 11, 2025 16:16

smelc added a commit that referenced this pull request Feb 12, 2025

cardano-testnet: remove check that is too conservative

d67ff61

The node itself, when it starts, doesn't require hashes of genesis files. So it seems overkill to require it to start a testnet. See also #6103 (comment)

smelc force-pushed the smelc/testnet-pass-config-files branch from 6ca79c5 to d67ff61 Compare February 12, 2025 10:38

smelc added a commit that referenced this pull request Feb 12, 2025

cardano-testnet: remove check that is too conservative

c78c34f

The node itself, when it starts, doesn't require hashes of genesis files. So it seems overkill to require it to start a testnet. See also #6103 (comment)

smelc force-pushed the smelc/testnet-pass-config-files branch from d67ff61 to c78c34f Compare February 12, 2025 10:39

smelc added a commit that referenced this pull request Feb 12, 2025

cardano-testnet: remove check that is too conservative

1f7c3d8

The node itself, when it starts, doesn't require hashes of genesis files. So it seems overkill to require it to start a testnet. See also #6103 (comment)

smelc force-pushed the smelc/testnet-pass-config-files branch from c78c34f to 1f7c3d8 Compare February 12, 2025 14:25

smelc added a commit that referenced this pull request Feb 12, 2025

cardano-testnet: remove check that is too conservative

6234a1a

The node itself, when it starts, doesn't require hashes of genesis files. So it seems overkill to require it to start a testnet. See also #6103 (comment)

smelc force-pushed the smelc/testnet-pass-config-files branch from 18c0d3e to 50849e1 Compare February 12, 2025 15:30

smelc added a commit that referenced this pull request Feb 12, 2025

cardano-testnet: remove check that is too conservative

38b11ab

The node itself, when it starts, doesn't require hashes of genesis files. So it seems overkill to require it to start a testnet. See also #6103 (comment)

smelc force-pushed the smelc/testnet-pass-config-files branch 3 times, most recently from 8117ffc to 3671f83 Compare February 18, 2025 12:14

smelc changed the title ~~cardano-testnet: allow to take node config and genesis files as input~~ cardano-testnet: allow to take node config file as input Feb 18, 2025

smelc added a commit that referenced this pull request Feb 18, 2025

cardano-testnet: remove possibility to programmatically pass a custom…

f95a273

… node configuration file (it was unused) This will be done differently in #6103

smelc mentioned this pull request Feb 18, 2025

cardano-testnet: remove possibility to programmatically pass a custom node configuration file (it was unused) #6118

Merged

3 tasks

smelc added a commit that referenced this pull request Feb 18, 2025

cardano-testnet: remove possibility to programmatically pass a custom…

0459379

… node configuration file (it was unused) This will be done differently in #6103

smelc force-pushed the smelc/testnet-pass-config-files branch from c25acc3 to 6dc7fb4 Compare February 18, 2025 14:52

Jimbo4350 reviewed Feb 18, 2025

View reviewed changes

smelc added a commit that referenced this pull request Feb 19, 2025

cardano-testnet: remove possibility to programmatically pass a custom…

14b138f

… node configuration file (it was unused) This will be done differently in #6103

smelc mentioned this pull request Feb 19, 2025

cardano-testnet: runTestnet: don't tie it with H.Integration #6122

Open

smelc force-pushed the smelc/testnet-pass-config-files branch 5 times, most recently from a9301e7 to 12c67b4 Compare February 21, 2025 15:44

smelc added 4 commits February 26, 2025 17:49

cardano-testnet: allow to take node configuration file as input, pres…

8779f40

…erve existing behavior too

Adapt golden files

eeefb12

Adapt cardano-testnet-test accordingly

0844d79

Test cardano-testnet executable

b26b198

smelc force-pushed the smelc/testnet-pass-config-files branch from 12c67b4 to b26b198 Compare February 26, 2025 16:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cardano-testnet: allow to take node config file as input #6103

cardano-testnet: allow to take node config file as input #6103

smelc commented Feb 5, 2025 •

edited

Loading

Jimbo4350 left a comment

Jimbo4350 Feb 7, 2025

smelc Feb 12, 2025

Jimbo4350 Feb 18, 2025

smelc Feb 19, 2025

smelc commented Feb 12, 2025 •

edited

Loading

smelc commented Feb 12, 2025 •

edited

Loading

smelc commented Feb 18, 2025

Jimbo4350 left a comment

Jimbo4350 Feb 18, 2025

Jimbo4350 Feb 18, 2025

Jimbo4350 Feb 18, 2025

Jimbo4350 Feb 18, 2025

Jimbo4350 Feb 18, 2025

Jimbo4350 Feb 18, 2025

Jimbo4350 Feb 18, 2025

Jimbo4350 Feb 18, 2025 •

edited

Loading

cardano-testnet: allow to take node config file as input #6103

Are you sure you want to change the base?

cardano-testnet: allow to take node config file as input #6103

Conversation

smelc commented Feb 5, 2025 • edited Loading

Description

How to trust this PR

Checklist

Jimbo4350 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smelc commented Feb 12, 2025 • edited Loading

smelc commented Feb 12, 2025 • edited Loading

smelc commented Feb 18, 2025

Jimbo4350 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jimbo4350 Feb 18, 2025 • edited Loading

Choose a reason for hiding this comment

smelc commented Feb 5, 2025 •

edited

Loading

smelc commented Feb 12, 2025 •

edited

Loading

smelc commented Feb 12, 2025 •

edited

Loading

Jimbo4350 Feb 18, 2025 •

edited

Loading