composing nodes with input/output ports in Nickel

[MMesch]

Hi all,

I would love getting something like this to work to compose workflows with input/output ports in Nickel:

let node1 = {inputs={number : Number}, outputs={number : Number}} in
let node2 = {inputs={number : Number}, outputs={number : Number = (inputs.number * 2)}} in
let compose = fun n1 n2 => n2 & {inputs=node1.outputs} in
compose node1 node2

The example could be extended to include partially composed nodes but even before that it currently it produces at least these two problems:

error: incompatible types
  ┌─ /home/matto/Projects/nickel/nickel-pipelines/pipeline.ncl:2:68
  │
2 │ let node2 = {inputs={number : Number}, outputs={number : Number = (inputs.number * 2)}} in
  │                                                                    ^^^^^^ this expression
  │
  = Expected an expression of type `{ number: _a ; _rrows_b }`
  = Found an expression of type `Dyn`
  = These types are not compatible

and

error: non mergeable terms
    ┌─ <stdlib/internals.ncl>:59:15
    │    
 59 │       "$record" = fun field_contracts tail_contract label value =>
    │ ╭─────────────────^
    │ │ ╭───────────────^
 60 │ │ │     if %typeof% value == 'Record then
 61 │ │ │       # Returns the sub-record of `left` containing only those fields which are not
 62 │ │ │       # present in `right`. If `left` has a sealed polymorphic tail then it will be
    · │ │
104 │ │ │     else
105 │ │ │       %blame% (%label_with_message% "not a record" label),
    │ ╰─│─────────────────────────────────────────────────────────^ cannot merge this expression
    │   ╰─────────────────────────────────────────────────────────^ with this expression
    │    
    ┌─ /home/matto/Projects/nickel/nickel-pipelines/pipeline.ncl:3:28
    │
  3 │ let compose = fun n1 n2 => n2 & {inputs=node1.outputs} in
    │                            --------------------------- originally merged here
    │
    = Both values have the same merge priority but they can't be combined.
    = Primitive values (Number, String, and Bool) or arrays can be merged only if they are equal.
    = Functions can never be merged.

Is it possible to do this in Nickel?

[yannham]

If you want to do this, I think you should trade either the = sign or the : sign for a | (contract annotation), depending on what you want to express.

Writing {inputs = {number : Number}} has the following effect:

• {number : Number} is a record type. Why it's not a record with an undefined field of type number is dictated by specific rules, that have to be in place because we decided to have a unified syntax for types, contracts and values. For more details, see the relevant section of the manual. But basically it has a field with a type annotation and without a definition.
• you use =, the field assignment operator. It means "put the value ... inside the inputs field". Here, the value is a type. As per the idea behind having a unified syntax, we can use types as normal values: in that case, they are converted to their corresponding contract. Because record types and record contracts have subtle but important differences, this isn't just a normal record, but some opaque function. The only thing you can do, mostly, is to use it in another type or contract annotation, or to apply it using std.contract.apply.

So, what you get is {inputs = some_obscure_internal_function_generated_by_the_interpreter}, which I imagine is not what you want, because it can't be merged with anything meaningfully.

You have several possible fixes:

1. inputs | {number : Number}: now, you don't give a value to inputs, but you're saying "this is the shape inputs should have", and provide a type. It's closer to what you want to express, but see next point.
2. inputs | {number | Number}: it's very close to the previous suggestion, but I think it make more sense in your context. Type annotation are useful to typecheck a block statically, but here, inputs doesn't have a definition. So the type will be converted to a contract, and having a static type annotation in the first place is both a bit misleading and not very useful, so I would advise to use | everywhere as soon as you are in "dynamic contract check" land anyway.
3. inputs = {number | Number}. This one would work as well. The difference with the previous solution is subtle, but the idea is that when you attach a contract to inputs directly using |, as in the previous solution inputs | {number | Number}, you say "I impose that the shape of inputs must be the following, whatever it is merged with, or overridden by". In particular you can't add a new field which is not number. Even if you override inputs with merging, the new value has to respect the contract. On the other hand, writing inputs = {number | Number} is to merely provide an incomplete initial value to inputs. If you're overriding number directly, the effect will be similar, but this is more permissive, as you can add other fields, or even override inputs with something totally different that isn't even a record.

I hope it's not too confusing, but I wanted to explain a bit the different possibilities, because it's not always obvious what to do, when you can chose either |, : or = in many places when defining such nested record contracts. In the end, I would go with .2, and more generally to use | everywhere it makes sense, because 2. works (as opposed to the original try), is more restrictive (you can't override inputs with something totally unrelated), and doesn't mix static types and dynamic contract in setting which will be mostly dynamic anyway.

let node1 = {inputs | {number | Number}, outputs | {number | Number}} in
let node2 = {inputs | {number | Number}, outputs | {number | Number} = {number = inputs.number * 2}} in
let compose = fun n1 n2 => n2 & {inputs=n1.outputs} in
compose node1 node2

Now, you get a better motivated "missing definition for outputs", because indeed node1.outputs is not defined:

error: missing definition for `outputs`
  ┌─ repl-input-1:3:44
  │
1 │ let node1 = {inputs | {number | Number}, outputs | {number | Number}} in
  │             --------------------------------------------------------- in this record
2 │ let node2 = {inputs | {number | Number}, outputs | {number | Number} = {number = inputs.number * 2}} in
3 │ let compose = fun n1 n2 => n2 & {inputs=n1.outputs} in
  │                                         ---^^^^^^^
  │                                         │  │
  │                                         │  required here
  │                                         accessed here

From there, you might have another problem which is that all your inputs will be merged together, which is probably not what you want. Maybe you'll have to "namespace" your inputs when merging, to make a dinstinction between "global resulting inputs after the merge", "old inputs of node1 now unified with global resulting inputs", "old inputs of node2 now unified with node1.outputs" (and same thing for outputs).

[yannham]

We've fiddled with that idea a bit more, and here is a more polished version of composition:

let node1 = {
  Input = { number | Number },
  inputs | Input,
  outputs | { number | Number } = {
      number =
        inputs.number + 1
    },
}
in
let node2 = {
  Input = { number | Number},
  inputs | Input,
  outputs | { number | Number } = {
      number =
        inputs.number * 2
    }
}
in

let compose = fun n1 n2 =>
  {
    Input = n1.Input,
    inputs | n1.Input,
    outputs = 
      let inputs' = inputs in
      let n1' = n1 & { inputs = inputs' } in
      (n2 & { inputs = n1'.outputs }).outputs,
  }
in 

let node3 = (compose node1 node2) in
let composed = (compose node3 node1) in
(composed & {inputs.number = 2}).outputs

==> { number = 7 }

In the end, it's "just" encoding functions as recursive record, which is in some sense a natural encoding. That's also the idea behind NixOS modules versus packages-as-functions, and the Package As Record Model proposed by Nickel-Nix RFC.

Anyway, I think we can consider the original question to be solved. Feel free to re-open if not (or open a new issue or discussion for a related but separate matter).