jq and yq for querying, filtering, and transforming YAML files

[grahamgower]

It sometimes happens that one wants to extract some specific bit of information from a YAML file, or modify it in some way. And reaching for a parser like demes-python can be unnecessary or awkward depending on the task at hand. The json querier jq and its YAML wrapper yq are well suited for such tasks. The jq docs require some patient reading (maybe a couple of hours rather than a couple of minutes), so here are some examples to indicate possible use cases.

Extracting a Demes block that is embedded in another yaml.

It has been proposed that we could/should embed the Demes demography into a more general simulator-specific YAML file. E.g. taking the YAML example from tskit-dev/msprime#1842 (comment)

# ex1.yaml
ancestry:
  sequence_length: 100000
  recombination_rate: 1e-8
  samples: {A: 100, B: 100}
  ploidy: 1
  model: hudson
  demography:
    time_units: generations
    demes:
      - name: X
        epochs: [{end_time: 1000, start_size: 2000}]
      - name: A
        ancestors: [X]
        epochs: [{start_size: 2000}]
      - name: B
        ancestors: [X]
        epochs: [{start_size: 2000}]

mutations:
  rate: 1e-8
  model: blosum62

$ yq '.ancestry.demography' ex1.yaml -y

time_units: generations
demes:
  - name: X
    epochs:
      - end_time: 1000
        start_size: 2000
  - name: A
    ancestors:
      - X
    epochs:
      - start_size: 2000
  - name: B
    ancestors:
      - X
    epochs:
      - start_size: 2000

Transforming a non-compliant Demes file into a compliant one

With demes-python 0.2.0, we modified pulses to use a list of sources and proportions. We can convert pre 0.2.0 yaml files to the new format with a short jq script (can be input in one line, but using a separate file is clearer).

# convert.jq
def convert:
  .sources = [.source]
  | .proportions = [.proportion]
  | del(.source)
  | del(.proportion);

.pulses |=
  if . == null then empty  # do nothing if there are no pulses
  else map(convert) end

Then, given an old yaml file ex2.yaml,

# ex2.yaml
time_units: generations
demes:
  - name: a
    epochs:
      - start_size: 1
  - name: b
    epochs:
      - start_size: 1
pulses:
  - {source: a, dest: b, time: 100, proportion: 0.1}
  - {source: b, dest: a, time: 50, proportion: 0.1}

$ yq -f convert.jq ex2.yaml -y

time_units: generations
demes:
  - name: a
    epochs:
      - start_size: 1
  - name: b
    epochs:
      - start_size: 1
pulses:
  - dest: b
    time: 100
    sources:
      - a
    proportions:
      - 0.1
  - dest: a
    time: 50
    sources:
      - b
    proportions:
      - 0.1

Rename demes

Suppose you want to rename the demes in your file(s)? No problem!

# rename.jq
def rename:
  if . == "a" then "A_adustus"
  elif . == "b" then "A_bellus"
  else . end;

.demes[].name |= rename
| .demes[].ancestors[]? |= rename
| .pulses[]?.sources[] |= rename
| .pulses[]?.dest |= rename
| .migrations[]?.source? |= rename
| .migrations[]?.dest? |= rename
| .migrations[]?.demes? |= rename

As input, let's pipe the output from convert.jq above.

$ yq -f convert.jq ex2.yaml -y | yq -f rename.jq -y

time_units: generations
demes:
  - name: A_adustus
    epochs:
      - start_size: 1
  - name: A_bellus
    epochs:
      - start_size: 1
pulses:
  - dest: A_bellus
    time: 100
    sources:
      - A_adustus
    proportions:
      - 0.1
  - dest: A_adustus
    time: 50
    sources:
      - A_bellus
    proportions:
      - 0.1

Caveats

If we want to use a fully-resolved Demes file as the input to yq, then we find that yaml .inf values are converted to 1.7976931348623157e+308 (jq converts infinity to the largest representable floating point number)

$ yq -f convert.jq ex2.yaml -y | demes parse - | yq '.' -y

time_units: generations
demes:
  - name: a
    start_time: 1.7976931348623157e+308
    epochs:
      - end_time: 0
        start_size: 1
        end_size: 1
        size_function: constant
        selfing_rate: 0
        cloning_rate: 0
  - name: b
    start_time: 1.7976931348623157e+308
    epochs:
      - end_time: 0
        start_size: 1
        end_size: 1
        size_function: constant
        selfing_rate: 0
        cloning_rate: 0
pulses:
  - sources:
      - a
    dest: b
    time: 100
    proportions:
      - 0.1
  - sources:
      - b
    dest: a
    time: 50
    proportions:
      - 0.1

But, this is no problem. We just transform those infinity values!

$ yq -f convert.jq ex2.yaml -y | demes parse - | yq 'walk(if isinfinite then "Infinity" else . end)' -y

time_units: generations
demes:
  - name: a
    start_time: Infinity
    epochs:
      - end_time: 0
        start_size: 1
        end_size: 1
        size_function: constant
        selfing_rate: 0
        cloning_rate: 0
  - name: b
    start_time: Infinity
    epochs:
      - end_time: 0
        start_size: 1
        end_size: 1
        size_function: constant
        selfing_rate: 0
        cloning_rate: 0
pulses:
  - sources:
      - a
    dest: b
    time: 100
    proportions:
      - 0.1
  - sources:
      - b
    dest: a
    time: 50
    proportions:
      - 0.1