cli: split fetch-recent-miner-measurements and evaluate-measurements

[juliangruber]

Usage:

$ node bin/fetch-recent-miner-measurements.js f0...
...
$ node bin/evaluate-measurements measurements-f0....ndjson
 → evaluating round 19117n
 → added 183 accepted measurements from this round
 → evaluating round 19118n
 → added 0 accepted measurements from this round
 → evaluating round 19127n
 → added 181 accepted measurements from this round
 → evaluating round 19128n
 → added 0 accepted measurements from this round
Found 364 accepted measurements.
  IPNI_ERROR_404                           364        (100%)
Wrote evaluation to evaluation-5bf6db8.txt

This is a first step towards evaluating arbitrary rounds. Next: #402

• After merging, update Spark troubleshooting docs

[bajtos]

I'll review this PR next week. I am afraid there will be a merge conflict with #396 😢

How should we proceed to handle that in the least painful way?

[bajtos]

Before your change, the default behaviour is to not show rejected measurements in the output file. IIUC the proposed version, the first step will produce an intermediate file that includes rejected measurements too. I think this can create confusion when people not familiar with the concept of accepted/rejected scores use these tools.

An idea to consider:

• The first step creates a file with a suffix like -raw or -all
• The second step either creates files with no suffix (i.e. preserve the current output filenames) or use a different suffix like -accepted or -evaluated.

WDYT?

[juliangruber]

Before your change, the default behaviour is to not show rejected measurements in the output file. IIUC the proposed version, the first step will produce an intermediate file that includes rejected measurements too. I think this can create confusion when people not familiar with the concept of accepted/rejected scores use these tools.

An idea to consider:

• The first step creates a file with a suffix like -raw or -all
• The second step either creates files with no suffix (i.e. preserve the current output filenames) or use a different suffix like -accepted or -evaluated.

WDYT?

Since e2b5663, the output file will have .evaluation.txt suffix, which I think is great. I don't think there's any expectation about the shape of the intermediary file, it is supposed to contain all measurements submitted (in my mental model) and that it does.

[juliangruber]

I'll review this PR next week. I am afraid there will be a merge conflict with #396 😢

How should we proceed to handle that in the least painful way?

The merge conflict isn't going to be hard to deal with if we reapply https://github.com/filecoin-station/spark-evaluate/pull/396/files on top of this PR. Therefore I propose to merge this one first.

[bajtos]

I'll review this PR next week. I am afraid there will be a merge conflict with #396 😢
How should we proceed to handle that in the least painful way?

The merge conflict isn't going to be hard to deal with if we reapply https://github.com/filecoin-station/spark-evaluate/pull/396/files on top of this PR. Therefore I propose to merge this one first.

I was thinking about this some more, and would like to rework #396 - see #396 (comment)

I agree to land this pull request first.

[bajtos]

I quickly skimmed through the changes and I don't see any obvious problems. Let's land this and then fix any issues later as we discover them.

[bajtos]

Before your change, the default behaviour is to not show rejected measurements in the output file. IIUC the proposed version, the first step will produce an intermediate file that includes rejected measurements too. I think this can create confusion when people not familiar with the concept of accepted/rejected scores use these tools.
An idea to consider:

• The first step creates a file with a suffix like -raw or -all
• The second step either creates files with no suffix (i.e. preserve the current output filenames) or use a different suffix like -accepted or -evaluated.

WDYT?

Since e2b5663, the output file will have .evaluation.txt suffix, which I think is great. I don't think there's any expectation about the shape of the intermediary file, it is supposed to contain all measurements submitted (in my mental model) and that it does.

I believe the old version included evaluation results in the measurements written to the per-miner file. After your change, the first script that fetches measurements produces a file that does not have information about the evaluation result. The second script produces only TXT file, so there is no way how to further analyse the processed per-miner measurements using jq or other tools.

Please correct me if I'm wrong.

I think this can be also iterated on in a follow-up pull request.

Please remember to update our docs (https://docs.filspark.com/troubleshooting-miner-score, the source is in Notion) after you land this change.

[juliangruber]

I believe the old version included evaluation results in the measurements written to the per-miner file. After your change, the first script that fetches measurements produces a file that does not have information about the evaluation result. The second script produces only TXT file, so there is no way how to further analyse the processed per-miner measurements using jq or other tools.

Please correct me if I'm wrong.

Now, the measurements fetching prints a summary to stdout and produces a .ndjson file of raw measurements.

The evaluation script prints a summary to stdout and produces a .txt file of per-measurement evaluation. I will add a .ndjson file so that this data is machine processable.

[juliangruber]

Please remember to update our docs (https://docs.filspark.com/troubleshooting-miner-score, the source is in Notion) after you land this change.

Updated in https://www.notion.so/spacemeridian/Troubleshooting-Miner-Score-664ccb2e5c264b39986df09db6b445a4 👍