RSR per allocator

[bajtos]

Provide retrieval-based RSR calculated on a per-allocator basis.

• spark-evaluate code writing the new data
• REST API in spark-stats exposing new data
• Visualisation on the Spark Public Dashboard
  • Spark Allocator Stats Table on https://dashboard.filspark.com/
  • Per-allocator page similar to https://dashboard.filspark.com/provider/f03239692

Pre-requisites:

• RSR per client #193

Related discussions:

• https://filecoinproject.slack.com/archives/C0807FBUKC3/p1731226294482699
  ^^ The goal is to calculate RSR for deals made by a particular data onramp provider. See https://app.hex.tech/protocol/app/ccbb785c-7fad-42b5-9553-609cde5c6acc/latest
• Break down retrieval success rate by Deals, Clients, Allocators, MetaAllocators #139
• Provide miner RSR per client and per allocator spark-checker#79

Notes:

• In spark-evaluate, we are (or will be) mapping retrievals to clients, see RSR per client #193.

• The missing piece is aggregating per-client data to per-allocator data. The important insight is that each client is linked to a single allocator only.

• fil-deal-ingester maintains a mapping between clients and allocators in the table allocator_clients.

• We need to access this mapping from spark-evaluate or spark-stats, depending on where we aggregate per-client to per-allocator stats.

  Possible options to consider:

  1. Enhance each retrieval task in the round details with the list of allocators in addition to the list of clients.
     • This may be the easiest path if we map clients to allocators in spark-evaluate.
  2. Implement a new REST API endpoint in spark-stats to allow spark-evaluate or spark-stats to map clients to allocators.
     • We must be careful about performance - we don't want to make one request for each client.
     • This endpoint can be useful for per-client dashboard as it will allow us to show the allocator from which the client has received DataCap.
     • spark-stats can provide a public facade for this endpoint using the same mechanism we have already in place for getting deals eligible for retrieval testing (source code).

[bajtos]

Since one allocator can have multiple clients, we should not combine per-client aggregated stats to produce per-allocator stats.

For example, one measurement can be linked to two clients from the same allocator. This should account for 1 total measurement. If we combine per-client aggregated stats, we would get 2 total measurements.

Based on the above, I think we need to produce per-allocator stats inside spark-evaluate using a similar algorithm to the one producing per-client stats.

1. Loop over all measurements.
2. Map each measurement to a list of clients of the deal(s) measured and then from clients to allocators. The goal is to get Map<allocator, measurement[]>.
3. For each allocator, aggregate the measurements to calculate the stats (total, successful, successful_http, etc.)