perf: optimise SQL query in observeRetrievalResultCodes

[bajtos]

Rework observeRetrievalResultCodes to execute a single SQL query to update all daily codes in one go.

Before this change, we would run ~1k individual queries, consume a lot of CPU and trigger a CPU throttling alert.

As part of this change, I am also adding a console log to tell us how many rows - tuples (day, code) - we are updating in each loop iteration.

[bajtos]

Cross-posting from https://space-meridian.slack.com/archives/C06S76341B8/p1739288083931459?thread_ts=1739273894.720459&cid=C06S76341B8

The CPU usage spike happened between 10:36:15-10:36:45 UTC. According to papertrail logs, Fly.io restarted the service at 10:36 UTC. Here are the logs from that time:

Feb 11 10:36:17Z  > start
Feb 11 10:36:17Z  > node bin/spark-observer.js
Feb 11 10:36:18Z  Selected JSON-RPC endpoint https://api.node.glif.io/rpc/v0
Feb 11 10:36:18Z  Querying scheduled rewards from impact evaluator
Feb 11 10:36:21Z  Block number too old, GLIF only provides last 2000 blocks, querying from -1900
Feb 11 10:36:21Z  Querying impact evaluator Transfer events after block 4696890
Feb 11 10:36:21Z  Scheduled rewards for 0x39656B13B4D7009C270E2CC6D6698A3C97Fc02D6 88143932255125704n
Feb 11 10:36:22Z  Loop "Retrieval result codes" took 3985ms
Feb 11 10:36:23Z  Found 0 Transfer events
Feb 11 10:36:23Z  Loop "Transfer events" took 4605ms
Feb 11 10:36:23Z  Scheduled rewards for 0xc57450fa57748a09b437A3E3116673f4C64e64Dd 6701134242610932n
Feb 11 10:36:23Z  Scheduled rewards for 0x6E5Afb67b5978b1F7e56358577C6bf93789fe572 81390879695710545n
Feb 11 10:36:23Z  Scheduled rewards for 0xCE53e6dEfbD1f0810BFcd691660ad311d0DB24f7 77135747466396775n

The log continues with hundreds of "Scheduled rewards for 0x..." messages.

I suspect when spark-observer starts and executes the first iteration of the "Retrieval result codes" loop or "Transfer events" loop, too much CPU is needed to process the data.

spark-observer is currently running on shared-cpu-4x ($7.78/month). The next step is shared-cpu-8x ($15.55). The extra cost is negligible, but I am not confident that a beefier machine would prevent the CPU throttling alert. If the VM upgrade does not prevent the alert, we can also stay on the current VM size.

I checked the source code of the loops and see some optimisation opportunities, but I am not convinced it's worth our effort right now.

• observeTransferEvents - we update last_checked_block only when a transfer happens - I think that's once per week? - which means that most of the time, last_checked_block is 1-7 days old and we need to query 1900 blocks into the past. I believe this is only one JSON RPC API call, it apparently returns 0 events, so this loop should not be CPU intensive.
• observeRetrievalResultCodes - it executes one SQL query for each tuple (day, result_code) found in the InfluxDB data. When I executed the query myself, I got 962 rows back. That means 962 sql queries. This can be CPU intensive, as the PG client has to process a lot of data exchanged over the wire.

[pyropy]

Great job on optimizing this! 👏🏻

[juliangruber]

😍

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ error: ON CONFLICT DO UPDATE command cannot affect row a second time observeRetrievalResultCodes(observer) View Issue
• ‼️ Error: Connection terminated unexpectedly observeRetrievalResultCodes(observer) View Issue

_{Did you find this useful? React with a 👍 or 👎}

[bajtos]

Unfortunately, this does not work.

1. The SQL query fails, as Sentry pointed out above.
2. The service startup is still triggering CPU throttling.

I am going to revert this pull request.

[bajtos]

Interesting! After a short investigation, I found the cause of the bug, I think we discovered a flaw in the InfluxDB query.

Instead of reverting this PR, I opened a new one to fix that flaw: #319