Technologies cleanup automation

[max-ostapenko]

I added two steps to a crawl pipeline:

• an assertion for the technologies data on staging (optional, doesn't block the downstream tasks) to track the scale of the issue
• cleanup for the crawl.pages table.

This should give us a clean monthly technology data until we'll find time to fix those few anomalies on the wptagent.

Closes #43

[tunetheweb]

Are the wappalyzer tables updated with every crawl? Is that planned to be continued after we remove the legacy tables? (I thought this WAS a legacy table!)

[max-ostapenko]

Those are updated with every repo update, see https://github.com/HTTPArchive/wappalyzer/blob/main/.github/workflows/upload.yml.

These are not dependent on crawl detections.
But they should represent the crawl configuration considering we don't merge changes during crawl.

[tunetheweb]

Any risk that a technology that is removed (or renamed) mid-crawl causes it to be deleted? I try not to merge things mid-crawl, but stranger things have happened. But not sure how to deal with that to be honest...

[max-ostapenko]

Sure. But I think monthly manual changes pose a bigger risk.

• I hope 🤞🏽 we'll find a solution to avoid corrupted values on wptagent during the next attempt.
• To avoid retrospective issues I expect we'll be able to extend reports.cwv_tect_technologies with clean historical data.
• We could also backup the original data for N months just in case.