Facebook crawler excessively hitting downloads endpoint

[achave11-ucsc]

The following displays the offensive crawler by IP and binned by hour:

---

CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-service-prod
start-time: 2025-01-07T10:52:59.965Z
end-time: 2025-01-07T21:27:21.206Z
query-string:

fields @message
| filter @message like 'Download of file'
| filter @message like 'facebook'
| parse `x-forwarded-for` '*, *' as source_ip, proxy_ip
| stats count(*) as num_requests, sum(file_size) / 1024 / 1024 / 1024 as volume by bin(1h) as datetime, source_ip, `user-agent`
| filter num_requests > 10
| sort datetime desc, num_requests desc
| display datetime, source_ip, num_requests, `user-agent`, volume

---

datetime	source_ip	num_requests	user-agent	volume
2025-01-07 19:00:00.000	57.141.5.12	12	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	0.4041
2025-01-07 19:00:00.000	57.141.0.10	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	0.7261
2025-01-07 19:00:00.000	57.141.5.16	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	0.3957
2025-01-07 19:00:00.000	57.141.5.1	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	0.5608
2025-01-07 18:00:00.000	57.141.5.11	13	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	1.5473
2025-01-07 18:00:00.000	57.141.5.10	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	0.4446
2025-01-07 18:00:00.000	57.141.5.14	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	1.0543
2025-01-07 17:00:00.000	57.141.5.7	14	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	8.2623
2025-01-07 17:00:00.000	57.141.5.1	14	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	3.6244
2025-01-07 17:00:00.000	57.141.0.2	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	1.5474
2025-01-07 16:00:00.000	57.141.5.8	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	4.3393
2025-01-07 14:00:00.000	57.141.5.2	16	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	7.0154
2025-01-07 14:00:00.000	57.141.5.6	13	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	0.4127
2025-01-07 14:00:00.000	57.141.5.3	12	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	15.7544
2025-01-07 14:00:00.000	57.141.5.14	12	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	13.893
2025-01-07 14:00:00.000	57.141.5.10	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	2.0579
2025-01-07 13:00:00.000	57.141.5.8	17	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	1.3593
2025-01-07 13:00:00.000	57.141.5.2	16	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	1.0003
2025-01-07 13:00:00.000	57.141.5.22	16	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	1.1724
2025-01-07 13:00:00.000	57.141.5.7	13	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	4.1591
2025-01-07 13:00:00.000	57.141.5.15	12	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	1.842
2025-01-07 13:00:00.000	57.141.0.5	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	1.5095
2025-01-07 13:00:00.000	57.141.5.17	11	meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)	0.1058

---