Improve Gateway Server throughput

[adriansmares]

Summary

This PR is a follow up of some issues observed while deploying Redis-based rate limiting.

Changes

• Make the source address firewall concurrency safer.
  • The protocol used to store the latest last seen timestamp is now concurrency safe.
• Do not put extra pressure on Redis when the rate limiting is triggered, by rate limiting logs in memory always.
• Make packet filtering (rate limiting and firewalling) parallel.
  • Currently, this is 'single goroutined'. This means that any throughput limitations here (like the rate limiting going from in memory to Redis, introducing roundtrip latency) lead to packet loss, as UDP has no retransmissions in case the socket buffer is full.
  • Now, we actually run the firewalls and packet filtering in a worker pool. This has the added advantage of actually making the packet loss visible in the worker pool metrics (because if we receive packets faster than we can filter them, we register a drop).

Testing

Steps

This will be tested on staging1 and other environments. There are no actual tests to be done manually, as this is mainly an optimization PR.

Results

We've been able to heavily reduce the number of Gateway Servers used due to the relief of the single packet processing bottleneck.

Regressions

N/A.

Checklist

• Scope: The referenced issue is addressed, there are no unrelated changes.
• Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in README.md for the chosen target branch.
• Documentation: Relevant documentation is added or updated.
• Testing: The steps/process to test this feature are clearly explained including testing for regressions.
• Infrastructure: If infrastructural changes (e.g., new RPC, configuration) are needed, a separate issue is created in the infrastructural repositories.
• Changelog: Significant features, behavior changes, deprecations and fixes are added to CHANGELOG.md.
• Commits: Commit messages follow guidelines in CONTRIBUTING.md, there are no fixup commits left.