fix(ingest): decrease ingest memory usage for large datasets

[anna-parker]

resolves #

preview URL: https://ingest-memory-fixes.loculus.org/

Summary

When dealing with organisms with a large number of sequences (e.g. influenza A has 1.3M sequences) even just reading it the entire metadata is highly memory intensive - this PR switches to streaming metadata where ever possible and is similar to the change @corneliusroemer made in #2277 to stream sequences.

Note that in the case where segments are grouped the entire file still must be read into a dictionary.

Screenshot

PR Checklist

• Improve batching: this still reads in both fasta and metadata and is thus the current memory bottleneck -> to remove this the input files need to be sorted
• extend tests to ndjson files