When too many query/ref pairs are reported, final_stats uses too much of memory due to its naive Python implementation #185

karel-brinda · 2022-09-23T13:27:41Z

Happens with eg 1M queries

This is the problematic part:
https://github.com/karel-brinda/mof-search/blob/e79b0c842ed919f1787a3071a52065d2317c8f71/scripts/final_stats.py#L109

Probably should be possible optimize by that the output is sorted according to ref (so it's sufficient to keep just stats for the last ref in memory)

karel-brinda added the bug Something isn't working label Sep 23, 2022

karel-brinda self-assigned this Sep 23, 2022

karel-brinda changed the title ~~When many query/ref pairs are reported, final_stats uses to much of memory~~ When many query/ref pairs are reported, final_stats uses too much of memory Sep 23, 2022

karel-brinda removed their assignment Dec 6, 2022

leoisl added the paper label Jan 10, 2023

karel-brinda changed the title ~~When many query/ref pairs are reported, final_stats uses too much of memory~~ When too many query/ref pairs are reported, final_stats uses too much of memory due to its naive Python implementation May 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When too many query/ref pairs are reported, final_stats uses too much of memory due to its naive Python implementation #185

When too many query/ref pairs are reported, final_stats uses too much of memory due to its naive Python implementation #185

karel-brinda commented Sep 23, 2022

When too many query/ref pairs are reported, final_stats uses too much of memory due to its naive Python implementation #185

When too many query/ref pairs are reported, final_stats uses too much of memory due to its naive Python implementation #185

Comments

karel-brinda commented Sep 23, 2022