Optimize ParquetReader

[mariosasko]

Optimize the ParquetReader component by reading files in chunks (of size 1000) and only decoding the content_key column (reads less bytes and makes the Arrow to Python conversion faster)

This Colab shows the difference in speed:
https://colab.research.google.com/drive/14c1lvasWYg0ScsIxeTZcqIUVV8ulCYUi?usp=sharing

[guipenedo]

Hi!
I suppose we can indeed increase the batch size a bit, I just don't want it to be too big so as to not considerably increase memory consumption. 1k is fine I guess, maybe we can make it configurable?

Regarding the columns, the general approach with readers is to dump the remaining columns on metadata as people might want to keep it (and even the id could come from the parquet file). Maybe also add an option to only load the text column?

Otherwise LGTM

[mariosasko]

The datasets lib has the default batch size of 10_000, and its primary focus is not processing data at scale (we got no complaints there regarding this 🙂), so I think 1000 here should be okay (we can expect users to have enough RAM).

[guipenedo]

just a small note that the yield should be outside the track_time block, otherwise other blocks' execution time will be taken into account here

[guipenedo]

Looks great, thanks a lot! Regarding the RAM comment, just as an example I'm currently running a data processing job on a few k cpus where each has ~2GB of ram, and the usage is almost the full 2GB of RAM per cpu, hence my concern for keeping usage low when possible. But with the configurable option we can now tune with a performance/memory trade off, so all good.
I've made a small change which addresses my yield comment above. Unfortunately it also adds a new list, if you find a better way to do it feel free to change, if not we can merge.