[FEAT] Explore AnyDataframe (spark, ray, and dask) integration

[AzulGarza]

Currently, hierarchicalforecast only supports a pandas dataframe as input. For the library to scale horizontally, we need to explore different alternatives on how to integrate frameworks such as spark, ray, and dask. This issue is intended to discuss possible approaches.

The main problem right now is that hierarchicalforecast takes a dataframe and then converts it to numpy arrays that are fed to the reconciliation methods. Because of the characteristic of the hierarchical reconciliation problem, those numpy arrays must be used at once in each reconciliation method.

Any ideas?

[AzulGarza]

@kvnkho

[elephaint]

My thoughts:

• Methods that don't require complex linear algebra (basically the simple ones) could be rewritten in DataFrame-only operations, which would allow to abstract away from Numpy and allow horizontal scaling.
• [FEAT] Add polars support #305 enables (implicit) support for a.o. Dask via Modin, but as long as we keep the numpy stuff we'd still rely on having a single machine executing the linear algebra at some point.
• A proper solution for the complex stuff woul be some form of batch-wise reconciliation strategies, similar to neural network training. I'd have to explore, though.

[jmoralez]

Dask has arrays, so we could try supporting dask dataframes, converting to arrays and using that API.