Best libraries for distributed computing with modelx

[bakerwy]

Hi,

I am looking for good libraries for distributed computing with modelx.

Of course, each library has their own pros and cons.

but if I want to use with modelx, ipyparallel seems the best because ipyparallel can send each exported model to each engines directly
without modification of model.

if I want to use other libraries such as Ray, Dask, Joblib, I need to manually adjust functions to send them to engines.
In case of Ray, Ray can send class to engines but it is hard to use with Windows due to stack overflow.
Actuarial model has thousands of functions so.. manually adjust all of them would be too much work for end users.

Are there good libraries for distributed computing or a way of use efficiently well known distributed computing libraries with modelx?

[fumitoh]

I'm not clear as to why you need to manually adjust functions to send them to engines when using Ray, Dask, Joblib. The latest versions of these libraries seem to support Python 3.11 or 3.12, so they should be able to run exported models without adjustments, right?

[bakerwy]

Hi, sorry my question was not clear.

For example, let's say Model A. A is exported as independent library.

In case of ipyparallel, we can run distributed computing with the code below. It uses whole model itself. so What I need to do is just run them.

code =
'''
import exported_A
import numpy

func
'''

rc[:].execute(code)

but if I want to use ray, ray needs decorator.
also in case of Dask, Dask needs their own executor for distributed computing. if I have thousands of functions, it would be not so easy to use them since I have to execute all of functions or have to select some of them for distributed computing.

Am I misunderstanding? if you have some idea, it would be very good.

Also, I tried to use Ray for class but ray actor dies immediately because of stack overflow with python 3.11. Exported model runs well with the version.

[fumitoh]

I think I now see your point.

[vineetgupta086]

Hi,

Just to add to the discussion, I myself have tried to look into something similar so that the computing can be distributed or parallelized while using modelx. I came across a library called modin, which appears to scale up pandas.
It appears that it is very easy to implement and works with the libraries you mentioned - ray and dask, and even enables you to choose the execution engine between the two.

Unfortunately, I've not been able to implement anything of sort yet, but I thought it was worth sharing.

[fumitoh]

How about Modin? It aims to be a drop-in replacement for pandas. So all you need to do is bind pd not to pandas but to modin.pandas instead. I tested this library with modelx a couple of years ago, but I couldn't make it work. It might work with exported models though.

[bakerwy]

Thanks for sharing your idea! I will look it up :)