-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
use asyncio to manager workers #100
Comments
So no real loss then? 😈 I briefly experimented with implementing a "ruffus-lite" based on asyncio, and then I tried again with either curio or trio. If you are serious I think it is worth looking at those as well, for reasons outlined in the article linked for trio. The biggest issue of all is that asyncio is not good for CPU bound Python code because it's single threaded, you can't escape with multithreading because of the GIL, and it doesn't have any equivalent to multiprocessing. That was okay for me when I tried my experiment, because ocrmypdf (my application that uses ruffus, and the only open source dependent of ruffus I know of) is mostly "process.wait()-bound". ocrmypdf kicks off other programs and waits for results. But as it matures, it's been accumulating more in house functionality and relies on ruffus+multiprocessing for performance. You can kick off an asyncio child process that also runs Python, but it's not a pretty sight. https://docs.python.org/3/library/asyncio-protocol.html#subprocess-protocols concurrent.futures might also be worth investigating too. |
Hi @jbarlow83 , my thinking was to replace the cooperative multitasking pool in gevent with something based on asyncio - not to change the workflow part. |
This will remove dependency on gevent, but this will break python 2.7 compatibility.
The text was updated successfully, but these errors were encountered: