Possible to post updates from a remote server?

[CodyCBakerPhD]

Let's say I'm running an operations in the cloud (on AWS EC2 or something) and inside that instance I would conceptually have various TQDM bars running

But peeking into such an instance can be a pain

Would it be possible to host a dedicated website on our end, kind of like how neurosift works, that we can tell our TQDM updates inside the remote instance to forward updates to? (the bars would look just like the client HTML pages do in our demos, it's just the Python server wouldn't be local to the machine and wouldn't forward to localhost)

Without loss of generality this could also work for a local Python server but the progress updates on the webpage could then be sharable with other devices

How hard do you think this would be? @garrettmflynn

[garrettmflynn]

Can you elaborate on how the monitored progress bars are spawned by the user of these EC2 instances?

If it's similar to the GUIDE (e.g. triggered by an endpoint like /run_conversion), then the "pain" you're describing would be minimal because the client already knows the URL to subscribe to updates directly from. So there's nothing stopping them from sending a request to /events too.

Would it be possible to host a dedicated website on our end, kind of like how neurosift works, that we can tell our TQDM updates inside the remote instance to forward updates to? (the bars would look just like the client HTML pages do in our demos, it's just the Python server wouldn't be local to the machine and wouldn't forward to localhost)

We could technically do this. But we'd need to assume that the server URL is static/known (which would remove the need for this approach, as described above) and will then need to pass additional metadata about our progress bar to make any of this intelligible.

1. Some client_id to only pass back updates that are relevant to a particular client, as the server would otherwise indiscriminately pass all the updates attached to the N current clients.
   • However, this again assumes that a client requested this action using a REST API, which would again limit the need for this approach. If this doesn't happen, then there's a chance we'd send a ton of superfluous updates to all clients.
2. Some description of the task being run. Without this, there's no way for a client to know what the process is about—thereby reducing the utility of knowing it at all.

Without loss of generality this could also work for a local Python server but the progress updates on the webpage could then be sharable with other devices

I understand that this could be a benefit and, as far as I've understood, would be the main reason why this approach would be desirable. But what's a concrete use-case where we'd want to observe progress updates triggered from a different client?

[CodyCBakerPhD]

But what's a concrete use-case where we'd want to observe progress updates triggered from a different client?

Concrete use cases include remote processing in the cloud (unrelated to GUIDE but is related to other projects)

I also personally see it as a great improvement over classic TQDM stdout usage, which can have problems with the position and leave rendering on various systems (commonly resulting on progress updates getting written to newlines, doesn't look great)

So philosophically, a step away from trying to make a terminal printout look decent in lieu of the proper solution, to let a pretty browser handle it

This is rather outside scope of GUIDE however, so I'm just asking out of curiosity

Can you elaborate on how the monitored progress bars are spawned by the user of these EC2 instances?

Let https://customtqdmprogress.com be the base URL (apparently someone already has prettyprogress.app lol)

First request to the site generates new hash for a subpage - this is sent back to me on the Python side and allows me to navigate to https://customtqdmprogress.com/<some UUID4> which at this point is completely blank or saying something like 'waiting for progress data from server...'

Second request sends the TQDM __init__ information (stuff like total iterations, desc, etc.) to the webpage which populates the empty progress bar

All further posts from the server to the website are increments called during update showing the progress in as close to real time as can be had with the lag that can now exist because everything is going over the web

Hope that helps, anything else I can clarify?

[garrettmflynn]

Ah, maybe I've misunderstood how generic your ask is.

From what I read above, your description omits how the progress request was made in the first place. Would it be correct to say that you'd want to, for instance, run a Dendro job that includes a progress bar. Once this request is submitted, the server would redirect you to https://customtqdmprogress.com/<some UUID4> where you could then see the progress bar in the browser?

In this and any other case, though, wouldn't it make sense to assume that Dendro renders the progress itself?

If https://customtqdmprogress.com/<some UUID4> request (which could also just be https://customtqdmprogress.com with a client ID sent with the request) is spawning the progress bar, we would pass updates back to the client in an identical way as we do with the GUIDE. Unless we lack a reliable EC2 instance URL (e.g. https://customtqdmprogress.com or any subpath) to request from, this wouldn't be a problem.

With this discussion, we may want to ensure desc can be transferred somehow in the general requests. But that's the main improvement that I see here.

[CodyCBakerPhD]

I'm not talking specifically about Dendro actually - though even for that, wouldn't it simplify Dendros life to just need an iframe that points to a central manager?

desc is (or should be) a part of the format_dict BTW, and is otherwise a part of the **kwargs passed into the tqdm.__init__

I guess my question is, do you know how to serve such a site and what would be required of the internals for handling the requests to/from the site?

Not a high priority ATM, this is just a 'do it in your free time' kind of thing

[garrettmflynn]

Got it, thanks for the clarification.

I'm not talking specifically about Dendro actually - though even for that, wouldn't it simplify Dendros life to just need an iframe that points to a central manager?

Actually no. In this case, I'd rather use a Lit / React component that can be shared across services like Dendro. This would simply require an initial events URL to subscribe to (e.g. new ProgressHandler({ url: "https://https://dendro.vercel.app", events: "events"}), and have a method (e.g. .run("/compute")) that would trigger a progress bar to be created on the service, which would pass updates through https://https://dendro.vercel.app/events and trigger a progress bar to display in the service client itself.

In either case, Dendro would have to update its backend to allow the updates to be forwarded by using a custom class from tqdm_publisher. I'd rather spell out how to add an /events endpoint (e.g. with the ProgressHandler class and our demos / documentation) than maintain centralized infrastructure for this.

This centralized infrastructure just requires a lot of tightly-coordinated networking requests to function at all, where putting the responsibility on the service itself to implement these things limits the possible points of failure here.

I guess my question is, do you know how to serve such a site and what would be required of the internals for handling the requests to/from the site?

I haven't built a dynamic site like this before with custom subpaths, but it doesn't appear to be that complicated. As I've been communicating, though, this architecture is intrinsically complex—more than it really needs to be.

Though I may just not be in agreement about how much responsibility for progress management that a service like Dendro would want to defer to us.

[CodyCBakerPhD]

What was the link to that repo you showed me for WebTQDM? Happy to close this soon as I find it

[garrettmflynn]

The Python library and application repositories are contained in a separate tqdme organization at the moment.