Issue #182 Fixes a problematic loop in produce_simulation_results

[bjohnson5]

Resolves Issue #182 by removing set.join_next() from the tokio::select! statement and waiting for tasks to finish outside of the loop. Also does some error handling in the run function in case we encounter an error after starting data collection threads.

[carlaKC]

let's squash this commit 👍

[carlaKC]

Thanks for the fix (and detection!!), only one real comment here.

Going to tag a patch once this goes in so that we can get a fresh docker image out.

[carlaKC]

Let's pull this out into a closure to de-dup it a little?

[bjohnson5]

Any suggestions on what that closure should look like? It will have to be async which is always tricky to me.

[carlaKC]

Async closures are dark magic indeed :') Since we're also using it in produce_simulation_results it can probably just be a utility function instead.

Closure could have been something like this:

let wait_for_tasks = |mut tasks: JoinSet<()>| async move {
        while let Some(res) = tasks.join_next().await {
            if let Err(e) = res {
                log::error!("Task exited with error: {e}.");
            }
        }
};

[carlaKC]

TIL that you can return a value with break in rust - perhaps we can do this here?

let result = loop .... and then break Ok(()) / break Err(()) and then go on to the joinset wait at the end. Has the advantage of always waiting for everything to close out cleanly vs current approach where we only wait on Ok() exits.

[bjohnson5]

Yes, that is a great idea! Just added that in the latest commit.

[carlaKC]

tACK, last comment non-blocking (only saves us a few lines).

CPU looking muuch better 😻