Index issue with generating data script [discussion]

[ncclementi]

See dask/dask#8983

[jrbourbeau]

Thanks for bringing this up, I had forgotten about this issue. Do you have a sense for how big a problem this is for us when running these benchmarks? My guess is that if, for example, partitions are all 103 MB instead of 100 MB, that won't impact us significantly.

[ncclementi]

When we took a look at this with Ian, it was between 5-10% increase in memory footprint. See comment dask/dask#8983 (comment)

[jrbourbeau]

Okay, good to know. My question is still does this lead to problems when running h2o benchmarks? Do we need to devote time to resolving dask/dask#8983 in order to make progress on h2o benchmarks, or is it just a related issue to be aware of (i.e. a valid bug, but not mission critical)? I think it's not a blocking issue, but I could be wrong

[ncclementi]

My question is still does this lead to problems when running h2o benchmarks?

Hard to know, we haven't compared runs with and without this problem. Things will still run, as seen in the runtime, but we don't know how it affects.

Do we need to devote time to resolving dask/dask#8983 in order to make progress on h2o benchmarks, or is it just a related issue to be aware of (i.e. a valid bug, but not mission critical)? I think it's not a blocking issue, but I could be wrong

I don't think it's a blocking issue either, to get started, but it's something to be aware.