Total memory limitation for PCA metrics ?

[b-grimaud]

Whenever I try to calculate PCA metrics on HD-MEA data (lots of units, lots of channels), I always run out of memory before the computation can finish, on a machine with ~100 GB of available RAM.

I know that the job kwargs have a "total_memory" argument, but as far as I can see that only applies to operations performed on recordings (I'm assuming filtering, spike sorting...).

Would it be feasible to progressively calculate metrics then only keep the output in memory ? Or maybe temporarily write the arrays to disk at the expense of speed ?

[zm711]

We are super bad about checking the discussions. Sorry. I think in the end the goal is to extend the job_kwargs to the PCA but you're right that for now it doesn't have the same benefits (it's a problem we have on Windows workstations too). But maybe @alejoe91 would be better to comment here.

[b-grimaud]

Thanks ! I checked the issue you tagged me in, it seems like having sklearn's IPCA as a backup could actually help.

Surprisingly, while testing #3670 I found out that when using the concatenated mode, the computation is actually small enough to fit in memory. I'm assuming the size of the array scales with the number of channels when using the other modes ?