GPU tests failing (and taking a long time to run)

[ivirshup]

The GPU tests seem to be failing, and taking at ~1.5 hours to do so, which is a little expensive

• Failing job: https://github.com/scverse/anndata/actions/runs/7797158098/job/21263231065
• Previous working job: https://github.com/scverse/anndata/actions/runs/7710103461/job/21012752331

It looks like this was triggered when #1354 was merged, but I doubt this is the cause. I suspect it has to do with CUDA versions

Rapids single cell's tests aren't failing, but also has some slight different libraries installed. Includes: libcublas and python

TODO:

• Set faster timeout on the GPU CI job (don't need to pay for an hour and half if the job should take <10 min when working added yaml for gpu_ci #1359
• Fix environment, potentially need to pin some stuff added yaml for gpu_ci #1359
• Re-enable recent cuda version for testing

cc: @flying-sheep @Intron7

---

Failure messages look like:

FAILED anndata/tests/test_views.py::test_view_of_view[cupy_csc-spmatrix_bool_subset-spmatrix_bool_subset] - RuntimeError: Runtime compilation failed

[ivirshup]

Diff between environments: https://www.diffchecker.com/VUIKrubT/

Interestingly, only the build number for cupy and cupy-core are different. numba, llvmlite, libgcc, and a few other libs are notably different. These could be related since it's a compiler error.

[ivirshup]

We've changed how cuda is installed and pinned cuda to 11.8, which was released in 2022.

Things still seem to fail with the new installation method and cuda 12.3 and 12.2.

I've merged the change to use 11.8 (plus a timeout and better version reporting) as a temporary fix for now.

[flying-sheep]

Please remove the toplevel .ci directory that holds one file. That file can live in .github.

[ivirshup]

That directory will hold more files with the minimum version test job

[flying-sheep]

OK, but currently, that PR calls the directory ci, not .ci. Please unify that then.

[Intron7]

@flying-sheep #1363 does that

[flying-sheep]

Awesome! Also thanks for helping fix this!