[NO MRG] Test CI with NumPy 2

[jakirkham]

Follow up to PR: #1375

Requires PR: rapidsai/cudf#16300

As noted previously ( #1375 ), there is a cycle between Dask-CUDA and cuDF when it comes to testing. Due to this it is not possible to test either Dask-CUDA or cuDF with NumPy 2 without both packages being built to support NumPy 2. However doing both at the same time is not possible

So we chose to break the cycle by updating Dask-CUDA first in PR ( #1375 ). This was done in light of Dask-CUDA's lighter usage of NumPy's API. cuDF touches more of NumPy's API so we want to test that more thoroughly. Admittedly there has been some offline testing as part of this bringup process. Still we would like to reconfirm in CI

Now that Dask-CUDA packages are up and are installable with NumPy 2, we can incorporate Dask-CUDA into cuDF's CI testing with NumPy 2, which is now happening in PR ( rapidsai/cudf#16300 ) as explained in comment ( rapidsai/cudf#16300 (comment) ). Once cuDF CI testing is complete and the cuDF PR is merged, we will have NumPy 2 compatible cuDF packages. These can then be installed with Dask-CUDA in an environment using NumPy 2 on CI for testing Dask-CUDA

This PR will serve the purpose of testing cuDF + Dask-CUDA + NumPy 2 in CI here once cuDF compatible NumPy 2 binaries are available

[jakirkham]

Closing and reopening to restart CI now that cuDF is done: rapidsai/cudf#16300

[jakirkham]

One build failed with this unrelated error about CUDA initialization issues (see details below). Unfortunately this error seems to be showing up in different places ( for example: rapidsai/cudf#16300 (comment) ). Wonder if there is a misconfigured CI node

/opt/conda/envs/test/lib/python3.11/site-packages/cudf/utils/_ptxcompiler.py:64: UserWarning: Error getting driver and runtime versions:

stdout:



stderr:

Traceback (most recent call last):
  File "/opt/conda/envs/test/lib/python3.11/site-packages/numba/cuda/cudadrv/driver.py", line 254, in ensure_initialized
    self.cuInit(0)
  File "/opt/conda/envs/test/lib/python3.11/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/opt/conda/envs/test/lib/python3.11/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [999] Call to cuInit results in CUDA_ERROR_UNKNOWN

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/opt/conda/envs/test/lib/python3.11/site-packages/numba/cuda/cudadrv/driver.py", line 292, in __getattr__
    self.ensure_initialized()
  File "/opt/conda/envs/test/lib/python3.11/site-packages/numba/cuda/cudadrv/driver.py", line 258, in ensure_initialized
    raise CudaSupportError(f"Error at driver init: {description}")
numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: Call to cuInit results in CUDA_ERROR_UNKNOWN (999)


Not patching Numba
  warnings.warn(msg, UserWarning)
Traceback (most recent call last):
  File "/__w/dask-cuda/dask-cuda/dask_cuda/benchmarks/local_cudf_shuffle.py", line 27, in <module>
    import cudf
  File "/opt/conda/envs/test/lib/python3.11/site-packages/cudf/__init__.py", line 20, in <module>
    validate_setup()
  File "/opt/conda/envs/test/lib/python3.11/site-packages/cudf/utils/gpu_utils.py", line 55, in validate_setup
    raise e
  File "/opt/conda/envs/test/lib/python3.11/site-packages/cudf/utils/gpu_utils.py", line 52, in validate_setup
    gpus_count = getDeviceCount()
                 ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.11/site-packages/rmm/_cuda/gpu.py", line 102, in getDeviceCount
    raise CUDARuntimeError(status)
rmm._cuda.gpu.CUDARuntimeError: cudaErrorUnknown: unknown error

Will restart that job after CI completes

Otherwise cuDF tests appear to be passing with NumPy 2. So think this will clear after a restart

[jakirkham]

Can confirm NumPy 2 is installed with the latest cuDF and all tests pass! 🥳

Closing as completed