Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] dask-cuda v24.08 #1370

Merged
merged 18 commits into from
Aug 7, 2024
Merged

[RELEASE] dask-cuda v24.08 #1370

merged 18 commits into from
Aug 7, 2024

Conversation

raydouglass
Copy link
Member

❄️ Code freeze for branch-24.08 and v24.08 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-24.08 until release (merging of this PR).

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-24.08 into main for the release

AyodeAwe and others added 17 commits May 20, 2024 21:12
@AyodeAwe
DOC v24.08 Updates [skip ci]
f1a1405
@GPUtester
Merge pull request #1345 from rapidsai/branch-24.06
037a4c8 
Forward-merge branch-24.06 into branch-24.08
@bdice
Adopt CI/packaging codeowners (#1347)
4fc6df2
@vyasr
Remove text builds of documentation (#1346)
c1e27de 
This PR removes text builds of the documentation, which we do not currently use for anything. Contributes to rapidsai/build-planning#71.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Ray Douglass (https://github.com/raydouglass)

URL: #1346
@jameslamb
use rapids-build-backend (#1343)
abded3a 
Contributes to rapidsai/build-planning#31
Contributes to rapidsai/dependency-file-generator#89

Proposes introducing `rapids-build-backend` as this project's build backend, to reduce the complexity of various CI/build scripts.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Lawrence Mitchell (https://github.com/wence-)
  - Ray Douglass (https://github.com/raydouglass)

URL: #1343
@jameslamb
make conda recipe data-loading stricter (#1349)
098109a 
Contributes to rapidsai/build-planning#72

Proposes using `[]` subsetting instead of `.get()` in templating statements in the conda recipe that read data out of `pyproject.toml`. That'll ensure that we get a big loud build error if changes to `pyproject.toml` remove some sections that the conda recipe expects to exist.

## Notes for Reviewers

### How I tested this

Rendered the recipe.

```shell
git fetch upstream --tags

RAPIDS_DATE_STRING="2408" \
RAPIDS_PACKAGE_VERSION="24.8.0" \
conda render \
  -c conda-forge \
  -c rapidsai-nightly \
  conda/recipes/dask-cuda
```

<details><summary>It looks correct to me (click for details)</summary>

```text
--------------
Hash contents:
--------------
{}
----------
meta.yaml:
----------
package:
  name: dask-cuda
  version: 24.8.0
source:
  path: /Users/jlamb/repos/dask-cuda
build:
  entry_points:
    - dask-cuda-worker = dask_cuda.cli:worker
    - dask-cuda-config = dask_cuda.cli:config
  number: '10'
  script:
    - /Users/jlamb/miniforge3/conda-bld/dask-cuda_1718216576022/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/python
      -m pip install . -vv
  string: py310_2408_g3a04719_10
requirements:
  host:
    - bzip2 1.0.8 h93a5062_5
    - ca-certificates 2024.6.2 hf0a4a13_0
    - libffi 3.4.2 h3422bc3_5
    - libzlib 1.3.1 hfb2fe0b_1
    - ncurses 6.5 hb89a1cb_0
    - python_abi 3.10 4_cp310
    - tzdata 2024a h0c530f3_0
    - xz 5.2.6 h57fd34a_0
    - yaml 0.2.5 h3422bc3_2
    - libsqlite 3.46.0 hfb93653_0
    - openssl 3.3.1 hfb2fe0b_0
    - readline 8.2 h92ec313_1
    - tk 8.6.13 h5083fa2_1
    - python 3.10.14 h2469fbe_0_cpython
    - attrs 23.2.0 pyh71513ae_0
    - packaging 24.1 pyhd8ed1ab_0
    - pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1
    - pyyaml 6.0.1 py310h2aa6e3c_1
    - rpds-py 0.18.1 py310h947b723_0
    - setuptools 70.0.0 pyhd8ed1ab_0
    - tomlkit 0.12.5 pyha770c72_0
    - wheel 0.43.0 pyhd8ed1ab_1
    - zipp 3.19.2 pyhd8ed1ab_0
    - importlib_resources 6.4.0 pyhd8ed1ab_0
    - pip 24.0 pyhd8ed1ab_0
    - referencing 0.35.1 pyhd8ed1ab_0
    - jsonschema-specifications 2023.12.1 pyhd8ed1ab_0
    - jsonschema 4.22.0 pyhd8ed1ab_0
    - rapids-dependency-file-generator 1.13.11 py_0
    - rapids-build-backend 0.3.1 py_0
  run:
    - pynvml>=11.0.0,<11.5
    - numpy>=1.23,<2.0a0
    - python_abi 3.10.* *_cp310
    - click >=8.1
    - rapids-dask-dependency==24.8.*,>=0.0.0a0
    - numba>=0.57
    - python >=3.10,<3.11.0a0
    - pandas>=1.3
    - zict>=2.0.0
test:
  commands:
    - dask cuda --help
    - dask-cuda-worker --help
    - dask cuda worker --help
    - dask-cuda-config --help
    - dask cuda config --help
  imports:
    - dask_cuda
about:
  dev_url: https://github.com/rapidsai/dask-cuda
  doc_url: https://docs.rapids.ai/api/dask-cuda/stable/
  home: https://github.com/rapidsai/dask-cuda
  license: Apache 2.0
  license_file:
    - ../../../LICENSE
  summary: Utilities for Dask and CUDA interactions
extra:
  copy_test_source_files: true
  final: true
```

</details>

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #1349
@jameslamb
remove .gitattributes (#1350)
7363b0c 
Contributes to rapidsai/build-planning#31

Removes `.gitattributes` file.

That was added in #88 for use with `versioneer`. Per the `git` docs ([link](https://git-scm.com/docs/gitattributes#_export_subst)), setting the attribute `export-subst` on a file via a `.gitattributes` tell `git` to replace placeholders in the file with some `git` information.

This is no longer done in `_version.py` files in this project, and this project no longer uses `versioneer`. `rapids-build-backend` handles storing git commit information in the published packages.

## Notes for Reviewers

Created based on this conversation: rapidsai/kvikio#369 (comment)

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Mike Sarahan (https://github.com/msarahan)

URL: #1350
@pentschev
Update cuDF's assert_eq import (#1353)
d8d87f3 
rapidsai/cudf#16063 has updated the import location of `assert_eq` to the public `cudf.testing.assert_eq`, this change updates imports accordingly.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)

URL: #1353
@pentschev
Allow disabling RMM in benchmarks (#1352)
0a3cc7f 
Allows disabling RMM in benchmarks via a new option `--disable-rmm`. This change makes benchmarks a little more similar to RMM setup in `LocalCUDACluster`/`dask cuda worker`, where not specifying `rmm-pool-size` or specifying `None` as its value entirely disables setting up RMM as the default allocator. Since for benchmarks it's desired that the default is having an RMM pool we cannot change the default `--rmm-pool-size` to `None` as that would make benchmarks run much slower by default, therefore `--disable-rmm` is the closest we can make this to the rest of Dask-CUDA.

Additionally add `--rmm-maximum-pool-size` for benchmarks.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - Ayush Dattagupta (https://github.com/ayushdg)
  - Ray Douglass (https://github.com/raydouglass)

URL: #1352
@jakirkham
Drop setup.py (#1354)
3e0f7c3 
This is just calling `setup` from `setuptools`, which should already happen with the `setuptools` backend. So go ahead and drop `setup.py`.

Authors:
  - https://github.com/jakirkham

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - James Lamb (https://github.com/jameslamb)

URL: #1354
@rjzamora
Fix partitioning in explicit-comms shuffle (#1356)
fe23e45 
Closes #1355

Current version of the explicit-comms shuffle does not produce partitioning that is consistent with `dask.dataframe`.

Authors:
  - Richard (Rick) Zamora (https://github.com/rjzamora)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #1356
@KyleFromNVIDIA
Build and test with CUDA 12.5.1 (#1357)
c31aaac 
This PR updates the latest CUDA build/test version 12.2.2 to 12.5.1.

Contributes to rapidsai/build-planning#73

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #1357
@KyleFromNVIDIA
Use workflow branch 24.08 again (#1359)
b279200 
After updating everything to CUDA 12.5.1, use `shared-workflows@branch-24.08` again.

Contributes to rapidsai/build-planning#73

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #1359
@KyleFromNVIDIA
Use verify-alpha-spec hook (#1360)
fa226b1 
With the deployment of rapids-build-backend, we need to make sure our dependencies have alpha specs.

Contributes to rapidsai/build-planning#31

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - https://github.com/jakirkham
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #1360
@pentschev
Add arguments to enable cuDF spilling and set statistics (#1362)
d6cafc1 
Add arguments to enable cuDF spilling and set statistics in `dask cuda worker`/`LocalCUDACluster`. This is implemented as a Dask plugin, and does not require users anymore to rely on `client.run` to do that.

Closes #1280

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Richard (Rick) Zamora (https://github.com/rjzamora)
  - Mads R. B. Kristensen (https://github.com/madsbk)

URL: #1362
@jameslamb
split up CUDA-suffixed dependencies in dependencies.yaml (#1364)
13a5f47 
Contributes to rapidsai/build-planning#31

In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`.

This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`.

For more details, see:
* rapidsai/build-planning#31 (comment)
* rapidsai/cudf#16183

## Notes for Reviewers

### Why target 24.08?

This is targeting 24.08 because:

1. it should be very low-risk
2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #1364
@jameslamb
consolidate cuda_suffixed=false blocks in dependencies.yaml, fix upda…
064e254 
…te-version.sh (#1367)

Contributes to rapidsai/build-planning#31.

Follow-up to #1364.

Implements some of the suggestions made in rapidsai/cudf#16183 (after #1364 was already merged):

* removing `cuda_suffixed: "false"` blocks in `dependencies.yaml` wherever they're identical to each other and the fallback matrix
* changing `dependencies.yaml` anchors with names like `*_conda` to `*_unsuffixed`, to reflect the fact that they're not conda-specific
* checking that `update-version.sh` catches all changes to versions

## Notes for Reviewers

### How I tested this

Looked for `update-versions.sh` issues manually like this:

```shell
git fetch upstream --tags
ci/release/update-version.sh '24.10.0'
git grep -E '24\.8|24\.08|0\.39'
```

The did find a few problems (like UCX dependency versions not being updated). This fixes those issues.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #1367
@raydouglass raydouglass requested review from a team as code owners July 26, 2024 14:43
@github-actions github-actions bot added python python code needed conda conda issue ci labels Jul 26, 2024
@raydouglass raydouglass merged commit 520c8df into main Aug 7, 2024
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
ci conda conda issue python python code needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@raydouglass @AyodeAwe @GPUtester @bdice @vyasr @jameslamb @pentschev @jakirkham @rjzamora @KyleFromNVIDIA