Repository navigation
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces the ResourceRequest class and updates the Ray Data execution operators and the V2 cluster autoscaler to support exact resource shape scaling. Instead of relying on aggregate cluster-wide utilization ratios, the V2 autoscaler can now collect exact resource requests from active tasks and actors across the execution topology, allowing for more precise scaling decisions based on specific resource shapes (including custom resources). Comprehensive unit tests have been added to verify the correctness of resource request tracking, aggregator demand reporting, and autoscaler behavior. I have no additional feedback to provide as the implementation is clean and well-tested.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
dragongu
force-pushed
the
fix/coordinator-resource-shape-matching
branch
from
October 2, 2026 00:38
952cfa6 to
fef13fe
Compare
ray-gardener
Bot
added
data
dragongu
force-pushed
the
fix/coordinator-resource-shape-matching
branch
from
October 2, 2026 10:37
fef13fe to
dffe0c9
Compare
dragongu
force-pushed
the
fix/coordinator-resource-shape-matching
branch
2 times, most recently
from
October 3, 2026 01:02
e9f323f to
6ad6f44
Compare
iamjustinhsu
reviewed
Oct 5, 2026
iamjustinhsu
left a comment
iamjustinhsu
left a comment
Contributor
There was a problem hiding this comment.
Left some initial feedback, will take a look at the rest of the files later
dragongu
force-pushed
the
fix/coordinator-resource-shape-matching
branch
from
October 10, 2026 04:43
6ad6f44 to
1993655
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 1993655. Configure here.
The rate-based cluster autoscaler built its resource request from each
operator's logical ExecutionResources, which cannot express custom
resources. A task or actor asking for {"CPU": 1, "worker_group_a": 1},
or a GPU task with accelerator_type="A10G", was flattened into a shape
the workload never uses, and different shapes from one operator were
mixed into a single bundle. The autoscaler therefore grew the wrong
worker groups: the groups providing the custom resource never grew, and
the batches that needed them stayed unscheduled while the request named
nodes the workload cannot use.
Build the request from the exact resource request of every in-flight
task and actor, reported through a single
PhysicalOperator.get_resource_requests() hook, and scale up only the
shapes whose demand has crossed their utilization threshold, measured
against the worker groups that can host the shape. The throughput
solver's delta and the object-store padding are unchanged.
Also fixes two keep-alive bugs found on the way: demand for
already-finished tasks was kept alive for up to
low_util_request_release_delay_s, and a later scale-up cancelled a copy
that was still inside its window.
Tests: python/ray/data/tests/test_rate_based_cluster_autoscaler.py,
python/ray/data/tests/unit/test_exact_resource_requests.py
Signed-off-by: dragongu <andrewgu@vip.qq.com>
dragongu
force-pushed
the
fix/coordinator-resource-shape-matching
branch
from
October 10, 2026 06:48
549a6a8 to
d00bf84
Compare
dragongu
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The rate-based cluster autoscaler builds its request from each operator's logical
ExecutionResources,which carries only the four standard dimensions — CPU, GPU, memory and object store memory — so it can
express neither custom resources nor labels. Tasks and actors that need different resources are
therefore flattened into one logical shape that carries none of their custom resources. The autoscaler
grows the wrong worker groups, and the work that needs the missing resource stays pending.
This PR requests the exact shape of every in-flight task and actor, reported through a new
PhysicalOperator.get_resource_requests()hook, and scales up only the shapes whose own demand hascrossed its utilization threshold, measured against the worker groups that can host the shape. The
throughput solver's delta and the object-store padding are unchanged.
Scope: custom resources. Labels are left to a separate PR — they keep working through Ray
scheduling and the coordinator's existing interfaces, but they do not take part in the shape grouping
here.