Skip to content

[data][1/n] Add the push-based streaming_split coordinator #66465

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
goutamvenkat-anyscale merged 16 commits into from
Oct 14, 2026
Merged

[data][1/n] Add the push-based streaming_split coordinator #66465

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
9ef534c
[data] Add push-based streaming_split
xinyuangui2 Sep 24, 2026
674e76d
[data] Keep push-based split out of the Dataset API for now
xinyuangui2 Sep 24, 2026
1183794
[data] Narrow the first PR to the push-split coordinator
xinyuangui2 Sep 24, 2026
4cc7a4e
[data] Address review: condition-based epoch barrier, validate split_idx
xinyuangui2 Sep 24, 2026
ae78856
[data] Release the push-split barrier even if teardown fails; fix pyr…
xinyuangui2 Sep 24, 2026
6ad886e
Merge branch 'master' into push-split-coordinator
xinyuangui2 Oct 8, 2026
3f2438c
[data] Address review: finished-split reports, dataset tag, stats parity
xinyuangui2 Oct 8, 2026
163ef0e
[data] Put the push-based split overview in the coordinator module do…
xinyuangui2 Oct 8, 2026
45a6728
[data] Move the StreamingSplit dataset helper to ray.data._internal.util
xinyuangui2 Oct 8, 2026
1a476c6
[data] Address review: raise teardown failures to every split; docume…
xinyuangui2 Oct 9, 2026
8eed180
Merge branch 'master' into push-split-coordinator
xinyuangui2 Oct 9, 2026
0157c00
[data] Build Dataset.streaming_split's split dataset with the shared …
xinyuangui2 Oct 10, 2026
787d2c1
[data] Ignore pushes recorded after a split finished
xinyuangui2 Oct 10, 2026
52e6df9
[data] Hold sent block refs until consumed, so Ray Data counts them
xinyuangui2 Oct 11, 2026
aca1c8e
[data] Give the push-split coordinator tests a larger Bazel timeout
xinyuangui2 Oct 12, 2026
a2b0551
[data] Address review: Condition.wait_for in the barrier, single-loop…
xinyuangui2 Oct 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions python/ray/data/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1883,6 +1883,20 @@ py_test(
],
)

py_test(
name = "test_push_split_coordinator",
size = "medium",
srcs = ["tests/test_push_split_coordinator.py"],
tags = [
"exclusive",
"team:data",
],
deps = [
":conftest",
"//:ray_lib",
],
)

py_test(
name = "test_shuffle_diagnostics",
size = "large",
Expand Down
Loading
Loading