Skip to content

fix: prevent integer overflow in shuffle buffer offsets during resize - #1061

Merged
fzhedu merged 4 commits into
bytedance:mainfrom
zhangxffff:fix/shuffle_int_overflow
Oct 14, 2026
Merged

fzhedu merged 4 commits into
bytedance:mainfrom
zhangxffff:fix/shuffle_int_overflow

Conversation

@zhangxffff

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

In BoltShuffleWriter, when resizing a partition buffer with preserveData=true, the 64-bit binary
value offset is stored in a local variable declared as auto valueOffset = 0,
which deduces int. This narrows offsets above INT32_MAX and corrupts them
when converted back to uint64_t, cause coredump after
resize.

Issue Number: N/A

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

Description

  • Use uint64_t for the temporary valueOffset to preserve the full binary
    buffer offset during partition buffer resize.

Performance Impact

No measurable overhead is expected: this only changes the temporary offset's
type to match the existing 64-bit field. No benchmarks were run.

Release Note

Fix integer overflow in Spark shuffle binary buffer offsets when resizing
partition buffers while preserving data.

Validation

  • Validate in real world workload

Checklist (For Author)

  • I have added/updated unit tests (ctest).
  • I have verified the code with local build (Release/Debug).
  • I have run clang-format / linters.
  • (Optional) I have run Sanitizers (ASAN/TSAN) locally for complex C++ changes.
  • No need to test or manual test.

Breaking Changes

  • No
  • Yes

@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3f0d0b60-b6ea-427d-a167-000bc1d77e3d
📥 Commits

Reviewing files that changed from the base of the PR and between 6e11a9f and b975cab.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Summary

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of binary value-buffer offsets during shuffle buffer resizing.

BoltShuffleWriter::resizePartitionBuffer now declares the binary value-buffer offset local as uint64_t instead of inferring its type from the integer literal 0. The existing preserve-data path still retains the current binary offset.

Suggested reviewers: guhaiyan0221

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to d80e4

Large shuffle offsets now remain intact during preserve-data resizing. A focused regression test is recommended, but the available source evidence shows no current functional failure, so the change is mergeable with normal checks.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments: 1
bolt/shuffle/sparksql/BoltShuffleWriter.cpp: 1

1978-1981: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a regression test for large binary offsets.

No existing test covers a binary valueOffset above INT32_MAX. A derived fixture can access the protected resize state and use a small ResizableBuffer test double that records the requested size without allocating it. Assert that preserve-data resizing retains the offset. Reverting the local declaration to int must fail this assertion.


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 87a5a928-5725-43b1-925e-fe17b256f3c3
📥 Commits

Reviewing files that changed from the base of the PR and between 42f76ad and d80e467.

📒 Files selected for processing: 1
  • bolt/shuffle/sparksql/BoltShuffleWriter.cpp

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

@zhangxffff
zhangxffff requested a review from fzhedu October 14, 2026 08:47

@fzhedu fzhedu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fzhedu
fzhedu enabled auto-merge October 14, 2026 10:57
@fzhedu
fzhedu added this pull request to the merge queue Oct 14, 2026
Merged via the queue into bytedance:main with commit 19cd307 Oct 14, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants