Repository navigation
fix: prevent integer overflow in shuffle buffer offsets during resize - #1061
Conversation
|
Important Review skippedReview was skipped as selected files did not have any reviewable changes. ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 Summary
Priority: ⬇️ Low Merge Risk: ⚪ Minimal · up to
|
There was a problem hiding this comment.
🧹 Nitpick comments: 1
bolt/shuffle/sparksql/BoltShuffleWriter.cpp: 1
1978-1981: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a regression test for large binary offsets.
No existing test covers a binary
valueOffsetaboveINT32_MAX. A derived fixture can access the protected resize state and use a smallResizableBuffertest double that records the requested size without allocating it. Assert that preserve-data resizing retains the offset. Reverting the local declaration tointmust fail this assertion.
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
87a5a928-5725-43b1-925e-fe17b256f3c3
📒 Files selected for processing: 1
bolt/shuffle/sparksql/BoltShuffleWriter.cpp
Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.
What problem does this PR solve?
In BoltShuffleWriter, when resizing a partition buffer with
preserveData=true, the 64-bit binaryvalue offset is stored in a local variable declared as
auto valueOffset = 0,which deduces
int. This narrows offsets aboveINT32_MAXand corrupts themwhen converted back to
uint64_t, cause coredump afterresize.
Issue Number: N/A
Type of Change
Description
uint64_tfor the temporaryvalueOffsetto preserve the full binarybuffer offset during partition buffer resize.
Performance Impact
No measurable overhead is expected: this only changes the temporary offset's
type to match the existing 64-bit field. No benchmarks were run.
Release Note
Fix integer overflow in Spark shuffle binary buffer offsets when resizing
partition buffers while preserving data.
Validation
Checklist (For Author)
Breaking Changes