z_order max_spill_size parameter incorrectly documented#2205 Closed wjones127 opened this issue Feb 22, 2024· 2 comments · Fixed by #2835 CommentsCollaborator wjones127 commented Feb 22, 2024 The parameter says that is max bytes spilled to disk: delta-rs/crates/core/src/operations/optimize...
- Z-Order with larger dataset resulting in memory error [\#2284](https://github.com/delta-io/delta-rs/issues/2284) - Successful writes return error when using concurrent writers [\#2279](https://github.com/delta-io/delta-rs/issues/2279) - Rust writer should raise when decimal types are...
max_spill_size had misleading documentation that seemed to indicate the limit on disk. However, the value was a limit on memory to avoid out-of-memory and to spill to disk on such case. This issue was raised in #2205 Related Issue(s) closes z_order max_spill_size parameter incorrectly ...
| Optimize - Z-order | ![done] | ![done] | Place similar data into the same file | | Merge | ![done] | ![done] | Merge a target Delta table with source data | | Update | ![done] | ![done] | Update values from a table | | Add Column | ![done] | ![done] | Add ...
When we run the following code it fails with _internal.DeltaError: Failed to parse parquet: Parquet error: Z-order failed while scanning data: ArrowError(ExternalError(General("ParquetObjectReader::get_byte_ranges error: Generic MicrosoftAzure error: request or response body error: operation timed...
%K to Z Transformation klev = (1:41).'; nk = numel(klev); zmid = (2.5:5:202.5).'; nd = numel(GliderDoYs); placecounter =1; tempMat =nan(nk,365); fordii =1:nd%loop over the glider days thisDayIdx = binIdx==dii;
| Optimize - Z-order | ![done] | ![done] | Place similar data into the same file | | Merge | ![done] | ![done] | Merge a target Delta table with source data | | Update | ![done] | ![done] | Update values from a table | | Add Column | ![done] | ![done] | Add ...
| Optimize - Z-order | ![done] | ![done] | Place similar data into the same file | | Merge | ![done] | ![done] | Merge a target Delta table with source data | | Update | ![done] | ![done] | Update values from a table | | Add Column | ![done] | ![done] | Add ...