How it works
The first runs on AWS showed each hour's task bound to one of the worker's four cores, with the Parquet write its largest stage. These changes spread the work over the cores and stopped copying it; each row links its pull request and the measurement table in its description.
One gharchive hour on AWS
c7g.xlarge workers, about 85,000 events an hour, median of 24 hours.
| Stage | Before (run 36) | After (run 48) |
|---|---|---|
| task | 1,094 ms | 680 ms |
| download | 172 ms | 130 ms |
| gunzip | 210 ms | 154 ms |
| parse | 160 ms | 134 ms |
| write | 570 ms | 256 ms |
| commit | 486 ms | 683 ms |
Commit time grows with the table's snapshot count: run 133, after the latest release, measured 1,207 ms for the commit with the other stages unchanged. Under investigation.
The changes
| Change | What it did | Measured (Apple M4) | PR |
|---|---|---|---|
| mojo-census #4 | Streams each hour through gunzip and parsing as it downloads, and writes Parquet on the workers. | Six hours locally, 10.3 s to 4.5 s with 4 tasks | #4 |
| parquet #59 | Streams row groups, splits string pages by size, encodes pages on threads. | 50,000-row batch encode 198 to 81 ms (4 threads), peak 828 to 463 MB | #59 |
| iceberg #71 | Writes a batch without copying it. | One hour 838–855 to 783–804 ms, peak 1.30–1.51 to 0.96–0.97 GB | #71 |
| iceberg #72 | Writes through the streaming writer on every core. | One hour to 375 ms, peak about 0.5 GB (with #59, 4 threads) | #72 |
| deflate #10, #12 | Streaming inflate, and a faster decode loop. | Gunzip of an hour 553 to 336 ms (1.46 GB/s) | #10, #12 |
| iceberg #70 + orca #114 | Workers hand their data files to the step to commit (append_files), which removes the census's own DataFile codec. | – | iceberg #70, orca #114 |