Content-defined chunking for OCI images via Puzzlefs
InfrastructureComments
Does the Puzzlefs implementation use a global index or a per-image map for those chunks? I am wondering if the lookup time scales linearly with the number of images stored on the host.
A significant upside here is the potential for lazy pulling. By using a chunk-based model, the runtime could fetch only the specific blocks required for the initial process execution instead of downloading entire layers.
fastcdc minimizes duplication but won't eliminate it if binary offsets shift due to minor header changes.
This mirrors early attempts at content-addressable storage for VMs. We eventually found that the metadata index became the bottleneck, not the data transfer itself.
Most of us are just fighting massive base image bloat in CI pipelines. If this actually speeds up pulls by only grabbing new chunks, it solves a real bandwidth bottleneck for remote sites.
If the workload consists of immutable production deployments rather than active CI, the bandwidth savings might be negligible compared to the risk of introducing a non-standard filesystem driver.
The deduplication could be massive for multi-arch builds... imagine sharing the same shared libraries across different base OS versions... that would save so much storage!
Similar to ZFS deduplication. It saves disk space, but usually degrades performance unless you have enough RAM to cache the deduplication table.