Struct CompressionEstimator 

pub struct CompressionEstimator { /* private fields */ }

A streaming estimator for the size of compressed data.

For many compression algorithms, there is a large amount of buffering and state during compression. This allows compression algorithms to better compress data by finding patterns across the current and previous inputs, as well as amortize how often they write compressed data to the output stream, increasing the potential efficiency of the related function or system calls to do so.

However, this presents a problem when there is a need to ensure that the size of the compressed data does not exceed a certain threshold. As many inputs can be written to the compressor before the next chunk of compressed data is output, it is possible to write enough data that the compressed output exceeds the threshold. Further, many compression algorithms/implementations do not provide a way to query the size of the compressed data without expensive operations that either require doing multiple compression passes on different slices of the data, or early flushing of compressed data, potentially leading to abnormally low compression ratios.

This estimator provides a way to estimate the size of the compressed data by combining both the known size of data written to the compressor’s output stream, as well as the inputs written to the compressor. We track the state changes of the compressor, observing when it writes compressed data to the output stream. We additionally track every write in terms of its uncompressed size. In combining the two, we estimate the worst-case size of the compressed data based on what we know has been compressed so far and what we’ve written since the last time the compressed flush to the output stream.

TODO: We should probably move this into Compressor itself, because it will also make it easier to do per-compression-algorithm tweaks to the estimation logic if that’s a path we want to take, and it also would be cleaner and let us avoid any footguns around forgetting to update the necessary estimator state, etc.

Implementations

impl CompressionEstimator

pub fn track_write<W>(&mut self, compressor: &W, uncompressed_len: usize)

where W: WriteStatistics,

Tracks a write to the compressor.

pub fn reset(&mut self)

Resets the estimator.

pub fn estimated_len(&self) -> usize

Returns the estimated length of the compressor.

This figure is the sum of the total bytes written by the compressor to the output stream and the number of uncompressed bytes written to the compressor since the last time the compressor wrote to the output stream when factoring in the estimated compression ratio over the overall output stream.

pub fn would_write_exceed_threshold(&self, len: usize, threshold: usize) -> bool

Estimates if writing len bytes to the compressor would cause the final compressed size to exceed threshold bytes.

Trait Implementations

impl Debug for CompressionEstimator

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CompressionEstimator

fn default() -> CompressionEstimator

Returns the “default value” for a type.