zrip

Pure Rust zstd codec. The decoder reads standard zstd blocks and frames produced at any zstd compression level. The encoder supports levels -8 through 4 for fast transfer pipelines. First-class dictionary support.

Why zrip

Fastest pure-Rust zstd encoder. Consistently outperforms other pure-Rust zstd implementations on both encode and decode across all supported levels. See the benchmarks below.

Negative levels (-8 through -1). Unlocks zstd's fastest compression tiers, useful when throughput matters more than ratio. L-8 is zrip's own addition beyond C zstd's range: raw literals only, no Huffman table build, approaching LZ4-class encode speed while still producing standard zstd frames.

Memory safety. Unsafe is minimized and confined to small, auditable primitives modules. The paranoid feature eliminates all remaining unsafe. See SAFETY.md.

Small codebase. ~13k Rust SLOC. Levels above 4 add complexity for compression ratios that only matter in archival storage, not transfer pipelines.

Dictionary compression. COVER and FastCOVER training built in for small-message workloads (log lines, JSON records, RPC payloads).

no_std + alloc. Works in embedded and kernel contexts with the alloc feature; frame requires std.

WebAssembly. Available as @paddor/zrip on JSR. Auto-detects WASM SIMD support. 15% faster encode than C zstd compiled to WASM.

Performance

x86_64 details (per-file pipeline, scatter, matrix)

aarch64 (Apple M4)

wasm32 (wasmtime)

API

// One-shot (allocating)
let compressed = zrip::compress(input, 1)?;
let original   = zrip::decompress(&compressed)?;

// One-shot into caller buffer
let n = zrip::compress_into(input, &mut output_buf, 1)?;
zrip::decompress_into(&compressed, &mut output_vec)?;

// Reusable context (amortizes table allocation across calls)
let mut ctx = zrip::CompressContext::new(1)?;
let compressed = ctx.compress(input)?;

let mut dec = zrip::DecompressContext::new();
let original = dec.decompress(&compressed)?;

Streaming

use std::io::Write;

let mut enc = zrip::FrameEncoder::new(Vec::new(), 1)?;
enc.write_all(b"hello")?;
enc.write_all(b" world")?;
let compressed = enc.finish()?;

use std::io::Read;
let mut dec = zrip::FrameDecoder::new(&compressed[..]);
let mut out = String::new();
dec.read_to_string(&mut out)?;

FrameEncoder and FrameDecoder own persistent hash tables and workspace buffers. Call reset() to start a new frame while reusing all allocations:

let mut enc = zrip::FrameEncoder::new(Vec::new(), 1)?;
enc.write_all(b"first frame")?;
let first  = enc.reset(Vec::new())?;  // finishes frame, keeps buffers
enc.write_all(b"second frame")?;
let second = enc.finish()?;

Large-window and long distance matching

The normal match finder operates within a sliding window (512 KiB at L1). LDM finds matches at distances up to 1 << window_log bytes by sampling positions into a separate hash table. Useful for data with long-range repeats: log files, database dumps, source archives. See DESIGN.md for how LDM works.

let opts = zrip::Options::default().window_log(24).ldm(true);
let compressed = zrip::compress_opts(input, 1, &opts)?;

Streaming with LDM:

use std::io::Write;
let opts = zrip::Options::default().window_log(24).ldm(true);
let mut enc = zrip::FrameEncoder::with_options(Vec::new(), 1, &opts)?;
enc.write_all(input)?;
let compressed = enc.finish()?;

window_log controls the maximum match distance (24 = 16 MiB, 27 = 128 MiB). The main cost is memory: the encoder allocates an 8 MiB LDM hash table plus a window buffer of 1 << window_log bytes, and the decoder allocates a window buffer of the same size (declared in the frame header). At window_log=27 that is ~136 MiB on each side. On data without long-range repeats, LDM adds overhead with no ratio benefit.

Dictionary compression

let dict = zrip::Dictionary::from_bytes(&dict_bytes)?;
let compressed = zrip::compress_with_dict(input, 1, &dict)?;
let original   = zrip::decompress_with_dict(&compressed, &dict)?;

Streaming with a dictionary:

let mut enc = zrip::FrameEncoder::with_dict(Vec::new(), 1, dict.clone())?;
enc.write_all(input)?;
let compressed = enc.finish()?;

let mut dec = zrip::FrameDecoder::with_dict(&compressed[..], dict);
let mut output = Vec::new();
dec.read_to_end(&mut output)?;

CompressContext::with_dict() and DecompressContext::with_dict() provide the same reuse for one-shot compression.

Dictionary training

Build a dictionary from sample data using the built-in FastCOVER trainer. Requires the dict_builder feature.

use zrip::dict::{train_dict_fastcover, fastcover::FastCoverParams};

let samples: Vec<&[u8]> = messages.iter().map(|m| m.as_bytes()).collect();
let dict = train_dict_fastcover(&samples, 16384, FastCoverParams::default());
// Use with compress_with_dict() / decompress_with_dict()

Features

Feature	Default	Description
std	yes	Enables CompressContext, DecompressContext
frame	yes	Frame header parsing and writing; implies std
alloc	yes	no_std + heap via alloc crate
ldm	yes	Long distance matching for large-window compression
dict_builder	no	COVER/FastCOVER dictionary training
simd	yes	fearless_simd runtime dispatch (AVX2+BMI2, NEON, SIMD128)
paranoid	no	Pure safe Rust: forbid(unsafe_code) on all crates
nightly	no	#[optimize] attributes on hot functions

Safety

SAFETY.md documents the unsafe boundary and catalogs C zstd memory safety bugs that Rust prevents by construction.

All codec paths are fuzz-tested (16 targets, ~10.7M executions) and verified under Miri on both x86_64 and aarch64. Fuzz targets cover round-trip correctness, cross-validation against C zstd, streaming, dictionary modes, and corruption resistance (bitflip, splice, truncate, overwrite).

Design

DESIGN.md covers the encode/decode pipeline, performance architecture, SIMD dispatch, compile-time specialization, and divergences from C zstd.

Levels

zrip's encoder covers the fast end of the zstd speed/ratio curve. Negative levels favor throughput over ratio. Positive levels spend more match-finding work for better ratios while staying in the Fast/DFast range.

Level 0 maps to the library default, currently level 1.

L-8 is zrip-specific. Its purpose is to get as close to LZ4 encode speed as possible while still producing standard zstd frames. It does that by forcing raw literal blocks and using predefined sequence tables.

The decoder is level-independent and supports standard zstd frames produced by all zstd compression levels. See DESIGN.md for exact encoder level parameters and pipeline behavior.