lz4.c File Reference

Demonstrates how LZ4 compression levels and the shuffle filter affect NetCDF-4 compression ratio and I/O throughput. More...

#include <netcdf.h>
#include <netcdf_filter.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/time.h>

Macros
#define	H5Z_FILTER_LZ4   32004
#define	NZ   500
#define	NY   180
#define	NX   360
#define	TMP_FILE   "lz4_tmp.nc"
#define	CHUNK_Z   10
#define	CHUNK_Y   45
#define	CHUNK_X   90
#define	NVALS   ((size_t)NZ * NY * NX)
#define	UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}
#define	MIN_LEVEL   1
#define	MAX_LEVEL   9

Functions
static double	get_time (void)
	Return current wall-clock time in seconds.
static int	run_one (float *data, int level, int shuffle)
	Run one lz4/shuffle combination and print a CSV row.
int	main (void)
	Main entry point.

Detailed Description

Demonstrates how LZ4 compression levels and the shuffle filter affect NetCDF-4 compression ratio and I/O throughput.

NetCDF-4/HDF5 supports LZ4 compression via the HDF5 LZ4 filter (ID 32004) using the generic nc_def_var_filter() API. LZ4 is a fast lossless compression algorithm focused on compression and decompression speed. LZ4 offers a level range (1–9) where higher levels achieve better compression ratios at the cost of compression speed. Decompression speed remains consistently fast across all levels.

The shuffle filter (shuffle = 1) reorders bytes before compression: for IEEE 754 floating-point data the most-significant bytes of adjacent values are grouped together, which typically increases the compression ratio by 3–5× at little additional cost. The shuffle filter is enabled by calling nc_def_var_deflate(ncid, varid, 1, 0, 0) (shuffle only, no deflate) before applying the LZ4 filter via nc_def_var_filter().

This program iterates LZ4 levels 1–9 × shuffle {off, on} (18 combinations total) over a 500×180×360 NC_FLOAT temperature dataset (~129 MB uncompressed) and reports:

• Compressed file size on disk (bytes via stat())
• Compression ratio (uncompressed / compressed)
• Write time (seconds)
• Read time (seconds, full variable via nc_get_var_float())

Output: CSV printed to stdout with columns:

lz4_level,shuffle,compressed_bytes,ratio,write_s,read_s

Typical workflow:

./lz4 > lz4_results.csv
python3 plot_lz4.py   # produces lz4_performance.jpg

Learning Objectives:

• Understand LZ4 compression characteristics: fastest decompression of all filters
• Learn how to use the generic nc_def_var_filter() API with HDF5 filter IDs
• See how LZ4 levels (1–9) trade compression speed for ratio
• Recognize LZ4's strength: consistently fast decompression regardless of level
• Determine when LZ4 is preferred (read-heavy workloads, real-time access)

Key Concepts:

• LZ4: A fast lossless compression algorithm optimized for decompression speed; achieves lower ratios than deflate or zstd but decompresses 2–4× faster
• Generic Filter API: nc_def_var_filter(ncid, varid, id, nparams, params) enables any registered HDF5 filter; LZ4 uses filter ID 32004
• Decompression Speed: LZ4's main advantage — constant high-speed decode regardless of compression level; ideal for read-intensive applications
• Level Range (1–9): Higher levels spend more CPU on compression for marginally better ratios; decompression speed is unaffected
• Shuffle + LZ4: Shuffle pre-groups IEEE 754 bytes, improving LZ4's ratio by 3–5× at minimal cost to decompression speed

Prerequisites:

• simple_nc4.c - NetCDF-4 file creation basics
• chunking.c - Understanding chunked storage (required for compression)
• deflate.c - Comparison baseline (deflate is the traditional default)

Related Examples:

• deflate.c - Traditional deflate/zlib compression (better ratio, slower decompress)
• zstandard.c - Zstandard compression (better ratio, slightly slower decompress)
• bzip2.c - BZIP2 compression (best ratio, slowest)
• lossless.c - Unified comparison of all lossless filters

Key API functions:

• nc_def_var_filter() Enable LZ4 compression using HDF5 filter ID 32004
• nc_inq_var_filter() Query filter settings on an open variable
• nc_def_var_deflate() Enable shuffle filter without deflate (deflate=0)
• nc_def_var_chunking() Required before any compression filter
• nc_put_var_float() Write entire variable in one call
• nc_get_var_float() Read entire variable in one call

Note

The program is intended for local performance profiling. Build with ENABLE_BENCHMARKS=ON (CMake) or –enable-benchmarks (Autotools); it is excluded from regular CI.

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Macro Definition Documentation

CHUNK_X

#define CHUNK_X   90

CHUNK_Y

#define CHUNK_Y   45

CHUNK_Z

#define CHUNK_Z   10

Chunk shape: matches all v1.10.0 performance examples for consistency.

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

ERRCODE

#define ERRCODE   2

H5Z_FILTER_LZ4

#define H5Z_FILTER_LZ4   32004

HDF5 LZ4 filter ID.

MAX_LEVEL

#define MAX_LEVEL   9

MIN_LEVEL

#define MIN_LEVEL   1

Minimum and maximum LZ4 levels.

NVALS

#define NVALS   ((size_t)NZ * NY * NX)

Total number of float values in the dataset.

NX

#define NX   360

Longitude dimension: 360 degrees.

NY

#define NY   180

Latitude dimension: 180 degrees.

NZ

#define NZ   500

Time dimension: 500 time steps.

TMP_FILE

#define TMP_FILE   "lz4_tmp.nc"

Temporary NetCDF file created and removed for each measurement.

UNCOMPRESSED_BYTES

#define UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))

Uncompressed size in bytes.

Function Documentation

get_time()

static double get_time ( void  )

static

Return current wall-clock time in seconds.

Returns

Seconds since the epoch as a double.

main()

int main

(

void

)

Main entry point.

Allocates a 500×180×360 NC_FLOAT buffer with synthetic temperature data, then iterates LZ4 levels 1–9 × shuffle {0, 1} (18 rows total), printing one CSV row per combination.

Returns

0 on success, 1 on any error.

run_one()

static int run_one ( float *  data, int  level, int  shuffle  )

static

Run one lz4/shuffle combination and print a CSV row.

Creates lz4_tmp.nc with the given lz4 level and shuffle setting, writes a 500×180×360 NC_FLOAT temperature variable, stats the file to obtain the compressed size, reads the variable back, then removes the file.

When shuffle is 1, nc_def_var_deflate() is called with shuffle=1 and deflate=0 to activate the byte-shuffle filter without zlib compression; nc_def_var_lz4() is then called to add lz4 on top.

Parameters

data	Pre-allocated buffer of NVALS floats (synthetic data).
level	LZ4 compression level (1 to 9).
shuffle	1 to enable the shuffle filter, 0 to disable.

Returns

0 on success, 1 on any error.