deflate.c File Reference

Demonstrates how zlib deflate levels and the shuffle filter affect NetCDF-4 compression ratio and I/O throughput. More...

#include <netcdf.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/time.h>

Macros
#define	NZ   500
#define	NY   180
#define	NX   360
#define	TMP_FILE   "deflate_tmp.nc"
#define	CHUNK_Z   10
#define	CHUNK_Y   45
#define	CHUNK_X   90
#define	NVALS   ((size_t)NZ * NY * NX)
#define	UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

Functions
static double	get_time (void)
	Return current wall-clock time in seconds.
static int	run_one (float *data, int level, int shuffle)
	Run one deflate/shuffle combination and print a CSV row.
int	main (void)
	Main entry point.

Detailed Description

Demonstrates how zlib deflate levels and the shuffle filter affect NetCDF-4 compression ratio and I/O throughput.

NetCDF-4/HDF5 supports per-variable deflate compression via nc_def_var_deflate(). The level parameter (0 = no compression, 9 = maximum compression effort) trades CPU time for smaller files. The shuffle filter (shuffle = 1) reorders bytes before compression: for IEEE 754 floating-point data the most-significant bytes of adjacent values are grouped together, which typically increases the deflate ratio by 2–5× at little additional cost.

This program iterates all 20 combinations of deflate level (0–9) and shuffle (off/on) over a 500×180×360 NC_FLOAT temperature dataset (~129 MB uncompressed) and reports:

• Compressed file size on disk (bytes via stat())
• Compression ratio (uncompressed / compressed)
• Write time (seconds)
• Read time (seconds, full variable via nc_get_var_float())

Output: CSV printed to stdout with columns:

deflate_level,shuffle,compressed_bytes,ratio,write_s,read_s

Typical workflow:

./deflate > deflate_results.csv
python3 plot_deflate.py   # produces deflate_performance.jpg

Learning Objectives:

• Understand the deflate (zlib/gzip) compression algorithm in NetCDF-4
• Learn how compression level (0–9) trades CPU time for file size reduction
• See how the shuffle filter improves compression for floating-point data
• Measure and compare write time, read time, and compression ratio
• Make data-driven decisions about compression settings for production use

Key Concepts:

• Deflate (zlib): The default lossless compression in HDF5/NetCDF-4; uses LZ77 + Huffman coding; level 0 = store only, level 9 = max effort
• Shuffle Filter: Byte transposition that groups corresponding bytes of adjacent values (e.g., all MSBs together), exploiting IEEE 754 structure
• Level 1 Sweet Spot: For scientific float data, level 1 typically achieves 90–95% of level 9's compression ratio at 3–5× faster write speed
• Chunked Storage: Compression requires chunked layout; each chunk is compressed independently as a single block

Prerequisites:

• simple_nc4.c - NetCDF-4 file creation basics
• chunking.c - Understanding chunked storage (required for compression)

Related Examples:

• zstandard.c - Zstandard compression (wider level range, often faster)
• bzip2.c - BZIP2 compression (higher ratio, slower)
• lz4.c - LZ4 compression (fastest decompression)
• lossless.c - Unified comparison of all lossless filters
• compression.c - NetCDF-4 compression tutorial (non-benchmark)

Key API functions:

• nc_def_var_deflate() Enable deflate and/or shuffle filter on a variable
• nc_inq_var_deflate() Query deflate and shuffle settings
• nc_put_var_float() Write entire variable in one call
• nc_get_var_float() Read entire variable in one call

Note

The program is intended for local performance profiling. Build with ENABLE_BENCHMARKS=ON (CMake) or –enable-benchmarks (Autotools); it is excluded from regular CI.

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Macro Definition Documentation

CHUNK_X

#define CHUNK_X   90

CHUNK_Y

#define CHUNK_Y   45

CHUNK_Z

#define CHUNK_Z   10

Chunk shape: one time slab per chunk (good general-purpose shape for deflate).

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

ERRCODE

#define ERRCODE   2

NVALS

#define NVALS   ((size_t)NZ * NY * NX)

Total number of float values in the dataset.

NX

#define NX   360

Longitude dimension: 360 degrees.

NY

#define NY   180

Latitude dimension: 180 degrees.

NZ

#define NZ   500

Time dimension: 500 time steps.

TMP_FILE

#define TMP_FILE   "deflate_tmp.nc"

Temporary NetCDF file created and removed for each measurement.

UNCOMPRESSED_BYTES

#define UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))

Uncompressed size in bytes.

Function Documentation

get_time()

static double get_time ( void  )

static

Return current wall-clock time in seconds.

Returns

Seconds since the epoch as a double.

main()

int main

(

void

)

Main entry point.

Allocates a 500×180×360 NC_FLOAT buffer with synthetic temperature data, then iterates all 20 combinations of shuffle (0 and 1) and deflate level (0–9), printing one CSV row per combination.

Returns

0 on success, 1 on any error.

run_one()

static int run_one ( float *  data, int  level, int  shuffle  )

static

Run one deflate/shuffle combination and print a CSV row.

Creates deflate_tmp.nc with the given deflate level and shuffle setting, writes a 500×180×360 NC_FLOAT temperature variable, stats the file to obtain the compressed size, reads the variable back, then removes the file.

Parameters

data	Pre-allocated buffer of NVALS floats (synthetic data).
level	Deflate level (0–9).
shuffle	1 to enable the shuffle filter, 0 to disable.

Returns

0 on success, 1 on any error.