zstandard.c File Reference

Demonstrates how Zstandard compression levels and the shuffle filter affect NetCDF-4 compression ratio and I/O throughput. More...

#include <netcdf.h>
#include <netcdf_filter.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/time.h>

Macros
#define	NZ   500
#define	NY   180
#define	NX   360
#define	TMP_FILE   "zstandard_tmp.nc"
#define	CHUNK_Z   10
#define	CHUNK_Y   45
#define	CHUNK_X   90
#define	NVALS   ((size_t)NZ * NY * NX)
#define	UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}
#define	NLEVELS   (int)(sizeof(LEVELS) / sizeof(LEVELS[0]))

Functions
static double	get_time (void)
	Return current wall-clock time in seconds.
static int	run_one (float *data, int level, int shuffle)
	Run one zstd/shuffle combination and print a CSV row.
int	main (void)
	Main entry point.

Variables
static const int	LEVELS [] = {-7, -3, -1, 0, 1, 3, 6, 9, 12, 15, 19, 22}

Detailed Description

Demonstrates how Zstandard compression levels and the shuffle filter affect NetCDF-4 compression ratio and I/O throughput.

NetCDF-4/HDF5 supports Zstandard (zstd) compression via nc_def_var_zstandard(). Unlike zlib deflate (levels 0–9), zstd supports a much wider range: negative levels (-7 to -1) trade compression ratio for extreme write speed, level 1 is the fast default, and levels up to 22 push toward maximum compression at increasing CPU cost.

The shuffle filter (shuffle = 1) reorders bytes before compression: for IEEE 754 floating-point data the most-significant bytes of adjacent values are grouped together, which typically increases the compression ratio by 3–5× at little additional cost. Because nc_def_var_zstandard() does not accept a shuffle parameter directly, shuffle is enabled by calling nc_def_var_deflate(ncid, varid, 1, 0, 0) (shuffle only, no deflate) before nc_def_var_zstandard().

This program iterates a representative subset of 12 levels {-7, -3, -1, 0, 1, 3, 6, 9, 12, 15, 19, 22} × shuffle {off, on} (44 combinations total) over a 500×180×360 NC_FLOAT temperature dataset (~129 MB uncompressed) and reports:

• Compressed file size on disk (bytes via stat())
• Compression ratio (uncompressed / compressed)
• Write time (seconds)
• Read time (seconds, full variable via nc_get_var_float())

Output: CSV printed to stdout with columns:

zstd_level,shuffle,compressed_bytes,ratio,write_s,read_s

Typical workflow:

./zstandard > zstandard_results.csv
python3 plot_zstandard.py   # produces zstandard_performance.jpg

Learning Objectives:

• Understand Zstandard (zstd) compression and its wide level range (-7 to 22)
• Learn how negative levels trade compression ratio for extreme write speed
• See how the shuffle filter interacts with Zstandard for float data
• Compare Zstandard performance against deflate at equivalent quality points
• Make data-driven decisions about Zstandard level selection

Key Concepts:

• Zstandard (zstd): Modern compression algorithm by Facebook/Meta; uses asymmetric numeral systems (ANS) for faster decode than deflate at equal ratio
• Negative Levels (-7 to -1): Ultra-fast compression with lower ratios; useful when write speed is paramount (e.g., real-time data acquisition)
• Level 1 Default: Good balance of speed and compression; comparable ratio to deflate level 6 at much faster write speed
• High Levels (12–22): Push toward maximum compression at significant CPU cost; useful for archival of rarely-rewritten datasets
• Shuffle + Zstandard: The shuffle filter typically adds 3–5× improvement to zstd compression ratio for IEEE 754 float data

Prerequisites:

• simple_nc4.c - NetCDF-4 file creation basics
• chunking.c - Understanding chunked storage (required for compression)
• deflate.c - Comparison baseline (deflate is the traditional default)

Related Examples:

• deflate.c - Traditional deflate/zlib compression
• bzip2.c - BZIP2 compression (higher ratio, much slower)
• lz4.c - LZ4 compression (fastest decompression)
• lossless.c - Unified comparison of all lossless filters

Key API functions:

• nc_def_var_zstandard() Enable Zstandard compression at a given level
• nc_inq_var_zstandard() Query Zstandard settings on an open variable
• nc_def_var_deflate() Enable shuffle filter without deflate (deflate=0)
• nc_def_var_chunking() Required before any compression filter
• nc_put_var_float() Write entire variable in one call
• nc_get_var_float() Read entire variable in one call

Note

The program is intended for local performance profiling. Build with ENABLE_BENCHMARKS=ON (CMake) or –enable-benchmarks (Autotools); it is excluded from regular CI.

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Macro Definition Documentation

CHUNK_X

#define CHUNK_X   90

CHUNK_Y

#define CHUNK_Y   45

CHUNK_Z

#define CHUNK_Z   10

Chunk shape: matches all v1.10.0 performance examples for consistency.

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

ERRCODE

#define ERRCODE   2

NLEVELS

#define NLEVELS   (int)(sizeof(LEVELS) / sizeof(LEVELS[0]))

NVALS

#define NVALS   ((size_t)NZ * NY * NX)

Total number of float values in the dataset.

NX

#define NX   360

Longitude dimension: 360 degrees.

NY

#define NY   180

Latitude dimension: 180 degrees.

NZ

#define NZ   500

Time dimension: 500 time steps.

TMP_FILE

#define TMP_FILE   "zstandard_tmp.nc"

Temporary NetCDF file created and removed for each measurement.

UNCOMPRESSED_BYTES

#define UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))

Uncompressed size in bytes.

Function Documentation

get_time()

static double get_time ( void  )

static

Return current wall-clock time in seconds.

Returns

Seconds since the epoch as a double.

main()

int main

(

void

)

Main entry point.

Allocates a 500×180×360 NC_FLOAT buffer with synthetic temperature data, then iterates 12 representative zstd levels × shuffle {0, 1} (44 rows total), printing one CSV row per combination.

Returns

0 on success, 1 on any error.

run_one()

static int run_one ( float *  data, int  level, int  shuffle  )

static

Run one zstd/shuffle combination and print a CSV row.

Creates zstandard_tmp.nc with the given zstd level and shuffle setting, writes a 500×180×360 NC_FLOAT temperature variable, stats the file to obtain the compressed size, reads the variable back, then removes the file.

When shuffle is 1, nc_def_var_deflate() is called with shuffle=1 and deflate=0 to activate the byte-shuffle filter without zlib compression; nc_def_var_zstandard() is then called to add zstd on top.

Parameters

data	Pre-allocated buffer of NVALS floats (synthetic data).
level	Zstandard compression level (-7 to 22).
shuffle	1 to enable the shuffle filter, 0 to disable.

Returns

0 on success, 1 on any error.

Variable Documentation

LEVELS

const int LEVELS[] = {-7, -3, -1, 0, 1, 3, 6, 9, 12, 15, 19, 22}

static

Representative zstd levels covering the full range -7 to 22.