szip.c File Reference

Demonstrates how SZIP options_mask and pixels_per_block affect NetCDF-4 compression ratio and I/O throughput. More...

#include <netcdf.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/time.h>

Macros
#define	NZ   500
#define	NY   180
#define	NX   360
#define	TMP_FILE   "szip_tmp.nc"
#define	CHUNK_Z   10
#define	CHUNK_Y   45
#define	CHUNK_X   90
#define	NVALS   ((size_t)NZ * NY * NX)
#define	UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}
#define	N_PPB   5

Functions
static double	get_time (void)
	Return current wall-clock time in seconds.
static int	run_one (float *data, int pixels_per_block)
	Run one SZIP combination and print a CSV row.
int	main (void)
	Main entry point.

Variables
static const int	PPB_VALUES [] = {2, 6, 10, 18, 30}

Detailed Description

Demonstrates how SZIP options_mask and pixels_per_block affect NetCDF-4 compression ratio and I/O throughput.

SZIP (Lossless Scientific Data Compression) is a lossless compression algorithm developed at JPL for scientific floating-point data and integrated into HDF5 as filter ID 4. NetCDF-4 exposes it via nc_def_var_szip().

SZIP supports two coding methods: nearest-neighbor (NC_SZIP_NN) and entropy coding (NC_SZIP_EC). However, NC_SZIP_EC requires integer data with a defined bit depth and does not work with NC_FLOAT. This program therefore uses NC_SZIP_NN exclusively, which is the correct and commonly used mode for floating-point scientific data.

The pixels_per_block parameter (even values 2 to NC_MAX_PIXELS_PER_BLOCK = 32) controls the block size used by the SZIP encoder. Larger values give the encoder more context and often improve ratio, at some increase in CPU cost. For chunked storage, pixels_per_block must divide the chunk's fastest-varying dimension (CHUNK_X). With CHUNK_X=90, valid values are 2 (90/2=45) and any divisor of 90.

This program iterates 5 pixels_per_block values over a 500x180x360 NC_FLOAT temperature dataset (~129 MB uncompressed) and reports:

• Compressed file size on disk (bytes via stat())
• Compression ratio (uncompressed / compressed)
• Write time (seconds)
• Read time (seconds, full variable via nc_get_var_float())

Output: CSV printed to stdout with columns:

pixels_per_block,compressed_bytes,ratio,write_s,read_s

Typical workflow:

./szip > szip_results.csv
python3 plot_szip.py   # produces szip_performance.jpg

Learning Objectives:

• Understand SZIP compression designed specifically for scientific data
• Learn the pixels_per_block parameter and its constraints (even, divides chunk)
• Recognize the difference between NC_SZIP_NN (float-safe) and NC_SZIP_EC (integer-only)
• Measure SZIP performance characteristics vs other filters
• Understand SZIP's niche: good for integer sensor data and satellite imagery

Key Concepts:

• SZIP: Lossless compression developed at JPL for scientific data; integrated into HDF5 as filter ID 4; uses an entropy coding scheme optimized for correlated scientific data
• NC_SZIP_NN (Nearest-Neighbor): Coding mode safe for all data types including NC_FLOAT; the standard choice for floating-point data
• NC_SZIP_EC (Entropy Coding): Higher compression but requires integer data with defined bit depth; will fail on NC_FLOAT variables
• pixels_per_block: Block size for SZIP encoder (must be even, 2–32, and must divide the fastest-varying chunk dimension); larger values provide more context for better compression
• License: SZIP decompression is always available in HDF5; SZIP write (encoding) requires a build linked with an SZIP encoder library (libaec)

Prerequisites:

• simple_nc4.c - NetCDF-4 file creation basics
• chunking.c - Understanding chunked storage (required for compression)
• deflate.c - Comparison baseline

Related Examples:

• deflate.c - Deflate compression (more portable, no license concerns)
• zstandard.c - Zstandard compression (modern alternative)
• lossless.c - Unified comparison of all lossless filters including SZIP

Key API functions:

• nc_def_var_szip() Enable SZIP compression (NC_SZIP_NN) with given block size
• nc_inq_var_szip() Query SZIP settings on an open variable
• nc_def_var_chunking() Set chunked storage (required before any filter)
• nc_put_var_float() Write entire variable in one call
• nc_get_var_float() Read entire variable in one call

Note

SZIP write support requires a NetCDF-C/HDF5 build compiled with szlib.

NC_SZIP_EC is not used because it does not support NC_FLOAT data. Build with ENABLE_BENCHMARKS=ON (CMake) or –enable-benchmarks (Autotools); it is excluded from regular CI.

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Macro Definition Documentation

CHUNK_X

#define CHUNK_X   90

CHUNK_Y

#define CHUNK_Y   45

CHUNK_Z

#define CHUNK_Z   10

Chunk shape: CHUNK_X=90 is divisible by all tested pixels_per_block values {2, 6, 10, 18, 30}: the even divisors of 90 up to NC_MAX_PIXELS_PER_BLOCK. pixels_per_block must be even and divide CHUNK_X for NC_SZIP_NN.

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

ERRCODE

#define ERRCODE   2

N_PPB

#define N_PPB   5

NVALS

#define NVALS   ((size_t)NZ * NY * NX)

Total number of float values in the dataset.

NX

#define NX   360

Longitude dimension: 360 degrees.

NY

#define NY   180

Latitude dimension: 180 degrees.

NZ

#define NZ   500

Time dimension: 500 time steps.

TMP_FILE

#define TMP_FILE   "szip_tmp.nc"

Temporary NetCDF file created and removed for each measurement.

UNCOMPRESSED_BYTES

#define UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))

Uncompressed size in bytes.

Function Documentation

get_time()

static double get_time ( void  )

static

Return current wall-clock time in seconds.

Returns

Seconds since the epoch as a double.

main()

int main

(

void

)

Main entry point.

Allocates a 500x180x360 NC_FLOAT buffer with synthetic temperature data, then iterates 5 pixels_per_block values (5 rows total), printing one CSV row per value. All rows use NC_SZIP_NN coding.

Returns

0 on success, 1 on any error.

run_one()

static int run_one ( float *  data, int  pixels_per_block  )

static

Run one SZIP combination and print a CSV row.

Creates szip_tmp.nc with the given options_mask and pixels_per_block, writes a 500x180x360 NC_FLOAT temperature variable, stats the file to obtain the compressed size, reads the variable back, then removes the file.

Parameters

data	Pre-allocated buffer of NVALS floats (synthetic data).
pixels_per_block	Value that divides CHUNK_X=90.

Returns

0 on success, 1 on any error.

Variable Documentation

PPB_VALUES

const int PPB_VALUES[] = {2, 6, 10, 18, 30}

static

pixels_per_block values to test: even divisors of CHUNK_X=90.