lossless.c File Reference

Compare best-performing settings of all lossless compression filters. More...

#include <netcdf.h>
#include <netcdf_filter.h>
#include "nep.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/time.h>

Data Structures
struct	filter_config_t

Macros
#define	NZ   500
#define	NY   180
#define	NX   360
#define	TMP_FILE   "lossless_tmp.nc"
#define	CHUNK_Z   10
#define	CHUNK_Y   45
#define	CHUNK_X   90
#define	NVALS   ((size_t)NZ * NY * NX)
#define	UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}
#define	NUM_FILTERS   (sizeof(filters) / sizeof(filters[0]))

Functions
static double	get_time (void)
	Return current wall-clock time in seconds.
static int	apply_filter (int ncid, int varid, const filter_config_t *filter)
	Apply the specified filter to the variable.
static int	verify_filter (int ncid, int varid, const filter_config_t *filter)
	Verify that the filter was applied correctly.
static int	run_one_filter (float *data, const filter_config_t *filter)
	Run one filter test and print a CSV row.
int	main (void)
	Main entry point.

Variables
static const filter_config_t	filters []

Detailed Description

Compare best-performing settings of all lossless compression filters.

This program evaluates the optimal settings for each available lossless compression filter (DEFLATE, Zstandard, SZIP, LZ4, BZIP2) with the shuffle filter enabled, providing a unified comparison for scientific floating-point data. Results help users choose the right compression based on their priorities: maximum compression ratio or fastest I/O throughput.

Filters tested (all with shuffle enabled):

• DEFLATE (gzip): level 1 (good ratio, fast compression)
• Zstandard: level 1 (optimal ratio vs speed tradeoff)
• SZIP: NC_SZIP_NN with pixels_per_block=2 (best for NC_FLOAT)
• LZ4: level 1 (best ratio with shuffle)
• BZIP2: level 1 (optimal for this data type)

The shuffle filter is enabled for all tests via nc_def_var_deflate(ncid, varid, 1, 0, 0) before applying the primary compression filter. This has been empirically determined to significantly improve compression ratios for IEEE 754 floating-point data.

This program iterates over the 5 filters (1 row per filter) over a 500×180×360 NC_FLOAT temperature dataset (~129 MB uncompressed) and reports:

• Filter name and optimal setting (level or pixels_per_block)
• Compressed file size on disk (bytes via stat())
• Compression ratio (uncompressed / compressed)
• Write time (seconds)
• Read time (seconds, full variable via nc_get_var_float())

Output: CSV printed to stdout with columns:

filter,level_or_pixels,compressed_bytes,ratio,write_s,read_s

Typical workflow:

./lossless > lossless_results.csv
python3 plot_lossless.py   # produces lossless_performance.jpg

Learning Objectives:

• Compare all available lossless compression filters at their optimal settings
• Understand the performance trade-offs: ratio vs write speed vs read speed
• Learn which filter is best for different priorities (archival vs real-time)
• See the consistent benefit of the shuffle filter across all lossless compressors
• Make informed filter selection decisions based on empirical measurements

Key Concepts:

• Filter Comparison: Each lossless filter has different strengths:
  • DEFLATE: universal compatibility, moderate speed and ratio
  • Zstandard: best speed/ratio tradeoff for most workloads
  • SZIP: designed for scientific data, good for correlated integers
  • LZ4: fastest decompression, lowest ratio
  • BZIP2: highest ratio, slowest speed
• Optimal Settings: Each filter tested at its empirically best single setting (determined from individual filter benchmarks), not all possible levels
• Shuffle Baseline: All filters tested with shuffle enabled, since shuffle universally improves ratio for IEEE 754 float data at minimal cost
• Unified Methodology: Same dataset, chunk shape, and measurement approach across all filters for fair comparison

Prerequisites:

• simple_nc4.c - NetCDF-4 file creation basics
• chunking.c - Understanding chunked storage
• deflate.c - Understanding individual filter benchmarks

Related Examples:

• deflate.c - Detailed deflate level sweep
• zstandard.c - Detailed Zstandard level sweep
• bzip2.c - Detailed BZIP2 level sweep
• lz4.c - Detailed LZ4 level sweep
• szip.c - Detailed SZIP pixels_per_block sweep
• quantize.c - Lossy quantization combined with lossless filters

Key API functions:

• nc_def_var_deflate() Enable shuffle and/or DEFLATE compression
• nc_def_var_zstandard() Enable Zstandard compression
• nc_def_var_szip() Enable SZIP compression
• nc_def_var_lz4() Enable LZ4 compression
• nc_def_var_bzip2() Enable BZIP2 compression
• nc_inq_var_*() Query filter settings for verification
• nc_put_var_float() Write entire variable in one call
• nc_get_var_float() Read entire variable in one call

Note

The program is intended for local performance profiling. Build with ENABLE_BENCHMARKS=ON (CMake) or –enable-benchmarks (Autotools); it is excluded from regular CI.

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Macro Definition Documentation

CHUNK_X

#define CHUNK_X   90

CHUNK_Y

#define CHUNK_Y   45

CHUNK_Z

#define CHUNK_Z   10

Chunk shape: matches all v1.10.0 performance examples for consistency.

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

ERRCODE

#define ERRCODE   2

NUM_FILTERS

#define NUM_FILTERS   (sizeof(filters) / sizeof(filters[0]))

NVALS

#define NVALS   ((size_t)NZ * NY * NX)

Total number of float values in the dataset.

NX

#define NX   360

Longitude dimension: 360 degrees.

NY

#define NY   180

Latitude dimension: 180 degrees.

NZ

#define NZ   500

Time dimension: 500 time steps.

TMP_FILE

#define TMP_FILE   "lossless_tmp.nc"

Temporary NetCDF file created and removed for each measurement.

UNCOMPRESSED_BYTES

#define UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))

Uncompressed size in bytes.

Function Documentation

apply_filter()

static int apply_filter ( int  ncid, int  varid, const filter_config_t *  filter  )

static

Apply the specified filter to the variable.

Parameters

ncid	NetCDF file ID.
varid	Variable ID.
filter	Filter configuration.

Returns

0 on success, non-zero on error.

get_time()

static double get_time ( void  )

static

Return current wall-clock time in seconds.

Returns

Seconds since the epoch as a double.

main()

int main

(

void

)

Main entry point.

Allocates a 500×180×360 NC_FLOAT buffer with synthetic temperature data, then iterates over the 5 filters at their optimal settings (5 rows total), printing one CSV row per filter.

Returns

0 on success, 1 on any error.

run_one_filter()

static int run_one_filter ( float *  data, const filter_config_t *  filter  )

static

Run one filter test and print a CSV row.

Creates lossless_tmp.nc with the given filter at its optimal setting with shuffle enabled, writes a 500×180×360 NC_FLOAT temperature variable, stats the file to obtain the compressed size, reads the variable back, then removes the file.

Parameters

data	Pre-allocated buffer of NVALS floats (synthetic data).
filter	Filter configuration to test.

Returns

0 on success, 1 on any error.

verify_filter()

static int verify_filter ( int  ncid, int  varid, const filter_config_t *  filter  )

static

Verify that the filter was applied correctly.

Parameters

ncid	NetCDF file ID.
varid	Variable ID.
filter	Filter configuration.

Returns

0 on success, 1 on verification failure.

Variable Documentation

filters

const filter_config_t filters[]

static

Initial value:

= {
    {"deflate",   "DEFLATE",   1},
    {"zstandard", "ZSTANDARD", 1},
    {"szip",      "SZIP",      2},
    {"lz4",       "LZ4",       1},
    {"bzip2",     "BZIP2",     1},
}

Optimal filter configurations (determined empirically in prior sprints).