compression.c File Reference

Demonstrates NetCDF-4 compression filters with performance analysis. More...

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <math.h>
#include <netcdf.h>
#include <netcdf_filter.h>

Data Structures
struct	CompressionTest

Macros
#define	NTIME   50
#define	NLAT   90
#define	NLON   180
#define	NDIMS   3
#define	FILL_VALUE   -9999.0f
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

Functions
void	generate_temperature_data (float *data)
long	get_file_size (const char *filename)
void	create_compressed_file (CompressionTest *test, float *data)
void	read_compressed_file (CompressionTest *test, float *original_data)
int	check_zstd_support (void)
int	main ()

Detailed Description

Demonstrates NetCDF-4 compression filters with performance analysis.

This example explores NetCDF-4's built-in compression capabilities by creating multiple files with different compression settings and measuring their performance characteristics. Compression is essential for reducing storage requirements and I/O bandwidth for large scientific datasets.

The program generates realistic 3D temperature data (time×lat×lon) and creates files with various compression configurations: no compression, deflate (zlib) only, Zstandard (zstd) only, shuffle only, and shuffle+deflate or shuffle+zstd combinations at different compression levels. It measures write/read times, file sizes, and compression ratios.

Learning Objectives:

• Understand NetCDF-4 compression filters (deflate, shuffle, Zstandard)
• Learn to configure compression with nc_def_var_deflate() and nc_def_var_zstandard()
• Master compression level selection (zlib 1-9, zstd 1-22 tradeoff)
• Measure compression performance (time, size, ratio)
• Make informed compression decisions for different data types

Key Concepts:

• Deflate Filter: GZIP compression (levels 1-9, higher = better compression)
• Zstandard Filter: Modern zstd compression (levels 1-22, wider speed/ratio range)
• Shuffle Filter: Byte reordering to improve compression of typed data
• Compression Ratio: Original size / compressed size
• Compression Overhead: Extra CPU time for compression/decompression
• Filter Pipeline: Shuffle then deflate or shuffle then zstd for optimal results
• Zstandard is preferred: When available, Zstandard (zstd) is generally faster than zlib and provides better compression ratios. Use zstd level 3 as the default; higher levels yield diminishing returns at greater CPU cost.
• Deflate level 1 is best for zlib: For almost all real-world scientific data, deflate level 1 provides nearly the same compression ratio as higher zlib levels but at a fraction of the CPU cost. Use deflate only when zstd is unavailable.

Compression Strategies:

• No Compression: Fastest I/O, largest files, use for small datasets
• Deflate Only: Good compression, slower, use for mixed data types
• Shuffle Only: Minimal overhead, modest gains, use for fast I/O
• Shuffle+Deflate 1: Best zlib tradeoff; default when zstd is unavailable
• Shuffle+Zstandard 3: Best overall tradeoff; preferred when zstd is available
• High Deflate/Zstandard Levels (5-9): Marginally better compression, much slower, rarely worth the cost except for one-time archival of small datasets

When to Use Compression:

• Large datasets where storage/bandwidth is limited
• Floating-point data with spatial/temporal correlation
• Archival data where read performance is less critical
• Network transfers where bandwidth is constrained

Prerequisites:

• simple_nc4.c - NetCDF-4 format basics
• var4d.c - Multi-dimensional arrays

Related Examples:

• f_compression.f90 - Fortran equivalent
• chunking_performance.c - Chunking impacts compression
• simple_nc4.c - NetCDF-4 format introduction

Compilation:

gcc -o compression compression.c -lnetcdf -lm

Usage:

./compression
ls -lh compression_*.nc

Expected Output: Creates multiple files with compression analysis:

• compress_none.nc (uncompressed baseline)
• compress_deflate_N.nc (deflate levels 1, 5, 9)
• compress_shuffle.nc (shuffle only)
• compress_shuffle_deflate1.nc (combined, level 1)
• compress_zstd_N.nc (zstd levels 1, 3, 9)
• compress_shuffle_zstd_N.nc (combined, levels 3, 9) Note: Level 1 is the preferred deflate level; zstd level 3 is the best Zstandard tradeoff for most real-world data. Displays performance comparison table and a best zlib vs best zstd head-to-head.

Note

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Date

2026

Macro Definition Documentation

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

ERRCODE

#define ERRCODE   2

FILL_VALUE

#define FILL_VALUE   -9999.0f

NDIMS

#define NDIMS   3

NLAT

#define NLAT   90

NLON

#define NLON   180

NTIME

#define NTIME   50

Function Documentation

check_zstd_support()

int check_zstd_support

(

void

)

create_compressed_file()

void create_compressed_file	(	CompressionTest *	test,
		float *	data
	)

generate_temperature_data()

void generate_temperature_data

(

float *

data

)

get_file_size()

long get_file_size

(

const char *

filename

)

main()

int main

(

void

)

read_compressed_file()

void read_compressed_file	(	CompressionTest *	test,
		float *	original_data
	)