bzip2.c File Reference

Demonstrates how BZIP2 compression levels and the shuffle filter affect NetCDF-4 compression ratio and I/O throughput. More...

#include <netcdf.h>
#include <netcdf_filter.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/time.h>

Macros
#define	NZ   500
#define	NY   180
#define	NX   360
#define	TMP_FILE   "bzip2_tmp.nc"
#define	CHUNK_Z   10
#define	CHUNK_Y   45
#define	CHUNK_X   90
#define	NVALS   ((size_t)NZ * NY * NX)
#define	UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}
#define	MIN_LEVEL   1
#define	MAX_LEVEL   9

Functions
static double	get_time (void)
	Return current wall-clock time in seconds.
static int	run_one (float *data, int level, int shuffle)
	Run one bzip2/shuffle combination and print a CSV row.
int	main (void)
	Main entry point.

Detailed Description

Demonstrates how BZIP2 compression levels and the shuffle filter affect NetCDF-4 compression ratio and I/O throughput.

NetCDF-4/HDF5 supports BZIP2 compression via nc_def_var_bzip2(). BZIP2 is a block-sorting lossless compression algorithm that typically achieves higher compression ratios than LZ4 or Zstandard, but at the cost of significantly slower compression speed. BZIP2 offers a level range (1–9) where higher levels spend more CPU time searching for better compression patterns within each block.

The shuffle filter (shuffle = 1) reorders bytes before compression: for IEEE 754 floating-point data the most-significant bytes of adjacent values are grouped together, which typically increases the compression ratio by 3–5× at little additional cost. Because nc_def_var_bzip2() does not accept a shuffle parameter directly, shuffle is enabled by calling nc_def_var_deflate(ncid, varid, 1, 0, 0) (shuffle only, no deflate) before nc_def_var_bzip2().

This program iterates BZIP2 levels 1–9 × shuffle {off, on} (18 combinations total) over a 500×180×360 NC_FLOAT temperature dataset (~129 MB uncompressed) and reports:

• Compressed file size on disk (bytes via stat())
• Compression ratio (uncompressed / compressed)
• Write time (seconds)
• Read time (seconds, full variable via nc_get_var_float())

Output: CSV printed to stdout with columns:

bzip2_level,shuffle,compressed_bytes,ratio,write_s,read_s

Typical workflow:

./bzip2 > bzip2_results.csv
python3 plot_bzip2.py   # produces bzip2_performance.jpg

Learning Objectives:

• Understand BZIP2 block-sorting compression in NetCDF-4/HDF5
• Learn how BZIP2 levels (1–9) affect compression ratio and speed
• See how the shuffle filter interacts with BZIP2 for float data
• Compare BZIP2 trade-offs: highest ratio but slowest speed among filters
• Determine when BZIP2's ratio advantage justifies its speed penalty

Key Concepts:

• BZIP2: Block-sorting (Burrows-Wheeler) lossless compression; typically achieves the highest ratio among NetCDF-4 filters but at 5–10× slower compression speed compared to deflate or Zstandard
• Block Sorting: BZIP2 rearranges data blocks to group similar bytes, then applies move-to-front transform and Huffman coding
• Level Range (1–9): Higher levels use larger block sizes (100K–900K); level 9 gives marginally better compression at more memory use
• Shuffle + BZIP2: The shuffle filter pre-groups float bytes for the block sorter, typically adding 3–5× compression ratio improvement
• Use Case: Best for archival storage where compression time is acceptable and maximum file size reduction is the priority

Prerequisites:

• simple_nc4.c - NetCDF-4 file creation basics
• chunking.c - Understanding chunked storage (required for compression)
• deflate.c - Comparison baseline (deflate is the traditional default)

Related Examples:

• deflate.c - Traditional deflate/zlib compression (faster, lower ratio)
• zstandard.c - Zstandard compression (faster than BZIP2, similar ratio)
• lz4.c - LZ4 compression (fastest, lowest ratio)
• lossless.c - Unified comparison of all lossless filters

Key API functions:

• nc_def_var_bzip2() Enable BZIP2 compression at a given level
• nc_inq_var_bzip2() Query BZIP2 settings on an open variable
• nc_def_var_deflate() Enable shuffle filter without deflate (deflate=0)
• nc_def_var_chunking() Required before any compression filter
• nc_put_var_float() Write entire variable in one call
• nc_get_var_float() Read entire variable in one call

Note

The program is intended for local performance profiling. Build with ENABLE_BENCHMARKS=ON (CMake) or –enable-benchmarks (Autotools); it is excluded from regular CI.

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Macro Definition Documentation

CHUNK_X

#define CHUNK_X   90

CHUNK_Y

#define CHUNK_Y   45

CHUNK_Z

#define CHUNK_Z   10

Chunk shape: matches all v1.10.0 performance examples for consistency.

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

ERRCODE

#define ERRCODE   2

MAX_LEVEL

#define MAX_LEVEL   9

MIN_LEVEL

#define MIN_LEVEL   1

Minimum and maximum BZIP2 levels.

NVALS

#define NVALS   ((size_t)NZ * NY * NX)

Total number of float values in the dataset.

NX

#define NX   360

Longitude dimension: 360 degrees.

NY

#define NY   180

Latitude dimension: 180 degrees.

NZ

#define NZ   500

Time dimension: 500 time steps.

TMP_FILE

#define TMP_FILE   "bzip2_tmp.nc"

Temporary NetCDF file created and removed for each measurement.

UNCOMPRESSED_BYTES

#define UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))

Uncompressed size in bytes.

Function Documentation

get_time()

static double get_time ( void  )

static

Return current wall-clock time in seconds.

Returns

Seconds since the epoch as a double.

main()

int main

(

void

)

Main entry point.

Allocates a 500×180×360 NC_FLOAT buffer with synthetic temperature data, then iterates BZIP2 levels 1–9 × shuffle {0, 1} (18 rows total), printing one CSV row per combination.

Returns

0 on success, 1 on any error.

run_one()

static int run_one ( float *  data, int  level, int  shuffle  )

static

Run one bzip2/shuffle combination and print a CSV row.

Creates bzip2_tmp.nc with the given bzip2 level and shuffle setting, writes a 500×180×360 NC_FLOAT temperature variable, stats the file to obtain the compressed size, reads the variable back, then removes the file.

When shuffle is 1, nc_def_var_deflate() is called with shuffle=1 and deflate=0 to activate the byte-shuffle filter without zlib compression; nc_def_var_bzip2() is then called to add bzip2 on top.

Parameters

data	Pre-allocated buffer of NVALS floats (synthetic data).
level	BZIP2 compression level (1 to 9).
shuffle	1 to enable the shuffle filter, 0 to disable.

Returns

0 on success, 1 on any error.