chunking.c File Reference

Demonstrates how chunk shape selection affects NetCDF-4 I/O performance. More...

#include <netcdf.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>

Macros
#define	NZ   100
#define	NY   36
#define	NX   72
#define	CHUNK_Z_TIME   1
#define	CHUNK_Y_TIME   36
#define	CHUNK_X_TIME   72
#define	CHUNK_Z_COL   100
#define	CHUNK_Y_COL   1
#define	CHUNK_X_COL   1
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

Functions
static double	get_time (void)
	Return current wall-clock time in seconds.
static int	write_file (const char *path, size_t chunk_z, size_t chunk_y, size_t chunk_x)
	Create a NZ×NY×NX temperature file with the given chunk shape.
static int	time_access_time_slab (const char *path, const char *shape_label)
	Time the "time slab" access pattern: read all NZ horizontal fields.
static int	time_access_column (const char *path, const char *shape_label)
	Time the "column profile" access pattern: read all NY×NX point time series.
int	main (void)
	Main entry point.

Detailed Description

Demonstrates how chunk shape selection affects NetCDF-4 I/O performance.

Chunk shape is one of the most impactful performance decisions when creating a NetCDF-4 file. The optimal shape depends entirely on how the data will be accessed: a layout tuned for one access pattern is often worst-case for another.

This program creates a meteorologically realistic 3D temperature dataset (100 time steps × 36 latitudes × 72 longitudes) with two different chunk shapes and measures read performance across two contrasting access patterns:

Chunk shapes tested:

• Time-optimized (1×36×72): one complete horizontal field per chunk. Reading a time step loads exactly one chunk — ideal for NWP model output.
• Column-optimized (100×1×1): one complete time series per grid point. Extracting a station time series loads exactly one chunk — ideal for observation-style access.

Access patterns measured:

• Time slab: read all 100 horizontal fields sequentially (typical forecast output)
• Column profile: read all NY×NX = 2,592 point time series (one per grid point)

Output: CSV printed to stdout with columns: chunk_shape, access_pattern, elapsed_s, MB_per_s

The 2×2 contrast (two shapes × two patterns) demonstrates that a 10–100× performance difference is common between well-matched and mismatched configurations.

Plotting the output:

./chunking > results.csv
# Import results.csv into any spreadsheet or Python/R plotting tool

Learning Objectives:

• Understand how chunk shape determines which access patterns are fast or slow
• Learn to use nc_def_var_chunking() to set chunk shape at creation time
• Use nc_inq_var_chunking() to verify chunk shape on an existing variable
• Measure read throughput with gettimeofday() wall-clock timing
• See that neither chunk shape is universally better — match shape to access

Key Concepts:

• Chunk Shape: The dimensions of each storage block; determines how many chunks must be read for a given access pattern (fewer = faster)
• Time-Optimized Chunks (1×NY×NX): Each chunk holds one complete time slab; reading a time step loads exactly one chunk — ideal for spatial analysis
• Column-Optimized Chunks (NZ×1×1): Each chunk holds one complete time series; extracting a station record loads exactly one chunk — ideal for point analysis
• Access Pattern Mismatch: When chunk shape conflicts with access pattern, many extra chunks must be read and mostly discarded — 10–100× slower
• No Universal Best: The optimal chunk shape depends entirely on how the data will be read; design for the dominant access pattern

Prerequisites:

• simple_nc4.c - NetCDF-4 file creation basics
• chunking_performance.c - Broader chunking strategies overview

Related Examples:

• chunking_performance.c - Three chunking strategies compared (netcdf-4 tutorial)
• deflate.c - Compression performance (depends on chunk shape)
• cache_tuning.c - Chunk cache tuning (mitigates mismatch overhead)

Key API functions:

• nc_def_var_chunking() Set chunk shape at variable definition time
• nc_inq_var_chunking() Query chunk shape on an open variable
• nc_get_vara_float() Read a hyperslab (both access patterns)

Note

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Macro Definition Documentation

CHUNK_X_COL

#define CHUNK_X_COL   1

CHUNK_X_TIME

#define CHUNK_X_TIME   72

CHUNK_Y_COL

#define CHUNK_Y_COL   1

CHUNK_Y_TIME

#define CHUNK_Y_TIME   36

CHUNK_Z_COL

#define CHUNK_Z_COL   100

CHUNK_Z_TIME

#define CHUNK_Z_TIME   1

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

Error handling macro.

ERRCODE

#define ERRCODE   2

Error exit code.

NX

#define NX   72

Longitude dimension size

NY

#define NY   36

Latitude dimension size

NZ

#define NZ   100

Time dimension size

Function Documentation

get_time()

static double get_time ( void  )

static

Return current wall-clock time in seconds.

Returns

Seconds since the epoch as a double.

main()

int main

(

void

)

Main entry point.

Creates two NetCDF files with contrasting chunk shapes, runs two access patterns on each, and prints CSV results to stdout.

Returns

0 on success, 1 on any error.

time_access_column()

static int time_access_column ( const char *  path, const char *  shape_label  )

static

Time the "column profile" access pattern: read all NY×NX point time series.

Issues NY×NX nc_get_vara_float() calls, each requesting one complete [NZ, 1, 1] column. With column-optimized chunks this loads one chunk per call; with time-optimized chunks each call must touch NZ chunks.

Parameters

path	Path to the NetCDF file to read.
shape_label	Short string identifying the chunk shape (for CSV output).

Returns

0 on success, 1 on error.

time_access_time_slab()

static int time_access_time_slab ( const char *  path, const char *  shape_label  )

static

Time the "time slab" access pattern: read all NZ horizontal fields.

Issues NZ separate nc_get_vara_float() calls, each requesting one complete [1, NY, NX] slab. With time-optimized chunks this loads one chunk per call; with column-optimized chunks each call must load NY×NX chunks.

Parameters

path	Path to the NetCDF file to read.
shape_label	Short string identifying the chunk shape (for CSV output).

Returns

0 on success, 1 on error.

write_file()

static int write_file ( const char *  path, size_t  chunk_z, size_t  chunk_y, size_t  chunk_x  )

static

Create a NZ×NY×NX temperature file with the given chunk shape.

For time-optimized chunks (1×NY×NX) the file is written one time slab at a time, which aligns with the chunk boundaries. For column-optimized chunks (NZ×1×1) the file is written one column at a time, which aligns with those chunk boundaries and avoids the extreme slowness of writing column chunks via horizontal slabs.

Parameters

path	Path to the NetCDF file to create (overwritten if it exists).
chunk_z	Chunk size along the time dimension.
chunk_y	Chunk size along the latitude dimension.
chunk_x	Chunk size along the longitude dimension.

Returns

0 on success, 1 on error.