fill_values.c File Reference

Demonstrates fill value handling performance across classic and NetCDF-4 formats. More...

#include <netcdf.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/time.h>

Macros
#define	NZ   500
#define	NY   180
#define	NX   360
#define	NVALS   ((size_t)NZ * NY * NX)
#define	UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))
#define	FILL_VALUE   (9.9692099683868690e+36f)
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

Functions
static double	get_time (void)
	Return current wall-clock time in seconds.
static void	generate_data (float *data)
	Generate synthetic temperature data.
static int	run_test (const char *path, int format, int fill_mode, const char *format_name)
	Run one fill value test and print a CSV row.
int	main (void)
	Main entry point.

Detailed Description

Demonstrates fill value handling performance across classic and NetCDF-4 formats.

NetCDF supports fill values — a special value indicating missing or unwritten data. When fill mode is enabled (NC_FILL, the default), NetCDF automatically writes fill values to any data not explicitly written. This benchmark measures the performance impact of fill mode (ON vs OFF) across both classic (netCDF-3) and NetCDF-4/HDF5 formats.

This program creates a 500×180×360 NC_FLOAT temperature dataset (~129 MB) four times with different configurations and reports:

• Write time (seconds)
• Read time (seconds, full variable via nc_get_var_float())
• File size on disk (bytes via stat())

Output: CSV printed to stdout with columns:

format,fill_mode,write_s,read_s,file_bytes

Typical workflow:

./fill_values > fill_values_results.csv
python3 plot_fill_values.py   # produces fill_values_performance.jpg

Learning Objectives:

• Understand fill value behavior in NetCDF (automatic pre-filling of unwritten data)
• Learn the performance impact of fill mode ON vs OFF for different formats
• Recognize the difference between nc_set_fill() (file-level) and nc_def_var_fill() (variable-level) APIs
• Measure write overhead caused by fill pre-initialization
• Make informed decisions about when to disable fill mode for performance

Key Concepts:

• Fill Mode (NC_FILL): When enabled (default), NetCDF pre-initializes all allocated storage with the fill value before user data is written; ensures unwritten elements return a known sentinel value on read
• NC_NOFILL: Disables pre-initialization; unwritten elements contain arbitrary bytes — faster writes but dangerous if data has gaps
• Classic vs NetCDF-4: In classic format, fill pre-writes the entire variable as a contiguous block; in NetCDF-4/HDF5, fill applies per-chunk and the cost depends on chunk cache behavior
• **_FillValue Attribute**: The sentinel value used for unwritten data; default varies by type (e.g., 9.96921e+36 for NC_FLOAT); set via nc_put_att_*() or nc_def_var_fill()
• When to Disable Fill: Safe when you guarantee every element will be written; eliminates the write-before-write overhead (can save 30–50% on large datasets)

Prerequisites:

• simple_2D.c - Fill value basics (nc_def_var_fill, nc_inq_var_fill)
• simple_nc4.c - NetCDF-4 file creation
• chunking.c - Chunk-level storage understanding

Related Examples:

• deflate.c - Compression performance (fill affects compressibility)
• endianness.c - Another storage property performance benchmark
• chunking.c - Chunk shape performance

Key API functions:

• nc_set_fill() Enable/disable fill value mode
• nc_inq_fill() Query current fill mode setting
• nc_put_att_float() Set _FillValue attribute for classic files
• nc_def_var_fill() Define fill value for netCDF-4 variables
• nc_get_att_float() Query variable fill value

Note

The program is intended for local performance profiling. Build with ENABLE_BENCHMARKS=ON (CMake) or –enable-benchmarks (Autotools); it is excluded from regular CI.

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Macro Definition Documentation

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

Error handling macro.

ERRCODE

#define ERRCODE   2

Error exit code.

FILL_VALUE

#define FILL_VALUE   (9.9692099683868690e+36f)

NetCDF fill value for floats.

NVALS

#define NVALS   ((size_t)NZ * NY * NX)

Total number of float values in the dataset.

NX

#define NX   360

Longitude dimension: 360 degrees.

NY

#define NY   180

Latitude dimension: 180 degrees.

NZ

#define NZ   500

Time dimension: 500 time steps.

UNCOMPRESSED_BYTES

#define UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))

Uncompressed size in bytes.

Function Documentation

generate_data()

static void generate_data ( float *  data )

static

Generate synthetic temperature data.

Fills the buffer with synthetic data: 280.0 + t*0.1 + y*0.01 + x*0.001

Parameters

data	Buffer of NVALS floats to fill.

get_time()

static double get_time ( void  )

static

Return current wall-clock time in seconds.

Returns

Seconds since the epoch as a double.

main()

int main

(

void

)

Main entry point.

Runs four fill value tests:

1. Classic format with NC_FILL
2. Classic format with NC_NOFILL
3. NetCDF-4 format with NC_FILL
4. NetCDF-4 format with NC_NOFILL

Prints CSV header followed by one data row per test.

Returns

0 on success, 1 on any error.

run_test()

static int run_test ( const char *  path, int  format, int  fill_mode, const char *  format_name  )

static

Run one fill value test and print a CSV row.

Creates a file with the given format and fill mode, writes a 500×180×360 NC_FLOAT temperature variable, stats the file to obtain the on-disk size, reads the variable back, then removes the file.

Parameters

path	Path for temporary file (will be created and removed).
format	File format: NC_CLOBBER (classic) or NC_NETCDF4|NC_CLOBBER.
fill_mode	1 for NC_FILL, 0 for NC_NOFILL.
format_name	String for CSV output ("classic" or "nc4").

Returns

0 on success, 1 on any error.