cache_tuning.c File Reference

Demonstrates NetCDF-4 chunk cache configuration for optimal I/O performance. More...

#include <netcdf.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>

Macros
#define	NDIMS   3
#define	NX   360
#define	NY   180
#define	NZ   500
#define	CHUNK_X   90
#define	CHUNK_Y   45
#define	CHUNK_Z   10
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

Functions
int	main (int argc, char **argv)
	Main entry point for the cache tuning example.

Detailed Description

Demonstrates NetCDF-4 chunk cache configuration for optimal I/O performance.

This example explores NetCDF-4's chunk cache mechanism and demonstrates how proper cache configuration can dramatically improve read performance for chunked datasets. The chunk cache is an in-memory buffer that stores recently accessed data chunks, reducing disk I/O when the same chunks are read repeatedly.

The program creates a realistic 3D scientific dataset (temperature over time, latitude, longitude) with chunked storage, then performs four experiments:

1. Baseline measurement with default cache settings
2. Performance scaling across multiple cache sizes (4MB to 128MB)
3. Cache thrashing detection and mitigation
4. File-level vs variable-level cache configuration

What is Chunk Caching? When NetCDF-4 reads chunked data, it first checks the chunk cache. If the chunk is in cache, it's returned immediately without disk I/O. If not, the chunk is read from disk and stored in cache for future access. Proper cache sizing ensures frequently accessed chunks remain in memory.

Learning Objectives:

• Understand how the NetCDF-4 chunk cache improves read performance
• Learn to query current cache settings with nc_get_var_chunk_cache()
• Configure variable-level cache with nc_set_var_chunk_cache()
• Configure file-level default cache with nc_set_chunk_cache()
• Detect and resolve cache thrashing scenarios
• Select appropriate cache sizes for your datasets
• Query chunking configuration with nc_inq_var_chunking()

Key Concepts:

• Chunk Cache: In-memory buffer storing recently accessed chunks
• Cache Size: Total memory allocated for caching (default is typically 64MB)
• Cache Elements: Maximum number of chunks that can be cached (default ~1009)
• Preemption Policy: How the cache evicts chunks when full (0.0-1.0, default 0.75)
• Cache Hit: Data found in cache, no disk read needed
• Cache Miss: Data not in cache, must read from disk
• Thrashing: When working set exceeds cache size, causing constant eviction
• Variable-Level Cache: Per-variable cache settings override file defaults
• File-Level Cache: Default cache for all variables in newly opened files

When to Tune the Cache:

• Reading large chunked datasets (>100MB)
• Repeated access to same data regions (time series analysis)
• Multi-pass algorithms (filtering, normalization)
• Random or strided access patterns
• Performance-critical applications

Cache Size Guidelines:

• Small datasets (<100MB): Default 64MB cache is usually sufficient
• Medium datasets (100MB-2GB): 128MB-512MB cache recommended
• Large datasets (>2GB): 512MB-2GB+ cache for optimal performance
• Working set size: Cache should hold all chunks accessed repeatedly
• System memory: Don't exceed available RAM - leave headroom for OS

Prerequisites:

• chunking_performance.c - Understanding chunking concepts
• simple_nc4.c - NetCDF-4 file basics
• Understanding of C arrays and memory management

Related Examples:

• chunking_performance.c - Chunking strategies and their performance impact
• compression.c - Compression (requires chunking)
• f_chunking_performance.f90 - Fortran equivalent

Compilation:

gcc -o cache_tuning cache_tuning.c -lnetcdf

Usage (API demonstration only):

./cache_tuning

Usage (with performance benchmarks):

# Build with benchmarks enabled
cmake -DENABLE_BENCHMARKS=ON ..
make cache_tuning
./cache_tuning

Expected Output (without ENABLE_BENCHMARKS):

NetCDF-4 Chunk Cache Tuning Example
===================================
 
Default cache settings:
  cache_size: 67108864 bytes (64.0 MB)
  nelems: 1009
  preemption: 0.75
 
Dataset: 500x180x360 temperature, chunked 10x45x90 (~600 KB/chunk)
Access pattern: Reading all 800 chunks repeatedly
 
Note: Timing tests disabled. Rebuild with ENABLE_BENCHMARKS=ON to run performance tests.
 
Test 4: File-level cache via nc_set_chunk_cache() and nc_get_chunk_cache()
  Default file-level cache: 64.0 MB, nelems=1009, preemption=0.75
  After nc_set_chunk_cache(): 256.0 MB, nelems=1009, preemption=0.75
  Variable storage: chunked, chunks: 10x45x90
  File opened with variable cache_size: 64.0 MB
 
All cache tuning tests complete.
 
Key takeaways:
  - Default cache may be too small for large datasets
  - Variable-level cache: nc_set_var_chunk_cache() for specific variables
  - File-level cache: nc_set_chunk_cache() affects new file opens
  - Cache thrashing occurs when working set exceeds cache size
  - Proper tuning can provide significant performance improvement

Expected Output (with ENABLE_BENCHMARKS): CSV data suitable for plotting (cache size vs performance):

# CSV data for plotting cache size vs performance
CacheSizeMB,TimeSeconds,Speedup
# Test 1: Default cache - 5 full passes through dataset
64.0,8.234,1.00
# Test 2: Cache size scaling - testing 256MB, 512MB, 1GB, 2GB
256.0,6.543,1.26
512.0,4.321,1.91
1024.0,2.876,2.86
2048.0,2.123,3.88
# Test 3: Cache thrashing detection (50-chunk working set, 100 iterations)
16.0,8.901,1.00
52.8,0.345,25.80

The CSV output can be plotted with gnuplot:

$ ./cache_tuning | grep -v "^#" > cache_results.csv
$ gnuplot -e "set datafile separator ','; plot 'cache_results.csv' using 1:2 with lines title 'Time vs Cache Size'"

Note

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Date

2026

Macro Definition Documentation

CHUNK_X

#define CHUNK_X   90

CHUNK_Y

#define CHUNK_Y   45

CHUNK_Z

#define CHUNK_Z   10

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

ERRCODE

#define ERRCODE   2

NDIMS

#define NDIMS   3

NX

#define NX   360

NY

#define NY   180

NZ

#define NZ   500

Function Documentation

main()

int main	(	int	argc,
		char **	argv
	)

Main entry point for the cache tuning example.

Creates a test dataset if needed, then runs four cache experiments:

1. Baseline with default cache
2. Performance scaling across cache sizes
3. Thrashing detection
4. File-level cache configuration