endianness.c File Reference

Demonstrates byte order (endianness) handling performance in NetCDF-4/HDF5. More...

#include <netcdf.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/time.h>

Macros
#define	NZ   500
#define	NY   180
#define	NX   360
#define	NVALS   ((size_t)NZ * NY * NX)
#define	UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))
#define	ERRCODE   2
#define	ERR(e)   {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

Functions
static double	get_time (void)
	Return current wall-clock time in seconds.
static void	generate_data (float *data)
	Generate synthetic temperature data.
static int	run_test (const char *path, int endian, const char *endian_name)
	Run one endianness test and print a CSV row.
int	main (void)
	Main entry point.

Detailed Description

Demonstrates byte order (endianness) handling performance in NetCDF-4/HDF5.

NetCDF-4 supports explicit byte order control via nc_def_var_endian(). By default, data is stored in the platform's native byte order (NC_ENDIAN_NATIVE). However, users can specify NC_ENDIAN_LITTLE or NC_ENDIAN_BIG to ensure cross-platform compatibility or match requirements of downstream tools.

This program creates a 500×180×360 NC_FLOAT temperature dataset (~129 MB uncompressed) three times with different byte orders and reports:

• Write time (seconds)
• Read time (seconds, full variable via nc_get_var_float())
• File size on disk (bytes via stat())

Output: CSV printed to stdout with columns:

endian_mode,write_s,read_s,file_bytes

Typical workflow:

./endianness > endianness_results.csv
python3 plot_endianness.py   # produces endianness_performance.jpg

Learning Objectives:

• Understand byte order (endianness) in scientific data storage
• Learn to control byte order with nc_def_var_endian()
• Measure the performance cost of byte-swapping on non-native endianness
• Recognize when explicit endianness control is beneficial for portability
• Make informed decisions about endianness for cross-platform data exchange

Key Concepts:

• Endianness: Byte order of multi-byte values; little-endian (LE) stores least-significant byte first (x86/ARM), big-endian (BE) stores MSB first
• NC_ENDIAN_NATIVE: Store in the platform's native byte order (default); fastest I/O since no byte-swapping is needed on the writing platform
• NC_ENDIAN_LITTLE: Force little-endian storage regardless of platform; ensures portability and fast reads on x86/ARM systems
• NC_ENDIAN_BIG: Force big-endian storage; matches legacy formats and network byte order (useful for interoperability with older tools)
• Byte-Swapping Cost: When reading non-native endianness, HDF5 must swap bytes for every value — typically adds 5–15% overhead for large arrays

Prerequisites:

• simple_nc4.c - NetCDF-4 file creation basics
• chunking.c - Understanding chunked storage in NetCDF-4

Related Examples:

• deflate.c - Compression performance (endianness affects compressibility slightly)
• fill_values.c - Another storage property that affects write performance
• chunking.c - Chunk shape performance (orthogonal to endianness)

Key API functions:

• nc_def_var_endian() Set byte order for a variable
• nc_inq_var_endian() Query current byte order setting
• nc_put_var_float() Write entire variable in one call
• nc_get_var_float() Read entire variable in one call

Note

The program is intended for local performance profiling. Build with ENABLE_BENCHMARKS=ON (CMake) or –enable-benchmarks (Autotools); it is excluded from regular CI.

Companion code for "The NetCDF Developer's Handbook: The Authoritative Guide to Writing High-Performance Programs for Scientific Data Management, Second Edition" (https://www.amazon.com/dp/B0H7Q1Z75L)

Author

Edward Hartnett, Intelligent Data Design, Inc.

Macro Definition Documentation

ERR

#define ERR

(

e

)

{printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

Error handling macro.

ERRCODE

#define ERRCODE   2

Error exit code.

NVALS

#define NVALS   ((size_t)NZ * NY * NX)

Total number of float values in the dataset.

NX

#define NX   360

Longitude dimension: 360 degrees.

NY

#define NY   180

Latitude dimension: 180 degrees.

NZ

#define NZ   500

Time dimension: 500 time steps.

UNCOMPRESSED_BYTES

#define UNCOMPRESSED_BYTES   ((double)NVALS * sizeof(float))

Uncompressed size in bytes.

Function Documentation

generate_data()

static void generate_data ( float *  data )

static

Generate synthetic temperature data.

Fills the buffer with synthetic data: 280.0 + t*0.1 + y*0.01 + x*0.001

Parameters

data	Buffer of NVALS floats to fill.

get_time()

static double get_time ( void  )

static

Return current wall-clock time in seconds.

Returns

Seconds since the epoch as a double.

main()

int main

(

void

)

Main entry point.

Runs three endianness tests:

1. NC_ENDIAN_NATIVE (platform native byte order)
2. NC_ENDIAN_LITTLE (explicit little-endian)
3. NC_ENDIAN_BIG (explicit big-endian)

Prints CSV header followed by one data row per test.

Returns

0 on success, 1 on any error.

run_test()

static int run_test ( const char *  path, int  endian, const char *  endian_name  )

static

Run one endianness test and print a CSV row.

Creates a file with the given byte order, writes a 500×180×360 NC_FLOAT temperature variable, stats the file to obtain the on-disk size, reads the variable back, then removes the file.

Parameters

path	Path for temporary file (will be created and removed).
endian	Byte order: NC_ENDIAN_NATIVE, NC_ENDIAN_LITTLE, or NC_ENDIAN_BIG.
endian_name	String for CSV output ("native", "little", or "big").

Returns

0 on success, 1 on any error.