HDF5

HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and is extensible, allowing applications to evolve in their use of HDF5. The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.

Installed versions

  • Use module avail lib/hdf5 to view installed version

How to?

  • load the latest HDF5 module
$ module load lib/hdf5

Manipulate data from your code

Example : Create Dataset

dataSet.c
/*
 *   Creating and closing a dataset.
 */
 
#include "hdf5.h"
#define FILE "dset.h5"
int main() {
 
   hid_t       file_id, dataset_id, dataspace_id;  /* identifiers */
   hsize_t     dims[2];
   herr_t      status;
 
   /* Create a new file using default properties. */
 
   file_id = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
 
   /* Create the data space for the dataset. */
 
   dims[0] = 4; 
   dims[1] = 6; 
   dataspace_id = H5Screate_simple(2, dims, NULL);
 
   /* Create the dataset. */
 
   dataset_id = H5Dcreate(file_id, "/dset", H5T_STD_I32BE, dataspace_id, 
 
                          H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
 
   /* End access to the dataset and release resources used by it. */
 
   status = H5Dclose(dataset_id);
 
   /* Terminate access to the data space. */ 
 
   status = H5Sclose(dataspace_id);
 
   /* Close the file. */
   status = H5Fclose(file_id);
}
  • Compile your code
$ [icc|ifort|gcc|mpicc] myApplication.c -o myProgram.exe -L$(HDF5_HOME)/lib -I$(HDF5_HOME)/include -lz -lhdf5 -lm

Python example

We need to load python first 

$ module load lang/python

Wrire HDF5 

#!/usr/bin/env python
import h5py
 
# Create random data
import numpy as np
data_matrix = np.random.uniform(-1, 1, size=(10, 3))
 
# Write data to HDF5
data_file = h5py.File('file.hdf5', 'w')
data_file.create_dataset('group_name', data=data_matrix)
data_file.close()

Read HDF5 

import h5py
filename = 'file.hdf5'
f = h5py.File(filename, 'r')
 
# List all groups
print("Keys: %s" % f.keys())
a_group_key = list(f.keys())[0]
 
# Get the data
data = list(f[a_group_key])

Using H5 tools

Commands are located here

Links