Page tree

The purpose of this page is to briefly describe the new HDF5 Collective Metadata I/O feature and provide a gateway to available documentation. The page includes the following sections:

Collective Metadata I/O Overview

Calls for HDF5 metadata can result in many small reads and writes. On metadata reads, collective metadata I/O can improve performance by allowing the library to perform optimizations when reading the metadata, by having one rank read the data and broadcasting it to all other ranks.

Collective metadata I/O improves metadata write performance through the construction of an MPI derived datatype that is then written collectively in a single call.

Collective Metadata I/O User and Resource Documents

HDF5 Collective Metadata I/O User Document (This document is not yet available.)

Until an HDF5 Collective Metadata I/O User Document becomes available, users may find the following resources helpful:

RFC: Collective Metadata Reads (PDF)

RFC: Collective Metadata Writes (PDF)Taken together, these papers discuss the motivation, design, implementation, and API for HDF5’s Collective Metadata I/O feature.

HDF5 Library APIs

New Collective Metadata I/O Functions
H5P_SET_COLL_METADATA_WRITE  Establishes I/O mode property setting, collective or independent, for metadata writes
H5P_GET_COLL_METADATA_WRITERetrieves I/O mode property setting for metadata writes
H5P_SET_ALL_COLL_METADATA_OPS  Establishes I/O mode, collective or independent, for metadata read operations
H5P_GET_ALL_COLL_METADATA_OPSRetrieves I/O mode for metadata read operations

Additional API Reference

Functions with No Access Property List Parameter that May Generate Metadata Reads

This page lists the functions associated with HDF5 operations that can issue metadata reads from the metadata cache, but that currently provide no access property list parameter by which to set a collective requirement individually for those read operations. The need for this document is discussed in H5P_SET_ALL_COLL_METADATA_OPS.


--- Last Modified: April 06, 2018 | 01:55 PM