The HDF Group
- HDF Software Priority Support
- HDF Consulting
- Downloads
- Documentation
- Community Forum
- Help Desk
- Licenses
Got HDF5?
Curious to see what’s inside? Try this
Collective Metadata I/O
The purpose of this page is to briefly describe the new HDF5 Collective Metadata I/O feature and provide a gateway to available documentation. The page includes the following sections:
Collective Metadata I/O Overview
Calls for HDF5 metadata can result in many small reads and writes. On metadata reads, collective metadata I/O can improve performance by allowing the library to perform optimizations when reading the metadata, by having one rank read the data and broadcasting it to all other ranks.
Collective metadata I/O improves metadata write performance through the construction of an MPI derived datatype that is then written collectively in a single call.
Collective Metadata I/O User and Resource Documents
HDF5 Collective Metadata I/O User Document (This document is not yet available.)
Until an HDF5 Collective Metadata I/O User Document becomes available, users may find the following resources helpful:
RFC: Collective Metadata Reads (PDF)
RFC: Collective Metadata Writes (PDF)
Taken together, these papers discuss the motivation, design, implementation, and API for HDF5’s Collective Metadata I/O feature.
HDF5 Library APIs
New Collective Metadata I/O Functions
| H5Pset_coll_metadata_write | Establishes I/O mode property setting, collective or independent, for metadata writes |
| H5Pget_coll_metadata_write | Retrieves I/O mode property setting for metadata writes |
| H5Pset_all_coll_metadata_ops | Establishes I/O mode, collective or independent, for metadata read operations |
| H5Pget_all_coll_metadata_ops | Retrieves I/O mode for metadata read operations |
Additional API Reference
Functions with No Access Property List Parameter that May Generate Metadata Reads
Currently there are several operations in HDF5 that can issue metadata reads from the metadata cache, but that take no property list. It is therefore not possible set a collective requirement individually for those operations. The only solution with the HDF5 1.10.0 release is to set the collective requirement globally on H5Fopen or H5Fcreate for all metadata operations to be collective.
The following is a list of those functions in the HDF5 library. This list is integral to the discussion in the H5Pset_all_coll_metadata_ops entry:
In addition, most deprecated functions fall into this category.
The HDF Group may address the above limitation in a future major release, but no decision has been made at this time. Such a change might, for example, include adding new versions of some or all the above functions with an extra property list parameter to allow an individual setting for the collective calling requirement.