Page tree

The license could not be verified: License Certificate has expired!

This page is under construction and subject to change.

 HDF5 1.12 introduces several new features in the HDF5 library.

Dynamically Loadable Virtual File Drivers – VFDs  (RFC)

Dynamically loadable Virtual File Drivers (VFDs) enable virtual file drivers to be included with HDF5, without requiring a rebuild of the HDF5 library.  This greatly simplifies the release of VFDs, especially proprietary VFDs where the source code should not be publicly visible. This feature is critical for distribution of the S3 and HDFS drivers, as well as drivers created by the community.

H5P_GET_FAPL_HDFSGets the information of the given Read-Only HDFS virtual file driver
H5P_GET_FAPL_ROS3Gets the information of the given Read-Only S3 virtual file driver


H5Fdelete and Changes to the Virtual File Layer (VFL) (RFC)

With the new virtual object layer (VOL), HDF5 "files" can map to arbitrary storage schemes such as object stores and relational database tables. The data created by these implementations may be inconvenient for a user to remove without a detailed knowledge of the storage scheme. The H5Fdelete() API was introduced to give VOL connector authors the ability to add connector-specific delete code to their connectors so that users can remove these "files" without detailed knowledge of the storage scheme.

H5F_DELETEDeletes an HDF5 file

Since HDF5 storage can differ among the virtual file drivers, changes had to be made so that each Virtual File Driver (VFD) could have its own driver-specific cleanup code.

H5Sencode Changes

H5Sencode performance was improved by encoding the start/stride/count/block in most cases when the selection is regular

The previous format for H5Sencode enumerated every block in the selection. For example, if the selection was a checkerboard, the serialized selection described the location of every block independently, instead of specifying the pattern of the blocks. This created performance and file space issues if the selection used a large number of blocks.

VDS and Hyperslab Performance Improvements

VDS and Hyperslab performance have been improved dramatically.

References API (RFC)

While the previous H5R API supported both object and region references, it was lacking in functionality and flexibility: there was no support for attribute or external references, and it was not suitable for use outside of native HDF5 files, a requirement of the virtual object layer (VOL).  In order to support these features several functions were introduced:

  • Create (H5R_CREATE*) functions were added for each reference type: object, dataset region and attribute.
  • A function was added to release a reference (H5R_DESTROY). This is required because a region reference no longer modifies the original file.
  • Functions were added to query references (H5R_GET*).
  • Other functions were added to simplify or clarify the API.

The H5R functions that were introduced are described below.

H5R_COPYCopies an existing reference
H5R_CREATE_ATTRCreates an attribute reference
H5R_CREATE_OBJECTCreates an object reference
H5R_CREATE_REGIONCreates a region reference
H5R_DECODEDecodes a reference from a buffer
H5R_DESTROYCloses a reference
H5R_ENCODEEncodes a reference into a buffer
H5R_EQUALDetermines whether two references are equal
H5R_GET_ATTR_NAMERetrieves the attribute name for a referenced object
H5R_GET_FILE_NAMERetrieves the file name for a referenced object
H5R_GET_OBJ_NAMERetrieves the object name for a referenced object
H5R_GET_OBJ_TYPE3Retrieves the type of object that an object reference points to
H5R_GET_TYPERetrieves the type of reference
H5R_OPEN_ATTROpens the referenced HDF5 attribute
H5R_OPEN_OBJECTOpens the referenced HDF5 object
H5R_OPEN_REGIONSets up a dataspace and selection as specified by a region reference


Virtual Object Layer (RFC)

The Virtual Object Layer (VOL) is an abstraction layer within the HDF5 library that enables different methods for accessing data and objects that conform to the HDF5 data model. The VOL intercepts all HDF5 API calls that potentially modify data on disk and forwards those calls to a plugin "object driver". The data on disk can be a different format than the HDF5 format.

The plugins can actually store the objects in variety of ways. A plugin could, for example, have objects be distributed remotely over different platforms, provide a raw mapping of the model to the file system, or even store the data in other file formats (like native netCDF or HDF4 format). The user still gets the same data model where access is done to a single HDF5 “container”; however the plugin object driver translates from what the user sees to how the data is actually stored. Having this abstraction layer maintains the object model of HDF5 and allows better usage of new object storage file systems that are targeted for Exascale systems.

The following functions were introduced to support this feature:

H5VL_CLOSECloses a VOL connector identifier
H5VL_GET_CONNECTOR_IDRetrieves the identifier for a registered VOL connector

Retrieves the connector name for the VOL associated with the object or file identifier

H5VL_IS_CONNECTOR_REGISTEREDTests whether a VOL class has been registered or not
H5VL_REGISTER_CONNECTORRegisters a new VOL connector
H5VL_REGISTER_CONNECTOR_BY_NAMERegisters a new VOL connector by name
H5VL_REGISTER_CONNECTOR_BY_VALUERegisters a new VOL connector by connector value
H5VL_UNREGISTER_CONNECTORRemoves a VOL connector identifier from the library

--- Last Modified: August 19, 2019 | 11:12 AM