Page tree

This page describes the new HDF5 Page Buffering option and provides links to available reference manual entries. The page includes the following sections:

Overview of the Page Buffering Feature

Summary

Page buffering adds a layer immediately above the Virtual File Driver I/O layer within the library, which acts as a page cache for metadata and raw data accesses that are smaller than the page size for file space allocation. For most applications the HDF5 library generates very small amounts of metadata and raw data accesses in HDF5 datasets, even though the total amount of metadata in an HDF5 file varies between applications. These small and random I/O accesses on parallel file systems result in poor performance for applications.

The page buffering feature, in conjunction with paged aggregation, can be used to reduce the number of small accesses in HDF5. Together these features give an application control over minimizing the granularity and alignment of HDF5 I/O requests. See File Space Management for information on paged aggregation.

Background Information

See the RFC on this feature for complete details.

HDF5 Library APIs

Following are the new APIs introduced for the Page Buffering feature:

 

H5P_GET_FILE_SPACE_STRATEGYRetrieves the File Space Strategy for a file creation property list. See File Space Management for more details
H5P_SET_FILE_SPACE_STRATEGYH5F_FSPACE_STRATEGY_PAGE sets the file space handling strategy to use Page Buffering. See File Space Management for more details
H5P_GET_FILE_SPACE_PAGE_SIZERetrieves the file space page size
H5P_SET_FILE_SPACE_PAGE_SIZESets the file space page size when file space allocation is H5F_FSPACE_STRATEGY_PAGE
H5P_GET_PAGE_BUFFER_SIZERetrieves the maximum size for the page buffer and the minimum percentage for metadata and raw data pages
H5P_SET_PAGE_BUFFER_SIZESets the maximum size for the page buffer and the minimum percentage for metadata and raw data pages
H5F_GET_PAGE_BUFFERING_STATSGathers page buffer statistics regarding page access when it is enabled
H5F_RESET_PAGE_BUFFERING_STATSResets the page buffer statistics

 

 

 

Tools

h5repackThe following options were added to h5repack:

-G FS_PAGESIZE,--fs_pagesize=FS_PAGESIZE enables the file space page size to be changed to FS_PAGESIZE.

-P FS_PERSIST,--fs_persist=FS_PERSIST sets the persisting free space to persist (1) or to not persist (0).

-S FS_STRATEGY, --fs_strategy=FS_STRATEGY sets the file space management strategy.

-T FS_THRESHOLD, --fs_threshold=FS_THRESHOLD sets the free-space section threshold.

--- Last Modified: April 06, 2018 | 02:36 PM