Page tree

The purpose of this page is to list and briefly describe the documentation available to those who will use the Single-Writer/Multiple-Reader (SWMR) feature found in the HDF5 library. The following are the sections on this page:

SWMR User’s Guide

The HDF5 Single-Writer/Multiple-Reader User’s Guide describes HDF5’s SWMR feature and describes how to use the feature. The feature allows concurrent access of HDF5 files by writer and reader processes.

HDF5 Library APIs

H5F_START_SWMR_WRITEEnables SWMR writing mode for a file
H5DO_APPENDAppends data to a dataset along a specified dimension
H5P_GET_APPEND_FLUSHRetrieves the values of the append property that is set up in the dataset access property list
H5P_SET_APPEND_FLUSHSets two actions to perform when the size of a dataset’s dimension being appended reaches a specified boundary
H5P_GET_OBJECT_FLUSH_CBRetrieves the object flush property values from the file access property list
H5P_SET_OBJECT_FLUSH_CBSets a callback function to invoke when an object flush occurs in the file
H5O_DISABLE_MDC_FLUSHESPrevents metadata entries for an HDF5 object from being flushed from the metadata cache to storage
H5O_ENABLE_MDC_FLUSHESReturns the cache entries associated with an HDF5 object to the default metadata flush and eviction algorithm
H5O_ARE_MDC_FLUSHES_DISABLEDDetermines if an HDF5 object (dataset, group, committed datatype) has had flushes of metadata entries disabled

 

Tools

The tools listed below will be incorporated into the HDF5 Reference Manual when they are fully released.

h5watch

Allows users to output new records appended to a dataset as it grows

In other words, this tool can be used to watch data that is added to a dataset. The functionality is similar to the Unix user command tail with the follow option, which outputs appended data as the file grows.

h5format_convertAllows users to convert the indexing type of a chunked dataset made with a 1.10.x version of the HDF5 library to the 1.8.x version 1 B-tree indexing type
h5clearAllows users to clear the file consistency flags field in the superblock of a file

 

SWMR Design Documents

Metadata Cache Issues under SWMR and their SolutionsThe HDF5 library contains a cache that provides more efficient access to HDF5 file metadata. Although helpful in most instances, this cache can cause problems when the single-writer/multiple-reader (SWMR) data access pattern is used to access an HDF5 file. Additionally, there are even deeper I/O issues that can affect HDF5 file metadata operations. This document describes the cache-oriented problems that the HDF5 library must overcome in order to implement the SWMR feature as well as our solutions.
Improvements for SWMR File Access and Dataset AppendThis document describes changes to the HDF5 Library that improve the SWMR (Single-Writer/Multiple-Reader) file access model and provide better support for dataset append operations.
HDF5 Single-Writer/Multiple-Reader Feature Design and SemanticsThis document describes the design and semantics of HDF5’s single-writer/multiple-reader (SWMR) feature. This feature allows multiple reader processes to inspect a file that is concurrently being written to by a single writer process without requiring any inter-process communication.
File Locking under SWMR – Semantics, Programming Model, and ImplementationThis document describes the design and semantics of how file locking is used to help prevent inappropriate concurrent file access. This locking is only intended to help the user conform to SWMR semantics and is not a fundamental part of SWMR synchronization.

--- Last Modified: April 06, 2018 | 02:53 PM