Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Tuning parallel HDF5 for a specific application on a specific system requires playing with a lot of tunable parameters many of which are specific to certain platforms. Not all hints are applicable to all platforms, and some hints may be ignored even if they can be applied. The best practice here is to look at each system's webpage on how to tune I/O parameters. For example, Hopper, a Cray XE6 supercomputer at NERSC, has a webpage specifically on how to tune parallel I/O parameters for specific file systems:

...

http://www.nersc.gov/users/computational-systems/hopper/file-storage-and-i-o/

...

Here are some general parameters that users should consider tuning when they see slow I/O performance from HDF5:

...

  1. Chunk size and dimensions: If the application is using chunked dataset storage, performance usually varies depending on the chunk size and how the chunks are aligned with block boundaries of the underlying parallel filesystem. Extra care must be taken on how the application is accessing the data to be able to set the chunk dimensions.

  2. Metadata cache: it is usually a good idea to increase the metadata cache size if possible to avoid small writes to the file system. See:

    http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetMdcConfig

     H5P_SET_MDC_CONFIG

  3. Alignment properties: For MPI IO and other parallel systems, choose an alignment which is a multiple of the disk block size. See:

    http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetAlignment

     H5P_SET_ALIGNMENT

MPI-IO parameters:

There are several MPI-I/O parameters to tune. Usually it is done by setting info keys in the info object passed to HDF5. Some implementations might allow other ways to pass those hints to the MPI library. The MPI standard reserves some key values. An implementation is not required to interpret these key values, but if it does interpret the key value, it must provide the functionality described. The best thing to do again here is to consult with the specific MPI implementation and system used documentation to see what parameters are available to tune. For example, ROMIO in MPICH provides a user guide with a section describing the hints that are available to tune:

...

http://www.mcs.anl.gov/research/projects/romio/doc/users-guide.pdf

...

Here are some general parameters that are usually tunable:

...

For Blue Gene /P and /Q, one can set the environment variable BGLOCKLESSMPIO_F_TYPE to 0x47504653 (the GPFS file system magic number). ROMIO will then pretend GPFS is like PVFS and not issue any fcntl() lock commands.

Some IBM specific hints:

...

http://www-01.ibm.com/support/knowledgecenter/SSFK3V_1.3.0/com.ibm.cluster.pe.v1r3.pe500.doc/am107_ifopen.htm?lang=en

...

Some Cray specific hints:

...

https://fs.hlrs.de/projects/craydoc/docs/man/xe_mptm/51/cat3/intro_mpi.3.html

...