How do you build HDF5 on BlueGene/L?
Here are a few steps to get you started on building HDF5 on BlueGene/L:
These are the compilers that we have used to build HDF5 on this platform:
C: mpxlc C++: mpxlC F77: mpxlf F90: mpxlf90
First try building without C++/Fortran support, unless you need those. If you don't have these compilers available to you, you should find out from your sysadmin what the MPI compilers on the system are.
You should run the
./configure script through
yodconfigure like so:
Then, set the
RUNSERIAL environment variable. For example:
mpirun -np 1 -cwd /g/g23/xxxx/hdf5
Note that the cwd parameter should be set to whatever directory you're building HDF5 in. The
RUNSERIAL environment variable tells the configure script to execute all binaries (the configure script builds several dozen small test binaries to execute and see how them work in order to deduce the characteristics of the computing environment) on the batch nodes rather than on the login nodes. This is necessary because you ***MUST*** have configure execute its test binaries on the batch nodes since these have different characteristics than the login nodes. A binary compiled to work on the login nodes will NOT work on the batch nodes.
Also note that this is slightly different than running the entire configure script as a batch job. Although that might work (and would certainly achieve the same effect), on most systems the batch nodes have a very minimal computing environment and sometimes do not even have a shell installed (let alone compilers) so the configure script wouldn't run on them. If you're feeling adventurous, you can try just submitting the entire configure script as a batch job on a single node. Don't be surprised if that fails, though.
Lastly, use the following line to configure:
./configure --enable-parallel --enable-fortran --disable-cxx --disable-stream-vfd
You may want to disable fortran, if you don't need it. The stream vfd doesn't compile on BlueGene/L, so you'll want to turn that off as well. As you've probably guessed, forcing the configure script to run each individual binary as a separate process on the batch node will cause the configure script to take a VERY long time to execute (~45 minutes for me).
As one final note, if you're using fortran, you may need to set the following environment variable:
yodconfigure does is to simply prefix all serial executions with the contents of the
RUNSERIAL variable. It just rewrites part of the configure script, so you still have to run configure after using yodconfigure. So, for example, instead of executing:
after being run through
yodconfigure, the configure script will execute:
And if runserial is empty, it's equivalent to the original configure, BUT if
RUNSERIAL is set to something like
mpirun -np 1 the command would become:
mpirun -np 1 ./a.out
which ensures that the binary is run on a compute node rather than the login node.
If you find this isn't working, one quick way to test is to write a hello world MPI program and try to run it on a compute node like this:
mpirun -np 1 -cwd /g/g23/xxxx/test.out
If that doesn't work, either the command for launching compute node jobs is different (which you'd have to look up for your system) or the system does not allow arbitrary launching of compute node jobs (ie, everything has to be scheduled) in which case none of this will work for you and your best shot would be to try to run the entire configure script as a batch job. We have never tried that, though, and don't know if it would work, but the
yodconfigure technique requires the ability to launch jobs on compute nodes with a single command that returns when the binary has finished executing (and sends back the test binary its return value).
Cross-compiling systems are a huge pain to deal with.