The lab is an interactive development environment for working notebooks, code, and data (especially HDF data).
The lab works with most browsers, but is most thoroughly tested with Chome and Firefox.
The lab is hosted on Amazon AWS, in the us-west-2 region (Oregon, USA)
The lab is based on the Open Source JupyterLab project, but contains several extensions to facilitate working with HDF data:
- Lab users are configured with access to a co-located instance of Kita Server. The Kita Server provides efficient access to large HDF Collections
- In addition to standard packages for data analytics: pandas, xarray, bokeh, dask, etc. the kernel has packages for working with HDF data, either HDF5 files on the local disk (h5py), or content on Kita Server (h5pyd).
- Command line tools are provided for working with either HDF data. Either local HDF5 files (h5dump, h5ls, h5repack, etc.) or HDF content on the server (hsinfo, hsls, hsget, etc.)
- Sample data is provided with HDF server in the folder: hdf5://shared/.
- Example notebooks are provided: /home/jovyan/examples
Everyone gets a 30-day free trial of Kita Lab. After the free trial, there is a small subscription fee.
Your account gives you:
- 8GB RAM for running your Jupyter Notebooks
- A dedicated Xeon CPU core
- 10GB local disk
- 200GB of storage on Kita Server
Your 10GB disk is stored as an AWS EBS Volume. Content in Kita Server is stored in a S3 Bucket. Each user has a server folder where they have permissions to write to. The name of the folder is listed in the output of the "hsinfo" command (see "command line tools" below). Typically it is
"/home/myname" where "myname" is your login name prior to the "@" symbol. In addition there is a server folder names "/shared" that contains data files accessible to everyone.
Yes, click on the "upload" button in the UI () and select the file you wish to upload.
Use Shift+Right Click (Ctrl+Shift on Mac) and select copy from the popup menu.
Yes, in the directory "/home/jovyan/examples" you will find sample notebooks and a tutorial. Running the different notebooks in the tutorial is a good way to become familiar with the capabilities of Kita Server. The GitHub repository for the examples is located at https://github.com/HDFGroup/hdflab_examples.
The easiest method would be to create a Github repository and have other users clone the repo.
Yes, follow these steps:
- Upload the file to the lab using the upload button
- Upon a terminal window (click '+' and then select terminal from the launcher)
- Run the command:
$ hsinfo, this will display information about the Kita Server, including your "home" folder, e.g. /home/myname/
- Run the command:
$ hsload myfile.h5 /home/myname/
- Run the command:
$ hsls /home/myname/myfile.h5to verify the contents get stored on Kita
You can upload directly from your desktop. Follow these steps:
- Install h5pyd on your desktop:
$ pip install h5pyd
- Download the file /home/jovyan/.hscfg to the home directory on your desktop
No, not directly, though you could share via git or other such means.
Yes, use the hsacl tool to make your folder or domain publically readable. If the domain is: /home/myname/mydata.h5, run the command:
$ hsacl /home/myname/mydata.h5 +r default .
To enable specific users to read the folder or domain, replace "default" in the above command with the user's email address that they use when logging in to Kita Lab.
You can also enable other users to have write permission to the domain, see the
hsacl help info.
You can get free help from the HDF community at https://forum.hdfgroup.org.
These are listed below.
This tool returns basic information on the server state and validates the username and password saved in the .hscfg file. For example:
$ hsinfo endpoint: http://hsdshdflab.hdfgroup.org server name: HDF Server for Kita Lab server state: READY username: test_user1 password: ******************************** home: /home/test_user1/ server version: 0.3 up: 55 min 27 sec h5pyd version: 0.3.3
If you add a domain argument to hsinfo, summary information about that domain will be displayed. Example:
$ hsinfo /shared/tall.h5
last modified: 2019-01-17 15:39:36
num objects: 10
num chunks: 4
This tool list either folders and domains at the given path.
Rather than directories and files, Kita uses the terms folders and domains. A folder is used to store collections of folders and domains. A domain is the equivalent to a regular HDF5 file stored in a POSIX filesystem.
During installation, a home folder is created for each user in the path:
/home/<username> . If peterpan is a username, running
$hsls /home/peterpan/ will display:
peterpan folder 2018-06-25 17:01:24 /home/peterpan/ 1 items
As additional content is created in this folder, those items will be displayed by
This tool will upload an HDF5 from the client environment to Kita.
$ hsload <filename> <folder> to upload a file to the given folder (where it will be saved using the same name as the local file), or
$hsload <filename> <domain> , to store to the given domain.
hsload myhdf5file.h5 /home/bob/ # creates
hsload myhdf5file.h5 /home/bob/mycloudfile.h5 # creates
Note: There's no equivalent to the concept of "current working directory" for paths on the server. Always use absolute path names.
Note: By default
hsload doesn't produce any output. Use the
-v flag or use the
hsls tool to verify the load.
Note: The requesting user most have permissions to write to the folder that will be used to store the domain (see
Note: Non-HDF5 files cannot be loaded. Use
h5ls to verify that a file to uploaded is indeed an HDF5 file.
Note: If you need a test file, you can download this one: https://s3.amazonaws.com/hdfgroup/data/hdf5test/tall.h5.
This tool retrieves a domain from the server an stores it as a ordinary HDF5 file on the client system.
$ hsget <domain> <filename>
hsget /home/bob/myhdf5file.h5 myhdf5file.h5 # stores myhdf5file.h5 in the
This tool creates a new folder or a domain (which will contain just the root group) on the server.
$ hstouch <folder> to create a folder
$ hstouch <domain> to create a domain
This tool removes a folder or domain from the server.
$ hsrm <folder> to delete a folder
$ hsrm <domain> to delete a domain
This tool can be used to move a domain from one folder location to another.
$ hsmv <src_domain> <des_domain> to move a domain
$ hsmv <src_domain> <des_folder> to move domain into folder
hsmv /home/bob/folder1/myhdf5file.h5 /home/bob/folder2/
This tool copies all content from the given domain to a new domain.
$ hscp <src_domain> <des_domain>
hscp /home/bob/myhdf5file.h5 /home/bob/myhdf5file_copy.h5
This tool compares the content of an HDF5 file with a domain and prints a message if the two objects are different.
$ hsdiff <HDF5 file> <domain>
$ hsdiff myhdf5file.h5 /home/bob/myhdf5file.h5
This tools can be used to display or modify ACLs (Access Control List) for folders or domains. The ACL for a given user controls which actions ("create", "read", "update", "delete", "read ACL", "update ACL") can be performed (either via the CLI tools, or programmatically).
hsacl --help for details on usage.
Also, this blog post: https://www.hdfgroup.org/2015/12/serve-protect-web-security-hdf5/ describes the design and use of ACLs as well as well as other web security related issues.