URLs

This describes a URL scheme to reference hierarchical / chunked storage containers (hdf5, n5, and zarr), their groups / datasets, and attributes.

Basics

The URL consists of three parts, the container, the group, and the attribute:

container?group#attribute

Where the container specifies a path to the root of the container (usually file system or cloud storage), the group specifies a path to the group, relative to the container root, and the attribute specifies the path to an attribute relative to the group.

Syntax

The URL syntax is based on w3c's syntax for URI's, outlined by the diagram below.

A URL is formatted as:

• container?group#attribute

Which was chosen to align closely to general URIs, which are formatted as:

• scheme://userinfo@host:port/path?query#fragment
  • where the userinfo@host:port is called the authority
  • Having an authority is supported, but all current use cases have no authority

Our URL is such that

• the container is the scheme, authority, and path of the URI
• the group is the query of the URI
• the attribute is the fragment of the URI

For example:

URL:                              container                                group        attribute
          ____________________________|_____________________________   _______|______   ____|____
         /                                                          \ /              \ /         \
         s3://janelia-cosem-datasets/jrc_mus-kidney/jrc_mus-kidney.n5?/em/fibsem-uint8#multiscales
         \_/  \_____________________________________________________/ \______________/  \________/
          |                           |                                       |             |
URI:   scheme                        path                                   query       fragment

general URI flow diagram

Attribute paths

When using the N5-API, metadata attributes are generally refered to by a string valued name or "key". For example, the version of an n5 container is stored at the key "n5":

{ "n5" : "2.6.1" }

Note We will visualize attributes using JSON here, but the same principles will work for backends that do not store attributes with JSON (e.g. HDF5).

Attributes can be written to a container using N5Writers setAttribute method.

n5.setAttribute("group", "six", 6);

the result

{ "six" : 6 }

Where the examples here assume a variable n5 exists of type N5Writer.