URLs

This describes a URL scheme to reference hierarchical / chunked storage containers (hdf5, n5, and zarr), their groups / datasets, and attributes.

Basics

The URL consists of three parts, the container, the group, and the attribute:

container?group#attribute

Where the container specifies a path to the root of the container (usually file system or cloud storage), the group specifies a path to the group, relative to the container root, and the attribute specifies the path to an attribute relative to the group.

Syntax

The URL syntax is based on w3c's syntax for URI's, outlined by the diagram below.

A URL is formatted as:

• container?group#attribute

Which was chosen to align closely to general URIs, which are formatted as:

• scheme://userinfo@host:port/path?query#fragment
  • where the userinfo@host:port is called the authority
  • Having an authority is supported, but all current use cases have no authority

Our URL is such that

• the container is the scheme, authority, and path of the URI
• the group is the query of the URI
• the attribute is the fragment of the URI

For example:

URL:                              container                                group        attribute
          ____________________________|_____________________________   _______|______   ____|____
         /                                                          \ /              \ /         \
         s3://janelia-cosem-datasets/jrc_mus-kidney/jrc_mus-kidney.n5?/em/fibsem-uint8#multiscales
         \_/  \_____________________________________________________/ \______________/  \________/
          |                           |                                       |             |
URI:   scheme                        path                                   query       fragment

general URI flow diagram

Attribute paths

Code for the examples below can be found here.

Summary:

• The root attribute can be referenced with a single forward slash
  • "/"
• Fields by object can be referenced by their name
  • "atrrname"
  • "/attrname" (equivalent to the above)
• Reference sub-fields by name after delimiting with a forward slash /
  • "attrname/sub-attribute"
• Reference array elements with an index inside square brackets
  • "array[1]"
  • "array/[1]" (equivalent to the above)

Basics

When using the N5-API, metadata attributes are generally referred to by a string valued name or "key". For example, the version of an n5 container is stored at the key "n5":

{ "n5" : "2.6.1" }

Note We will visualize attributes using JSON here, but the same principles will work for backends that do not store attributes with JSON (e.g. HDF5).

Attributes can be written to a container using N5Writer's setAttribute( String group, String key, Object value) method and read using N5Reader's getAttribute( String group, String key, Class class) method.

n5.setAttribute("group", "six", 6);
n5.getAttribute("group", "six", int.class); // returns 6

the result

{ "six" : 6 }

where the examples here assume a variable n5 exists of type N5Writer. Keys may be long and contain whitespace:

n5.setAttribute("group", "The Answer to the Ultimate Question of Life", 42);
n5.getAttribute("group", "The Answer to the Ultimate Question of Life", int.class); // returns 42
n5.getAttribute("group", "The Answer to the Ultimate Question of Life", String.class); // returns "42"

Note that the requested output type need not be the same as the input type, it is only required that the requested type be We see above that int types may be interpreted as Strings, but the reverse is not possible, in general.

n5.setAttribute("group", "name", "Marie Daly");
n5.getAttribute("group", "name", String.class); // returns "Marie Daly"
n5.getAttribute("group", "name", int.class);    // returns null

n5.setAttribute("group", "year", "1921");       // write the year as a string
n5.getAttribute("group", "name", String.class); // returns "1921"
n5.getAttribute("group", "name", int.class);    // returns 1921

Setting the value of an existing attribute will overwrite the previous value.

n5.setAttribute("group", "animal", "aardvark");
n5.setAttribute("group", "animal", new String[]{"bat", "cat", "dog"}); // overwrites "animal"
n5.getAttribute("group", "name", String[].class); // ["bat", "cat", "dog"]

The entire json object may be referenced with the "root key" /. Setting the value of the root key will replace all attributes, and so should be done with care.

n5.gettAttribute("group", "/", JsonObject.class ); // 
n5.setAttribute("group", "/", new JsonObject()); // write the empty object
n5.gettAttribute("group", "/", JsonObject.class ); // {}

Arrays

The value of a given attribute can be a more complex type, such an array.

n5.setAttribute(group, "array", new double[]{ 5, 6, 7, 8 });
n5.getAttribute(group, "array", double[].class))); // returns [5.0, 6.0, 7.0, 8.0]

Individual elements of the array can be retrieved by adding [i] after the key, where i is an integer (zero-based indexing). N5 will return null for indexes outside the bounds of the array, including for negative values.

n5.getAttribute(group, "array[0]", double.class);  // returns 5.0
n5.getAttribute(group, "array[2]", double.class);  // returns 7.0
n5.getAttribute(group, "array[9]", double.class);  // returns null
n5.getAttribute(group, "array[-1]", double.class); // returns null

This notation may be used to set array values as well. Arrays will grow in size if they are too small to fit the requested (positive) index. Numeric arrays will be filled with zero. Non-numeric arrays will be filled with null. Setting the value at a negative array index does nothing.

n5.setAttribute(group, "array[1]", 0.6);    // array is now [ 5.0, 0.6, 7.0, 8.0 ]
n5.setAttribute(group, "array[6]", 99.99);  // array is now [ 5.0, 0.6, 7.0, 8.0, 0.0, 0.0, 99.99 ]
n5.setAttribute(group, "array[-5]", -5);    // array is now [ 5.0, 0.6, 7.0, 8.0, 0.0, 0.0, 99.99 ]

N5's setAttribute will always do what is requested when possible, even if it will overwrite data. If safety is necessary, developers should manually check if an attribute key is present. Use of the type JsonElement type is the most safe, because a non-null JsonElement will be returned if data of any type is present at the requested key.

n5.setAttribute(group, "array", new String[]{"destroy"});   // array is now [ "destroy" ]

if( n5.getAttribute( group, "array", JsonElement.class ) == null )
    n5.setAttribute(group, "array", new String[]{});   // array is still [ "destroy" ]

if( n5.getAttribute( group, "array", double[].class ) == null )
    n5.setAttribute(group, "array", new String[]{});   // array is now []

Objects

Objects are structures with "fields" that can be referenced by their String name. One way to set objects is by using a Map.

Map a = Collections.singletonMap("a", "A");
Map b = Collections.singletonMap("b", "B");
Map c = Collections.singletonMap("c", "C");

n5.setAttribute(group, "obj", a ); 
n5.getAttribute(group, "obj", Map.class);   // returns {"a": "A"}

The value for an object's field can be any type, even another object. Individual fields for an object can be accessed by appending /<field-name> to the attribute name. For example:

n5.setAttribute(group, "obj/a", b);
n5.getAttribute(group, "obj", Map.class);   // returns {"a": {"b": "B"}}
n5.getAttribute(group, "obj/a", Map.class); // returns {"b": "B"}

n5.setAttribute(group, "obj/a", b);
n5.getAttribute(group, "obj", Map.class);   // returns {"a": {"b": "B"}}
n5.getAttribute(group, "obj/a", Map.class); // returns {"b": "B"}

n5.setAttribute(group, "obj/a/b", c);
n5.getAttribute(group, "obj", Map.class);     // returns {"a": {"b": {"c": "C"}}}
n5.getAttribute(group, "obj/a", Map.class);   // returns {"b": {"c": "C"}}
n5.getAttribute(group, "obj/a/b", Map.class); // returns {"c": "C"}

Notice that it is possible to repeatedly access subfields of nested objects. In fact, the set of all attributes in an N5 group is usually itself an object! We call it the "root object" and access it with the the path "/"

n5.getAttribute(group, "/", Map.class);     // returns {"obj": {"a": {"b": {"c": "C"}}}}

Besides Maps, one can set at attribute's value using general structured data, as an object. For example consider the Pet type with String name and int age.

Definition of `Pet`

class Pet {
    String name;
    int age;

    public Pet(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public String toString() {
        return String.format("pet %s is %d", name, age);
    }
}

n5.setAttribute(group, "pet", new Pet("Pluto", 93));
n5.getAttribute(group, "pet", Pet.class);   // returns Pet("Pluto", 93)
n5.getAttribute(group, "pet", Map.class);   // {"name": "Pluto", "age": 93}

One can add fields to an attribute by setting the desired field.

n5.setAttribute(group, "pet/likes", new String[]{"Micky"});
n5.getAttribute(group, "pet", Map.class);   // {"name": "Pluto", "age": 93, "likes": ["Micky"]}

One does not need to manually create every level of nested objects, because are automatically created even if they do not exist. Arrays values are automatically filled with zeros if setting them to a numeric value, or null otherwise. This examples sets the value of an integer inside several nested arrays and objects.

n5.removeAttribute(group, "/");
// attributes.json contains:
// null

n5.setAttribute(group, "one/[2]/three/[4]", 5);
// attributes.json contains:
// {"one":[null,null,{"three":[0,0,0,0,5]}]}

Removing attributes and nulls

The removeAttribute methods can also be used to remove attributes. The first variant takes the group and attribute key as arguments, and returns nothing. The second variant also takes a Class<T> argument and will return an object of type T if possible. If the value of the attribute cannot be parsed into the requested type, the attribute will not be removed, even if the key exists.

n5.setAttribute(group, "cow", "moo");
n5.setAttribute(group, "dog", "woof");
n5.setAttribute(group, "sheep", "baa");
// attributes.json contains:
// {"sheep":"baa","cow":"moo","dog":"woof"}

n5.removeAttribute(group, "cow"); // void method
// attributes.json contains:
// {"sheep":"baa","dog":"woof"}

String theDogSays = n5.removeAttribute(group, "dog", String.class); // return "woof"
// attributes.json contains:
// {"sheep":"baa"}

n5.removeAttribute(group, "sheep", int.class); // returns null because the value of "sheep" is not an int
// attributes.json contains:
// {"sheep":"baa"}

String theSheepSays = n5.removeAttribute(group, "sheep", String.class)); // return "baa" 
// attributes.json contains:
// {}

By default, setting the value of an attribute to null will remove that attribute (i.e. the attribute's key will be removed).

n5.setAttribute(group, "attr", "value");
// attributes.json contains:
// {"attr":"value"}

n5.setAttribute(group, "attr", null);
// attributes.json contains: 
// {}

n5.setAttribute(group, "foo", 12);
// attributes.json contains:
// {"foo":12}

n5.removeAttribute(group, "foo", "bar");
// attributes.json contains:
// {}

In cases where it is useful to write the value null into the attributes.json file, you must create an N5Writer using a GsonBuilder with serializeNulls enabled.

N5FSWriter n5 = new N5FSWriter( rootPath, new GsonBuilder().serializeNulls() );

n5.setAttribute(group, "attr", "value");
// attributes.json contains:
// {"attr":"value"}

n5.setAttribute(group, "attr", null);
// attributes.json contains: 
// {"attr":null}


n5.setAttribute(group, "foo", 12);
// attributes.json contains:
// {"attr":null,"foo":12}

n5.removeAttribute(group, "foo", "bar");
// attributes.json contains:
// {"attr":null}

Details

Special characters

⚠️ Warning We strongly recommend against using forward and back slashes as key names.

While we recommend against it, is it possible to use forward slash (/) or backslash \ as field names for attributes. Since / is reserved to refer to the root attribute, it must be escaped with a backslash to refer to the literal string "/".

Note In the java example below, backslash must itself be escaped, so the string "\\" refers to the single backslash character \.

n5.setAttribute(group, "\\/", "fwdSlash");
n5.getAttribute(group, "\\/", String.class );   // returns "fwdSlash"

n5.setAttribute(group, "\\\\", "bckSlash");
n5.getAttribute(group, "\\\\", String.class );   // returns "bckSlash"