Skip to content

URLs

Jump to bottom
John Bogovic edited this page Apr 5, 2023 · 12 revisions

This describes a URL scheme to reference hierarchical / chunked storage containers (hdf5, n5, and zarr), their groups / datasets, and attributes.

Basics

The URL consists of three parts, the container, the group, and the attribute:

container?group#attribute

Where the container specifies a path to the root of the container (usually file system or cloud storage), the group specifies a path to the group, relative to the container root, and the attribute specifies the path to an attribute relative to the group.

Syntax

The URL syntax is based on w3c's syntax for URI's, outlined by the diagram below.

A URL is formatted as:

  • container?group#attribute

Which was chosen to align closely to general URIs, which are formatted as:

  • scheme://userinfo@host:port/path?query#fragment
    • where the userinfo@host:port is called the authority
    • Having an authority is supported, but all current use cases have no authority

Our URL is such that

  • the container is the scheme, authority, and path of the URI
  • the group is the query of the URI
  • the attribute is the fragment of the URI

For example:

URL:                              container                                group        attribute
          ____________________________|_____________________________   _______|______   ____|____
         /                                                          \ /              \ /         \
         s3://janelia-cosem-datasets/jrc_mus-kidney/jrc_mus-kidney.n5?/em/fibsem-uint8#multiscales
         \_/  \_____________________________________________________/ \______________/  \________/
          |                           |                                       |             |
URI:   scheme                        path                                   query       fragment

general URI flow diagram

URI flow diagram

Attribute paths

Basics

When using the N5-API, metadata attributes are generally refered to by a string valued name or "key". For example, the version of an n5 container is stored at the key "n5":

{ "n5" : "2.6.1" }

Note We will visualize attributes using JSON here, but the same principles will work for backends that do not store attributes with JSON (e.g. HDF5).

Attributes can be written to a container using N5Writer's setAttribute( String group, String key, Object value) method and read using N5Reader's getAttribute( String group, String key, Class class) method.

n5.setAttribute("group", "six", 6);
n5.getAttribute("group", "six", int.class); // returns 6
the result 
{ "six" : 6 }

where the examples here assume a variable n5 exists of type N5Writer. Keys may be long and contain whitespace:

n5.setAttribute("group", "The Answer to the Ultimate Question of Life, The Universe, and Everything", 42);
n5.getAttribute("group", "The Answer to the Ultimate Question of Life, The Universe, and Everything", int.class); // returns 42
n5.getAttribute("group", "The Answer to the Ultimate Question of Life, The Universe, and Everything", String.class); // returns "42"

Note that the requested output type need not be the same as the input type, it is only required that the requested type be We see above that int types may be interpreted as Strings, but the reverse is not possible, in general.

n5.setAttribute("group", "name", "Marie Daly");
n5.getAttribute("group", "name", String.class); // returns "Marie Daly"
n5.getAttribute("group", "name", int.class);    // returns null

n5.setAttribute("group", "year", "1921");       // write the year as a string
n5.getAttribute("group", "name", String.class); // returns "1921"
n5.getAttribute("group", "name", int.class);    // returns 1921

Arrays

The value of a given attribute can be a more complex type, such an array.

n5.setAttribute(group, "array", new double[]{ 5, 6, 7, 8 });
n5.getAttribute(group, "array", double[].class))); // returns [5.0, 6.0, 7.0, 8.0]

Individual elements of the array can be retrieved by adding [i] after the key, where i is an integer (zero-based indexing). N5 will return null for indexes outside the bounds of the array, including for negative values.

n5.getAttribute(group, "array[0]", double.class);  // returns 5.0
n5.getAttribute(group, "array[2]", double.class);  // returns 7.0
n5.getAttribute(group, "array[9]", double.class);  // returns null
n5.getAttribute(group, "array[-1]", double.class); // returns null

This notation may be used to set array values as well. Arrays will grow in size if they are too small to fit the requested (positive) index. Numeric arrays will be filled with zero. Non-numeric arrays will be filled with null. Setting the value at a negative array index does nothing.

n5.setAttribute(group, "array[1]", 0.6);    // array is now [ 5.0, 0.6, 7.0, 8.0 ]
n5.setAttribute(group, "array[6]", 99.99);  // array is now [ 5.0, 0.6, 7.0, 8.0, 0.0, 0.0, 99.99 ]
n5.setAttribute(group, "array[-5]", -5);    // array is now [ 5.0, 0.6, 7.0, 8.0, 0.0, 0.0, 99.99 ]

N5's setAttribute will always do what is requested when possible, even if it will overwrite data. If safety is necessary, developers should manually check if an attribute key is present. Use of the type JsonElement type is the most safe, because a non-null JsonElement will be returned if data of any type is present at the requested key.

n5.setAttribute(group, "array", new String[]{"destroy"});   // array is now [ "destroy" ]

if( n5.getAttribute( group, "array", JsonElement.class ) == null )
    n5.setAttribute(group, "array", new String[]{});   // array is still [ "destroy" ]

if( n5.getAttribute( group, "array", double[].class ) == null )
    n5.setAttribute(group, "array", new String[]{});   // array is now []

Objects

Objects are structures with "fields" that can be referenced by their String name. One way to set objects is by using a Map.

Map a = Collections.singletonMap("a", "A");
Map b = Collections.singletonMap("b", "B");
Map c = Collections.singletonMap("c", "C");

n5.setAttribute(group, "obj", a ); 
n5.getAttribute(group, "obj", Map.class);   // returns {"a": "A"}

The value for an object's field can be any type, even another object. Individual fields for an object can be accessed by appending /<field-name> to the attribute name. For example:

n5.setAttribute(group, "obj/a", b);
n5.getAttribute(group, "obj", Map.class);   // returns {"a": {"b": "B"}}
n5.getAttribute(group, "obj/a", Map.class); // returns {"b": "B"}

n5.setAttribute(group, "obj/a", b);
n5.getAttribute(group, "obj", Map.class);   // returns {"a": {"b": "B"}}
n5.getAttribute(group, "obj/a", Map.class); // returns {"b": "B"}

n5.setAttribute(group, "obj/a/b", c);
n5.getAttribute(group, "obj", Map.class);     // returns {"a": {"b": {"c": "C"}}}
n5.getAttribute(group, "obj/a", Map.class);   // returns {"b": {"c": "C"}}
n5.getAttribute(group, "obj/a/b", Map.class); // returns {"c": "C"}

Notice that it is possible to repeatedly access subfields of nested objects. In fact, the set of all attributes in an N5 group is usually itself an object! We call it the "root object" and access it with the the path "/"

n5.getAttribute(group, "/", Map.class);     // returns {"obj": {"a": {"b": {"c": "C"}}}}

Besides Maps, one can set at attribute's value using general structured data, as an object. For example consider the Pet type with String name and int age.

Definition of `Pet` 
class Pet {
    String name;
    int age;

    public Pet(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public String toString() {
        return String.format("pet %s is %d", name, age);
    }
}
n5.setAttribute(group, "pet", new Pet("Pluto", 93));
n5.getAttribute(group, "pet", Pet.class);   // returns Pet("Pluto", 93)
n5.getAttribute(group, "pet", Map.class);   // {"name": "Pluto", "age": 93}

One can add fields to an attribute by setting the desired field.

n5.setAttribute(group, "pet/likes", new String[]{"Micky"});
n5.getAttribute(group, "pet", Map.class);   // {"name": "Pluto", "age": 93, "likes": ["Micky"]}
Clone this wiki locally