+
Figure 1 shows a conceptual representation of dataset D with uncompressed data. The desired region and the chunks that contain it are shown in green and yellow, respectively.
-
+
Figure 1: Conceptual representation of dataset D
In Figure 1, the chunks and region of interest are represented by the yellow and green areas of the diagram. Figure 2 shows an enlarged view of the region and chunks, with labels added. The dashed lines delineate individual elements in the dataset. Elements in the region of interest have been labeled so they can be traced through the pipeline process.
-
+
Figure 2: Conceptual representation of region and chunks in dataset D
Figure 3 shows a more accurate depiction of the chunks and elements in the region as they could be laid out on disk. Note that data in each chunk is stored contiguously on disk, and that the chunks have unequal sizes due to compression of the data.
-
+
+
Figure 3: Conceptual representation of chunks and region elements on disk
@@ -114,7 +116,7 @@ Therefore:
In this example, the application includes a value transformation in the data transfer property list for the H5Dread call. The transformation specifies that the integer value “2” should be added to each element in the region of interest before it is copied to the application’s memory buffer.
-
+
#### Example A: H5Dread setup and call
***
@@ -155,7 +157,7 @@ Total cells in array equals: 2 * 16 = 32
Size of array is: 32 * 8 bytes = 256 bytes
No value transformation is applied in this example.
-
+
#### Example B: H5Dread setup and call
***
@@ -196,7 +198,7 @@ If one or more filters were applied when the dataset was written, as they were i
+
**Filters in HDF5**
The HDF5 library allows applications to specify filters to apply when a dataset is written to an HDF5 file via the H5Pset\_filter call. These filters perform operations such as compression, encryption, a checksum computation. Each filter operation applied when a dataset is written must be “reversed” when the dataset is read. For instance, if a dataset was compressed when written, it must be uncompressed when read. @@ -208,7 +210,7 @@ In dataset D, two filters were applied when the data was written. The DEFLATE co
+
Checksum error detection
Checksum error detection is enabled by default. H5Pset\_edc\_check can be used to disable checksum error detection
@@ -218,7 +220,7 @@ Again using memory in the application’s memory space (heap) that is managed by
+
HDF5 chunk cache
Every HDF5 dataset with the chunked storage format has an HDF5 chunk cache associated with it. The HDF5 chunk cache for the dataset is allocated from the application’s memory and managed by the HDF5 library. As its name suggests, the cache remains allocated across multiple calls, and is used to provide performance optimizations in accessing data.
@@ -229,7 +231,7 @@ If there is sufficient space in dataset D’s chunk cache, the data for the curr
In the given examples, the uncompressed data for the current chunk will be stored. A chunk cache of at least 64 bytes is needed to hold a single chunk of uncompressed data for dataset D.
-
+
Temporary buffer size
The default size of the temporary buffer used for datatype conversion and/or value transformation is 1 MB in Release 1.8.2. The size can be controlled with H5Pset\_buffer.
@@ -258,7 +260,7 @@ Unprocessed elements in the region of interest are gathered from the chunk cache
Considering chunk A in the examples, eight of the nine elements that are of interest will fit into the temporary buffer. Figure 5 depicts the temporary buffer at this stage of the pipeline.
-
+
Figure 5: Temporary buffer with first eight elements in region
@@ -267,7 +269,7 @@ If the memory representation is not the same as the disk representation, datatyp
In the examples, the values will be converted from 32-bit little-endian integers into 64‐bit big‐endian integers. Figure 6 illustrates the contents of the temporary buffer after the datatype conversion.
-
+
Figure 6: Temporary buffer with first eight elements after datatype conversion
@@ -282,14 +284,14 @@ The HDF library scatters elements from the temporary buffer into the application
Figure 7 represents the contents of the application’s memory buffer for Example A after this step completes the first time. The elements in the application’s memory buffer have been converted into the memory datatype and have had the value transformation applied.
-
+
Figure 7: Application's memory buffer after first pass through Step 8 for Example A
Figure 8 represents the contents of the application’s memory buffer for Example B after Step 8 completes the first time. The elements in the application’s memory buffer have been converted into the memory datatype. No value transformation is applied in Example B.
-
+
Figure 8: Application's memory buffer after first pass through step 8 in Example B
@@ -316,18 +318,18 @@ With the requested data in the application’s memory buffer, and the memory use
Figure 9 shows the contents of the application’s memory buffer when H5Dread returns for Example A, and Figure 10 shows the results for Example B.
-
+
Figure 9: Application's memory buffer when H5Dread returns for Example A
-
+
Figure 10: Application's memory buffer when H5Dread returns for Example B
+
Figure 11: H5Dread activity diagram