Merge pull request #246 from HDFGroup/adding_topics

Add CollectiveMetadataIO, FileSpaceManagement, FineTuningMetadataCache

documentation/hdf5-docs/advanced_topics/CollectiveMetadataIO.md

@@ -0,0 +1,101 @@
+The purpose of this page is to briefly describe the new HDF5 Collective Metadata I/O feature and provide a gateway to available documentation. The page includes the following sections:
+
+* [Collective Metadata I/O Overview 
+
+* [Collective Metadata I/O User and Resource Documents 
+
+* [HDF5 Library APIs
+
+## Collective Metadata I/O Overview
+
+Calls for HDF5 metadata can result in many small reads and writes. On metadata reads, collective metadata I/O can improve performance by allowing the library to perform optimizations when reading the metadata, by having one rank read the data and broadcasting it to all other ranks.
+
+Collective metadata I/O improves metadata write performance through the construction of an MPI derived datatype that is then written collectively in a single call.
+
+## Collective Metadata I/O User and Resource Documents
+
+HDF5 Collective Metadata I/O User Document (This document is not yet available.)
+
+Until an *HDF5 Collective Metadata I/O User Document* becomes available, users may find the following resources helpful:
+
+RFC: Collective Metadata Reads ([PDF](https://docs.hdfgroup.org/hdf5/rfc/RFC-CollectiveMetadataReads.pdf))
+
+RFC: Collective Metadata Writes ([PDF](https://docs.hdfgroup.org/hdf5/rfc/RFC-CollectiveMetadataWrites.pdf))
+
+Taken together, these papers discuss the motivation, design, implementation, and API for HDF5’s Collective Metadata I/O feature.
+
+## HDF5 Library APIs
+
+### New Collective Metadata I/O Functions
+
+|                   |                                                              |
+| ----------------- | ------------------------------------------------------------ |
+| [H5Pset_coll_metadata_write](https://docs.hdfgroup.org/hdf5/v1_14/group___f_a_p_l.html#ga6380f9929cf42c8203813f7e72dde35c) | Establishes I/O mode property setting, collective or independent, for metadata writes |
+| [H5Pget_coll_metadata_write](https://docs.hdfgroup.org/hdf5/v1_14/group___f_a_p_l.html#gac83ab4e788a5b6e0d578f40ca67d8d00) | Retrieves I/O mode property setting for metadata writes | 
+| [H5Pset_all_coll_metadata_ops](https://docs.hdfgroup.org/hdf5/v1_14/group___g_a_p_l.html#ga5bc0b1fc7e1e3961bf0b441e722949eb) | Establishes I/O mode, collective or independent, for metadata read operations | 
+| [H5Pget_all_coll_metadata_ops](https://docs.hdfgroup.org/hdf5/v1_14/group___g_a_p_l.html#ga8792cbe7eeace1382f588ed12a92092a) | Retrieves I/O mode for metadata read operations |
+
+### Additional API Reference
+
+#### Functions with No Access Property List Parameter that May Generate Metadata Reads
+
+Currently there are several operations in HDF5 that can issue metadata reads from the metadata cache, but that take no property list. It is therefore not possible set a collective requirement individually for those operations. The only solution with the HDF5 1.10.0 release is to set the collective requirement globally on H5Fopen or H5Fcreate for all metadata operations to be collective.
+
+The following is a list of those functions in the HDF5 library. This list is integral to the discussion in the H5Pset_all_coll_metadata_ops entry:
+
+[H5Awrite](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_a.html#gab70871e205d57450c83efd9912be2b5c)
+[H5Aread](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_a.html#gaacb27a997f7c98e8a833d0fd63b58f1c)
+[H5Arename](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_a.html#ga490dcd6db246c1fda7295badfce28203)
+[H5Aiterate2](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_a.html#ga9315a22b60468b6e996559b1b8a77251)
+[H5Adelete](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_a.html#gada9fa3d6db52329f1fd55662de6ff6ba)
+[H5Aexists](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_a.html#ga293b5be270d90cd5e47f782ca9aec80b)
+
+[H5Dget_space_status](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_d.html#ga7639ef5c12cb906c71670ce73b856a4c)
+[H5Dget_storage_size](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_d.html#gafb249479a493e80891f0c7f5d8a91b00)
+[H5Dset_extent](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_d.html#gad31e1e0129f4520c531ce524de2a056f)
+[H5Ddebug](url)
+[H5Dclose](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_d.html#gae47c3f38db49db127faf221624c30609)
+[H5Dget_get_create_plist](url)
+[H5Dget_space](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_d.html#gad42a46be153d895d8c28a11ebf5a0d0a) (when dataset is a virtual dataset)
+
+[H5Gget_create_plist](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_g.html#ga0b959a53cbffa48f5d68ce33b43b7ed8)
+[H5Gget_info](https://docs.hdfgroup.org/hdf5/v1_14/group___j_h5_g.html#gaece6d1057c42630d336dad0f3915b337)
+[H5Gclose](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_g.html#ga8dbe20b390d2504f0bd3589ed8f4e221)
+
+[H5Literate](https://docs.hdfgroup.org/hdf5/v1_14/group___j_h5_l.html#ga4e9e84159546db7f17d3d0c6ee709371)
+[H5Lvisit](https://docs.hdfgroup.org/hdf5/v1_14/group___t_r_a_v.html#gac0558936502924d9e898d8b6e041ed69)
+
+[H5Rcreate](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_r.html#ga0ac6997b5de26b11d91a95de2869950d)
+[H5Rdereference2](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_r.html#ga9b09586f7b6ec708434dd8f95f58a9b7) (when reference is an object reference)
+[H5Rget_region](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_r.html#ga1702d609e85b9edd3d1e526a0276484f)
+[H5Rget_obj_type2](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_r.html#ga766e39a76bcdd68dc514425353eff807)
+[H5Rget_name](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_r.html#ga4c112c388f697324270fd085100dccaa)
+
+[H5Ocopy](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_o.html#gaa94449be6f67f499be5ddd3fc44f4225)
+[H5Oopen_by_addr](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_o.html#ga137f3823adab4daaaf8fe87b40453fa2)
+[H5Oincr_refcount](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_o.html#ga2086bad6c3cd2a711c306a48c093ff55)
+[H5Odecr_refcount](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_o.html#ga60c20da5e244c28a653d4fa23d316b44)
+[H5Oget_info](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_o.html#gaf4f302a33faba9e1c2b5f64c62ca4ed5)
+[H5Oset_comment](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_o.html#ga8b5cf8e916204e29616516046121f631)
+[H5Ovisit](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_o.html#ga5ce86255fcc34ceaf84a62551cd24233)
+
+[H5Fis_hdf5](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#ga6055c2ea3438bd4aaf221eba66843225)
+[H5Fflush](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#gae686870f0a276c4d06bbc667b2c24124)
+[H5Fclose](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#gac55cd91d80822e4f8c2a7f04ea71b124)
+[H5Fget_file_image](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#gadc53f4e76b1199cb5d2a8cb7fbb114ad)
+[H5Freopen](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#ga3f213eb05c5419d63ba168c30036e47b)
+[H5Fget_freespace](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#ga3ef2673183567543346668a8f1eca2e9)
+[H5Fget_info2](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#gaced8c09c1559636a9c3f33dff3f4520e)
+[H5Fget_free_sections](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#gab9cbf1a45f9dcda34b43f985b7848434)
+[H5Fmount](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#ga7c4865fd36ee25d839725252150bb53b)
+[H5Funmount](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#gae8f807d3f04a33f132ffb6c5295e897f)
+
+[H5Iget_name](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_i.html#ga9c84a8dc29566b82b6d1ff7a6e6828f1)
+
+[H5Tget_create_plist](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_t.html#ga6802c22c6e90216aa839a4a83909a54c)
+[H5Tclose](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_t.html#gafcba4db244f6a4d71e99c6e72b8678f0)
+
+[H5Zunregister](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_z.html#ga6b8bcdde70c9256c50c7c62ba66380f8)
+In addition, most deprecated functions fall into this category.
+
+The HDF Group may address the above limitation in a future major release, but no decision has been made at this time. Such a change might, for example, include adding new versions of some or all the above functions with an extra property list parameter to allow an individual setting for the collective calling requirement.

documentation/hdf5-docs/advanced_topics/DSChunkingIssues.md

@@ -1,7 +1,7 @@
 ---
-title: Chunking in HDF5
+title: Dataset Chunking Issues
 redirect_from:
-  - https://support.hdfgroup.org/HDF5/doc/H5.user/Chunking.html
+  - 
 ---
 
 # Dataset Chunking Issues

documentation/hdf5-docs/advanced_topics/FileSpaceManagement.md

@@ -0,0 +1,85 @@
+---
+title: File Space Management
+redirect_from:
+  - display/HDF5/File%2BSpace%2BManagement
+---
+
+This page briefly describes the documentation available to those who use the file space management feature found in the HDF5 library.
+
+The HDF5 library's file space management activities encompass both the allocation of file space and the management of free space. When an HDF5 object (group, dataset, etc.) is created and written, file space is allocated for storing its metadata and raw data. When an object is removed, the space associated with the object becomes free space.
+
+The following are the sections on this page:
+
+File Space Management Strategies
+File Space Management User’s Guide  (not yet available)
+HDF5 Library APIs
+Tools
+Differences between HDF5-1.10 and HDF5-1.8
+How to Remove the Free Space in an Existing File
+
+File Space Management Strategies
+The HDF5 library uses three mechanisms to manage space in an HDF5 file. They are:
+
+Free-space managers that track free-space sections of various sizes in the file that are not currently allocated.
+Aggregators, which are contiguous blocks of free space in the file.
+Virtual file drivers, which use the virtual file driver interface to request additional space from the file driver associated with the file.
+There are four file space-handling strategies available to users that use these mechanisms:
+
+H5F_FSPACE_STRATEGY_FSM_AGGR
+This strategy has always been available in HDF5 and is the default. The mechanisms used for this strategy are free-space managers, aggregators, and virtual file drivers.
+
+H5F_FSPACE_STRATEGY_PAGE
+The current HDF5 file space allocation accumulates small pieces of metadata and raw data in aggregator blocks. However, these blocks are not page aligned and vary widely in sizes. The paged aggregation feature provides efficient paged access of these small pieces of metadata and raw data. It accumulates metadata and raw data into well-aligned pages called file space pages. The library defines a default file space page size but a user can set the page size via a new public routine.
+
+The mechanisms used for this strategy are free-space managers with embedded paged aggregation and virtual file drivers.
+
+See the RFC on this feature for complete details.
+
+H5F_FSPACE_STRATEGY_AGGR
+With this strategy the library will request space from either the metadata or raw data aggregator depending on the file space type. If the request is not satisfied, the library will request space from the virtual file driver.
+
+The mechanisms used for this strategy are aggregators and virtual file drivers. It does not use the free-space manager.
+
+H5F_FSPACE_STRATEGY_NONE
+This strategy will request space from the virtual file driver. The only mechanism used is the virtual file driver. It does not use the free-space manager.
+
+File Space Management User’s Guide
+(This document is not yet available.)
+
+HDF5 Library APIs
+The APIs listed below from the HDF5 Reference Manual provide a means for users to directly manage the file space management feature.
+
+[H5Fget_free_sections](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#gab9cbf1a45f9dcda34b43f985b7848434)Retrieves free-space section information for a file
+[H5Fget_freespace](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#ga3ef2673183567543346668a8f1eca2e9) Returns the amount of free space in a file
+[H5Fget_info2](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_f.html#gaced8c09c1559636a9c3f33dff3f4520e) Returns global information for a file
+[H5Pget_file_space_strategy](https://docs.hdfgroup.org/hdf5/v1_14/group___f_c_p_l.html#ga54cf6ca4f897ba9ee3695a15fe8e6029) Retrieves the File Space Strategy for a file creation property list
+[H5Pset_file_space_strategy](https://docs.hdfgroup.org/hdf5/v1_14/group___f_c_p_l.html#ga167ff65f392ca3b7f1933b1cee1b9f70) Sets the File Space Strategy for a file creation property list
+[H5Pget_file_space_page_size](https://docs.hdfgroup.org/hdf5/v1_14/group___f_c_p_l.html#gaab5e8c08e4f588e0af1d937fcebfc885) Retrieves the file space page size for paged aggregation
+[H5Pset_file_space_page_size](https://docs.hdfgroup.org/hdf5/v1_14/group___f_c_p_l.html#gad012d7f3c2f1e1999eb1770aae3a4963) Sets the file space page size for paged aggregation
+Tools
+The tools listed below have been modified to preserve or modify file freepace information appropriately when processing files employing this feature.
+
+h5dump	When printing the file creation property information for the superblock via the -B option, h5dump includes the block size obtained via H5Pget_file_space_page_size.
+h5stat	When printing the file space information via the -S option, h5stat includes the block size obtained via H5Pget_file_space_page_size.
+h5repack	The following options were added to h5repack:
+-G FS_PAGESIZE,--fs_pagesize=FS_PAGESIZE enables the file space page size to be changed to FS_PAGESIZE.
+
+-P FS_PERSIST,--fs_persist=FS_PERSIST sets the persisting free space to persist (1) or to not persist (0).
+
+-S FS_STRATEGY, --fs_strategy=FS_STRATEGY sets the file space management strategy.
+
+-T FS_THRESHOLD, --fs_threshold=FS_THRESHOLD sets the free-space section threshold.
+
+
+
+Differences between HDF5-1.10 vs HDF5-1.8
+HDF5-1.10
+File space management strategies were introduced via H5Pset_file_space_strategy to manage the unused space in a file.
+
+While a file is open, HDF5 tracks and re-uses the unused space in the file according to the strategy used. If using a strategy that uses the free space manager, then free space can be tracked across file opens by use of the "persist" flag and a minimum free space threshold can be specified. If not specifying a strategy that uses the free space manager, then when the file is closed, any free space is lost and will remain in the file.
+
+HDF5-1.8
+File space management only occurs between the HDF5 file open and close, and the free space is NOT tracked beyond file closed. In other words, when you delete a dataset, the space used by the dataset becomes free space that can be re-used as long as the file is open. Once the file is closed, the free space is lost and will remain in the file.
+
+How to Remove the Free Space in an Existing File
+The h5repack utility can be used to remove the unused space in a file, by writing the file to a new file. This utility comes with the HDF5 binary distribution.