07-S3-and-the-Cloud.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b0bf0f0a-b19e-4a50-8a13-820ec7439437",
   "metadata": {},
   "source": [
    "# Reading Data Directly from the Cloud\n",
    "\n",
    "In the previous notebook, we showed how the library can use a VOL connector to access files stored in a cloud backend through HSDS. But this is not the only way to use HDF5 in the cloud. HDF5 files stored in an S3 bucket can be accessed directly from the library without any external connector or drivers. This is possible through the Read-Only S3 VFD (ROS3 VFD). \n",
    "\n",
    "## Enabling the ROS3 VFD\n",
    "\n",
    "The ROS3 VFD is packaged with the library, and only needs to be enabled during the build process. When using CMake to build HDF5, this is done by setting the variable `HDF5_ENABLE_ROS3_VFD` to `ON`.\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "The cloning and compilation takes, depending on the machine type, about three minutes.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a7406c3-96f8-4a65-8cdc-69090897543a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git clone https://github.com/HDFGroup/hdf5.git build/hdf5\n",
    "mkdir -p build/hdf5/build\n",
    "cd build/hdf5/build\n",
    "cmake -DCMAKE_INSTALL_PREFIX=/home/vscode/.local -DHDF5_ENABLE_ROS3_VFD=ON -DBUILD_STATIC_LIBS=OFF -DBUILD_TESTING=OFF -DHDF5_BUILD_EXAMPLES=OFF -DHDF5_BUILD_TOOLS=ON -DHDF5_BUILD_UTILS=OFF ../ 2>&1 > /dev/null\n",
    "make -j 4 2>&1 > /dev/null\n",
    "make install 2>&1 > /dev/null\n",
    "cd ../../.."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "defa2db2-6da3-4057-ba43-5b5f904cdc9d",
   "metadata": {},
   "source": [
    "For this demonstration, we will use a copy of the `ou_process.h5` file that has been uploaded to an S3 bucket. We can use the command line tool `h5dump` to examine this file and see that it is the same file we're familiar with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "082b1665-2780-4d45-9cc0-9e72aec7a207",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "/home/vscode/.local/bin/h5dump -pBH --vfd-name ros3 --s3-cred=\"(,,)\" https://s3.us-east-2.amazonaws.com/docs.hdfgroup.org/hdf5/h5/ou_process.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d8d3599-4f70-41d4-aff4-05842fd14efd",
   "metadata": {},
   "source": [
    "Now we will use the ROS3 VFD to read the file. Similarly to other VFDs, the ROS3 VFD is enabled at file access time with a File Access Property List (FAPL) modified via `H5Pset_fapl_ros3()`. This function also requires some S3-specific parameters in the form of the `H5FD_ros3_fapl_t` structure. Because this file is publicly accessible without any authentication, all we need to provide is the region it is located in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bd871fd-7521-4d58-b234-b7627a292b85",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/ou_ros3.c\n",
    "#include \"hdf5.h\"\n",
    "#include <array>\n",
    "#include <cassert>\n",
    "#include <cstring>\n",
    "#include <iomanip>\n",
    "#include <iostream>\n",
    "#include <string>\n",
    "#include <vector>\n",
    "\n",
    "using namespace std;\n",
    "\n",
    "int main(void)\n",
    "{\n",
    "H5FD_ros3_fapl_t s3_params;\n",
    "\n",
    "// Set the AWS region and the S3 bucket name\n",
    "strcpy(s3_params.aws_region, \"us-east-2\");\n",
    "string file_uri = \"https://s3.us-east-2.amazonaws.com/docs.hdfgroup.org/hdf5/h5/ou_process.h5\";\n",
    "// The version of this ROS3 FAPL parameter structure\n",
    "s3_params.version = 1;\n",
    "// This file permits anonymous access, so authentication is not needed\n",
    "// secret_id and secret_key auth fields that would be required for protected S3 buckets\n",
    "s3_params.authenticate = 0; \n",
    "\n",
    "// Open and read from the file\n",
    "auto fapl = H5Pcreate(H5P_FILE_ACCESS);\n",
    "H5Pset_fapl_ros3(fapl, &s3_params);\n",
    "auto file = H5Fopen(file_uri.c_str(), H5F_ACC_RDONLY, fapl);\n",
    "auto dset = H5Dopen(file, \"dataset\", H5P_DEFAULT);\n",
    "\n",
    "// Get the dataspace details from the file\n",
    "auto dspace = H5Dget_space(dset);\n",
    "assert(H5Sget_simple_extent_ndims(dspace) == 2);\n",
    "array<hsize_t, 2> dims;\n",
    "H5Sget_simple_extent_dims(dspace, dims.data(), NULL);\n",
    "vector<double> read_buffer(dims[0] * dims[1]);\n",
    "\n",
    "H5Dread(dset, H5T_IEEE_F64LE, H5S_ALL, H5S_ALL, H5P_DEFAULT, (void*) read_buffer.data());\n",
    "\n",
    "for (size_t i = 0; i < 10; i++) {\n",
    "    size_t row_idx = i * dims[0] / 10;\n",
    "    size_t col_idx = i * dims[1] / 10;\n",
    "    auto element = read_buffer[row_idx * dims[1] + col_idx];\n",
    "    cout << \"Element at index (\" << row_idx << \", \" << col_idx << \") = \"\n",
    "         << std::setprecision(16) << element << endl;\n",
    "}\n",
    "\n",
    "H5Dclose(dset);\n",
    "H5Sclose(dspace);\n",
    "H5Pclose(fapl);\n",
    "H5Fclose(file);\n",
    "\n",
    "return 0;\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8b45b90-3ec9-4af6-8536-256eb4c06bd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "g++ -Wall -pedantic -I/home/vscode/.local/include -L/home/vscode/.local/lib ./src/ou_ros3.c -o ./build/ou_ros3 -lhdf5 -Wl,-rpath=/home/vscode/.local/lib\n",
    "./build/ou_ros3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0351df0-5336-416d-86e3-c40ee8a1b1c6",
   "metadata": {},
   "source": [
    "The ROS3 VFD is also transparently accessible through h5py! (remember Tutorial 04?) The default installation of h5py does not have th ROS3 VFD enabled. To enable it, we build h5py from source against an HDF5 library that already has the ROS3 VFD enabled. Then, we can access our file in the cloud by providing the same ROS3 arguments as before to `h5py.File()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8da438a-7df4-4860-90af-e031f86fed7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git clone https://github.com/h5py/h5py\n",
    "cd h5py\n",
    "HDF5_DIR=/home/vscode/.local pip install .\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f70bbf3d-5cd3-41fb-820d-dad47a7a41b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py\n",
    "\n",
    "dims = [100, 1000]\n",
    "\n",
    "s3_uri = \"https://s3.us-east-2.amazonaws.com/docs.hdfgroup.org/hdf5/h5/ou_process.h5\"\n",
    "kwargs = {}\n",
    "kwargs['mode'] = 'r'\n",
    "kwargs['driver'] = \"ros3\"\n",
    "kwargs['aws_region'] = (\"us-east-2\").encode(\"utf-8\")\n",
    "# kwargs['authenticate'] = 0\n",
    "\n",
    "f = h5py.File(s3_uri, **kwargs)\n",
    "\n",
    "dset = f[\"dataset\"]\n",
    "\n",
    "data = dset[:]\n",
    "\n",
    "for i in range(10):\n",
    "    row_idx = int(i * dims[0] / 10)\n",
    "    col_idx = int(i * dims[1] / 10)\n",
    "    print(f\"dataset[{row_idx}, {col_idx}] = {data[row_idx, col_idx]}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c6994af-1120-4870-9c5d-c2e7c4bf0df8",
   "metadata": {},
   "source": [
    "## Cloud-Optimized\n",
    "\n",
    "While it is possible to read HDF5 files in their native format directly from the Cloud, this introduces some inefficiencies. The access patterns of the library were optimized for filesystem I/O, not cloud access and the delays it brings. These concerns motivated the development of the [HDF5 Cloud-Optimized Read-Only Library (H5Coro)](https://github.com/ICESat2-SlideRule/h5coro). H5Coro a pure Python implementation of a subset of the HDF5 specification that has been optimized for reading data out of S3. For large files, H5Coro can speed up access times by [two orders of magnitude](https://www.hdfgroup.org/wp-content/uploads/2021/05/JPSwinski_H5Coro.pdf). This is achieved primarily through using a larger cache to avoid repeated small reads to the cloud."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}