Description

This issue tracks the implementation of the create_vectors_dataset and upload_index tasks - these tasks handle downloading vectors from a Remote Store repository, and uploading the knn index to the Remote Store repository, respectively. The build_index task implementation will be tracked in a separate issue. All 3 of these tasks are utilized by the Remote Vector Index Builder Service to construct a knn index on hardware accelerators (such as GPUs), for an Opensearch cluster. For more background on the Remote Vector Index Builder Service, please read: opensearch-project/k-NN#2293

The tasks are part of the core Remote Vector Index Builder Python library.In the initial implementation, only the s3 Remote Store repository will be supported. However, the tasks must be designed to support other repositories (such as Azure Blob Storage, GCP) in the future.

Sub-tasks

• Create a draft PR implementation, get feedback from others
• Integrate feedback, add unit tests, and do benchmarking - finalize implementation