Code

S3 support#2

Code

Merged

cgoina merged 127 commits intomasterJaneliaSciComp/jacs-storage:masterfrom s3-supportJaneliaSciComp/jacs-storage:s3-supportCopy head branch name to clipboardJan 24, 2025

+7,582-9,198

Commits

Commits on Jul 30, 2024

• Prepare next build

  

  Cristian Goina

  committed
  c4c6637

  Copy full SHA for c4c6637

Commits on Sep 20, 2024

• removed support for tar bundles

  

  Cristian Goina

  committed
  5610920

  Copy full SHA for 5610920

• added services to support reading from S3 or from fileservice; renamed original implementations so that I can override them

  

  Cristian Goina

  committed
  dd03c94

  Copy full SHA for dd03c94

Commits on Sep 24, 2024

• implemented the agent get by path endpoint; fixed the unit tests to run

  

  Cristian Goina

  committed
  a545513

  Copy full SHA for a545513

Commits on Sep 26, 2024

• removed overflow volume; added a generic S3 volume that allows null values for root storage uri; handle the null root storage uri

  

  Cristian Goina

  committed
  2786b79

  Copy full SHA for 2786b79

• the storage capacity is only computed by the localvolumemanager

  

  Cristian Goina

  committed
  c51b003

  Copy full SHA for c51b003

• implemented support for content metadata; complete agent endpoints by path or storage URI

  

  Cristian Goina

  committed
  7026450

  Copy full SHA for 7026450

• started changes on volumestorageresource

  

  Cristian Goina

  committed
  9747c72

  Copy full SHA for 9747c72

Commits on Sep 27, 2024

• no longer expose contentstorage provider - methods added to DataContentService; implemented two endpoints from VolumeStorageResource and fixed unit tests

  

  Cristian Goina

  committed
  da6a1b5

  Copy full SHA for da6a1b5

• handle no content found

  

  Cristian Goina

  committed
  069ff11

  Copy full SHA for 069ff11

• upgraded apache commons and replaced RandomUtils with UniformRandomGenerator

  

  Cristian Goina

  committed
  2a8851d

  Copy full SHA for 2a8851d

Commits on Sep 29, 2024

• changed data bundle endpoint; fixed unit tests; s3contentservice always takes bucket as an arg so the path should no longer contain the bucket

  

  Cristian Goina

  committed
  04f4f27

  Copy full SHA for 04f4f27

• removed StorageRelativePath class; N5resource still needs to be changed to support S3

  

  Cristian Goina

  committed
  a57a1ee

  Copy full SHA for a57a1ee

Commits on Sep 30, 2024

• implemented support for N5

  

  Cristian Goina

  committed
  525f343

  Copy full SHA for 525f343

• fixed codecs

  

  Cristian Goina

  committed
  dc66ffc

  Copy full SHA for dc66ffc

• fixed some NPEs

  

  Cristian Goina

  committed
  d32b268

  Copy full SHA for d32b268

Commits on Oct 1, 2024

• added method to JADEURI to get only the content key without the bucket; upgrade mongo driver

  

  Cristian Goina

  committed
  56ce4e2

  Copy full SHA for 56ce4e2

• removed the last uses of org.janelia.jacsstorage.model.jacsstorage.OriginalStoragePathURI

  

  Cristian Goina

  committed
  bcc55b8

  Copy full SHA for bcc55b8

• removed the use of the original DataStorageService

  

  Cristian Goina

  committed
  6d64dfd

  Copy full SHA for 6d64dfd

• removed original DataStorageService

  

  Cristian Goina

  committed
  6e94fc3

  Copy full SHA for 6e94fc3

• fixed dao integrationtests

  

  Cristian Goina

  committed
  57a53af

  Copy full SHA for 57a53af

• upgrade wrapper

  

  Cristian Goina

  committed
  81294e9

  Copy full SHA for 81294e9

• NPE fixes

  

  Cristian Goina

  committed
  0820a62

  Copy full SHA for 0820a62

Commits on Oct 2, 2024

• fixed the path sent to the agent - use absolute path

  

  Cristian Goina

  committed
  1728f1f

  Copy full SHA for 1728f1f

• use jade storage in datapathparam

  

  Cristian Goina

  committed
  c7c8631

  Copy full SHA for c7c8631

• debug logging

  

  Cristian Goina

  committed
  29cdf1f

  Copy full SHA for 29cdf1f

• more logging

  

  Cristian Goina

  committed
  71a3628

  Copy full SHA for 71a3628

• debugging

  

  Cristian Goina

  committed
  2c0e324

  Copy full SHA for 2c0e324

• more debug

  

  Cristian Goina

  committed
  e4588ab

  Copy full SHA for e4588ab

• fixed null scheme

  

  Cristian Goina

  committed
  f9520d0

  Copy full SHA for f9520d0

• fixed webdav endpoint

  

  Cristian Goina

  committed
  e59cb4f

  Copy full SHA for e59cb4f

• default storageType to FileSystem; marshall storageroot location

  

  Cristian Goina

  committed
  f3f99cc

  Copy full SHA for f3f99cc

Commits on Oct 3, 2024

• renamed JSON property for storageRootLocation

  

  Cristian Goina

  committed
  7b6558f

  Copy full SHA for 7b6558f

• set mimetype

  

  Cristian Goina

  committed
  f2b45ba

  Copy full SHA for f2b45ba

• improve exist implementation

  

  Cristian Goina

  committed
  af8eec2

  Copy full SHA for af8eec2

• only return files from filesystemstorage

  

  Cristian Goina

  committed
  7400561

  Copy full SHA for 7400561

• return the content size in a few situations

  

  Cristian Goina

  committed
  ee66942

  Copy full SHA for ee66942

• renamed contentfilter

  

  Cristian Goina

  committed
  d86f52f

  Copy full SHA for d86f52f

Commits on Oct 4, 2024

• refactored content reading to support size estimates so that we can return content-length header as it is today in production; the feature is needed by horta

  

  Cristian Goina

  committed
  13c8c79

  Copy full SHA for 13c8c79

• refactored content reading to support size estimates so that we can return content-length header as it is today in production; the feature is needed by horta

  

  Cristian Goina

  committed
  3ccd14f

  Copy full SHA for 3ccd14f

• comment

  

  Cristian Goina

  committed
  2b894a5

  Copy full SHA for 2b894a5

• set storage type when looking up volumes

  

  Cristian Goina

  committed
  a1352b4

  Copy full SHA for a1352b4

• add aws region config

  

  Cristian Goina

  committed
  84015f5

  Copy full SHA for 84015f5

• changed checkAccess to use listV2 query; head request works only for existing objects

  

  Cristian Goina

  committed
  994ac68

  Copy full SHA for 994ac68

Commits on Oct 6, 2024

• client changes to support s3 locations

  

  Cristian Goina

  committed
  b370b99

  Copy full SHA for b370b99

• removed unused code

  

  Cristian Goina

  committed
  ff448fe

  Copy full SHA for ff448fe

• moved contentaccessparam to services module

  

  Cristian Goina

  committed
  2067bfa

  Copy full SHA for 2067bfa

• change version to 1.22.0.RC1

  

  Cristian Goina

  committed
  70dee77

  Copy full SHA for 70dee77

• fixed NPEs

  

  Cristian Goina

  committed
  aee1780

  Copy full SHA for aee1780

• more client fixes to support s3 locations

  

  Cristian Goina

  committed
  1514f8e

  Copy full SHA for 1514f8e

• logging

  

  Cristian Goina

  committed
  53c77dd

  Copy full SHA for 53c77dd

Commits on Oct 7, 2024

• use offset to skip records

  

  Cristian Goina

  committed
  92b74f8

  Copy full SHA for 92b74f8

• small change to return immediately if there are no content nodes

  

  Cristian Goina

  committed
  1e28b36

  Copy full SHA for 1e28b36

• option to send storage path as a path param or query param; this is to see if I can avoid the double slash merge that happens in the api-gateway

  

  Cristian Goina

  committed
  db76865

  Copy full SHA for db76865

• created a different endpoint that can take a query for storagepath

  

  Cristian Goina

  committed
  ddf71b7

  Copy full SHA for ddf71b7

• created endpoints to get or lookup content using query parameters to avoid the problem caused by nginx merging slashes (path separators)

  

  Cristian Goina

  committed
  b5af3b8

  Copy full SHA for b5af3b8

Commits on Oct 9, 2024

• object to for holding storage options such as aws credentials or other options in case we need to add other storage types (google for example)

  

  Cristian Goina

  committed
  49b45e4

  Copy full SHA for 49b45e4

• update the clients api to use jadestorageoptions

  

  Cristian Goina

  committed
  fa20364

  Copy full SHA for fa20364

• update version

  

  Cristian Goina

  committed
  f8634e9

  Copy full SHA for f8634e9

Commits on Oct 11, 2024

• modified client api to take JadeStorageAttributes; implemented JadeDataLocation (actually moved the implementation from jacs-model:jacs-model-rendering"

  

  Cristian Goina

  committed
  d1bca02

  Copy full SHA for d1bca02

Commits on Oct 12, 2024

• added the clientproxy and client provider that used to be in jacs-model-rendering

  

  Cristian Goina

  committed
  22385af

  Copy full SHA for 22385af

• renamed listresultwrapper

  

  Cristian Goina

  committed
  f35e414

  Copy full SHA for f35e414

Commits on Oct 14, 2024

• ignore unknow pros in jaderesults

  

  Cristian Goina

  committed
  a089668

  Copy full SHA for a089668

• fixed nodeaccess field

  

  Cristian Goina

  committed
  4b97c73

  Copy full SHA for 4b97c73

• ignore unknown

  

  Cristian Goina

  committed
  c120457

  Copy full SHA for c120457

Commits on Oct 15, 2024

• created agent webdav endpoint that takes the datapath from a query param

  

  Cristian Goina

  committed
  5853ce5

  Copy full SHA for 5853ce5

• try not to set headers

  

  Cristian Goina

  committed
  d09638b

  Copy full SHA for d09638b

• pass containerrequestcontext

  

  Cristian Goina

  committed
  2ef9750

  Copy full SHA for 2ef9750

• put headers back

  

  Cristian Goina

  committed
  f6810a7

  Copy full SHA for f6810a7

Commits on Oct 16, 2024

• handle prefixes differently and tested access to open buckets; I added a chaincredentials provider that includes anonymouscredentialsprovider

  

  Cristian Goina

  committed
  75c9b43

  Copy full SHA for 75c9b43

• messed up the branch for selecting the appropriate N5Reader

  

  Cristian Goina

  committed
  174f461

  Copy full SHA for 174f461

• changed the uri method

  

  Cristian Goina

  committed
  1f8c642

  Copy full SHA for 1f8c642

• fixed the n5 queryifexists; attempt to fix the date parser

  

  Cristian Goina

  committed
  a291ad5

  Copy full SHA for a291ad5

Commits on Oct 17, 2024

• changed the S3 N5 keyvalue access

  

  Cristian Goina

  committed
  382ef86

  Copy full SHA for 382ef86

• more N5 changes - list directories took too long

  

  Cristian Goina

  committed
  119c6b2

  Copy full SHA for 119c6b2

• bad condition

  

  Cristian Goina

  committed
  6f22d3e

  Copy full SHA for 6f22d3e

• upgrade n5 imglib2

  

  Cristian Goina

  committed
  c6d8502

  Copy full SHA for c6d8502

• added a discoverandparse in the n5service

  

  Cristian Goina

  committed
  665a0a2

  Copy full SHA for 665a0a2

• try all metadata parsers

  

  Cristian Goina

  committed
  a0b6eca

  Copy full SHA for a0b6eca

• upgraded n5-imglib2 dependency; changed the N5 dataset discovery mechanism because N5NodeTree and the N5Discovery is no longer available since n5-imglib2:6.0.0

  

  Cristian Goina

  committed
  5f38b5b

  Copy full SHA for 5f38b5b

• Created object for dataset attributes because the compression is not serializable

  

  Cristian Goina

  committed
  de1b449

  Copy full SHA for de1b449

• populate children on the last level

  

  Cristian Goina

  committed
  b91c255

  Copy full SHA for b91c255

Commits on Oct 18, 2024

• ignore empty children and metadata

  

  Cristian Goina

  committed
  6a62451

  Copy full SHA for 6a62451

• moved jackson annotation

  

  Cristian Goina

  committed
  9589c38

  Copy full SHA for 9589c38

• exception handling

  

  Cristian Goina

  committed
  996b002

  Copy full SHA for 996b002

• logging

  

  Cristian Goina

  committed
  b4f0ceb

  Copy full SHA for b4f0ceb

• check if attributes.json exists before loading dataset properties

  

  Cristian Goina

  committed
  953476c

  Copy full SHA for 953476c

• throw N5KeyNotFound exception if key is missing - it looks like that exception is handled by the n5keyvalue reader

  

  Cristian Goina

  committed
  211ca4e

  Copy full SHA for 211ca4e

• replaced the headerparams for accesskey and secretkey with containerrequest context because I don't want those values logged

  

  Cristian Goina

  committed
  312104e

  Copy full SHA for 312104e

• replaced n5-imglib2 with n5-universe and added the N5TreeNode back

  

  Cristian Goina

  committed
  2b3f301

  Copy full SHA for 2b3f301

Commits on Oct 23, 2024

• fixed query param

  

  Cristian Goina

  committed
  8120979

  Copy full SHA for 8120979

Commits on Dec 5, 2024

• fix 2 NPEs

  

  Cristian Goina

  committed
  699a391

  Copy full SHA for 699a391

Commits on Dec 6, 2024

• handle prefixes and have an option to return only prefixes

  

  Cristian Goina

  committed
  fadbdd0

  Copy full SHA for fadbdd0

• consider that prefix and virtual path could be null

  

  Cristian Goina

  committed
  0b79cf7

  Copy full SHA for 0b79cf7

• always sort nodes by the key

  

  Cristian Goina

  committed
  3a3b393

  Copy full SHA for 3a3b393

• retrieve the top folder as well when listing nodes from S3

  

  Cristian Goina

  committed
  46829cd

  Copy full SHA for 46829cd

• fix the jade object relative path; bump up version

  

  Cristian Goina

  committed
  a199c95

  Copy full SHA for a199c95

• update model

  

  Cristian Goina

  committed
  6e01b0a

  Copy full SHA for 6e01b0a

Commits on Dec 9, 2024

• changed maxdepth handling

  

  Cristian Goina

  committed
  a014319

  Copy full SHA for a014319

• changed entry relative path handling

  

  Cristian Goina

  committed
  ead7600

  Copy full SHA for ead7600

Commits on Dec 11, 2024

• handle region; when getting the content get it as a byte array - is faster than returning it as inputstream

  

  Cristian Goina

  committed
  2c2f2f9

  Copy full SHA for 2c2f2f9

Commits on Dec 12, 2024

• unit tests logging setup

  

  Cristian Goina

  committed
  8e68651

  Copy full SHA for 8e68651

• changed the contentstreamreader interface

  

  Cristian Goina

  committed
  32c600a

  Copy full SHA for 32c600a

Commits on Dec 15, 2024

• added async implementation for S3StorageService; fixed some inefficiency that added prefixes to be search even if they had greater depth than allowed

  

  Cristian Goina

  committed
  e124619

  Copy full SHA for e124619

• removed commented code

  

  Cristian Goina

  committed
  b4f89f8

  Copy full SHA for b4f89f8

Commits on Dec 16, 2024

• tried to get the async client to work with internal bucket

  

  Cristian Goina

  committed
  6c72aeb

  Copy full SHA for 6c72aeb

Commits on Dec 18, 2024

• got the benchmarks working again; added method to retrieve content for a single object

  

  Cristian Goina

  committed
  553eb76

  Copy full SHA for 553eb76

• fixed benchmark params

  

  Cristian Goina

  committed
  87b78d9

  Copy full SHA for 87b78d9

• renamed jadestorageoptions

  

  Cristian Goina

  committed
  30d55d8

  Copy full SHA for 30d55d8

Commits on Jan 6, 2025

• updated gitignore; some updates to the build file

  

  Cristian Goina

  committed
  d453ada

  Copy full SHA for d453ada

Commits on Jan 8, 2025

• fixed retrieval of files that have the same prefix such as '1' and '10'

  

  Cristian Goina

  committed
  cf27052

  Copy full SHA for cf27052

Commits on Jan 9, 2025

• default async access to false

  

  Cristian Goina

  committed
  6f235d4

  Copy full SHA for 6f235d4

• update version in gradle build

  

  Cristian Goina

  committed
  91009df

  Copy full SHA for 91009df

Commits on Jan 10, 2025

• change log level

  

  Cristian Goina

  committed
  f6eb0d9

  Copy full SHA for f6eb0d9

• clean up

  

  Cristian Goina

  committed
  fa5d4d9

  Copy full SHA for fa5d4d9

• property for default async access

  

  Cristian Goina

  committed
  6a44a1b

  Copy full SHA for 6a44a1b

Commits on Jan 14, 2025

• update jackson

  

  Cristian Goina

  committed
  704e13a

  Copy full SHA for 704e13a

Commits on Jan 16, 2025

• redirect region too

  

  Cristian Goina

  committed
  327b2ac

  Copy full SHA for 327b2ac

• log request headers

  

  Cristian Goina

  committed
  c422d6c

  Copy full SHA for c422d6c

• always ignore authorization

  

  Cristian Goina

  committed
  26c479c

  Copy full SHA for 26c479c

Commits on Jan 17, 2025

• flag to be able to use anonymous access first in the aws credentials providers chain

  

  Cristian Goina

  committed
  759ee49

  Copy full SHA for 759ee49

• the credentials provider was never set for async client so it was always using default which does not include anonymous provider; separated the code that creates sync and async clients

  

  Cristian Goina

  committed
  65c9715

  Copy full SHA for 65c9715

Commits on Jan 18, 2025

• use netty based async client

  

  Cristian Goina

  committed
  6c30a30

  Copy full SHA for 6c30a30

• update aws hoping it would fix the NPE exception but it didn't

  

  Cristian Goina

  committed
  dc2d6c4

  Copy full SHA for dc2d6c4

Commits on Jan 24, 2025

• changes from the review: throw exception when file not found; changed the S3 adapter cache key

  

  Cristian Goina

  committed
  14aa1f9

  Copy full SHA for 14aa1f9

• Merge branch 'master' into s3-support

  

  Cristian Goina

  committed
  4b8d095

  Copy full SHA for 4b8d095

• moved S3StorageServiceTest to integration tests

  

  Cristian Goina

  committed
  84fa4ab

  Copy full SHA for 84fa4ab