Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: crypt4gh #223

Draft
wants to merge 162 commits into
base: main
Choose a base branch
from
Draft

feat: crypt4gh #223

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
162 commits
Select commit Hold shift + click to select a range
d41d0e4
Add crypt4gh example keys and encrypted BAM file for test purposes only
brainstorm Feb 28, 2023
b3c22d4
Add a bit more CLI tests on crypt4gh /cc @mmalenic
brainstorm Feb 28, 2023
358a54d
Merge branch 'main' into crypt4gh
brainstorm Jun 15, 2023
f20fc97
Sketching Crypt4GH traits in the context of htsget-rs storage layer.
brainstorm Jun 29, 2023
ed9643a
Implementing AsyncRead Cryptor future to iterate over encrypted/decry…
brainstorm Jul 3, 2023
a39612f
Introduced (encrypted) block reader and implement the poll_fill_buf/p…
brainstorm Jul 4, 2023
2335ecc
fix: build errors and code tidy
mmalenic Jul 16, 2023
9d6a3a3
refactor: add state for block decoding, and implement header info dec…
mmalenic Jul 17, 2023
d579e78
refactor: pass header packets count to the next state
mmalenic Jul 17, 2023
cb58fa9
feat: add dedicated error type
mmalenic Jul 17, 2023
9fe410f
feat: implement header packet splitting in decoder state
mmalenic Jul 17, 2023
c7618cc
refactor: move decode code into separate functions
mmalenic Jul 19, 2023
9b6c171
test: add decode header info test
mmalenic Jul 19, 2023
d4b5a05
refactor: rename `BlockType` to `DecodedBlock`
mmalenic Jul 19, 2023
38eac99
test: add test for header packets
mmalenic Jul 20, 2023
85c878f
test: add test for data block
mmalenic Jul 20, 2023
762be3e
refactor: create decrypt module
mmalenic Jul 23, 2023
58f32fd
refactor: move stream decryptor structs
mmalenic Jul 23, 2023
ffd5bce
feat: implement decryptor for header packets
mmalenic Jul 24, 2023
45beda7
fix: use session keys and body decrypt in data block decryptor
mmalenic Jul 24, 2023
b71af33
test: rearrange test functions and add data block decryptor test
mmalenic Jul 24, 2023
10b2bbc
test: add header packet decryptor test
mmalenic Jul 24, 2023
cb17e0d
refactor: move around some files and introduce polling functions
mmalenic Jul 25, 2023
ff26397
refactor: decrypt all header packets at once in the header packet dec…
mmalenic Jul 26, 2023
4cc14ab
wip: header info future
mmalenic Jul 26, 2023
696b0ef
refactor: decode all header packets in one go
mmalenic Jul 26, 2023
ef072aa
wip: last block of decoder
mmalenic Jul 27, 2023
540c01c
fix: decode end of file properly and add test for the last data block
mmalenic Jul 27, 2023
ff60068
fix: working Crypt4GH Stream and tests
mmalenic Jul 28, 2023
7905fb9
feat(crypt4gh): add builder for reader
mmalenic Aug 7, 2023
f68bfd0
feat(crypt4gh): implementation of AsyncRead and AsyncBufRead
mmalenic Aug 8, 2023
add2851
test(crypt4gh): test AsyncRead with noodles
mmalenic Aug 8, 2023
c2c8c9a
test(crypt4gh): update to check for all records
mmalenic Aug 8, 2023
a5e14eb
feat(crypt4gh): implement edit list parsing when decrypting
mmalenic Aug 15, 2023
73d5b1f
refactor: move crypt4gh code to a new crate
mmalenic Aug 15, 2023
4a71c1c
refactor(search)!: remove init_reader function from search and refact…
mmalenic Aug 15, 2023
ccb237f
refactor(search)!: rename read_index to get_index
mmalenic Aug 15, 2023
2bb6c53
refactor(config)!: split resolver module into multiple files
mmalenic Aug 16, 2023
962aba1
feat(config): add Crypt4GH config in resolver and update docs
mmalenic Aug 16, 2023
a0b8ce2
Merge branch 'main' of https://github.com/umccr/htsget-rs into crypt4gh
mmalenic Aug 16, 2023
dc1c91e
feat(config): add crypt4gh key config
mmalenic Aug 18, 2023
35748d3
build: bump dependencies
mmalenic Aug 18, 2023
36580cb
feat: apply crypt4gh file parsing to search
mmalenic Aug 18, 2023
2613e85
feat(crypt4gh): add positions to decrypter stream
mmalenic Aug 23, 2023
363bbbf
bug(test): incorrect optional dependency for http_tests feature flag …
mmalenic Aug 23, 2023
68869e0
test(crypt4gh): add header length test
mmalenic Aug 23, 2023
4fef66c
feat(crypt4gh): add current block size and tests
mmalenic Aug 23, 2023
d17d249
feat(crypt4gh): track current data block position and next data block…
mmalenic Aug 24, 2023
d64cb2d
test(crypt4gh): add tests for current and next position, fix header l…
mmalenic Aug 25, 2023
ad953d7
feat(crypt4gh): add inner reader getter functions
mmalenic Aug 25, 2023
e5d4c87
feat(crypt4gh): add seek operation
mmalenic Aug 29, 2023
94ec3bc
refactor(crypt4gh): refactor session key polling and implement AsyncS…
mmalenic Aug 29, 2023
d82e908
refactor(crypt4gh): AsyncSeek properly implemented and tests added
mmalenic Aug 29, 2023
64f4674
feat(crypt4gh): add length hint for determining the end of the stream
mmalenic Aug 29, 2023
5dccb47
refactor(crypt4gh): recompute stream length when building decrypter
mmalenic Aug 30, 2023
0209e07
feat(crypt4gh): implement advance trait
mmalenic Aug 30, 2023
3d0daae
feat(crypt4gh): add future for advance function
mmalenic Aug 31, 2023
27468c4
test(crypt4gh): add tests for advance trait
mmalenic Aug 31, 2023
e504af7
refactor(crypt4gh): replace seek trait implementation with a simpler …
mmalenic Sep 4, 2023
d879bac
refactor(search): add general object type enum and crypt4gh object as…
mmalenic Oct 20, 2023
e4dd65a
refactor(config): add crypt4gh decryption key to object type
mmalenic Oct 22, 2023
70dad36
refactor(config): add crypt4gh config to tls
mmalenic Oct 28, 2023
a98800e
feat(crypt4gh): add set stream length to builders
mmalenic Oct 28, 2023
1e912c1
test(crypt4gh): additional tests for overriding stream length
mmalenic Oct 28, 2023
c760dd0
test(crypt4gh): seek to 0 for first data block
mmalenic Oct 28, 2023
d99392e
test(crypt4gh): additional tests for seeking
mmalenic Oct 28, 2023
e32d355
test(crypt4gh): strengthen noodles test to include reference sequences
mmalenic Oct 28, 2023
81470ae
feat(crypt4h): add conversion from unencrypted position to encrypted …
mmalenic Oct 28, 2023
ce9eab9
feat(crypt4h): add unencrypted seek
mmalenic Oct 28, 2023
4b82712
feat(crypt4h): add unencrypted advance
mmalenic Oct 28, 2023
da1ffb8
test(crypt4h): unencrypted seek and advance
mmalenic Oct 29, 2023
5db80ba
feat(crypt4gh): add util functions to convert to encrypted positions
mmalenic Oct 31, 2023
6b0f173
feat(config): add endpoints to url storage config
mmalenic Oct 31, 2023
e733236
Merge branch 'main' of https://github.com/umccr/htsget-rs into crypt4gh
mmalenic Oct 31, 2023
a01bc08
document request urls for UrlStorage
blankdots Oct 31, 2023
93992ce
Merge pull request #211 from umccr/docs/crypt4gh-docs-requests
mmalenic Oct 31, 2023
928e6b9
feat(search): move url config options to url storage
mmalenic Oct 31, 2023
2f18bf7
refactor(crypt4gh): rename endpoint_header to endpoint_file
mmalenic Nov 1, 2023
4c5a40d
refactor(search): replace data uris with byte ranges because they are…
mmalenic Nov 1, 2023
18afeeb
Merge branch 'main' of https://github.com/umccr/htsget-rs into crypt4gh
mmalenic Nov 1, 2023
8c23099
fix: build errors
mmalenic Nov 1, 2023
1f15dce
fix: tests
mmalenic Nov 1, 2023
af54dd9
feat(crypt4gh): remove libsodium and switch to RustCrypto impl - all …
mmalenic Nov 1, 2023
90e41ac
feat(crypt4gh): implement translating byte positions to encrypted byt…
mmalenic Nov 1, 2023
40f8749
feat(search): make url joining more robust
mmalenic Nov 2, 2023
b954b4a
fix(config): check for gzi file ending too
mmalenic Nov 2, 2023
db31d04
feat(search): unencrypted byte ranges working
mmalenic Nov 2, 2023
82be158
fix(search): convert header using le bytes
mmalenic Nov 2, 2023
1eaafbf
fix(crypt4gh): update util functions
mmalenic Nov 2, 2023
3b5a5aa
test(crypt4gh): util function tests
mmalenic Nov 2, 2023
78289a4
test: fix crypt4gh url storage tests
mmalenic Nov 2, 2023
326f4e4
test: add some tests for crypt4gh url storage
mmalenic Nov 2, 2023
b06aadc
refactor: move endpoints into its own struct
mmalenic Dec 4, 2023
8c4a2cd
refactor: add public key to crypt4gh config
mmalenic Dec 5, 2023
e0defda
feat(config): add Crypt4GHGenerate option for dynamically creating en…
mmalenic Dec 5, 2023
f840959
refactor(config): make the generate keys option tagged
mmalenic Dec 5, 2023
628beb2
refactor(config): add unimplemented for using Crypt4GH with private k…
mmalenic Dec 5, 2023
880df18
refactor(config)!: add object type to query
mmalenic Dec 5, 2023
27f3b14
refactor(config)!: remove public key endpoint because the public key …
mmalenic Dec 5, 2023
2792f96
feat(search): add object to get options and bytes position options
mmalenic Dec 5, 2023
82b5669
feat(search): map crypt4gh objects to url path parameter
mmalenic Dec 5, 2023
c2f2030
refactor(search)!: outline url storage functionality within the conte…
mmalenic Dec 6, 2023
5d6ce29
Merge branch 'main' of https://github.com/umccr/htsget-rs into crypt4gh
mmalenic Dec 6, 2023
33e4b94
build(deps): update noodles
mmalenic Dec 6, 2023
dd302bc
build(deps): update noodles to 0.60, use cram get_ref for underlying …
mmalenic Jan 2, 2024
d353d84
Merge branch 'main' of https://github.com/umccr/htsget-rs into crypt4gh
mmalenic Jan 2, 2024
8a77c2b
Merge branch 'main' of https://github.com/umccr/htsget-rs into crypt4gh
mmalenic Jan 2, 2024
47b4f70
fix(config): build errors
mmalenic Jan 2, 2024
24c4c7d
feat(async): add some convenience functions
mmalenic Jan 2, 2024
1e05ffc
feat(async): implement edit list calculation
mmalenic Jan 3, 2024
1640e33
test(async): fix edit lists position calculation and add tests
mmalenic Jan 3, 2024
cb34954
refactor(async): rearrange to use reader in edit list function
mmalenic Jan 3, 2024
a6894f7
refactor(async): store header packet length bytes when decoding
mmalenic Jan 3, 2024
af956e2
test(async): working edit lists
mmalenic Jan 4, 2024
5cf4fec
feat: crypt4gh logic in url storage using edit lists
mmalenic Jan 5, 2024
74528ad
refactor(search): get public key from query headers
mmalenic Jan 5, 2024
d66ed78
feat(async): generate key pairs
mmalenic Jan 5, 2024
70d7364
fix(search): url storage expects base64 public key in the header
mmalenic Jan 5, 2024
f9290f2
fix(search): use either type for the reader in `UrlStorage`
mmalenic Jan 6, 2024
d4f2d10
fix(async): parse public and private keys when generating them
mmalenic Jan 6, 2024
c50eaa8
refactor(search): use headers to pass public key
mmalenic Jan 6, 2024
17ade16
fix(search): avoid returning crypt4gh header and instead fetch a byte…
mmalenic Jan 7, 2024
e0b8c47
test: real test with edit lists
mmalenic Jan 9, 2024
261286b
fix(async): add edit list partition function
mmalenic Jan 9, 2024
174978f
test(async): edit list partition test
mmalenic Jan 9, 2024
57581e2
fix: broken edit list function by using merged positions
mmalenic Jan 10, 2024
6999d5d
fix(async): apply edits directly instead of using broken `body_decryp…
mmalenic Jan 14, 2024
1abbd36
test: working url storage test with encrypted data
mmalenic Jan 15, 2024
2564d7f
fix(search): use encrypted file size when the file requested in an en…
mmalenic Jan 15, 2024
cab6afb
docs: add architecture and diagrams
mmalenic Jan 16, 2024
db59fbe
docs: update diagram
mmalenic Jan 16, 2024
38dc327
refactor: remove head endpoint, use client-public-key header
mmalenic Jan 16, 2024
decf34f
fix(build): build errors, clippy warnings, for all combinations of fe…
mmalenic Feb 1, 2024
14de005
feat(search): allow specifying key pair in config rather than generat…
mmalenic Feb 11, 2024
e530911
test(search): add test for predefined key pairs
mmalenic Feb 12, 2024
fb7e74b
refactor(search): add through header content length and add additiona…
mmalenic Feb 13, 2024
428b2cd
test(search): new protocol with client and server additional bytes
mmalenic Feb 13, 2024
eb3fb01
test(search): add tests, fix compile errors
mmalenic Feb 13, 2024
2ab1d71
test(config): update conflicting object types test
mmalenic Feb 14, 2024
1acef6a
docs: update architecture diagram and config docs
mmalenic Feb 14, 2024
83290af
fix(search): allow optional server-additional-bytes and convert heade…
mmalenic Feb 14, 2024
52bb063
test(search): broken test with byte range
mmalenic Feb 14, 2024
35b67c2
fix(config): url_storage config example
mmalenic Feb 22, 2024
3fcf744
refactor(search): add more log statements
mmalenic Feb 27, 2024
4d7c1e0
refactor(search): more log statements in edit lists
mmalenic Feb 27, 2024
2c6c456
fix(search): read client public key using crypt4gh library
mmalenic Feb 27, 2024
4f30ef9
merge from main
mmalenic Mar 11, 2024
77a011d
feat: add user agent to url storage
mmalenic Mar 11, 2024
a5f891a
fix(search): tests and byte ranges, use main branch of crypt4gh
mmalenic Mar 12, 2024
b734e93
fix(search): tests working, remove temp directory when reading/genera…
mmalenic Mar 12, 2024
10f7624
feat(search): send_encrypted_to_client option, fix tests, and compile…
mmalenic Mar 12, 2024
bd456e7
fix(search): fix full byte range search
mmalenic Mar 15, 2024
40ecafd
search(url): base64 encode key and add header lines
mmalenic Mar 24, 2024
194457b
test(search): fix test assumptions
mmalenic Mar 24, 2024
c7ecefb
fix(search): remove server-public-key, overwrite user-agent and clien…
mmalenic Apr 12, 2024
de7e9f8
fix(search): add correct headers to head requests, including updated …
mmalenic Apr 12, 2024
31c72e2
Merge branch 'main' of https://github.com/umccr/htsget-rs into crypt4gh
mmalenic Apr 17, 2024
7e5ee23
test(search): integration tests for crypt4gh and fix CRAM eof range
mmalenic Apr 18, 2024
1fab8e7
fix(deploy): exclude async-crypt4gh from docker ignore
mmalenic Apr 18, 2024
df675cd
fix(search): overwrite user-agent when requesting index file, replace…
mmalenic Apr 26, 2024
a0aba09
feat(search): add override for validation certificates for testing pu…
mmalenic Apr 26, 2024
c06b550
Merge branch 'main' of https://github.com/umccr/htsget-rs into crypt4gh
mmalenic May 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ jobs:
push: true
tags: |
ghcr.io/umccr/htsget-rs:latest

## SOCI (Seekable OCI) support. Only enable when and if docker layers surpass 10MB in the future, see:
# https://github.com/awslabs/soci-snapshotter/issues/100
# - name: Install aws SOCI
Expand Down
Loading
Loading