Opinion on how to implement a remote storage backend.

[Zilog8-duplicate]

Problem: I want to download a 100gb torrent, but my server only has 10gb of disk space
Solution: Write a remote storage back end. (Did it in the past for a different torrent client jackpal/Taipei-Torrent@1b09582 )
Goals for Efficiency:

1. Keep a local cache to reduce round-trips for uploading.
2. Allow "verification" hashing (resuming downloads, etc.) to be done at the remote end.

(1) is relatively trivial; I could just adapt the cached storage backend and SFTP client I made for Taipei-Torrent, but (2) requires refactoring where piece hashing is done (also a different transfer protocol altogether; although SFTP's spec includes an optional "check-file" extension, few servers implement it and it's not flexible enough to hash pieces that span more than one file).

I've been getting familiar with anacrolix/torrent code over the weekend, but want to get someone else's opinion on how I plan to implement the anacrolix/torrent side of this:

Step 1:
In anacrolix/torrent/storage/interface.go ; Create a new interface: type SelfHashing interface { SelfHash() metainfo.Hash }

Step 2:
Refactor the contents of func (t *Torrent) hashPiece(piece pieceIndex) (ret metainfo.Hash, err error) (in anacrolix/torrent/torrent.go), out into new function func (p Piece) SumHash (ret metainfo.Hash, err error) (in anacrolix/torrent/storage/wrappers.go). hashPiece would basically just call and return the results from p.SumHash

Step 3:
func (p Piece) SumHash pseudocode:
if p implements SelfHashing { return p.SelfHash() } Else { calculate the hash using the code moved during step 2 }

This should result in storage backends being able to override how hashing is done if they want to (by implementing the SelfHashing interface), but maintaining backwards compatibility for all storage backends who don't (without modification)

Opinions?

[Zilog8]

Ok, GitHub just made me a duplicate account, weird. Anyway, this is my main account.

[anacrolix]

Thank you for contributing this discussion. Yes this is definitely something I've been considering for a while. It's definitely a shame that you can't currently hash data without downloading it to the torrent client for verification. Many backend types can do the hash with special primitives there, or are just nicer without having to transmit the data to the client.

I think currently the best way to contribute this would be as an optional interface in storage, as you've described here.

Note that as part of this implementation, you can stream the downloaded data to SFTP, you don't have to use that synchronously as part of your storage implementation. Furthermore, you could create a storage implementation that caches data for local verification, and only after it passes, transmits it on to SFTP. You could also just use the existing "caching" storage implementations in anacrolix/torrent, and stream the data directly from a torrent.File.NewReader to SFTP without having to implement anything inside anacrolix/torrent.