Some ideas I have for OpenDAL

[Xuanwo]

This isn't a proposal, request, or plan—just some ideas I've been considering for a while. I'm still unsure if they are valid, feasible, or worth implementing. However, I think it's beneficial to start a discussion and share these thoughts with the entire community instead of keeping them to myself.

Feel free to leave your comment if you like this idea or suggesting more.

List With Glob/Pattern

Initially reported at #1251, and subsequently at #3500.

This makes sense as it's very common for users to list items using a pattern. For example, list all pdf files or all files that match 2024/**/result.json.

Additionally, some services such as Google Cloud Storage offer native features like matchGlob for enhanced functionality. This can reduce a lot of API requests if the prefix is large.

I'm struggling with how to design the API.

• Just support glob?
  • Like op.list(path).glob("2024/**/result.json").await
• How to implement fallback if services doesn't support glob?
• Services might have different behavior for the same glob pattern, how to support them?
• Do we need to support regex?
  • Like op.list(path).regex(r"(?m)^([^:]+):([0-9]+):(.+)$").await

Expose HttpClient API

Initially reported at #4471.

This is an interesting idea that allows users to choose the HTTP client they prefer. Users can do more things like metrics, logging inside the client by set the client for service.

I'm struggling with the API naming: HttpClient doesn't look like a good name for trait. We need better one. I want we can have a trait X, and a type called HttpClient = Arc<dyn X>.

Additionally, we should consider the internal handling of requests and responses. It's better to keep these processes hidden and not require users to implement them.

opendal_tokio_fs and opendal_std_fs

Initially reported at #4194

We can have two new crates that allowing users to use similiar API like tokio::fs, std::fs but backed by OpenDAL.

I'm struggling with how to design the API: tokio::fs and std::fs are provide globally, how users init opendal operator first?

My current idea is to have a set_fs(op: Operator) function or macro.