Use the /dbfs/put API endpoint to upload smaller DBFS files

[shreyas-goenka]

Changes

The DBFS put API is much more efficient and faster than the streaming upload API. It's recommended for files smaller than 2 GB.

This PR modifies the DBFS filer client to use the PUT API for local files that are smaller than the 2 GB threshold.

This PR utilizes a reader and a writer from io.Pipe() to make streaming uploads work.

Why don't we use the PUT API for non-local files?

The most important use case for the databricks fs cp command is uploading local files to DBFS. Thus I did not invest time into making that use case work well with this optimization since it'll lead to additional complexity.

Tests

Unit and integration tests. Also manually.

Before:

venv➜  bundle-playground git:(master) ✗ databricks fs cp databricks.yml dbfs:/Users/[email protected]/foo.yml -p dogfood --debug
12:41:12 DEBUG POST /api/2.0/dbfs/create
> {
>   "path": "/Users/[email protected]/foo.yml"
> }
< HTTP/2.0 200 OK
< {
<   "handle": 1648410211255868
< } pid=66693 sdk=true
12:41:12 DEBUG POST /api/2.0/dbfs/add-block
> {
>   "data": "IyB5YW1sLWxhbmd1YWdlLXNlcnZlcjogJHNjaGVtYT0uL3NjaGVtYS5qc29uCmJ1bmRsZToKICBuYW1lOiBidW5kbGUtcGxh... (144 more bytes)",
>   "handle": 1648410211255868
> }
< HTTP/2.0 200 OK
< {} pid=66693 sdk=true
12:41:12 DEBUG POST /api/2.0/dbfs/close
> {
>   "handle": 1648410211255868
> }
< HTTP/2.0 200 OK
< {} pid=66693 sdk=true
databricks.yml -> dbfs:/Users/[email protected]/foo.yml

After:

.venv➜  bundle-playground git:(master) ✗ cli  fs cp databricks.yml dbfs:/Users/[email protected]/bar.yml -p dogfood --debug
18:48:22 DEBUG POST /api/2.0/dbfs/put
> <io.Reader>
< HTTP/2.0 200 OK
< {} pid=97432 sdk=true

[eng-dev-ecosystem-bot]

Test Details: go/deco-tests/12129581169

[shreyas-goenka]

@denik We can skip proper automated benchmarks here. DBFS as a platform feature is almost deprecated and users are highly encouraged to use UC Volumes.

The legacy Databricks CLI used the PUT API (https://github.com/databricks/databricks-cli), but this new CLI does not, so this PR is really meant to handle that regression.

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 1951
• Commit SHA: e9b0afb337ec152a79235449a4baec9322b6c96c

Checks will be approved automatically on success.

[pietern]

This new approach really belongs in the SDK.

There is an existing interface for dealing with DBFS files (that is called here).

The details of streaming a file or doing a single put call can be abstracted there and surfaced here with a dedicated FileMode to indicate whether it should be a single call or multiple calls.

The size of a file can be retrieved through the io.Seeker interface.

The change here should really be limited to determining the file mode and not the implementation.

The SDK guarantees the correctness of the implementation in either streaming or single-call mode.

[shreyas-goenka]

Given that DBFS is a service that will be deprecated, at least the public-facing part I think we should just keep it in the CLI rather than invest time to define the interfaces on the SDK side and using them here.

This PR is really meant to address the regression from the legacy Databricks CLI.

[shreyas-goenka]

Happy to move it to the SDK if you disagree.