Skip mtime checks during upload when force_overwrite_to_cloud is set (#…

…380) (#384) * Skip mtime checks during upload when force_overwrite_to_cloud is set * Lint fix * Updated HISTORY.md Co-authored-by: Daniel Oriyan <[email protected]>
drivendataorg · Dec 11, 2023 · 41aa53d · 41aa53d
1 parent c5eaf7e
commit 41aa53d
Show file tree

Hide file tree

Showing 2 changed files with 11 additions and 6 deletions.
diff --git a/HISTORY.md b/HISTORY.md
@@ -2,6 +2,7 @@
 
 ## Unreleased
 - Fix `S3Client` cleanup via `Client.__del__` when `S3Client` encounters an exception during initialization. (Issue [#372](https://github.com/drivendataorg/cloudpathlib/issues/372), PR [#373](https://github.com/drivendataorg/cloudpathlib/pull/373))
+- Skip mtime checks during upload when force_overwrite_to_cloud is set to improve upload performance. (Issue [#379](https://github.com/drivendataorg/cloudpathlib/issues/379), PR [#380](https://github.com/drivendataorg/cloudpathlib/pull/380), thanks to [@Gilthans](https://github.com/Gilthans))
 
 ## v0.16.0 (2023-10-09)
  - Add "CloudPath" as return type on `__init__` for mypy issues. ([Issue #179](https://github.com/drivendataorg/cloudpathlib/issues/179), [PR #342](https://github.com/drivendataorg/cloudpathlib/pull/342))

diff --git a/cloudpathlib/cloudpath.py b/cloudpathlib/cloudpath.py
@@ -1172,17 +1172,21 @@ def _upload_file_to_cloud(
         """Uploads file at `local_path` to the cloud if there is not a newer file
         already there.
         """
+        if force_overwrite_to_cloud:
+            # If we are overwriting no need to perform any checks, so we can save time
+            self.client._upload_file(
+                local_path,
+                self,
+            )
+            return self
+
         try:
             stats = self.stat()
         except NoStatError:
             stats = None
 
-        # if cloud does not exist or local is newer or we are overwriting, do the upload
-        if (
-            not stats  # cloud does not exist
-            or (local_path.stat().st_mtime > stats.st_mtime)
-            or force_overwrite_to_cloud
-        ):
+        # if cloud does not exist or local is newer, do the upload
+        if not stats or (local_path.stat().st_mtime > stats.st_mtime):
             self.client._upload_file(
                 local_path,
                 self,