[BUG] Duplicati docker container taking 13GB of space...

[825i]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Duplicati version: 2.0.7.1_beta_2023-05-25

I have looked for reasons for this but cannot find any.

After installing Duplicati and running a single backup to OneDrive Cloud (of about 576GB in size).

My container grew to 13GB in size:

I have a flat limit of 25GB for my docker images. Even with a bunch of other services, nothing takes even 10% as much space as Duplicati is.

Checking the container settings via Unraid, it should not be saving anything "into" the image that increases the size of the container. The dir for temp files is on my storage array (not inside the /appdata folder).

Due to the lack of free space, other containers I have are now failing to load.

Expected Behavior

The Duplicati container should be no more than a few hundred megabytes in size.

Steps To Reproduce

1. Install Duplicati via Unraid
2. Create a backup job to OneDrive
3. Select the files to upload
4. Files are uploaded after a few hours
5. Next day the Docker container is 13GB in size

Environment

- OS: Unraid 6.12.5
- How docker service was installed: via Unraid

CPU architecture

x86-64

Docker creation

https://i.imgur.com/xoEfZKw.png

Container logs

at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.JSONWebHelper.GetResponse (Duplicati.Library.Utility.AsyncHttpRequest req, System.Object requestdata) [0x000c9] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T] (Duplicati.Library.Utility.AsyncHttpRequest req, System.Object requestdata) [0x00000] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T] (System.Net.HttpWebRequest req, System.Object requestdata) [0x00007] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.OAuthHelper.GetTokenResponse[T] () [0x00040] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.OAuthHelper.get_AccessToken () [0x0002f] in <430fb651568b4158b0aeeaccca31dfe8>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.OAuthHelper.get_AccessToken () [0x00171] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.OAuthHelper.CreateRequest (System.String url, System.String method, System.Boolean noAuthorization) [0x00027] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.OAuthHelper.CreateRequest (System.String url, System.String method) [0x00000] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T] (System.String url, System.Object requestdata, System.String method) [0x00020] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.Backend.GoogleCloudStorage.GoogleCloudStorage+<>c__DisplayClass16_0.<List>b__0 () [0x00000] in <5115d0feca4246e4811e886c5ac3d194>:0 
  at Duplicati.Library.Backend.GoogleCloudStorage.GoogleCloudStorage.HandleListExceptions[T] (System.Func`1[TResult] func) [0x00000] in <5115d0feca4246e4811e886c5ac3d194>:0 
  at Duplicati.Library.Backend.GoogleCloudStorage.GoogleCloudStorage+<List>d__16.MoveNext () [0x00070] in <5115d0feca4246e4811e886c5ac3d194>:0 
  at Duplicati.Library.Interface.BackendExtensions.TestList (Duplicati.Library.Interface.IBackend backend) [0x00017] in <9385304baed84a39b0ffc49ae589b75e>:0 
  at Duplicati.Library.Backend.GoogleCloudStorage.GoogleCloudStorage.Test () [0x00000] in <5115d0feca4246e4811e886c5ac3d194>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.TestConnection (System.String url, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x000b7] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x00091] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00289] in <30a34d71126b48248d040dda634ddad9>:0 
Duplicati.Library.Interface.UserInformationException: Failed to authorize using the OAuth service: Invalid authid in query. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=gcs ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.JSONWebHelper.GetResponse (Duplicati.Library.Utility.AsyncHttpRequest req, System.Object requestdata) [0x000c9] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T] (Duplicati.Library.Utility.AsyncHttpRequest req, System.Object requestdata) [0x00000] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T] (System.Net.HttpWebRequest req, System.Object requestdata) [0x00007] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.OAuthHelper.GetTokenResponse[T] () [0x00040] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.OAuthHelper.get_AccessToken () [0x0002f] in <430fb651568b4158b0aeeaccca31dfe8>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.OAuthHelper.get_AccessToken () [0x00171] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.OAuthHelper.CreateRequest (System.String url, System.String method, System.Boolean noAuthorization) [0x00027] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.OAuthHelper.CreateRequest (System.String url, System.String method) [0x00000] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T] (System.String url, System.Object requestdata, System.String method) [0x00020] in <430fb651568b4158b0aeeaccca31dfe8>:0 
  at Duplicati.Library.Backend.GoogleCloudStorage.GoogleCloudStorage+<>c__DisplayClass16_0.<List>b__0 () [0x00000] in <5115d0feca4246e4811e886c5ac3d194>:0 
  at Duplicati.Library.Backend.GoogleCloudStorage.GoogleCloudStorage.HandleListExceptions[T] (System.Func`1[TResult] func) [0x00000] in <5115d0feca4246e4811e886c5ac3d194>:0 
  at Duplicati.Library.Backend.GoogleCloudStorage.GoogleCloudStorage+<List>d__16.MoveNext () [0x00070] in <5115d0feca4246e4811e886c5ac3d194>:0 
  at Duplicati.Library.Interface.BackendExtensions.TestList (Duplicati.Library.Interface.IBackend backend) [0x00017] in <9385304baed84a39b0ffc49ae589b75e>:0 
  at Duplicati.Library.Backend.GoogleCloudStorage.GoogleCloudStorage.Test () [0x00000] in <5115d0feca4246e4811e886c5ac3d194>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.TestConnection (System.String url, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x000b7] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x00091] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00289] in <30a34d71126b48248d040dda634ddad9>:0 
System.Net.WebException: Error: NameResolutionFailure
  at System.Net.WebConnection.Connect (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00044] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x000cc] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetRequestStream (System.Int64 contentlength) [0x00068] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.Backend.WEBDAV.ListWithouExceptionCatch () [0x0004a] in <1aa1326b13664b9f92f75d7048721a2d>:0 
  at Duplicati.Library.Backend.WEBDAV.List () [0x000a5] in <1aa1326b13664b9f92f75d7048721a2d>:0 
  at Duplicati.Library.Backend.WEBDAV.Test () [0x00000] in <1aa1326b13664b9f92f75d7048721a2d>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.TestConnection (System.String url, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x000b7] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x00091] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00289] in <30a34d71126b48248d040dda634ddad9>:0 
System.Net.WebException: Error: NameResolutionFailure
  at System.Net.WebConnection.Connect (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00044] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x000cc] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetRequestStream (System.Int64 contentlength) [0x00068] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.Backend.WEBDAV.ListWithouExceptionCatch () [0x0004a] in <1aa1326b13664b9f92f75d7048721a2d>:0 
  at Duplicati.Library.Backend.WEBDAV.List () [0x000a5] in <1aa1326b13664b9f92f75d7048721a2d>:0 
  at Duplicati.Library.Backend.WEBDAV.Test () [0x00000] in <1aa1326b13664b9f92f75d7048721a2d>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.TestConnection (System.String url, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x000b7] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x00091] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00289] in <30a34d71126b48248d040dda634ddad9>:0 
System.Net.WebException: Error: SecureChannelFailure (Remote prematurely closed connection.) ---> System.IO.IOException: Remote prematurely closed connection.
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000e3] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at Mono.Net.Security.AsyncProtocolRequest.StartOperation (System.Threading.CancellationToken cancellationToken) [0x0008b] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <a8a996a78a804d888710c9e2575d78c8>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <a8a996a78a804d888710c9e2575d78c8>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <ae4a5c68f78041539efc9420d056bd22>:0 
  at Duplicati.Library.Backend.WEBDAV.CreateFolder () [0x00022] in <1aa1326b13664b9f92f75d7048721a2d>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.CreateFolder (System.String uri, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x0000c] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x0007f] in <30a34d71126b48248d040dda634ddad9>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00289] in <30a34d71126b48248d040dda634ddad9>:0

[github-actions]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[j0nnymoe]

You've likely configured a backup job to store files within the container that aren't getting removed.

[825i]

Thanks for the mega fast reply.

You can see it has indeed backed everything up to the Cloud (all 500+GB) so I don't get why it would be storing files inside the container (except for the DB)?

This is the only job I have:

It should not be backing up anything locally at all. Everything should be sent to the cloud and that /source/ folder is another share outside of the container. It's my //TOWER/storage location.

I just noticed that I need to change the backup retention. But that should be on the cloud host side. Not on the local side.

Can you tell me where/how it's storing things inside the container? It only has max 13GB of space (out of my 25GB total limit).

[j0nnymoe]

Temporary files maybe, as duplicate encrypts / zips the files before they get sent.

[825i]

Temporary files maybe, as duplicate encrypts / zips the files before they get sent.

Any way to remove these automatically so this doesn't happen?

[j0nnymoe]

You would need to check duplicati documentation. It's only a guess mind, you need to look through the config to make sure there's no rogue location that it's dumping files inside the container.

[825i]

Ok I found these. Looks like this is what's eating space. What are these? They are in /tmp/

Why do they need to be 1.9GB each?

[825i]

Whatever they are, I removed them all.

Size looks a lot more sane now. I'll see if this happens again in the future:

I'll close this for now.

[j0nnymoe]

Because you probably set the chunk size to that.

[825i]

I didn't set any "chunk" size anywhere. The file storage size is meant to be 50MB on the cloud end. As for the rest of the settings, you saw them all. Everything else is just set to default settings. I still don't understand why it would need to create temporary 1.9GB files inside the container for anything. Other than the DB it shouldn't be storing anything in the container. It even has use of a /tmp/ directory outside of the container for all the encryption etc.

Anyway. Thanks for helping me narrow down the issue.

[825i]

Ok it seems this happens when enabling the:

enable-http-buffering option. It's supposed to potentially speed up performance but for whatever reason it starts making these 1.9GB files in the /tmp location inside the image. It's disabled by default though, so I am just not going to use it.

No problems so far with it off. I have 3 jobs running to 2 different clouds and the largest file in the /tmp loation is 4.8MB.