503 error when using xarray.concat

[MatthewJA]

I'm getting a 503 error with the following server-side error log:

Traceback (most recent call last):
  File "/env/lib/python3.6/site-packages/tornado/tcpclient.py", line 143, in on_connect_done
    stream = future.result()
tornado.iostream.StreamClosedError: Stream is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/env/lib/python3.6/site-packages/tornado/web.py", line 1703, in _execute
    result = await result
  File "/env/lib/python3.6/site-packages/jupyter_server_proxy/websocket.py", line 97, in get
    return await self.http_get(*args, **kwargs)
  File "/env/lib/python3.6/site-packages/jupyter_server_proxy/handlers.py", line 359, in http_get
    return await self.proxy(port, proxied_path)
  File "/env/lib/python3.6/site-packages/jupyter_server_proxy/handlers.py", line 225, in proxy
    response = await client.fetch(req, raise_error=False)
  File "/env/lib/python3.6/site-packages/tornado/iostream.py", line 1226, in connect
    self.socket.connect(address)
OSError: [Errno 99] Cannot assign requested address
[I 2020-08-24 07:09:09.191 SingleUserNotebookApp kernelmanager:179] Kernel started: 33c4e177-ff22-4b83-915c-6af0374e1a54, name: python3

It occurs only when I use xarray.concat on a reasonably large dataset. I'm using the XL server and expect to have sufficient memory to store the whole thing.

[whatnick]

Possibly related to pod status not arriving in Hub cleanly : jupyterhub/kubespawner#233 . Needs further work outside of DEA scope in JupyterHub upstream.

[MatthewJA]

concat produced a ~30-60 GiB array, but spiked to 131 GB during operation. On the XXL sandbox this was fine and no 503 error was observed.

[whatnick]

@MatthewJA can we resolve this ?

[robbibt]

@MatthewJA Just a heads-up: xr.concat is an extremely memory hungry function which will typically cause memory spikes twice the total memory of its inputs as it combines them... there's often not much alternative to using it (until recently it was one of the main problems with the load_ard function), but if you're using it for large datasets you might want to make sure all your data types are as efficient as possible (e.g. using float32 instead of float64, or ideally int16s)