Permission errors with docker while launching higlass-manage

[ddepierre]

Dear higlass team,

I followed instructions to install higlass-manage:
Prerequisites:

$ python3 --version
Python 3.10.6

Docker installed following https://docs.docker.com/desktop/install/ubuntu/ and I can launch Docker desktop

$ docker compose version
Docker Compose version v2.15.1

$ docker --version
Docker version 20.10.22, build 3a2c30b

$ docker version
Client: Docker Engine - Community
 Cloud integration: v1.0.29
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:04 2022
 OS/Arch:           linux/amd64
 Context:           desktop-linux
 Experimental:      true

Server: Docker Desktop 4.16.1 (95567)
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Higlass is installed with pip install higlass-manage

$ higlass-manage version
0.8.0

But when I try to launch higlass, I hav the following errors:

$ higlass-manage start
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/home/david/.local/lib/python3.10/site-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/david/.local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/home/david/.local/lib/python3.10/site-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/david/.local/lib/python3.10/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/david/.local/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/david/.local/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/david/.local/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/david/.local/bin/higlass-manage", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/higlass_manage/start.py", line 134, in start
    _start(
  File "/home/david/.local/lib/python3.10/site-packages/higlass_manage/start.py", line 177, in _start
    client = docker.from_env()
  File "/home/david/.local/lib/python3.10/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/home/david/.local/lib/python3.10/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/david/.local/lib/python3.10/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

Could you help me and tell me what did I miss ?
Thanks!
David

[pkerpedjiev]

You probably need to allow docker run as a non-root user. Have you tried this?

https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user

[ddepierre]

Ok thanks, it seems to work better but I had an other issue that seemed to come from cython:

$ higlass-manage view /tmp/sample.mcool 
state True
temp_dir: /tmp/higlass-docker
Inferred filetype: cooler
Inferred datatype: matrix
Traceback (most recent call last):
  File "/home/david/.local/bin/higlass-manage", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/higlass_manage/view.py", line 157, in view
    from higlass.client import Track, View, ViewConf
  File "/home/david/.local/lib/python3.10/site-packages/higlass/__init__.py", line 3, in <module>
    from .server import Server
  File "/home/david/.local/lib/python3.10/site-packages/higlass/server.py", line 5, in <module>
    import cytoolz as toolz
  File "/home/david/.local/lib/python3.10/site-packages/cytoolz/__init__.py", line 3, in <module>
    from .functoolz import *
ImportError: /home/david/.local/lib/python3.10/site-packages/cytoolz/functoolz.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

So I updated cython as suggested here Toblerity/Fiona#1043, which may be incompatible with cooler but it allows me to open an higlass instance and visualise the sample.cool.

 pip install --upgrade cytoolz
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: cytoolz in /home/david/.local/lib/python3.10/site-packages (0.10.1)
Collecting cytoolz
  Downloading cytoolz-0.12.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 7.7 MB/s eta 0:00:00
Requirement already satisfied: toolz>=0.8.0 in /home/david/.local/lib/python3.10/site-packages (from cytoolz) (0.12.0)
Installing collected packages: cytoolz
  Attempting uninstall: cytoolz
    Found existing installation: cytoolz 0.10.1
    Uninstalling cytoolz-0.10.1:
      Successfully uninstalled cytoolz-0.10.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cooler 0.8.11 requires cytoolz<0.11, but you have cytoolz 0.12.1 which is incompatible.
Successfully installed cytoolz-0.12.1

However, higlass-manage browse command does not seem to exist.
So I could open the app with http://localhost:8989/app, find my track in the list, but can't visualise it (.cool file, 10kb res).
I could visualise however public data listed but this would quite, even after increase docker memory limits https://www.thorsten-hans.com/limit-memory-for-docker-containers/ ( I guess it could change something but not sure, I'm super new to docker)

Also, can you confirm higlass is the right tool for a fast and smooth exploration Hi-C data in .cool or .mcool format on the fly, like IGV is for other -seq ? I was used to work with .hic file from Juicer tools, and visualisation with Juicebox is actually quite fast, even at high res while higlass seems lagging a lot.
Would it be faster running through python/jupyter notebook ?

Thanks for your support,
David

[pkerpedjiev]

So I could open the app with http://localhost:8989/app, find my track in the list, but can't visualise it (.cool file, 10kb res).

HiGlass requires a multi-res cooler file. To go from single resolution to multires is easy: cooler zoomify my.cool.

Hard to say if higlass is the right tool. It's a tool that some people like and others don't. As you said, Juicer works well as well. The main factor that affects higlass performance is the resolutions that are present in your mcool file. If you use cooler zoomify you'll get the best performance.