** (MatchError) no match of right hand side value: {:error, :enoent} 360

[michaelalang]

---

Describe the bug
Is there any possibility to increase the debug level in asciinema-server ?
I am hitting following issue and cannot identify, what asciinema wants ...

11:00:00.354 [info] {"args":{},"attempt":1,"event":"job:start","id":17,"max_attempts":20,"meta":{},"queue":"default","source":"oban","system_time":1728471600354472616,"tags":[],"worker":"Asciinema.Streaming.GC"}
11:00:00.354 [info] {"args":{},"attempt":1,"event":"job:start","id":16,"max_attempts":20,"meta":{},"queue":"default","source":"oban","system_time":1728471600354419328,"tags":[],"worker":"Asciinema.GC"}
11:00:00.358 [info] {"args":{},"attempt":1,"duration":174,"event":"job:stop","id":16,"max_attempts":20,"meta":{},"queue":"default","queue_time":14875,"source":"oban","state":"success","tags":[],"worker":"Asciinema.GC"}
11:00:00.361 [info] {"args":{},"attempt":1,"duration":499,"event":"job:stop","id":17,"max_attempts":20,"meta":{},"queue":"default","queue_time":14875,"source":"oban","state":"success","tags":[],"worker":"Asciinema.Streaming.GC"}
11:00:33.428 request_id=F_zDyvLYowjCtksAAAtB [info] POST /api/asciicasts
11:00:33.639 request_id=F_zDyvLYowjCtksAAAtB [info] Sent 500 in 211ms
11:00:33.639 [error] #PID<0.3048.0> running AsciinemaWeb.Endpoint (connection #PID<0.3047.0>, stream id 1) terminated
Server: asciinema.apps.example.com:80 (http)
Request: POST /api/asciicasts
** (exit) an exception was raised:
** (MatchError) no match of right hand side value: {:error, :enoent}
(asciinema 1.0.0) lib/asciinema/recordings.ex:300: Asciinema.Recordings.save_file/2
(asciinema 1.0.0) lib/asciinema/recordings.ex:287: anonymous fn/2 in Asciinema.Recordings.do_create_asciicast/2
(ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:1358: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection 2.6.0) lib/db_connection.ex:1710: DBConnection.run_transaction/4
(asciinema 1.0.0) lib/asciinema/recordings.ex:277: Asciinema.Recordings.do_create_asciicast/2
(asciinema 1.0.0) lib/asciinema/recordings.ex:144: Asciinema.Recordings.create_asciicast/3
(asciinema 1.0.0) lib/asciinema_web/controllers/api/recording_controller.ex:12: AsciinemaWeb.Api.RecordingController.create/2
(asciinema 1.0.0) lib/asciinema_web/controllers/api/recording_controller.ex:1: AsciinemaWeb.Api.RecordingController.action/2

To Reproduce
Steps to reproduce the behavior:

1. deploy asciinema-server in k8s
2. use CNPG as database provider
3. create/login with a user
4. authorize your endpoint device for uploading
5. upload the cast
6. See error

Expected behavior
output of more hintful messages to identify what should be fixed

Versions:

• ghcr.io/asciinema/asciinema-server:latest
• OCP 4.14
• Browser Chrome 124.0.6367.118
• asciinema server: Running 20170920135754

[michaelalang]

meanwhile I identified the culrpit, the pod requires to run with privileges so the moment one add's an privileged SCC to the ServiceAccount the deployment starts working as expected.

I was able to identify that by reading the Database oban_jobs table.error column which is pointing towards the source

{"{\"at\": \"2024-10-09T10:46:28.340170Z\", \"error\": \"** (MatchError) no match of right hand side value: {:error, :enoent}\\n    (asciinema 1.0.0) lib/asciinema/recordings.ex:300: Asciinema.Recordings.save_file/2\\n 
...

so I verified if the default file storage space needs to be available but that is also not the case, which means (most likely) temporary upload to S3 space not accessible, so I added the privileged SCC.

It would be really nice to get asciinema-server running without the need to escalate privileges.

[ku1ik]

This error comes from Asciinema.Recordings.save_file which is trying to save the file in the recordings directory on the filesystem. This operation does 2 things: 1. creates a nested directory structure for the file (equivalent of mkdir -p), 2. writes the file to that directory. It's hard to tell which of those two operations failed here, but either way, the filesystem must be writable and the container must be able to create new directories within the storage path (see https://docs.asciinema.org/manual/server/self-hosting/configuration/#local-filesystem). asciinema server doesn't require any special privileges here, so maybe the problem is on the k8s side?

[michaelalang]

@ku1ik thanks for your reply... from my POV it looks not to be an issue of the kubernetes deployment per-se as the container image itself expects to run as root ...

$ podman run -ti --rm --name asciinema --entrypoint /usr/bin/id ghcr.io/asciinema/asciinema-server:latest
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)

that might be coming from the need to have privileges when installing the dependencies as well as when changing the ownership of /opt/app both would not work with unprivileged users in the container ...

$ podman inspect ghcr.io/asciinema/asciinema-server:latest | jq -r '.[0].History[]|.created_by' 
/bin/sh -c #(nop) ADD file:7625ddfd589fb824ee39f1b1eb387b98f3676420ff52f26eb9d975151e889667 in / 
/bin/sh -c #(nop)  CMD ["/bin/sh"]
RUN /bin/sh -c apk add --no-cache   libstdc++   tini   bash   ca-certificates   rsvg-convert   ttf-dejavu   pngquant # buildkit
WORKDIR /opt/app
COPY /opt/app/_build/prod/rel/asciinema . # buildkit
RUN /bin/sh -c chgrp -R 0 /opt/app && chmod -R g=u /opt/app # buildkit
COPY .iex.exs . # buildkit
ENV PORT=4000 ADMIN_BIND_ALL=1 DATABASE_URL=postgresql://postgres@postgres/postgres RSVG_FONT_FAMILY=Dejavu Sans Mono PATH=/opt/app/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENTRYPOINT ["/sbin/tini" "--"]
CMD ["/opt/app/bin/server"]
EXPOSE map[4000/tcp:{}]

my suggestion is to add a line USER 1001 # or which ever id one likes
to the Dockerfile in front of the line ENTRYPOINT

[ku1ik]

Thanks for investigating further. I think we only need root in the builder image. In the final image it should be possible to use a regular user indeed.