Leftover socket file interferes with startup when wait-for-db initContainer is enabled #295
Labels
bug
Something isn't working
Comments
rainest
pushed a commit
that referenced
this issue
Feb 25, 2021
#295 requires a workaround that behaves differently depending on whether a DB-backed instance has or does not have stream listens configured. At the end of wait-for-db, this workaround removes the problem socket file if it exists, or does nothing if it does not. test5-values.yaml is a new DB-backed test that does not include stream listens, as the only existing DB-backed test did include stream listens.
rainest
pushed a commit
that referenced
this issue
Feb 25, 2021
#295 requires a workaround that behaves differently depending on whether a DB-backed instance has or does not have stream listens configured. At the end of wait-for-db, this workaround removes the problem socket file if it exists, or does nothing if it does not. test5-values.yaml is a new DB-backed test that does not include stream listens, as the only existing DB-backed test did include stream listens.
rainest
pushed a commit
that referenced
this issue
Feb 26, 2021
#295 requires a workaround that behaves differently depending on whether a DB-backed instance has or does not have stream listens configured. At the end of wait-for-db, this workaround removes the problem socket file if it exists, or does nothing if it does not. test5-values.yaml is a new DB-backed test that does not include stream listens, as the only existing DB-backed test did include stream listens.
rainest
pushed a commit
that referenced
this issue
Mar 1, 2021
#295 requires a workaround that behaves differently depending on whether a DB-backed instance has or does not have stream listens configured. At the end of wait-for-db, this workaround removes the problem socket file if it exists, or does nothing if it does not. test5-values.yaml is a new DB-backed test that does not include stream listens, as the only existing DB-backed test did include stream listens.
|
Per the latest comment upstream, Kong 2.4 will fix this. We may be a bit stuck though, since removing the workaround would break 2.3 compatibility, and it seems a bit early to do that. |
ubergesundheit
pushed a commit
to giantswarm/kong-app
that referenced
this issue
Jun 14, 2021
Kong/charts#295 requires a workaround that behaves differently depending on whether a DB-backed instance has or does not have stream listens configured. At the end of wait-for-db, this workaround removes the problem socket file if it exists, or does nothing if it does not. test5-values.yaml is a new DB-backed test that does not include stream listens, as the only existing DB-backed test did include stream listens.
rainest
pushed a commit
that referenced
this issue
Dec 1, 2021
Use a temporary prefix directory for wait-for-db instead of removing problem files. For unclear reasons, versions of Kong that should include https://trac.nginx.org/nginx/ticket/753 are still leaving old socket files behind in the prefix after wait-for-db completes. Additional sockets in newer Kong versions result in a new variant of #295, preventing startup when using Postgres. Since we don't actually need anything wait-for-db writes to the prefix, we can write to a temporary prefix instead to avoid this class of issue entirely.
rainest
pushed a commit
that referenced
this issue
Dec 1, 2021
Use a temporary prefix directory for wait-for-db instead of removing problem files. For unclear reasons, versions of Kong that should include https://trac.nginx.org/nginx/ticket/753 are still leaving old socket files behind in the prefix after wait-for-db completes. Additional sockets in newer Kong versions result in a new variant of #295, preventing startup when using Postgres. Since we don't actually need anything wait-for-db writes to the prefix, we can write to a temporary prefix instead to avoid this class of issue entirely.
|
Somehow, we still experience this even with versions that should include the fix in https://trac.nginx.org/nginx/ticket/753 aka nginx/nginx@9c3ac44: newer versions will fail without the workaround. Not sure why sockets are still left behind by the init container, since manually SIGQUITing a process does not leave these around anymore. It's possible that the init container is getting SIGKILLed for some reason. |
2 tasks
rainest
pushed a commit
that referenced
this issue
Dec 1, 2021
Use a temporary prefix directory for wait-for-db instead of removing problem files. For unclear reasons, versions of Kong that should include https://trac.nginx.org/nginx/ticket/753 are still leaving old socket files behind in the prefix after wait-for-db completes. Additional sockets in newer Kong versions result in a new variant of #295, preventing startup when using Postgres. Since we don't actually need anything wait-for-db writes to the prefix, we can write to a temporary prefix instead to avoid this class of issue entirely.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The Kong container within our Deployment uses the same prefix emptyDir volume as its wait-for-db initContainer.
Because of Kong/kong#6827, wait-for-db will leave a stream_rpc.sock file in the prefix, which prevents the Kong container from starting.
We can work around this by manually removing the file at the end of wait-for-db, but want to remove the workaround once the underlying bug is fixed.
The text was updated successfully, but these errors were encountered: