Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 container sometimes doesn't resolve from PHP container #761

Open
1 task
mikelittle opened this issue Nov 27, 2024 · 8 comments
Open
1 task

S3 container sometimes doesn't resolve from PHP container #761

mikelittle opened this issue Nov 27, 2024 · 8 comments
Labels
bug Existing functionality isn't behaving as expected developer advocacy Developer Advocacy related tasks

Comments

@mikelittle
Copy link
Contributor

Sometimes, whether from a brand-new stack or an existing stack, uploading images results in an error that boils down to the PHP container being unable to resolve the MinIO s3 emulation container.

Here's a brain dump of notes from a brief investigation into the problem. Note, sometimes the S3 container does resolve, but the Tachyon one doesn't. This has only occurred on one machine. A previous macBook does not exhibit the problem (but then my current one didn't used to either). It doesn't occur (for me) on Linux (Ubuntu 24.04).

  • Found fatal error uploading an image as author
qatest-php  | [07-Aug-2024 09:35:43 UTC] PHP Warning:  Uncaught exception 'Aws\S3\Exception\S3Exception' with message 'Error executing "ListObjectsV2" on "https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777"; AWS HTTP error: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777'
qatest-php  |
qatest-php  | GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777 in /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:275
qatest-php  | Stack trace:
qatest-php  | #0 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(205): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array)
qatest-php  | #1 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(157): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
qatest-php  | #2 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(267): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
qatest-php  | #3 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(180): GuzzleHttp\Handler\CurlMultiHandler->processMessages()
qatest-php  | #4 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(206): GuzzleHttp\Handler\CurlMultiHandler->tick()
qatest-php  | #5 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
qatest-php  | #6 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn()
qatest-php  | #7 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #8 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #9 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #10 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #11 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #12 /usr/src/app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
qatest-php  | #13 /usr/src/app/vendor/aws/aws-sdk-php/src/ResultPaginator.php(138): Aws\AwsClient->execute(Object(Aws\Command))
qatest-php  | #14 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(52): Aws\ResultPaginator->valid()
qatest-php  | #15 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(69): Aws\map(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #16 [internal function]: Aws\flatmap(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #17 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-stream-wrapper.php(695): Generator->valid()
qatest-php  | #18 [internal function]: S3_Uploads\Stream_Wrapper->dir_readdir()
qatest-php  | #19 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-plugin.php(676): scandir('s3://s3-qatest/...')
qatest-php  | #20 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): S3_Uploads\Plugin->get_files_for_unique_filename_file_list(NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #21 /usr/src/app/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters(NULL, Array)
qatest-php  | #22 /usr/src/app/wordpress/wp-includes/functions.php(2667): apply_filters('pre_wp_unique_f...', NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #23 /usr/src/app/wordpress/wp-admin/includes/file.php(984): wp_unique_filename('s3://s3-qatest/...', 'zed1-img_1777.j...', NULL)
qatest-php  | #24 /usr/src/app/wordpress/wp-admin/includes/file.php(1105): _wp_handle_upload(Array, Array, '2024-08-07 09:3...', 'wp_handle_uploa...')
qatest-php  | #25 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(1322): wp_handle_upload(Array, Array, '2024-08-07 09:3...')
qatest-php  | #26 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(269): WP_REST_Attachments_Controller->upload_from_file(Array, Array, '2024-08-07 09:3...')
qatest-php  | #27 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(158): WP_REST_Attachments_Controller->insert_attachment(Object(WP_REST_Request))
qatest-php  | #28 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1230): WP_REST_Attachments_Controller->create_item(Object(WP_REST_Request))
qatest-php  | #29 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1063): WP_REST_Server->respond_to_request(Object(WP_REST_Request), '/wp/v2/media', Array, NULL)
qatest-php  | #30 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(439): WP_REST_Server->dispatch(Object(WP_REST_Request))
qatest-php  | #31 /usr/src/app/wordpress/wp-includes/rest-api.php(420): WP_REST_Server->serve_request('/wp/v2/media')
qatest-php  | #32 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): rest_api_loaded(Object(WP))
qatest-php  | #33 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters(false, Array)
qatest-php  | #34 /usr/src/app/wordpress/wp-includes/plugin.php(565): WP_Hook->do_action(Array)
qatest-php  | #35 /usr/src/app/wordpress/wp-includes/class-wp.php(418): do_action_ref_array('parse_request', Array)
qatest-php  | #36 /usr/src/app/wordpress/wp-includes/class-wp.php(813): WP->parse_request('')
qatest-php  | #37 /usr/src/app/wordpress/wp-includes/functions.php(1336): WP->main('')
qatest-php  | #38 /usr/src/app/wordpress/wp-blog-header.php(16): wp()
qatest-php  | #39 /usr/src/app/index.php(21): require('/usr/src/app/wo...')
qatest-php  | #40 {main}
qatest-php  |
qatest-php  | Next Aws\S3\Exception\S3Exception: Error executing "ListObjectsV2" on "https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777"; AWS HTTP error: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777 in /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:196
qatest-php  | Stack trace:
qatest-php  | #0 /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php(97): Aws\WrappedHttpHandler->parseError(Array, Object(GuzzleHttp\Psr7\Request), Object(Aws\Command), Array)
qatest-php  | #1 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(209): Aws\WrappedHttpHandler->Aws\{closure}(Array)
qatest-php  | #2 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(174): GuzzleHttp\Promise\Promise::callHandler(2, Array, NULL)
qatest-php  | #3 /usr/src/app/vendor/guzzlehttp/promises/src/RejectedPromise.php(49): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}(Array)
qatest-php  | #4 /usr/src/app/vendor/guzzlehttp/promises/src/TaskQueue.php(52): GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}()
qatest-php  | #5 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(167): GuzzleHttp\Promise\TaskQueue->run()
qatest-php  | #6 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(206): GuzzleHttp\Handler\CurlMultiHandler->tick()
qatest-php  | #7 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
qatest-php  | #8 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn()
qatest-php  | #9 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #10 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #11 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #12 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #13 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #14 /usr/src/app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
qatest-php  | #15 /usr/src/app/vendor/aws/aws-sdk-php/src/ResultPaginator.php(138): Aws\AwsClient->execute(Object(Aws\Command))
qatest-php  | #16 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(52): Aws\ResultPaginator->valid()
qatest-php  | #17 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(69): Aws\map(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #18 [internal function]: Aws\flatmap(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #19 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-stream-wrapper.php(695): Generator->valid()
qatest-php  | #20 [internal function]: S3_Uploads\Stream_Wrapper->dir_readdir()
qatest-php  | #21 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-plugin.php(676): scandir('s3://s3-qatest/...')
qatest-php  | #22 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): S3_Uploads\Plugin->get_files_for_unique_filename_file_list(NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #23 /usr/src/app/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters(NULL, Array)
qatest-php  | #24 /usr/src/app/wordpress/wp-includes/functions.php(2667): apply_filters('pre_wp_unique_f...', NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #25 /usr/src/app/wordpress/wp-admin/includes/file.php(984): wp_unique_filename('s3://s3-qatest/...', 'zed1-img_1777.j...', NULL)
qatest-php  | #26 /usr/src/app/wordpress/wp-admin/includes/file.php(1105): _wp_handle_upload(Array, Array, '2024-08-07 09:3...', 'wp_handle_uploa...')
qatest-php  | #27 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(1322): wp_handle_upload(Array, Array, '2024-08-07 09:3...')
qatest-php  | #28 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(269): WP_REST_Attachments_Controller->upload_from_file(Array, Array, '2024-08-07 09:3...')
qatest-php  | #29 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(158): WP_REST_Attachments_Controller->insert_attachment(Object(WP_REST_Request))
qatest-php  | #30 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1230): WP_REST_Attachments_Controller->create_item(Object(WP_REST_Request))
qatest-php  | #31 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1063): WP_REST_Server->respond_to_request(Object(WP_REST_Request), '/wp/v2/media', Array, NULL)
qatest-php  | #32 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(439): WP_REST_Server->dispatch(Object(WP_REST_Request))
qatest-php  | #33 /usr/src/app/wordpress/wp-includes/rest-api.php(420): WP_REST_Server->serve_request('/wp/v2/media')
qatest-php  | #34 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): rest_api_loaded(Object(WP))
qatest-php  | #35 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters(false, Array)
qatest-php  | #36 /usr/src/app/wordpress/wp-includes/plugin.php(565): WP_Hook->do_action(Array)
qatest-php  | #37 /usr/src/app/wordpress/wp-includes/class-wp.php(418): do_action_ref_array('parse_request', Array)
qatest-php  | #38 /usr/src/app/wordpress/wp-includes/class-wp.php(813): WP->parse_request('')
qatest-php  | #39 /usr/src/app/wordpress/wp-includes/functions.php(1336): WP->main('')
qatest-php  | #40 /usr/src/app/wordpress/wp-blog-header.php(16): wp()
qatest-php  | #41 /usr/src/app/index.php(21): require('/usr/src/app/wo...')
qatest-php  | #42 {main}
qatest-php  |   thrown in /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php on line 196
qatest-php  | [07-Aug-2024 09:35:43 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 41943040 bytes) in /usr/src/app/vendor/humanmade/s3-uploads/inc/class-plugin.php on line 676
  • Further testing.
  • Tried connecting to S3 from the docker shell - didn't work. Same from v19
$ composer server ssh
www-data@a7c1f17b35b0:/usr/src/app$ curl "https://altis19.altis.dev/s3-altis19/"
curl: (7) Failed to connect to altis19.altis.dev port 443 after 42 ms: Couldn't connect to server
www-data@a7c1f17b35b0:/usr/src/app$
  • testing inside qa-test v20 gave this
www-data@08674edaddfe:/usr/src/app$ curl -if 'http://s3:9000/minio/health/live'
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 0
Content-Security-Policy: block-all-mixed-content
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Bucket-Region: us-east-1
X-Amz-Request-Id: 17E97BFE1103A8B1
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Date: Wed, 07 Aug 2024 15:31:03 GMT

www-data@08674edaddfe:/usr/src/app$
  • testing qatest v20 again. FQDN for S3 does not work
www-data@08674edaddfe:/usr/src/app$ curl "http://s3-qatest.altis.dev:9000/s3-qatest/"
curl: (7) Failed to connect to s3-qatest.altis.dev port 9000 after 4 ms: Couldn't connect to server
  • but short name does work
www-data@08674edaddfe:/usr/src/app$ curl "http://s3:9000/s3-qatest/"
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>s3-qatest</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter></Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>www-data@08674edaddfe:/usr/src/app$
  • same problem with qatest. Also, s3-uploads command broken too
$ composer server cli -- s3-uploads upload-directory tmp uploads
Error: Error executing "PutObject" on "https://s3-qatest.altis.dev/s3-qatest/uploads/zed1-img_1922.jpg"; 
  AWS HTTP error: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: 
  Couldn\'t connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/uploads/zed1-img_1922.jpg

Acceptance criteria:

  • ...
@mikelittle mikelittle added the bug Existing functionality isn't behaving as expected label Nov 27, 2024
@roborourke
Copy link
Contributor

@mikelittle I wouldn't expect the FQDN to work within the containers so I think that's ok. There might be a configuration issue then, by which WP is trying to use the FQDN to interact with S3 rather than the container link name and port (the short name). The FQDN is purely for use via the host. S3 Uploads should be using the short URL or s3.localhost to speak to Minio from the PHP container, if it isn't then that's the problem.

Looks like it should be available at s3.localhost from the PHP container too. I recall the the S3 Uploads plugin and AWS SDK had some odd behaviour around how it resolve S3 URLs, it's possible this issue has crept in with an AWS SDK update.

@mikelittle
Copy link
Contributor Author

Ah, OK. That was my expectation too, but I couldn't figure out where it was picking up the FQDN from. I assumed it had always used the full name and worked somehow before.
If it might be something an AWS SDK update has done, that gives me a pointer to track this down. Still, odd that it works on a brand-new installation on an old Mac, and continues to work on new installations on linux VMs.

@roborourke
Copy link
Contributor

It might be worth tinkering with this code a bit

https://github.com/humanmade/altis-local-server/blob/master/inc/namespace.php#L28-L37

I would try adding some breakpoints with xdebug to see which URL its trying to use. It might arrive at the FQDN as part of some attempts to retry or try different versions of the S3 endpoint.

I can't actually remember why the S3 endpoint is set to the FQDN, I thought it should be the internal container link URL e.g. s3 or s3.localhost. It was a fairly complex piece to get working. Odd that it could be inconsistent like this.

Maybe could factor in docker engine versions? Are they same between Mac and Linux?

@abhishek-kaushik is this happening on all your Altis instances or only some? What about on a clean install?

@missjwo missjwo added the developer advocacy Developer Advocacy related tasks label Dec 16, 2024
@abhishek-kaushik
Copy link
Member

Yes @roborourke I am facing this issue on all of my stacks

@ferschubert-hm
Copy link
Contributor

In addition to the issue reported here the image in use is an older version:

Fetching s3 logs...
estudo-s3  |
estudo-s3  |  You are running an older version of MinIO released 3 years ago
estudo-s3  |  Update: Run `mc admin update`
estudo-s3  |
estudo-s3  |
estudo-s3  | API: http://172.21.0.7:9000  http://172.19.0.4:9000  http://127.0.0.1:9000
estudo-s3  |
estudo-s3  | Console: http://172.21.0.7:9001 http://172.19.0.4:9001 http://127.0.0.1:9001
estudo-s3  |
estudo-s3  | Documentation: https://docs.min.io

@ferschubert-hm
Copy link
Contributor

Could not reproduce the error, actions performed:

  1. New local server env created
  2. composer create-project altis/skeleton estudo
  3. composer server start
  4. Added images to the library as superadmin -> ok
  5. Added images to a post as author -> ok

Additional detail:

  • Altis version: 21
  • composer server status output:

image.png

@ferschubert-hm
Copy link
Contributor

@abhishek-kaushik could you provide some info on the Altis version and a an output for composer server status? Thanks!

@mikelittle
Copy link
Contributor Author

Remarkably, this has disappeared for me. With the same repository still checked out. Whilst there have been many package updates and many cycles of composer server start/destroy, I think the most significant change is that I changed my DNS over the holiday break.
I had previously been using Tailscale which does takes over DNS lookups before passing them on as normal. Note: I used Tailscale for over 18 months before this problem started happening. Over the break, I finally implemented a couple of pi-hole servers and directed both my home network and Tailscale to query the pi-hole servers first. Now the S3 container lookup does not fail.
@abhishek-kaushik Do you have any kind of VPN (standard or mesh), system level ad-blocker, or other thing that may insert itself in the DNS lookup process?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Existing functionality isn't behaving as expected developer advocacy Developer Advocacy related tasks
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@roborourke @mikelittle @missjwo @abhishek-kaushik @ferschubert-hm