Ensure download destination directory exists

[rlaphoenix]

Otherwise, a FileNotFound exception will be raised on the open() call.
This issue may be somewhat related to Pull Request #23 I'm not sure.

It wasn't as simple as making the env.srcdir, as the env.remote_images[src] typically contains at least one folder stem. e.g., '_video_thumbnail/657905289.jpg'. This is why it takes the full dst, gets its parent directory, and makes the directory of that including its parents' leafs.

I've personally encountered this on readthedocs in a poetry build environment.
I can confirm the rtd environment did already have env.srcdir made, but not the further _video_thumbnail/ folder from env.remote_images[src].

[jonascj]

@rlaphoenix Thank you for contributing this and at the same time raising an issue I wasn't aware of.

Would you mind contributing a minimal working example which replicates the error?

[rlaphoenix]

Hi, sorry for the wait. I've made a minimal reproducible example repository here: https://github.com/rlaphoenix/rtd-dir-reproducible-example
(https://rtd-dir-reproducible-example.readthedocs.io/en/latest/)

It seems the root of the issue may actually be caused by sphinxcontrib-youtube 1.2.0 (the new thumbnail feature it has).

You can see I progressively added further advanced usage of sphinxcontrib-images and it built correctly up until I added sphinxcontrib-youtube 1.2.0.

The build failed on rtd at the final stage:

/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/envs/latest/bin/python -m sphinx -T -E -b html -d _build/doctrees -D language=en . _build/html

Log of the call (long)

Running Sphinx v4.5.0
loading translations [en]... done
making output directory... done
Initiated sphinxcontrib-images backend: sphinxcontrib_images_lightbox2.lightbox2:LightBox2
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index

Downloading remote images...[ 20%] https://www.lipsum.com/images/banners/black_234x60.gif
https://www.lipsum.com/images/banners/black_234x60.gif -> /home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/_images/7843118d28263d43152f8e1380b7a9c50c0a38dd (downloading)
Downloading remote images...[ 40%] https://www.lipsum.com/images/banners/grey_234x60.gif
https://www.lipsum.com/images/banners/grey_234x60.gif -> /home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/_images/c419a2ad7676d87e6fddae63ce7b3e2914908948 (downloading)
Downloading remote images...[ 60%] https://www.lipsum.com/images/banners/white_234x60.gif
https://www.lipsum.com/images/banners/white_234x60.gif -> /home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/_images/24e5061ca764d033cde60a7b0c29ed6fceeb9865 (downloading)
Downloading remote images...[ 80%] https://vumbnail.com/657905289.jpg
https://vumbnail.com/657905289.jpg -> /home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/_video_thumbnail/657905289.jpg (downloading)
/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/index.rst:: WARNING: image file not readable: _images/7843118d28263d43152f8e1380b7a9c50c0a38dd
/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/index.rst:: WARNING: image file not readable: _images/c419a2ad7676d87e6fddae63ce7b3e2914908948
/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/index.rst:: WARNING: image file not readable: _images/24e5061ca764d033cde60a7b0c29ed6fceeb9865

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/envs/latest/lib/python3.8/site-packages/sphinx/events.py", line 94, in emit
results.append(listener.handler(self.app, *args))
File "/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/envs/latest/lib/python3.8/site-packages/sphinxcontrib/images.py", line 256, in download_images
with open(dst, 'wb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/_video_thumbnail/657905289.jpg'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/envs/latest/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
app.build(args.force_all, filenames)
File "/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/envs/latest/lib/python3.8/site-packages/sphinx/application.py", line 330, in build
self.builder.build_update()
File "/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/envs/latest/lib/python3.8/site-packages/sphinx/builders/init.py", line 286, in build_update
self.build(to_build,
File "/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/envs/latest/lib/python3.8/site-packages/sphinx/builders/init.py", line 300, in build
updated_docnames = set(self.read())
File "/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/envs/latest/lib/python3.8/site-packages/sphinx/builders/init.py", line 413, in read
for retval in self.events.emit('env-updated', self.env):
File "/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/envs/latest/lib/python3.8/site-packages/sphinx/events.py", line 102, in emit
raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function download_images at 0x7f94ce18e820> for event 'env-updated' threw an exception (exception: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/_video_thumbnail/657905289.jpg')

Extension error (sphinxcontrib.images):
Handler <function download_images at 0x7f94ce18e820> for event 'env-updated' threw an exception (exception: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/_video_thumbnail/657905289.jpg')

[rlaphoenix]

You can see in the log it made a download of the thumbnail to a _video_thumbnail directory, yet normally it saves to _static, which was registered in the config.

Downloading remote images...[ 60%] https://www.lipsum.com/images/banners/white_234x60.gif
https://www.lipsum.com/images/banners/white_234x60.gif -> /home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/_images/24e5061ca764d033cde60a7b0c29ed6fceeb9865 (downloading)
Downloading remote images...[ 80%] https://vumbnail.com/657905289.jpg
https://vumbnail.com/657905289.jpg -> /home/docs/checkouts/readthedocs.org/user_builds/rtd-dir-reproducible-example/checkouts/latest/docs/_video_thumbnail/657905289.jpg (downloading)

[jonascj]

@rlaphoenix Thank you for the example. I've studied it and the behavior of the build.

I find the same thing as you, with sphinxcontrib-youtube added things fail with the current version 0.9.4 of sphinxcontrib-images (sphinx 5.1.1 and sphinxcontrib-youtube 1.2.0, likely with other version combinations as well).

The error I get, when building with 'make html' is this:

Extension error (sphinxcontrib.images):
Handler <function download_images at 0x7f346e5a85e0> for event 'env-updated' threw an exception (exception: [Errno 2] No such file or directory: 'contrib-images-31/rtd-dir-reproducible-example/docs/_video_thumbnail/657905289.jpg')

And what I do not after looking at things this afternoon is why this extension should process files in _video_thumbnail/. That directory has to do with sphinxcontrib-youtube, not sphinxcontrib-images.

So currently I can not shake the feeling that this extension should ignore those files but do not. Making an extra directory (as you PR#31 does) might very well keep the extension from failing, but I'm not convinced it is the right place to fix the error.

I'll look at it again later.

[12rambau]

Hi everyone, I'm just passing by for some context. I'm responsible for the conflict between sphinxcontrib-youtube and sphinxcontrib-image as I inspired myself from this lib to download image thumbnails for youtube and vimeo videos.

Normally this conflict is solved since sphinx-contrib/youtube#40 if it is of any help. I'll listen to this thread if any further help is needed.

[rlaphoenix]

Yeah I can confirm it's now fixed by sphinx-contrib/youtube#40, now we are just waiting for a new release.