DOC: Add interactive notebooks to pages in the "Usage Examples" section

[agriyakhetarpal]

Description

Tip

Those interested can try it at https://pywavelets--741.org.readthedocs.build/en/741/regression/index.html

The Usage Examples section, following #728 and as requested in #737 (comment), is rendered interactive through the changes in this PR through the use of jupyterlite-sphinx and Markdown-based notebooks that are executed by MyST-NB. Please read below for a granular overview of all of these details:

Which issue does this PR solve/reference?

Addresses a part of #706

Key changes made

1. All .rst based files under doc/source/regression converted to Markdown and reformatted as notebooks
2. Jupytext frontmatter added to all of the notebooks so that they can be executed at the time of building the documentation
3. Code cells that raise an error marked with cell-based tags
4. MyST-NB is configured in conf.py
5. NotebookLite directive used from jupyterlite-sphinx to run the notebooks under WASM in the same tab (and something like Add the option to open JupyterLite window in new tab jupyterlite/jupyterlite-sphinx#165 can be added for this directive as well)
6. A custom, minimal Sphinx extension added to execute the Jupytext CLI before the sources and toctrees are read, and to selectively ignore the .ipynb-based notebooks that get generated during the process in order to keep the documentation build warning-free. Generating them at build time is for the NotebookLite directive to be able to access them and load them, since the NotebookLite directive currently does not load .md files or notebook files in other formats.
   i. I imagine this will be helpful for DOC: stats: Convert sampling tutorial to MyST-md scipy/scipy#20303 as well where it is required to load the notebooks in an interactive manner under a specific folder. The notebooks are not executed by Jupytext at the time of conversion, and therefore, based on past experiments, it takes ~10 seconds to convert 30 or so notebooks.

Additional context

This is just a pilot run of how notebook-based examples can be configured for Sphinx-based documentation websites, so there are a few corner cases that I have noticed so far:

1. The styling of the NotebookLite directive can be improved because it takes way too much of the screen's space, perhaps through a user-provided option in conf.py just like how the TryExamples directive can be configured.
2. References to the API for a package (in this case, public classes and methods for PyWavelets) are not linked to when running the notebook in its inline frame, because the notebook is not running under Sphinx and cannot access Sphinx's generated HTML pages. This can be worked around via better writing the notebooks by exploring other ways to reference pages and sections from the API (say, by adding Markdown-based headings or links, etc.).

A brief to-do list

Besides the points mentioned above, smaller tasks can be looked into:

• Hide reST-based syntax (at least the irrelevant top parts of each page) using MyST
• Improve the styling in other ways
• Provide a button to download notebooks (both .md and .ipynb)

[agriyakhetarpal]

Quite strange. The documentation is building without any problems locally...

Edit: I can reproduce locally if I delete a few of the previously generated files – this is coming from jupyterlite-sphinx. I think that Read the Docs is using a cached version of the documentation and is not purging the files properly, or it isn't allowing the subprocess to execute at the config-inited builder hook.

Edit, again: they were failing because the jupytext command was failing silently and was not converting any notebooks. Removing the hardcoded path fixed it.

[agriyakhetarpal]

I added a basic extension to clean up the generated IPython notebooks in 6bc9c60 using nbformat, works quite well. I am now exploring a method to add the directives based on #741 (comment) automatically – it requires some configuration but I feel that it can be achieved at the source-read event before the Markdown files are processed.

[agriyakhetarpal]

Processing Markdown programmatically doesn't bear good fruit – it would have been easier if we used IPyNB, and we don't plan to. I would suggest that we should merge this for now, @rgommers.

The issue with Markdown was that there isn't a clean way to add both the Jupytext frontmatter and the notebook directives and that requires some obtrusive, prone-to-failure file manipulation by processing the contents as a string (we can keep the notebook directives at the bottom and the frontmatter intact, but it's easier to provide the download buttons at the top of the page rather than at the bottom).

[agriyakhetarpal]

Weird, the local docs build and RTD both produce cleaner downloadable notebooks, but the NotebookLite directive on RTD doesn't have the cleaner notebooks (they still have the Sphinx directives). This does not happen locally. I think RTD has some caching troubles, because I removed the redundant module headings from the notebooks (such as (reg-dwt-idwt)= and so on), but they are still there on RTD.

Local build

Read the Docs

Edit: some notebooks are as expected, and some don't.

Edit 2: this is most likely the case of a cached build or some updates that don't trickle down the RTD, the logs say that all the notebooks are converted. I can't reproduce locally, and I feel that the issue will go away automatically on a fresh build.

[agriyakhetarpal]

I had a revelation just now and I think 07a0926 should fix the synchronisation issue for good – the RTD build is just buggy for some reason or the other and the local documentation builds where the issue doesn't arise can be trusted upon. Ready for review and further proceedings.

[agriyakhetarpal]

Tagging @melissawm and @steppi for a review as discussed during the 14/06/2024 interactive docs meeting – thanks, both!

[melissawm]

This seems to work really well- I love the buttons and dropdowns.

How flexible can we make this to share with other projects? Ideally, we could have an extension (or as a part of jupyterlite-sphinx) that adds the preprocess notebooks event, common settings and looks without having to turn them on manually for each project?

[steppi]

This seems to work really well- I love the buttons and dropdowns.

How flexible can we make this to share with other projects? Ideally, we could have an extension (or as a part of jupyterlite-sphinx) that adds the preprocess notebooks event, common settings and looks without having to turn them on manually for each project?

I had a discussion with @agriyakhetarpal offline where we discussed how this preprocessing could be added to jupyterlite-sphinx. I think he's working on it.

[steppi]

Thanks @agriyakhetarpal. Don't forget to set jupyterlite_silence = True again now that we're done debugging. After that I think this is good to merge.

cc @rgommers

[rgommers]

Don't forget to set jupyterlite_silence = True again now that we're done debugging. After that I think this is good to merge.

Looks like that last action was done, and everything is green. Are you happy with this PR as is @agriyakhetarpal?

[agriyakhetarpal]

In the last few comments, I bumped to the latest https://github.com/jupyterlite/pyodide-kernel/releases/tag/v0.5.1, which brings https://github.com/pyodide/pyodide/releases/tag/0.27.1.

[agriyakhetarpal]

Now that all tests are passing, this PR should be ready – thanks to @gabalafou's help with the styling 🙈

cc: @melissawm and @Carreau; tagging you here as I can't request a review from GitHub directly.

[melissawm]

Hi all! Because you pinged me, here's a few notes that you can feel free to ignore:

• There is no documentation on installing the docs requirements and they are not listed on pyproject.toml, only in the readthedocs requirements file;
• The regression/README file is not rendered in the final docs - maybe that's intentional, but it has useful information in the absence of other docs on how to build docs and how the notebook machinery works;
• When I tried building locally the "Open as a notebook" button opens a new tab pointing to an ipynb file but nothing is actually executed/there is no jupyterlite window. Perhaps this could use a note in the docs (that it doesn't work locally?) It is working nicely on the ReadTheDocs build.

Cheers!

[gabalafou]

Thanks @melissawm for this great feedback!

I will take it on me to address them:

• I ran into this too, I will open a PR to add a readme to the doc folder and link to it from the contributing doc.
• Oh interesting! I definitely intended to exclude regressions/README from the docs, as that section of the docs is aimed at end users rather than contributors. I wonder if I copied the first two paragraphs of the readme to the index file and then added a link in the index file directly to the GitHub readme along the lines of "if you're curious to learn more about the machinery behind dual doc/notebook pages"
• Hmm... opening the notebook works for me locally, I wonder if this is the result of no documentation on how to actually build the docs locally. Would you be open to trying again once I get that PR up with the instructions?

[gabalafou]

Here's the PR on how to build the docs (786)

[gabalafou]

I created another pull request on top of this one to add some of the info from the regression folder readme into the index page: agriyakhetarpal#5

[rgommers]

This is looking quite good now! Seems about ready once the last changes from @gabalafou have been folded in.

One question: is it expected that on the RTD preview, there is no caching of wheels? If I try multiple interactive examples on different pages, I'm always waiting 15-20 seconds for import pywt to complete, on a machine with a 300 Mbps connection.

[agriyakhetarpal]

One question: is it expected that on the RTD preview, there is no caching of wheels?

I think so. IIRC, only downloads from PyPI are cached by piplite, and not those from jsDelivr, so we'll need to rely on the caching available from the CDN and download them everytime.

[agriyakhetarpal]

If I try multiple interactive examples on different pages

This is also a point that separately needs some attention – currently, jupyterlite-sphinx will spawn multiple kernels for a page when spinning up multiple interactive examples, and I've seen that the runtime can hang when doing so. I've found that it's better to test out one or two examples at a time at the moment.

I have jupyterlite/jupyterlite-sphinx#242 open to see if there's a one-kernel-per-page solution we can implement, though it's going to be a reasonable amount of effort.

[gabalafou]

For the record, in the preview build I am not seeing the discrepancy between the HTML notebook outputs versus the JupyterLite outputs that I pointed out before, so I'm guessing it had something to do with building the docs on my local machine versus building them with Read the Docs. Since I am not finding the discrepancy on Read the Docs, I am not sure if there is anything actionable here. (Perhaps @agriyakhetarpal already did something to fix this?)

For example, go to https://pywavelets--741.org.readthedocs.build/en/741/regression/wp2d.html, scroll all the way to the bottom and you should see:

[agriyakhetarpal]

Thanks, @gabalafou – I'll note didn't do anything to fix that AFAICS, and the machinery to build the docs wouldn't control numerical differences, of course, so I think it was just the case of some platform-specific behaviour that doesn't show up on RTD's Linux systems.

@rgommers, we are all good to merge, here. 🎉

[rgommers]

I think so. IIRC, only downloads from PyPI are cached by piplite, and not those from jsDelivr, so we'll need to rely on the caching available from the CDN and download them everytime.

That seems like a fairly large usability issue, is there an issue that discusses this?

[rgommers]

This looks pretty polished now, let's get this in! Thanks for the hard work @agriyakhetarpal, @gabalafou, @melissawm, @steppi and @Carreau!

[agriyakhetarpal]

I think so. IIRC, only downloads from PyPI are cached by piplite, and not those from jsDelivr, so we'll need to rely on the caching available from the CDN and download them everytime.

That seems like a fairly large usability issue, is there an issue that discusses this?

Sorry; I think I was not entirely correct, @rgommers. In piplite, we have https://github.com/jupyterlite/pyodide-kernel/blob/36aedb97d38cf283df8c3b0d65d7d6b8e575f0ab/packages/pyodide-kernel/py/piplite/piplite/piplite.py#L128-L147, which caches downloads from PyPI. micropip itself caches downloads from jsDelivr: jupyterlite/pyodide-kernel#60 (comment)

I profiled a sample notebook's execution from my local browser, which took 48.97 seconds for me:

So, the downloads are being appropriately cached; it is likely the number of dynamic libraries across packages being loaded that makes up for most of the time – it took a lot of time for me with a 150 Mbps connection:

Loading Pillow, contourpy, cycler, fonttools, kiwisolver, matplotlib, matplotlib-pyodide, numpy, openblas, pyparsing, python-dateutil, pytz, pywavelets, scipy

I assume a fix for this could be only on the Pyodide side (xref pyodide/pyodide#5264), and we can't do much with JupyterLite... if it's any respite, the scikit-learn notebook examples are taking upwards of 30 seconds to me at the moment, so a warning at the top of the notebooks about high bandwidth usage and wait times sounds like a good option to pursue. I'd like to extend jupyterlite-sphinx a bit for this and have it in mind already (for which I recently opened sphinx-gallery/sphinx-gallery#1427).

[rgommers]

Loading Pillow, contourpy, cycler, [...]

Does that output come from the Network tab in your browser (as in your screenshot), or is there another way to show this debug info?

I assume a fix for this could be only on the Pyodide side (xref pyodide/pyodide#5264),

Ah that is a good explanation. Preloading everything is quite bad - I hope there will be progress on that one.

Short term though there is a fix: the dependencies here are wrong. PyWavelets does not depend on Matplotlib and has a very tiny optional dependency on SciPy. SciPy should definitely not be loaded, it adds nothing. Matplotlib I can imagine loading, because it's used in the docs. Getting rid of SciPy should help a lot already though. Could you please have a look at that?

[agriyakhetarpal]

Does that output come from the Network tab in your browser (as in your screenshot), or is there another way to show this debug info?

Ah, I should have mentioned; the "Loading <package names>" message appears in the Console tab, next to the Network tab.

Preloading everything is quite bad - I hope there will be progress on that one.

In this case, there are a few dependencies that we should consider pre-loading for the notebooks (Matplotlib is one), as JupyterLite/Pyodide are not great with internal package imports (xref sympy/live#18 for a similar SymPy issue that I spent a bit of effort hacking through the package installation's async boundaries on recently, but left to come back to later). For notebooks that need plotting functionalities, I suggest that the import matplotlib statement should be present somewhere at the top (dynamically added, preferably).

Short term though there is a fix: the dependencies here are wrong. PyWavelets does not depend on Matplotlib and has a very tiny optional dependency on SciPy. SciPy should definitely not be loaded, it adds nothing. Matplotlib I can imagine loading, because it's used in the docs. Getting rid of SciPy should help a lot already though. Could you please have a look at that?

Side note: would it be possible to eliminate the dependency on SciPy here? It is not used much, though there are few TODO/FIXME items: https://github.com/search?q=repo:PyWavelets/pywt+scipy&type=code. There's even a comment on using NumPy's FFT, which might be possible to resolve with NumPy v2 now being available. I can try to reduce the dependency in a few places, at least, if not completely remove it in one go.

I agree that both Matplotlib and SciPy are extra, though – I'll drop them on the Pyodide side. The reason Matplotlib might have been included in PyWavelets's Pyodide recipe back when it was first packaged for Pyodide could be that it's used in a bunch of example scripts in the demo/ directory: https://github.com/search?q=repo:PyWavelets/pywt+matplotlib&type=code, which might have been confusing.

That said, Pyodide's format doesn't support optional dependencies very well, and the build system doesn't support multiple outputs, unlike conda-forge, so a few packages have historically included optional dependencies as required (statsmodels, I think, was one). In PyWavelets's case, I'll drop them.

[rgommers]

Side note: would it be possible to eliminate the dependency on SciPy here? It is not used much, though there are few TODO/FIXME items: https://github.com/search?q=repo:PyWavelets/pywt+scipy&type=code. There's even a comment on using NumPy's FFT, which might be possible to resolve with NumPy v2 now being available. I can try to reduce the dependency in a few places, at least, if not completely remove it in one go.

Yes, please go for it! I think it's fine now that we have three numpy 2.x feature releases. scipy was only used as a performance improvement in a single place.

You can leave scipy usage in create_dat.py alone and move that file into util/ - it's not used as part of pywt.