Feature request: per cell cacheing for python cells

[ejolly]

From the documentation site:

Note that for Jupyter, the cache for a document is invalidated if any of the code blocks change. For Knitr, invalidation occurs on a per-cell basis. (emphasis added)

It would be great it adding or modifying 1 python cell only invalidates the cache for that specific cell, rather than re-executing all code cells. This would be really nice if you wanted to say add a new plotting cell but don't want to rerun expensive computations in previous cells

I'm not sure if this is a limitation of jupyter-cache or if there are plans to support this in the future. Or maybe allow usage or incorporation of a different caching library, e.g. like ipycache

[jjallaire]

Yes we would love it to work this way. We are finishing up our v1.0 release and will take a look at this post v1.0.

[JanPalasek]

I would highlight importance of this issue. The current implementation makes working with more computationally more intensive notebooks impossible.

[jjallaire]

Agree this is important! We currently use jupyter-cache (https://github.com/executablebooks/jupyter-cache) for notebook caching. While they don't currently have a per-cell cache option they certainly may develop one.

Another approach we've seen for users with extremely expensive computations is to author within the Jupyter Notebook UI (where there is effectively a per-cell cache). Note that when rendering an ipynb Quarto does not re-execute it by default.

[JanPalasek]

Agree this is important! We currently use jupyter-cache (https://github.com/executablebooks/jupyter-cache) for notebook caching. While they don't currently have a per-cell cache option they certainly may develop one.

Another approach we've seen for users with extremely expensive computations is to author within the Jupyter Notebook UI (where there is effectively a per-cell cache). Note that when rendering an ipynb Quarto does not re-execute it by default.

Thanks. Yes, I know about that option. But while I really like working in qmd, I don't like working in ipynb with Quarto that much. Not sure about that jupyter-cache, it had last release 7 months ago and no significant activity since then. I suggested the cell-level cache in a issue there, but we'll see.

[JanPalasek]

@jjallaire How would you like the cache to be implemented?
For example I might add use NotebookClient from nb-client package (https://github.com/jupyter/nbclient/blob/main/nbclient/client.py#L60). I could use the hooks to implement the caching. This class is used by nbconvert when executing the notebook. Would Quarto be able to use this implementation?

[jjallaire]

Yes, we currently use NotebookClient for interacting with notebooks.

That said, I think that it would be of substantial benefit to try to collaborate with the https://github.com/executablebooks/jupyter-cache project on this. I think it would be a desirable feature there and a lot of expertise could be brought to bear if worked on collaboratively.

My biggest overall concern about per-cell caching is that it requires that the entire Python environment be serializable (e.g. pickle). There are many Python objects though that cannot be easily serialized (anything with a pointer into an external library, for example) so there would a lot of qualification around how and when the cache could be used and expected to work properly

[Unco3892]

Hi, even though you use jupyter-cache, per cell caching via #| cache: true still does not seem to work for python cells. Is this a feature in development or completely disregarded?

[mcanouil]

It can't work at cell level as explained before because Jupyter-cache only supports global notebook caching, unless it has changed recently.

Edit: it has not

• Cell-level caching executablebooks/jupyter-cache#89