Is it possible to programmatically inspect variables of a NotebookNode?

[Jacob-Stevens-Haas]

Asking in nbclient repo because nbconvert.preprocessors.ExecutePreprocessor gets it's execute_cell method from nbclient.client.NotebookClient, so I hope this is the right place.

I've been trying to run certain experiments in a Jupyter Notebook in order to take advantage of the nice layout of code and plots. However, in order to do so repeatably and store other metadata, I set up and run the notebook in memory and convert to HTML.

When I want to use the results of the experiment in some of the calling code, I currently print it in a nbformat.notebooknode.NotebookNode cell and then read a notebook cell's ["outputs"] value. But for objects whose repr isn't enough to reconstruct the object, is there a way to inspect the global scope of the NotebookNode?

MWE:

import nbformat
from nbconvert.preprocessors import ExecutePreprocessor

nb = nbformat.v4.new_notebook()
nb["cells"] = [nbformat.v4.new_code_cell(
    source="foo=1\n"
    "print(foo)"
)]
ep = ExecutePreprocessor(timeout=-1)
ep.preprocess(nb)

Currently, I run

result_string = nb["cells"][0]["outputs"][0]["text"][:-1]

Desired

result = ep.<some function>("foo")

[davidbrochart]

Nbclient runs the code in a kernel, which is a separate process, so (de)serialization is the only way to get a result back.
You could pickle the result and base64-encode it, so that you can retrieve the original object.

[Jacob-Stevens-Haas]

Sure, that's an option. But since there are IDEs that let you inspect variables in a running Jupyter notebook, I thought they might have some other way of doing this. Like a function for client/kernel communication using serialization over a socket or pipe, rather than manually mucking about with files.

[davidbrochart]

I'm not talking about files, cell outputs are broadcast over the IOPub channel, which is a ZMQ socket.
BTW I think IDEs inspecting variables only show a representation of the variables, not the actual objects as you are trying to do.

[Jacob-Stevens-Haas]

Ooh, ok thank you - that makes enough sense for me to see why it would exist, but it's new enough to me that I'm not 100% sure what to do with it. Apologies if some of my replies don't make sense - I'm not too accustomed to concepts like sockets and channels.

Since cell outputs can be repr's rather and are not necessarily serialized data, is it correct that I would need to have the KernelManager object send a richInspectVariables message? Or am I thinking about this wrong, and it would make more sense to add a code cell at the end of the notebook that somehow

• identifies the PyZMQ socket in use for the IOPub channel,
• pickles the desired variable into a io.BytesIO object,
• and sends it via the socket?

[davidbrochart]

I've not tried, but does this work?

import base64
import pickle
import nbformat
from nbconvert.preprocessors import ExecutePreprocessor

nb = nbformat.v4.new_notebook()
nb["cells"] = [nbformat.v4.new_code_cell(
    source="foo=1\n"
    "print(base64.b64encode(pickle.dumps(foo)))"
)]
ep = ExecutePreprocessor(timeout=-1)
ep.preprocess(nb)

result = pickle.loads(base64.b64decode(nb["cells"][0]["outputs"][0]["text"][:-1]))