Opening notebooks occasionally yields kernel error, KeyError

[glortho]

Observed behavior: Users periodically open notebooks to a red "Kernel Error" status marker at top right, along with the bomb icon by the Python 3 kernel indicator. In the devtools console is this stack:

[E 14:13:17.077 NotebookApp] Uncaught exception POST /notebook/api/sessions (10.255.0.4)
    HTTPServerRequest(protocol='http', host='<our host here>', method='POST', uri='/notebook/api/sessions', version='HTTP/1.1', remote_ip='<our remote ip>')
    Traceback (most recent call last):
      File "/anaconda/envs/py3/lib/python3.6/site-packages/tornado/web.py", line 1543, in _execute
        result = yield result
      File "/anaconda/envs/py3/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/anaconda/envs/py3/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
        yielded = self.gen.send(value)
      File "/anaconda/envs/py3/lib/python3.6/site-packages/notebook/services/sessions/handlers.py", line 67, in post
        model = yield gen.maybe_future(sm.get_session(path=path))
      File "/anaconda/envs/py3/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 170, in get_session
        return self.row_to_model(row)
      File "/anaconda/envs/py3/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 209, in row_to_model
        raise KeyError
    KeyError
[W 14:13:17.078 NotebookApp] Unhandled error

On page refresh everything works fine.

That trace points to this:

notebook/notebook/services/sessions/sessionmanager.py

Lines 204 to 206 in 4135e3f

	# The kernel was killed or died without deleting the session.
	# We can't use delete_session here because that tries to find
	# and shut down the kernel.

We cull idle notebooks after 24 hours. I don't know if that's related. Either way, is this the desired behavior, that they would just get a kernel error when they land on the page? Is there some way to avoid this or trigger auto-refresh of the session, etc.?

[glortho]

We've confirmed that this does appear to happen when you land directly on a notebook for the first time after its kernel was culled. Most recently we saw this:

[I 21:08:51.983 NotebookApp] Kernel shutdown: 89855d3f-873d-40a5-bbca-4c828e627d73

and then when landing on it 6 minutes later:

[E 21:14:53.524 NotebookApp] Uncaught exception POST /juno/notebook/api/sessions

[glortho]

I could be off-base but it seems like we'd either want to:

• delete any sessions with a cullable kernel when the kernel itself is deleted https://github.com/jupyter/notebook/blob/master/notebook/services/kernels/kernelmanager.py#L467

Or:

• automatically create a new session rather than raising a KeyError here: https://github.com/jupyter/notebook/blob/master/notebook/services/sessions/sessionmanager.py#L209

They could be behind config options if need be. Are there ramifications to those that would make them impractical/problematic?

[kevin-bates]

+1 on culling being the culprit as I've been looking at this as well.

I don't think we can go with the first option because culling could be taking place on an entirely different server than where the sessions are managed. This is the case with Kernel/Enterprise Gateway - which proxies the kernel management to a different server via NB2KG.

I was thinking more along the lines of combining the session_exists() and get_session() calls into one that can return None, then make the else: case a check on that return value - which creates the new session if None.

I believe get_session()could just be extended with an option that indicates a dangling session be removed. This parameter would, of course, default to today's behavior, but when present would perform the cleanup that row_to_model() does today. row_to_model() would then not perform any cleanup, but could still raise KeyError to preserve functionality in list_sessions().

If this sounds okay to others, I'd be happy to submit a PR.

[glortho]

Thank you kindly for your attention to this @kevin-bates. I'm not very familiar with the codebase, but an option for automatically creating a new session sounds good.

I don't suppose you have suggestions for a workaround in the meantime, do you? We have one interface into notebooks where we could conceivably preflight the session post, but all ears for any other ideas!

[kevin-bates]

I can't think of any workarounds that don't require code changes - aside from disabling culling - which doesn't appear to be an option for you.

[kevin-bates]

I should have something to take for a spin by tomorrow afternoon (PST).

[glortho]

Awesome Kevin!

…

On Thu, Nov 15, 2018 at 20:08 Kevin Bates ***@***.***> wrote: I should have something to take for a spin by tomorrow afternoon (PST). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4209 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAgHzJHYjZ9tiLQgy0PSQOB6ESbb9YOkks5uvhAcgaJpZM4Yb2nP> .