Handling concurrent requests on Kubernetes cluster

Hi,

I've implemented EG on Kubernetes with a single master and single node server. I've created my own application using JupyterLab services to run the kernels on the cluster.

For testing the implementation, I tried to create multiple kernels concurrently, however, these aren't handled parallelly and waits until the previous requests are completed. This makes the kernel pods creation time too high. Is there something I'm missing for it to handle concurrent requests efficiently?

Any help would be appreciated. Thanks.

[kevin-bates]

Hi, This is a known issue and documented in #86. Unfortunately, this is systemic to the jupyter framework - as noted in some of the comments in that issue. Since local process startup is much faster - on the order of a second or two vs. 10-20s - the concurrency issue is not as evident, but does exist.

Aah okay. Since I'll be using EG for my own use, the API changes for the existing system wouldn't hamper me so much.

So, can you maybe point me to the changes I need to do to make the calls asynchronous? Since you've already made substantial headway on the solution.

Thanks once again.

[kevin-bates]

Hmm - not sure where you got the idea that any headway has been made on async startup. It is non-trivial and the lower layers must be addressed first. If you wanted to tackle this, I would start with the references in jupyter_client and notebook that Matthias posted to #86. Once async startup in Notebook has been made to work, then we could focus on EG.

When I first wrote up #86, I was under the impression that kernel startup was already async because the start_kernel method was decorated as a coroutine, but, now that I know a bit more, that's where it stops and the call stack must be plumbed much deeper with coroutines. In addition, it probably should be done in a backwards compatible fashion.

One possible option is to see if the newer jupyter_server project plans on supporting async kernel startup - it probably should, but is currently based completely on the notebook startup logic. There's also a new kernel management proposal in the works, so perhaps that can try to address async startup?

I'd prefer to close this as a duplicate and just work from #86 since it already has some information.

Thank you for pointing out the options. I'll start looking into the client and notebook code as well.

[kevin-bates]

@parewa-aswin - Not sure if you watch this repo, but we've made progress on #86. I have built a candidate enterprise-gateway image that can be pulled using docker pull kbates/enterprise-gateway:async. This image incorporates my local builds for PR #580 coupled with those PRs from jupyter_client and notebook.

Since the EG PR requires updated releases for jupyter_client and notebook, I can't say when this would be officially available. However, it would a tremendous help to the project if you could take this for a spin to see if it helps address your issues relative to concurrent startups.

@kevin-bates Awesome. I've been meaning to work on the async support myself. I'll definitely look at this and get back to you. Thanks a ton.

@kevin-bates - I just found time to test the image. I can start multiple kernels asynchronously now. Yay! However, it seems the socket operations accept, recv are still blocking. Are there any plans to shift this to non-blocking mode?

[kevin-bates]

@parewa-aswin - Good catch. I hadn't noticed that, but, yeah, we should probably try to address that. I'm not familiar with async sockets, but a quick search yielded asyncore. Are you familiar with async socket handling and, if so, would you want to take a crack at this? If not, I'll take a look - let me know.

We'd probably need to separate out the response socket's use of select_socket() with its own method/code in _prepare_response_socket().

@kevin-bates - Unfortunately, I'm not familiar with async sockets or socket programming in general. So, it'd be better if you or anybody else would look into this 😄.