Hang after too many gl requests in Docker

[scjody]

From the tests documented starting at https://github.com/plotly/streambed/issues/9865#issuecomment-349995119:

When image-exporter is run as an imageserver in Docker, after a small number of gl requests (30-40), the image-exporter hangs completely. The request in progress times out, and the server won't accept any more connections and must be restarted.

Two examples:

• If test/image-make_baseline.js is used to run gl*, it hangs at gl3d_chrisp-nan-1.json.
• If test/image-make_baseline.js is used to run gl3d*, it makes it past gl3d_chrisp-nan-1.json and hangs at gl3d_snowden_altered.json.

This means that the issue is unlikely to be specific to any one plot, but rather some resource becomes exhausted or something builds up to the point where image generation can't proceed.

@etpinard @monfera FYI

[scjody]

Note that this blocks using the new imageservers in prod, in On-Prem, and also blocks plotly/plotly.js#1972

[monfera]

I plugged a bunch of memory leaks in gl plots about a year ago and we got information that those worked but there were other leaks too, so I'm not too surprised. As there are many types of gl plots I wonder if periodically restarting the server would be a feasible short-term option, as hacky as it sounds. Also, in the future, there may always come a new plot type that has unintended leaks so it'd be a good preventive step. Maybe the real solution would be a plotly.js test bench, separate from the rest due to its resource needs, that would go through a large number of plot API calls and monotonic memory growth could be identified.

[monfera]

Self-assigned it on info from @scjody that @jackparmer suggested I continue with the gl plot leak detection&fix work. Jody, when work is starting on this (I understand after the couple of issues assigned to me are fixed) I'll need some bag of representative plots that I can run the server through, to reproduce the issue locally, so that we don't miss something.

IOW running it through all public plots is an incredibly effective way for ferreting out all memory leaks that can happen on a single render pass (it doesn't solve leaks from interactions though).

[scjody]

@monfera The best way I've found is to run test/image/mocks/gl* or test/image/mocks/gl3d* (from plotly.js). Both sets of mocks will reproduce the issue, but at different places.

[scjody]

I wonder if periodically restarting the server would be a feasible short-term option

@monfera I'd like to do that as a workaround for #41, but I don't think it's feasible here since the server hangs so quickly. We observe hangs after between 30 and 40 gl plots, so to be safe we'd need to restart every 10 plots or so, and that would reduce server performance to an unacceptable level.

I also don't think this particular issue is caused by a memory leak. Memory usage does grow steadily (#41) but does not increase significantly after 30 to 40 plots (gl or otherwise). Some other kind of resource leak is certainly possible!

[monfera]

thanks @scjody for the added notes!

| Memory usage does grow steadily (#41) but does not increase significantly after 30 to 40 plots (gl or otherwise)

Reading #41 I got the impression that, even without the gl plots, memory consumption grows steadily throughout the first 1.5hrs you added, with the text above referring to 351 mocks, 0.8GB -> 1.6GB. I get your point about how the gl and non-gl may be separate issues, but thought the non-gl ones represent a problem for us too (the raison d'être of #41)

[scjody]

#41 is a problem, but I'm reasonably sure restarting every 1000 plots is an adequate workaround for that issue.

[scjody]

@monfera @etpinard I've added some debugging stuff to the Docker container in #44.

• Xvfb runs with X errors sent to STDOUT (instead of being ignored), and auditing is turned on. This means it prints a message for every X connection and disconnection, which shows if Electron is connecting and disconnecting (spoiler alert: it isn't).
• Xvfb's screen dimensions have been doubled.
• I added a VNC server, window manager, and a wrapper script.

Usage:

• Build the image (docker build -f deployment/Dockerfile -t isdebug .), or grab it from quay.
• Run the image as container isdebug and expose the VNC and imageserver ports: docker rm -f isdebug ; docker run -p 9091:9091 -p 5900:5900 --name isdebug -ti isdebug
• From a second window, connect and run the VNC wrapper: docker exec -ti isdebug /vnc
• Connect to the VNC display localhost:0 using your favourite client, such as gvncviewer on Ubuntu or Chicken of the VNC on OS X.
• From a third window, run whatever tests are needed, such as (in the plotly.js repo): node test/image/make_baseline.js gl* (you'll want to modify testContainerUrl in tasks/util/constants.js, like in https://github.com/plotly/plotly.js/tree/image-exporter-testing)

In my case the image exporter window where the work was occurring was in fluxbox tab 2. Some errors were printed during normal operation (at this point images were being generated successfully):

When the server hung, a different message was printed:

I have not had the chance to investigate the significance of either of these things.

[etpinard]

I might be worth trying to listen to

https://electronjs.org/docs/api/app#event-gpu-process-crashed

in the app code. I'll give this a shot this afternoon.

[scjody]

I setup an Ubuntu 16.04 VM (since that's what @etpinard uses as a desktop and I can't easily get gl renders working with XQuartz on OS X), opened up the X server to TCP connections, and ran image-export-server using that X server to see if the issues were caused by Xvfb.

With image-export-server running in Docker the tests hung at the usual place (testing gl* caused a hang at gl3d-chrisp-nan-1). With image-export-server running on my development machine the tests completed successfully.

So the issue is unlikely to be caused by Xvfb. What's different about the app (or Electron) when it's built and run inside Docker vs. outside?

[monfera]

@scjody just jumping in to see if I can help with this stuff but may just state the obvious (perhaps incorrectly assuming that your development machine uses the graphics drivers of that machine for WebGL content rendering.)

| What's different about the app (or Electron) when it's built and run inside Docker vs. outside?

One difference is that when Electron is used on a desktop, the WebGL API calls go through whatever graphics chip driver the desktop has, eg. from Nvidia, AMD or intel integrated graphics. In a Docker container or on CI systems in general, it's running in headless mode, still expecting a display driver, in this case, Xvfb.

Many of our WebGL plots run on stack.gl and gl-vis which, unlike regl, don't provide automatic resource management. The typical WebGL resources are the shader programs, and buffer bindings that link typed JS arrays to contents on the GPU, used by the shaders (uniforms, attributes, textures etc.). As there's no automatic resource management, a small issue in one specific trace type may yield state inconsistency, eg. no buffer is bound to an enabled attribute (see on your log) or sometimes the other way around.

Various drivers, by extension, Xvfb too, have different idiosyncrasies and tolerances for handling slightly out of spec WebGL state. As we did encounter such things in the past, it might be that Xvfb is more sensitive to some of those.

Something else: apparently we're running tests with ignore-gpu-blacklist which in turn might disable some WebGL extensions depending on the graphics stack; if we rely on one of these and a related warning gets swallowed, we might get seemingly unrelated state inconsistency reports just like the above. A candidate is OES_vertex_array_object. We seem to only use it via a polyfill but the same graphics card may process real VAO and polyfill differently, or not properly 'disusing' the polyfill may lead to inconsistencies too.