Crashed in production, supervisord process was still running

[fabian-rohr]

So basically it stopped working, even though the process in supervisord was showing up as running (since 60 days). After I restarted the process with supervisord it worked again. What could have happened and how could I avoid this in the future?

[tomcoonen]

Do you have any logs available? Without that it will be a bit hard to diagnose.

[rennokki]

Bump

[Tofandel]

@Hillcow It happened to me, your issue is likely that the server ran out of memory, if that happens all request will fail because php cannot allocate more memory but the server will still be considered running (as the process doesn't exit if the failed allocation is within a try, catch) so supervisor will not restart it

Solution 1: increase the allowed memory for the ws server, this is a missing option in this lib that could be done with ini_set but can also be done when launching the command using php -d MEMORY_LIMIT=1G (doesn't work on all setups)

Solution 2: make sure to catch allocation failure exception and exit the process when that happens

[rennokki]

@Tofandel I assume Solution 2 can be adapted directly in this package by using a signal listener: https://github.com/reactphp/event-loop#addsignal

The bad thing is that that it needs the pcntl extension, so Windows is not favored for this kind of change.

[Tofandel]

The weird thing is that the loop kept running when out of memory, that's a fatal error so it shouldn't be caught in try catch but somehow the process still managed to continue and in the on error handler of the logger you can see the caught messages of "Unable to allocate xxx bytes" multiple times, once after each request

[yurik94]

We resolved by restarting the process every week, very ugly but it's the only working solution for us as memory was below the max limit.
Use also used pcntl extension as, by default, the worst loop was in use.

[rennokki]

Ugly-ugly, but we gotta find what causes this huge increase in memory. Prolly some memory leaks caused by closed connections? 🤔

[Tofandel]

@rennokki I don't really see how this would solve the issue as that would listen for external signals (eg: sigterm, sigend) and not internal unrecoverable errors

Yes there likely is some memory leak in this package
I would suggest looking into the statistics part, last week I completely disabled the statistics by modifying some code because I didn't need them and didn't need the table as well, there is no memory increase over time anymore
Though I also added a gc_collect_cycles everytime a connection closes, so it might also be some cyclic reference issue and if the garbage collector is not called enough and you get a high volume of request before garbage collection (eg 200 requests in 5 minutes) then the memory just blows

There is a lot of circular references eg ->app = app() in all the loggers singleton so I wouldn't be surprised

[rennokki]

I understand that PHP cannot create exceptions because the memory limit is... well, memory limit. So maybe we can check with a timer if the total ram reached a percentage in the last x minutes and restart the server.

Again, if this seems to be due to statistics, i understand. There was a PR merged for 1.6.1 that fixes a decorator for websockets by cloning the self. Perhaps it needs some on the other decorators.

[Tofandel]

@rennokki Yeah that was me :p Well actually I think for loggers, singleton should be dropped at least for the message logger and instead a trait or abstract class should be added, it's a big BC but the websocket logger is causing other issues by decorating the class

One of them is interface implementation, ratchet will check your class for some implementations and provide some features in that case (Eg: supported protocol with interface WsServerInterface) and by decorating you lose the interface and so the feature, I think logging should be done within the class by extension of onMessage and onError rather than wrapping the class within a class

For memory you can indeed add a loop ticker every 30sec and check if you are at more than 95% of ini_get('memory_limit') if so stop the server

[rennokki]

I can drop the decorators for the 2.x. It is still in beta and it needs thoroughly tested for this point.