Server stops responding

[jetpax]

Thanks for building this great project!

Not sure if this is related to #8, so am opening a new issue.

I did a build with webui in LittleFS, and system works but is quite unstable and server stops responding after a while.

ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
ERROR: Too many SSE messages queued for [192.168.1.125]
[879753][V][WiFiGeneric.cpp:362] _arduino_event_cb(): STA Disconnected: SSID: Paxnet, BSSID: b4:fb:e4:d7:d4:ee, Reason: 200
[879754][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[879761][W][WiFiGeneric.cpp:955] _eventCallback(): Reason: 200 - BEACON_TIMEOUT
[879768][D][WiFiGeneric.cpp:975] _eventCallback(): WiFi Reconnect Running
WiFi Disconnected. Reason code=200
WiFi connection dropped, stopping NTP.
[879787][V][WiFiGeneric.cpp:343] _arduino_event_cb(): STA Stopped
[879789][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 3 - STA_STOP
Starting software access point
[881615][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 0 - WIFI_READY
[881617][V][WiFiGeneric.cpp:392] _arduino_event_cb(): AP Started
[881617][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 10 - AP_START
[881618][V][WiFiGeneric.cpp:97] set_esp_interface_ip(): Configuring SoftAP static IP: 192.168.4.1, MASK: 255.255.255.0, GW: 192.168.4.1
[881636][V][WiFiGeneric.cpp:143] set_esp_interface_ip(): SoftAP: 192.168.4.1 | Gateway: 192.168.4.1 | DHCP Start: 0.0.0.0 | Netmask: 255.255.255.0
[881649][V][WiFiGeneric.cpp:190] set_esp_interface_ip(): DHCP Server Range: 192.168.4.2 to 192.168.4.12
[882343][V][WiFiGeneric.cpp:395] _arduino_event_cb(): AP Stopped
[882343][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 11 - AP_STOP
[882344][V][WiFiGeneric.cpp:392] _arduino_event_cb(): AP Started
[882351][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 10 - AP_START
Starting captive portal on 192.168.4.1
Connecting to WiFi.
[900901][V][WiFiGeneric.cpp:340] _arduino_event_cb(): STA Started
[900901][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 2 - STA_START
[900901][V][WiFiGeneric.cpp:97] set_esp_interface_ip(): Configuring Station static IP: 0.0.0.0, MASK: 0.0.0.0, GW: 0.0.0.0
[900952][V][WiFiGeneric.cpp:355] _arduino_event_cb(): STA Connected: SSID: Paxnet, BSSID: f4:92:bf:a5:99:f1, Channel: 6, Auth: WPA2_PSK
[900953][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 4 - STA_CONNECTED
WiFi Connected.
[902371][V][WiFiGeneric.cpp:369] _arduino_event_cb(): STA Got Same IP:192.168.1.153
[902371][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 7 - STA_GOT_IP
[902374][D][WiFiGeneric.cpp:996] _eventCallback(): STA IP: 192.168.1.153, MASK: 255.255.255.0, GW: 192.168.1.1
WiFi Got IP. localIP=192.168.1.153, hostName=Svelte-RetroVMS
Got IP address, starting NTP Synchronization
Starting NTP...
[908358][E][vfs_api.cpp:105] open(): /littlefs/www/index.html does not exist, no permits for creation
[908389][E][vfs_api.cpp:105] open(): /littlefs/www/index.html does not exist, no permits for creation
[908419][E][vfs_api.cpp:105] open(): /littlefs/www/index.html does not exist, no permits for creation
[908488][E][vfs_api.cpp:105] open(): /littlefs/www/index.html does not exist, no permits for creation
[908518][E][vfs_api.cpp:105] open(): /littlefs/www/index.html does not exist, no permits for creation
[908548][E][vfs_api.cpp:105] open(): /littlefs/www/index.html does not exist, no permits for creation
[908644][E][vfs_api.cpp:105] open(): /littlefs/www/_app/immutable/entry/start.js does not exist, no permits for creation
[909278][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[909279][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[909279][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[909281][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[909286][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[909304][E][vfs_api.cpp:105] open(): /littlefs/www/favicon.png does not exist, no permits for creation
New client connected to Event Source: 1 Clients connected
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x44c
load:0x403c9700,len:0xbec
load:0x403cc700,len:0x2920
entry 0x403c98d8
[   317][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 0 - WIFI_READY
[   352][V][WiFiGeneric.cpp:340] _arduino_event_cb(): STA Started
[   353][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 2 - STA_START
[   355][E][WiFiGeneric.cpp:940] _eventCallback(): esp_wifi_set_ps failed
E (452) wifi_init_default: esp_wifi_get_mac failed with 12289
[   367][V][WiFiGeneric.cpp:343] _arduino_event_cb(): STA Stopped
[   373][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 3 - STA_STOP
Running Firmware Version: 0.2.2
Connecting to WiFi.
[   542][D][esp32-hal-rmt.c:615] rmtInit():  -- TX RMT - CH 0 - 1 RAM Blocks - pin 48
[   555][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 0 - WIFI_READY
[   557][V][WiFiGeneric.cpp:340] _arduino_event_cb(): STA Started
[   557][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 2 - STA_START
[   557][V][WiFiGeneric.cpp:97] set_esp_interface_ip(): Configuring Station static IP: 0.0.0.0, MASK: 0.0.0.0, GW: 0.0.0.0
Starting software access point
[   732][V][WiFiGeneric.cpp:392] _arduino_event_cb(): AP Started
[   732][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 10 - AP_START
[   733][V][WiFiGeneric.cpp:97] set_esp_interface_ip(): Configuring SoftAP static IP: 192.168.4.1, MASK: 255.255.255.0, GW: 192.168.4.1
[   746][V][WiFiGeneric.cpp:143] set_esp_interface_ip(): SoftAP: 192.168.4.1 | Gateway: 192.168.4.1 | DHCP Start: 0.0.0.0 | Netmask: 255.255.255.0
[   758][V][WiFiGeneric.cpp:190] set_esp_interface_ip(): DHCP Server Range: 192.168.4.2 to 192.168.4.12
[  1454][V][WiFiGeneric.cpp:395] _arduino_event_cb(): AP Stopped
[  1455][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 11 - AP_STOP
[  1455][V][WiFiGeneric.cpp:392] _arduino_event_cb(): AP Started
[  1462][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 10 - AP_START
Starting captive portal on 192.168.4.1
[  2455][V][WiFiGeneric.cpp:362] _arduino_event_cb(): STA Disconnected: SSID: Paxnet, BSSID: 68:ff:7b:06:87:15, Reason: 2
[  2455][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[  2462][W][WiFiGeneric.cpp:955] _eventCallback(): Reason: 2 - AUTH_EXPIRE
[  2469][D][WiFiGeneric.cpp:975] _eventCallback(): WiFi Reconnect Running
WiFi Disconnected. Reason code=2
WiFi connection dropped, stopping NTP.
[  2487][V][WiFiGeneric.cpp:362] _arduino_event_cb(): STA Disconnected: SSID: Paxnet, BSSID: f4:92:bf:a5:99:f1, Reason: 202
[  2493][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[  2493][V][WiFiGeneric.cpp:343] _arduino_event_cb(): STA Stopped
[  2500][W][WiFiGeneric.cpp:955] _eventCallback(): Reason: 202 - AUTH_FAIL
WiFi Disconnected. Reason code=202
WiFi connection dropped, stopping NTP.
[  2523][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 3 - STA_STOP
Connecting to WiFi.
[ 30548][V][WiFiGeneric.cpp:340] _arduino_event_cb(): STA Started
[ 30548][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 2 - STA_START
[ 30549][V][WiFiGeneric.cpp:97] set_esp_interface_ip(): Configuring Station static IP: 0.0.0.0, MASK: 0.0.0.0, GW: 0.0.0.0
[ 30734][V][WiFiGeneric.cpp:355] _arduino_event_cb(): STA Connected: SSID: Paxnet, BSSID: f4:92:bf:a5:99:f1, Channel: 6, Auth: WPA2_PSK
[ 30735][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 4 - STA_CONNECTED
WiFi Connected.
[ 31266][V][WiFiGeneric.cpp:369] _arduino_event_cb(): STA Got New IP:192.168.1.153
[ 31267][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 7 - STA_GOT_IP
[ 31270][D][WiFiGeneric.cpp:996] _eventCallback(): STA IP: 192.168.1.153, MASK: 255.255.255.0, GW: 192.168.1.1
WiFi Got IP. localIP=192.168.1.153, hostName=Svelte-RetroVMS
Got IP address, starting NTP Synchronization
Starting NTP...
New client connected to Event Source: 1 Clients connected
[ 39401][E][vfs_api.cpp:105] open(): /littlefs/www/_app/version.json does not exist, no permits for creation
Stopping captive portal
Stopping software access point
[ 40758][V][WiFiGeneric.cpp:395] _arduino_event_cb(): AP Stopped
[ 40758][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 11 - AP_STOP
[ 40759][V][WiFiGeneric.cpp:392] _arduino_event_cb(): AP Started
[ 40769][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 10 - AP_START
[ 40770][V][WiFiGeneric.cpp:395] _arduino_event_cb(): AP Stopped
[ 40782][D][WiFiGeneric.cpp:931] _eventCallback(): Arduino Event: 11 - AP_STOP
[ 42864][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
[ 42866][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
[ 42867][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
[ 42868][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
[ 45573][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 45936][E][vfs_api.cpp:105] open(): /littlefs/www/favicon.png does not exist, no permits for creation
[ 45979][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 47043][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 47043][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 47513][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 47514][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 48053][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 48053][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 48605][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 48605][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
New client connected to Event Source: 1 Clients connected
[105819][E][vfs_api.cpp:105] open(): /littlefs/www/index.html does not exist, no permits for creation
[105848][E][vfs_api.cpp:105] open(): /littlefs/www/index.html does not exist, no permits for creation
[105879][E][vfs_api.cpp:105] open(): /littlefs/www/index.html does not exist, no permits for creation
[106689][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[106689][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[106689][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[106691][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[107279][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[107279][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[107677][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[107677][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[107678][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[107680][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[107684][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108408][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108408][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108408][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108410][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108415][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108872][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108872][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108872][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108874][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[108879][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
New client connected to Event Source: 2 Clients connected
New client connected to Event Source: 3 Clients connected
[119216][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[119216][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[119216][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
E (127354) tsens: Do not configure the temp sensor when it's running!
E (157350) tsens: Do not configure the temp sensor when it's running!
[324237][I][WebSocketServer.h:144] onWSEvent(): [WebSocket Server] ws[/ws/lightState][1] connect
[342419][I][WebSocketServer.h:253] onWSEvent(): [WebSocket Server] ws[/ws/lightState][1] disconnect: 1070549080
Starting NTP...
Connecting to MQTT...
[509382][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
Connected to MQTT, without persistent session
Disconnected from MQTT reason: 0
Connecting to MQTT...
Connected to MQTT, without persistent session
[530068][I][WebSocketServer.h:144] onWSEvent(): [WebSocket Server] ws[/ws/lightState][2] connect
[537240][I][WebSocketServer.h:253] onWSEvent(): [WebSocket Server] ws[/ws/lightState][2] disconnect: 1070549080
[551064][I][WebSocketServer.h:144] onWSEvent(): [WebSocket Server] ws[/ws/lightState][3] connect
[557424][I][WebSocketServer.h:253] onWSEvent(): [WebSocket Server] ws[/ws/lightState][3] disconnect: 97
E (4545344) tsens: Do not configure the temp sensor when it's running!
E (4585341) tsens: Do not configure the temp sensor when it's running!
E (4775339) tsens: Do not configure the temp sensor when it's running!
[4805411][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
E (10791926) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (10791926) task_wdt:  - IDLE (CPU 0)
E (10791926) task_wdt: Tasks currently running:
E (10791926) task_wdt: CPU 0: Analytics Servi
E (10791926) task_wdt: CPU 1: IDLE
E (10791926) task_wdt: Aborting.

abort() was called at PC 0x4202f5d8 on core 0


Backtrace: 0x403778de:0x3fc962b0 0x4037cf39:0x3fc962d0 0x40383611:0x3fc962f0 0x4202f5d8:0x3fc96370 0x40378c55:0x3fc96390 0x4202a76f:0x3fca7280 0x4202a7df:0x3fca72b0 0x420254c6:0x3fca7300 0x4201676c:0x3fca7330 0x42016979:0x3fca7870

  #0  0x403778de:0x3fc962b0 in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:402
  #1  0x4037cf39:0x3fc962d0 in esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c:128
  #2  0x40383611:0x3fc962f0 in abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/abort.c:46
  #3  0x4202f5d8:0x3fc96370 in task_wdt_isr at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/task_wdt.c:176 (discriminator 3)
  #4  0x40378c55:0x3fc96390 in _xt_lowint1 at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/freertos/port/xtensa/xtensa_vectors.S:1111
  #5  0x4202a76f:0x3fca7280 in temp_sensor_read_raw at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/driver/esp32s3/rtc_tempsensor.c:122 (discriminator 2)
  #6  0x4202a7df:0x3fca72b0 in temp_sensor_read_celsius at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/driver/esp32s3/rtc_tempsensor.c:158
  #7  0x420254c6:0x3fca7300 in temperatureRead at /Users/jep/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-misc.c:73
  #8  0x4201676c:0x3fca7330 in AnalyticsService::_loop() at lib/framework/AnalyticsService.h:64 (discriminator 3)
  #9  0x42016979:0x3fca7870 in AnalyticsService::_loopImpl(void*) at lib/framework/AnalyticsService.h:48

[EvEggelen]

@jetpax
I also noticed that the wifi connection was very unstable with my aliexpress nodemcu-32s board. When I added a small capacitor ( e.g. 10uF+ ) on the 5V rail the server was with the same SW stable. I expect that ESP32-sveltekit uses a lot of power in small peaks. The inductance of my USB cable limited the peak current to the board. You could also say the capacitor on the cloned nodemcu-32s board is too small...

One solution is using break-outboard that has extra capacitors on the supply rail. I used the one below and added the capacitor leads in the left top in the screw connection. ( GND and VIN are next to each-other).

The log you have was similar to what I had without the capacitor. But I hope we can still look into the code and figure out why the server stops responding. Although the capacitor improved the stability of the WiFi in my case, I expect this also happens when the WiFi is really bad. In my view bad WiFi should not result in a deadlock of the server. Without the capacitor I was able to reproduce this with multiple devices connection over WiFi to the ESP32.

[EvEggelen]

@theelims

Would it be an idea add a setting to control the WiFi TX power in the WiFi setup. If I am not mistaken there is also a powersaving option for WiFi ( Default, Off, minimum and Maximum )

[theelims]

@jetpax please try to stabilize the supply voltage first. ESP's are known to be very power hungry. I had similar issues with one board as well. You need to be able to supply 500mA peaks. You can hook-up a scope on your power rail and see whether it drops every few ms.

The crash you observed is coming from issues with reading the temperature sensor. I haven't seen this in my tests. But I'll observe whether I can observe traces of this like E (127354) tsens: Do not configure the temp sensor when it's running! in my own builds. But since one ADC is tightly coupled to the WiFi hardware and this is likely used for all internal ADC measurements it could be a brownout as well.

@EvEggelen I don't know if this is the right approach to fix poor hardware design.

[EvEggelen]

@theelims
I agree with you that the supply voltage should be stable. This needs to be addressed.

Regarding adding a setting for changing the TX power. This is in that sense not related and not intended to solve this problem. To be honest I should have added this in a different ticket as feature request. My main motivation is to reduce radiation and power. When possible I reduce the WiFI power to the level that is required to function correctly ( with some margin ). I do this for my access point and so too. I see more IOT devices have this feature.

Regarding the dead-lock of the server. I expect this can also happen with a stable power supply when the WiFi signal is bad. Even with my stable power supply see errors that I do not understand. Luckily they do not result in a dead-lock of the server. When I have something concrete I will make a ticket of this issue.

[theelims]

There are a number of error messages coming from the AsnycTCP and AsnycWebServer libraries. Unfortunately they are largely unmaintained and rather buggy. I included the most stable versions and included all pull request I could gather from various forks that seemed usable. Plus I added some of my own. However, their queuing mechanism has a memory leak resulting in a crash if too much of these show up ERROR: Too many SSE messages queued for [192.168.1.125]. Same for websocket. Also increasing any message rate to > 20Hz is rather dangerous and results in a crash, too. But I couldn't locate the bug, yet.

[EvEggelen]

@theelims
Great to hear that you included the latest fixes. I quickly look at the AsyncTCP, and indeed it seems unmaintained. But I see forks popping-up left and right. One of them is : https://github.com/TienHuyIoT/AsyncTCP that has fixes of 2 days ago. I assume you are aware of the "new" forks. Do you know if this is just a collection of fixes in the unhandled pull requests, or do they include new fixes ?

But these kind of issues related to message rate are in my view not really nice. Not sure what the exact root cause is, but based on what you indicate, I would not be amazed that if the Wifi connection is really crappy, this happens at lower rates too.

As I wanted to use ESP32-sveltekit as foundation for my new project I was hoping for a stable and maintained core. Do you know if there are stable and maintained alternatives for AsnycTCP and AsnycWebServer ?

[theelims]

I do follow the what I think best maintained forks https://github.com/OttoWinter/AsyncTCP and https://github.com/esphome/ESPAsyncWebServer

Since Many big ESP open source projects use these libraries I'm not overly concerned. As long as you keep out of the danger zone of fast websocket and SSE the code should be pretty reliable.

[EvEggelen]

After testing I notice that the current SW is not stable. This is even with "low" amount of messages. I noticed that the WiFi connection is bad, seen CORRUPT HEAP and ERROR: Too many SSE messages queued for [�␌�?␟] in the log. I initially expected it was the HW or WiFi reception/HW. I tested different boards, different (lab) power supplies but this does not seem to fix the issue.

As a test I compiled the original ESP8266 React project for the nodemcu-32s board and flashed it on the same HW ( functionality is simular ). So the only thing what is different is the SW on the device ( WiFi signal, power supply, board and physical location are all the same). In this case the SW is rock stable. I had 4 devices connected, went crazy with browser refreshes and let it run for hours ( with the LightState toggle in main) . I could not find any problems with that SW.

I find it hard to reproduce the issues deterministic with ESP32-sveltekit. But refreshing the page, connecting with both AP and Wifi normally triggers issues ( so with 2 devices). I sometimes also notice that the number of connected devices keeps increasing while the number of connected devices is the same. Is there also code to disconnect devices ( release memory) ? Could the bad Wifi issue a scheduling problem that conflicts with WiFi reception framework ( and that this triggers other issues) ? I also noticed power consumption is very different than from ESP8266 React ( stable 130mA) while with ESP32-sveltekit I have it seen dropping to ~20mA.

I would love to solve this issue. What is best to take as a next step. I can create some more logs. Would that help ?

Added code for the automatic LightState toggle.

void loop()
{
    static bool bLocalState = true;
    lightStateService.update([&](LightState &state)
                             {
                                 if (bLocalState == state.ledOn)
                                 {
                                     return StateUpdateResult::UNCHANGED; // lights were already on, return UNCHANGED
                                 }

                                 state.ledOn = bLocalState;
                                 return StateUpdateResult::CHANGED; // notify StatefulService by returning CHANGED
                             },
                             "timer");

    usleep(2 * 1000000);
    bLocalState = !bLocalState;
    // Delete Arduino loop task, as it is not needed in this example
    // vTaskDelete(NULL);
}

Logs:

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12732
␛[33m=> 0x40078000: ?? ??:0␛[0m
ho 0 tail 12 room 4
load:0x40080400,len:2908
␛[33m=> 0x40080400: _init at ??:?␛[0m
entry 0x400805c4
␛[33m=> 0x400805c4: ?? ??:0␛[0m
[    34][D][esp32-hal-cpu.c:244] setCpuFrequencyMhz(): PLL: 480 / 2 = 240 Mhz, APB: 80000000 Hz
[    71][W][WiFiGeneric.cpp:1403] setTxPower(): Neither AP or STA has been started
[    92][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 0 - WIFI_READY
[   190][V][WiFiGeneric.cpp:340] _arduino_event_cb(): STA Started
[   193][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: [   STA][V][RTFiGener c94]p:][W] _iGeuinic.vpnt1042): evA tCalped
(): esp_wifi_set_ps failed
[   201][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 3 - STA_STOP
Running Firmware Version: 0.2.2
Connecting to WiFi.
[   532][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 0 - WIFI_READY
[   541][V][WiFiGeneric.cpp:340] _arduino_event_cb(): STA Started
[   542][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 2 - STA_START
[   543][V][WiFiGeneric.cpp:97] set_esp_interface_ip(): Configuring Station static IP: 0.0.0.0, MASK: 0.0.0.0, GW: 0.0.0.0
Starting software access point
[   581][V][WiFiGeneric.cpp:392] _arduino_event_cb(): AP Started
[   582][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 10 - AP_START
[   586][V][WiFiGeneric.cpp:97] set_esp_interface_ip(): Configuring SoftAP static IP: 192.168.4.1, MASK: 255.255.255.0, GW: 192.168.4.1
[   597][V][WiFiGeneric.cpp:143] set_esp_interface_ip(): SoftAP: 192.168.4.1 | Gateway: 192.168.4.1 | DHCP Start: 0.0.0.0 | Netmask: 255.255.255.0
[   608][V][WiFiGeneric.cpp:190] set_esp_interface_ip(): DHCP Server Range: 192.168.4.2 to 192.168.4.12
[  1079][V][WiFiGeneric.cpp:395] _arduino_event_cb(): AP Stopped
[  1080][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 11 - AP_S[ P1
81][V][WiFiGeneric.cpp:392] _arduino_event_cb(): AP Started
[  1087][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 10 - AP_START
Starting captive portal on 192.168.4.1
[  2884][V][WiFiGeneric.cpp:362] _arduino_event_cb(): STA Disconnected: SSID: SpeedTouch12, BSSID: b8:d5:26:d1:78:e1, Reason: 2
[  2885][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[  2892][W][WiFiGeneric.cpp:1057] _eventCallback(): Reason: 2 - AUTH_EXPIRE
[  2899][D][WiFiGeneric.cpp:1077] _eventCallback(): WiFi Reconnect Running
WiFi Disconnected. Reason code=2
WiFi connection dropped, stopping NTP.
[  3383][V][WiFiGeneric.cpp:362] _arduino_event_cb(): STA Disconnected: SSID: SpeedTouch12, BSSID: b8:d5:26:d1:78:e1, Reason: 203
[  3384][D[ Wi3iGe[V]iWicpGen035c cppent3a]lbark():o_eduito Eve: S 5 S STAedI
CONNECTED
[  3391][W][WiFiGeneric.cpp:1057] _eventCallback(): Reason: 203 - ASSOC_FAIL
WiFi Disconnected. Reason code=203
WiFi connection dropped, stopping NTP.
[  3409][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 3 - STA_STOP
Connecting to WiFi.
[ 30514][V][WiFiGeneric.cpp:340] _arduino_event_cb(): STA Started
[ 30514][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 2 - STA_START
[ 30515][V][WiFiGeneric.cpp:97] set_esp_interface_ip(): Configuring Station static IP: 0.0.0.0, MASK: 0.0.0.0, GW: 0.0.0.0
[ 31143][V][WiFiGeneric.cpp:355] _arduino_event_cb(): STA Connected: SSID: SpeedTouch12, BSSID: b8:d5:26:d1:78:e1, Channel: 3, Auth: WPA2_PSK
[ 31145][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 4 - STA_CONNECTED
WiFi Connected.
[ 32174][V][WiFiGeneric.cpp:369] _arduino_event_cb(): STA Got New IP:192.168.1.24
[ 32174][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 7 - STA_GOT_IP
[ 32177][D][WiFiGeneric.cpp:1098] _eventCallback(): STA IP: 192.168.1.24, MASK: 255.255.255.0, GW: 192.168.1.1
WiFi Got IP. localIP=192.168.1.24, hostName=esp32
Got IP address, starting NTP Synchronization
Starting NTP...
[ 33108][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 33108][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 33108][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 33110][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 33115][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 33120][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
New client connected to Event Source: 1 Clients connected
[ 37742][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
Stopping captive portal
Stopping software access point
[ 41056][V][WiFiGeneric.cpp:395] _arduino_event_cb(): AP Stopped
[ 41058][V][WiFiGeneric.cpp:392] _arduino_event_cb(): AP Startedt: 11 - AP_STOP
[ 41064][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 10 - AP_START
[ 41071][V][WiFiGeneric.cpp:395] _arduino_event_cb(): AP Stopped
[ 41077][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 11 - AP_STOP
[ 99875][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
New client connected to Event Source: 1 Clients connected
CORRUPT HEAP: Bad head at 0x3ffdb172. Expected 0xabba1234 got 0x00103ffd

Log 2

WiFi Connected.
[421063][V][WiFiGeneric.cpp:369] _arduino_event_cb(): STA Got Same IP:192.168.1.148
[421063][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 7 - STA_GOT_IP
[421066][D][WiFiGeneric.cpp:1098] _eventCallback(): STA IP: 192.168.1.148, MASK: 255.255.255.0, GW: 192.168.1.1
WiFi Got IP. localIP=192.168.1.148, hostName=esp32.home
Got IP address, starting NTP Synchronization
Starting NTP...
[458360][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[458361][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[458362][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[458369][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
New client connected to Event Source: 1 Clients connected
[461813][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
CORRUPT HEAP: Bad tail at 0x3ffdff68. Expected 0xbaad5678 got 0xbaad569e
CORRUPT HEAP: Bad tail at 0x3ffdf2dc. Expected 0xbaad5678 got 0xbaad569e
[492229][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[495183][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
[495189][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
[495191][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
New client connected to Event Source: 1 Clients connected

[ 45190][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 45190][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 45191][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 45203][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 45675][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 45675][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 45675][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 46158][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[ 48638][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
[ 49638][W][AsyncTCP.cpp:999] _poll(): rx timeout 4
New client connected to Event Source: 3 Clients connected
New client connected to Event Source: 4 Clients connected
New client connected to Event Source: 5 Clients connected
New client connected to Event Source: 6 Clients connected
New client connected to Event Source: 7 Clients connected
CORRUPT HEAP: Bad tail at 0x3ffe17bc. Expected 0xbaad5678 got 0xbaad573d
CORRUPT HEAP: Bad tail at 0x3ffdfff0. Expected 0xbaad5678 got 0xbaad569f
CORRUPT HEAP: Bad tail at 0x3ffe2734. Expected 0xbaad5678 got 0xbaad573f
CORRUPT HEAP: Bad tail at 0x3ffe130c. Expected 0xbaad5678 got 0xbaad573f
CORRUPT HEAP: Bad tail at 0x3ffccf10. Expected 0xbaad5678 got 0xbaad569f
CORRUPT HEAP: Bad tail at 0x3ffdf67c. Expected 0xbaad5678 got 0xbaad5736
CORRUPT HEAP: Bad tail at 0x3ffdffd0. Expected 0xbaad5678 got 0xbaad5736
[4962027][W][AsyncTCP.cpp:976] _poll(): pcb is NULL
[4962028][W][AsyncTCP.cpp:976] _poll(): pcb is NULL

Log 3

[2762402][V][WiFiGeneric.cpp:362] _arduino_event_cb(): STA Disconnected: SSID: SpeedTouch12, BSSID: b8:d5:26:d1:78:e1, Reason: 2
[2762402][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[2762410][W][WiFiGeneric.cpp:1057] _eventCallback(): Reason: 2 - AUTH_EXPIRE
WiFi Disconnected. Reason code=2
WiFi connection dropped, stopping NTP.
[2762429][V][WiFiGeneric.cpp:343] _arduino_event_cb(): STA Stopped
[2762429][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 3 - STA_STOP
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for [�␌�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [�␌�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [�␌�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [�␌�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for [�␂�?␟]
ERROR: Too many SSE messages queued for [�      �?␟]
ERROR: Too many SSE messages queued for [�      �?␟]
ERROR: Too many SSE messages queued for [�      �?␟]
ERROR: Too many SSE messages queued for [�      �?␟]
ERROR: Too many SSE messages queued for [�      �?␟]
ERROR: Too many SSE messages queued for [�      �?␟]
ERROR: Too many SSE messages queued for [�      �?␟]
ERROR: Too many SSE messages queued for [�      �?␟]
ERROR: Too many SSE messages queued for [�      �?␟]
ERROR: Too many SSE messages queued for [$�?␟]
ERROR: Too many SSE messages queued for [$�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [�␌�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [�␌�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [<      �?␟]
ERROR: Too many SSE messages queued for [<      �?␟]
ERROR: Too many SSE messages queued for [<      �?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for []
ERROR: Too many SSE messages queued for [�␌�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [�␌�?␟]
ERROR: Too many SSE messages queued for [P
�?␟]
ERROR: Too many SSE messages queued for [P

[theelims]

@EvEggelen No need to do more tests. I think I know where the issue is. I make use of AsyncWebServers Server Send Messages (esp8266-react doesn't). This seems not so heavily used and still quite buggy. It will happen with Websockets as well. It's the queuing mechanism which does not release memory properly. At least once you start to stream analog sensor readings through it. I reviewed the code section many times, but couldn't find an obvious reason why it crashes.

I'm tracing this bug since a while, but couldn't get a hold of it yet.

[EvEggelen]

Currently I am testing the patch : esphome/ESPAsyncWebServer#24. In the comments I found the following remark

"I have the feeling the hole code is not perfect.
I can crash my ESP32 by just reloading the web site 5-6 times quickly.
So I think the goal currently is not perfect code...but just a bit better."

So I do not expect miracles. But so far it seems indeed an improvement. So far I did not see the error message ERROR: Too many SSE messages queued for. With 2 clients refreshing, I see connect clients go up to 4, but after some time it goes back to 2 again. But the

CORRUPT HEAP: Bad tail at 0x3ffe3ae0. Expected 0xbaad5678 got 0xbaad569d

still pops-up :-(. Strangely enough, it does not take down the application.

[jetpax]

I came across a discussion about ESPAsyncWebserver that I found interesting, with code that supposedly fixes its shortcomings.

It was written by Phil Bowles who alas has since passed away, but seems solid.

Maybe worth looking at...

[EvEggelen]

@jetpax Thanks for posting this information. I watched all Phil videos and get the impression he did quit some work on this. Actually, he reports the exact same issues we are seeing ( instability ). If his info is valid, these ASync libs currently used in ESP32-sveltekit are so riddled with issues that I prefer not to use them in framework that has the goal to be used as a stable foundation. Also I got the impression he put a lot of effort in rewriting the whole stack and making a new stable foundation as a "bug" fix here and there will not do the job in the Async stack of libraries. Unfortunately, if he passed away ( where did you see this ), I am wondering if it is realistic to expect that his work will be maintained / matured.

Did anyone ever consider using the : https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/esp_http_server.html. I did a quick test and it seems to work. I expect this version is stable and will get the maintenance when needed. So potentially this is more future proof ?

[jetpax]

Yes @theelims, I agree with you re maturity/long term support of H4, (even though Phil's work seems to have been adopted (philbowles/h4plugins@master...HamzaHajeir:h4plugins:master)) and think the espressif stack is the way to go, esp. as it has been around long enough to mature (IMO espressif tend to release early, release often)
They also support MQTT V5.0 with topic aliases I believe, which is great for reducing data usage in mobile applications (which I am personally interested in)
Can I ask why did you opt for Arduino framework rather than esp-idf, as I find it more solid in general?