Moving from 8266 to esp32

[MassiPi]

Hello all,
some context
some years ago, in the me-no-dev era, i used this wonderful library for a lot of things. among them, the controller of my daiking splits. It has a web interface talking with the esp8266 through websockets. As soon as the client connects, the server sends it 6 packages of data.
It talks with home assistant over mqtt. I'm now moving to an encrypted MQTT so i need to move to esp32. I have a esp32-c3-supermini.
the issue
I expected all, but not to find it impossible to make it work. I fixed the code, new libraries, some fixes here and there, but the point is that:

• the webserver seems far slower than the one on the 8266 (i mean, much slower)
• mostly, it crashes after sending the first websocket message. I tried changing queue, affinity, delay between messages, it still crashes after the first message

Since it can't be possible, i'm clearly missing something and this is the question: any big topic i'm missing moving from 8266 to 32?

The debug info, in case..

CORRUPT HEAP: Bad tail at 0x3fcb528c. Expected 0xbaad5678 got 0xbaad5600

assert failed: multi_heap_free multi_heap_poisoning.c:259 (head != NULL)
Core  0 register dump:
MEPC    : 0x40381d70  RA      : 0x40386622  SP      : 0x3fcb26a0  GP      : 0x3fc8dc00
  #0  0x40381d70 in panic_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:408

TP      : 0x3fc85e6c  T0      : 0x37363534  T1      : 0x7271706f  T2      : 0x33323130
S0/FP   : 0x3fcb2818  S1      : 0x0000007f  A0      : 0x3fcb2704  A1      : 0x3fc90319
A2      : 0x00000001  A3      : 0x00000029  A4      : 0x00000001  A5      : 0x3fc98000
A6      : 0x7a797877  A7      : 0x76757473  S2      : 0x3fcb26f8  S3      : 0x00000001
S4      : 0x3fcb26f8  S5      : 0x4038ba12  S6      : 0x00000000  S7      : 0x00000000
S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000
T3      : 0x6e6d6c6b  T4      : 0x6a696867  T5      : 0x66656463  T6      : 0x62613938
MSTATUS : 0x00001801  MTVEC   : 0x40380001  MCAUSE  : 0x00000007  MTVAL   : 0x00000000
MHARTID : 0x00000000

Stack memory:
3fcb26a0: 0x3fca1bdc 0x00000000 0x3c0f37d4 0x4038beec 0x00000001 0x00000001 0x3fcb27c0 0x3fc90318
  #0  0x4038beec in newlib_include_assert_impl at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c:97
  #1  0x3fc90318 in __global_pointer$ at ??:?

3fcb26c0: 0xffffffff 0x00000000 0x3fcb27c0 0x00393532 0xffffffff 0x3fc90400 0x3c0f37d4 0x3fc907ec
  #0  0x3fc90400 in __global_pointer$ at ??:?
  #1  0x3fc907ec in __global_pointer$ at ??:?

3fcb26e0: 0x3c0f383d 0x3fc90410 0x3fcb26cc 0x3fc90414 0x3c0f3818 0x3fc90318 0x00000000 0x00000000
  #0  0x3fc90410 in __global_pointer$ at ??:?
  #1  0x3fc90414 in __global_pointer$ at ??:?
  #2  0x3fc90318 in __global_pointer$ at ??:?

3fcb2700: 0xff000000 0x65737361 0x66207472 0x656c6961 0x6d203a64 0x69746c75 0x6165685f 0x72665f70
3fcb2720: 0x6d206565 0x69746c75 0x6165685f 0x6f705f70 0x6e6f7369 0x2e676e69 0x35323a63 0x68282039
3fcb2740: 0x20646165 0x4e203d21 0x294c4c55 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcb2760: 0x00000000 0x00000000 0x3c104000 0x3c105000 0x00000001 0x3fcb511c 0x3fcb528c 0x40048722
3fcb2780: 0x00000000 0x3fcb8a5c 0x00000000 0x00000000 0x00000000 0x3fcb8a64 0x3fc99050 0x3fcb27b4
3fcb27a0: 0x00000000 0x00000000 0x3c104000 0x4038bbce 0x3fcb509c 0x3fcb528c 0xbaad5678 0xbaad5600
  #0  0x4038bbce in verify_allocated_region at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c:116

3fcb27c0: 0x00000001 0xbaad5600 0xfa000000 0xdb4dbf99 0x3fc98000 0x00000000 0x3c104000 0x3c105000
3fcb27e0: 0x000000e8 0x3fcb5124 0x3fc99050 0x4038ba12 0x000000e8 0x00000001 0x3fcb5094 0x4200cbbe
  #0  0x4038ba12 in multi_heap_free at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c:266
  #1  0x4200cbbe in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)1>::_M_release() at c:\users\massi\.platformio\packages\toolchain-riscv32-esp\riscv32-esp-elf\include\c++\8.4.0\bits/shared_ptr_base.h:162       
      (inlined by) std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)1>::_M_release() at c:\users\massi\.platformio\packages\toolchain-riscv32-esp\riscv32-esp-elf\include\c++\8.4.0\bits/shared_ptr_base.h:148        

3fcb2800: 0x3fc98608 0x0000016c 0x3fcb3b64 0x42017fbe 0x00000001 0x00000001 0x3fcb3b64 0x42018748
  #0  0x3fc98608 in ?? at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:367
  #1  0x42017fbe in std::deque<AsyncWebSocketMessage, std::allocator<AsyncWebSocketMessage> >::pop_front() at c:\users\massi\.platformio\packages\toolchain-riscv32-esp\riscv32-esp-elf\include\c++\8.4.0\bits/stl_deque.h:1596
      (inlined by) AsyncWebSocketClient::_clearQueue() at .pio/libdeps/wiredDaikin32/ESPAsyncWebServer/src/AsyncWebSocket.cpp:291
  #2  0x42018748 in AsyncWebSocketClient::_onAck(unsigned int, unsigned long) at .pio/libdeps/wiredDaikin32/ESPAsyncWebServer/src/AsyncWebSocket.cpp:332

3fcb2820: 0x00000000 0x00000000 0x3fcb3b80 0x4202d701 0x3c105000 0x3fcb4998 0x3fcb34b8 0x420c23f4
  #0  0x4202d701 in __pinMode at C:/Users/Massi/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-gpio.c:110
  #1  0x420c23f4 in AsyncClient::_sent(tcp_pcb*, unsigned short) at .pio/libdeps/wiredDaikin32/AsyncTCP/src/AsyncTCP.cpp:1054

3fcb2840: 0x00000000 0x00000000 0x00000000 0x0000016c 0x3fcb3b64 0x3fcb34b8 0x0000016c 0x000000e8
3fcb2860: 0x3c105000 0x3fcb4998 0x3fcb4998 0x420c2dae 0x6e6d6c6b 0x00000000 0x3fc97a84 0x420c2f80
  #0  0x420c2dae in AsyncTCP_detail::handle_async_event(lwip_tcp_event_packet_t*) at .pio/libdeps/wiredDaikin32/AsyncTCP/src/AsyncTCP.cpp:300
  #1  0x420c2f80 in _async_service_task(void*) at .pio/libdeps/wiredDaikin32/AsyncTCP/src/AsyncTCP.cpp:338

3fcb2880: 0x00000000 0x00000000 0x00000000 0x00000001 0x00000000 0x00000000 0x00000000 0x00000000
3fcb28a0: 0x00000000 0x00000000 0x00000000 0x403893ee 0x00000000 0x00000000 0x00000000 0x00000000
  #0  0x403893ee in _prvTaskExitError at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/riscv/port.c:126
      (inlined by) prvTaskExitError at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/riscv/port.c:140

3fcb28c0: 0x00000000 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xbaad5678 0x00000160 0xabba1234
3fcb28e0: 0x00000154 0x3fcb27e0 0x000041ae 0x3fc941f0 0x3fc941f0 0x3fcb28e4 0x3fc941e8 0x0000000f
3fcb2900: 0x3fcb27f0 0x00003bcb 0x3fcb28e4 0x00000000 0x0000000a 0x3fcae8d4 0x6e797361 0x63745f63
3fcb2920: 0x1ebf0070 0x00cb2900 0x00000000 0x3fcb28d0 0x0000000a 0x00000001 0x3fcaccac 0x42038f22
  #0  0x42038f22 in pthread_local_storage_thread_deleted_callback at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/pthread/pthread_local_storage.c:129

3fcb2940: 0x00000000 0x3fc99a54 0x3fc99abc 0x3fc99b24 0x00000000 0x00000000 0x00000001 0x00000000
3fcb2960: 0x00000000 0x00000000 0x420cc7ae 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
  #0  0x420cc7ae in _cleanup_r at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/riscv32-esp-elf/src/newlib/newlib/libc/stdio/findfp.c:229

3fcb2980: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcb29a0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcb29c0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcb29e0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcb2a00: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcb2a20: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0xbaad5678 0x00000018
3fcb2a40: 0xabba1234 0x0000000c 0x3fcb28e4 0x00000000 0x00000000 0xbaad5678 0x00000020 0xabba1234
3fcb2a60: 0x00000010 0x3fcac7b0 0x00000000 0x00000000 0x3fcacaa8 0xbaad5678 0x3fcb2a54 0x00000060
3fcb2a80: 0xabba1234 0x00000054 0x3fcb2a88 0x3fcb2a88 0x3fcb2a88 0x3fcb2a88 0x00000000 0x3fcb2aa0

ELF file SHA256: cc5bd61659794e12

[mathieucarbou]

Hi,
Clearly there is something wrong in your code.
Do not forget that you have 2 cores for esp32 so concurrency has to be thought about.
You can share your code or, at least, work on a MRE showing how to reproduce that. Doing this effort will allow you to find the cause or help other user help you find the cause.

[MassiPi]

Remember that since asyncws is asynchronous and esp32 multi-core, any callback should also not block the async_tcp task.

ok wait :) not sure i got this point. Should i change my mind on any point compared with the functioning of the original asyncwebserver on ESP8266? the common point i always tried to follow is: in callbacks the very minimum of calculation (in my case it commonly creates jsons / read jsons) and if something needs to be done it is done in the loop (using vars to trigger)

Your stack trace is weird and unusual to me. I don't know fro where you have copy pasted that or how you have parsed it, but its the first time I see one formatted like that (you might use another tool / ide maybe ?)

It comes directly from the monitor windows in VSC+platformio. Enabled with monitor_filters = esp32_exception_decoder in platformio.ini
Again, if i'm doing anything wrong it's very very welcome :)

Also, the lines do not match with this project, both for ESPAsyncWebServer and AsyncTCP. I don't know from where you sourced them, but please make sure you use the official ones and the latest versions also.

this is the very scary part of the message.
My platformio.ini includes this:

lib_deps=
    ESP32Async/AsyncTCP
    ESP32Async/ESPAsyncWebServer

My main.cpp includes this

#include <AsyncTCP.h>
#include <ESPAsyncWebServer.h>

i hope i'm in the correct place :) and i can guarantee i did not touch any code!
Also from versions it seems i'm ok

|-- AsyncTCP @ 3.4.10
|-- ESPAsyncWebServer @ 3.9.6`

(yes i'm giving hope to this and to a stupid mistake to see my code come back to life 🤣 )

[mathieucarbou]

lib_compat_mode = strict
lib_ldf_mode = chain
lib_deps =
  ESP32Async/AsyncTCP @ 3.4.10
  ESP32Async/ESPAsyncWebServer @ 3.9.6

It is possible that you have a lib that brings also another version... look inside your .pio/libdeps diectory.

monitor_filters = esp32_exception_decoder <= yes that is totally OK.

you definitely need to work on a MRE I think...

Is it possible otherwise that you are queuing too many websocket messages, or too big and you do not have enough heap left ?

[MassiPi]

It is possible that you have a lib that brings also another version... look inside your .pio/libdeps diectory.

I've just tried to clean the code from other async libraries, but same behavior. I'll check all libraries i have in libdeps.. thanks

Is it possible otherwise that you are queuing too many websocket messages, or too big and you do not have enough heap left ?

yesterday trying to debug the issue i got the heap at each step, and honestly it does not seem to be the problem (the client receives only the first send, the config

Websocket client connection received
Send 1: Free heap: 175476 Min free: 169128 Max alloc: 143348
Sending config to client
Send 2: Free heap: 175868 Min free: 169128 Max alloc: 143348
Sending sensor data to client
Send 3: Free heap: 175484 Min free: 169128 Max alloc: 143348
Sending compressor data to client
Send 4: Free heap: 175224 Min free: 169128 Max alloc: 143348
Sending info to client
Send 5: Free heap: 174772 Min free: 169128 Max alloc: 143348
Sending start time to client
Send 6: Free heap: 174536 Min free: 169128 Max alloc: 143348
Sending rssi to 1 clients
Over: Free heap: 174320 Min free: 169128 Max alloc: 143348
CORRUPT HEAP: Bad tail at 0x3fcb3ff0. Expected 0xbaad5678 got 0xbaad5600

The sizes of the json sent: with new libraries i can no longer declare the size of jsonducuments, but in the old 8266 code all the buffers together were below 2kb, again i think this is not the issue.

[mathieucarbou]

Here:

https://github.com/MassiPi/DaikinS21/blob/d6f7e9d6db6ef019a3612d11fc4c4639ae5893d6/src/main.cpp#L241-L243

  AsyncWebSocketMessageBuffer * buffer = ws.makeBuffer(len); //  creates a buffer (len + 1) for you.
  if (buffer) {
    serializeJson(root, (char *)buffer->get(), len + 1);

You are overflowing the array.

should be:

  AsyncWebSocketMessageBuffer * buffer = ws.makeBuffer(len);
  if (buffer) {
    serializeJson(root, buffer->get(), len);

[MassiPi]

Since my MRE is as short as hell and still crashes, i was already thinking about json management, also looking at this:
https://github.com/ESP32Async/ESPAsyncWebServer/blob/main/examples/Json/Json.ino
am i still doing right using the ArduinoJson library? (now in version 7, first time i install it)
I wonder because i see i also have an asyncjson included, the example seems to use jsons very differently than me and now you are pointing out a different code for an historical serializeJson, so easy easy i'm doing wrong the management of json itself :) do i still need the arduinojson?

btw, your fix seems to work, i can't understand if it's a change in the function due to arduinojson v7 or it worked for years (and is working right now in my daikin splits) for no reason :)

[mathieucarbou]

makeBuffer implementation was changed over time, and a very long time ago. When changing to another org or upgrading major digit you cannot assume an immediate backward compatibility.