Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ESP32-S* Safe Mode when using HTTPS / TLS TCP server #9937

Open
anecdata opened this issue Jan 4, 2025 · 4 comments
Open

ESP32-S* Safe Mode when using HTTPS / TLS TCP server #9937

anecdata opened this issue Jan 4, 2025 · 4 comments
Labels
bug crash espressif applies to multiple Espressif chips
Milestone
9.2.x

Comments

@anecdata
Copy link
Member

anecdata commented Jan 4, 2025 •
edited
Loading

CircuitPython version

Adafruit CircuitPython 9.2.1-45-ge596261dc1-dirty on 2024-12-29; Adafruit Feather ESP32-S3 TFT with ESP32S3

Adafruit CircuitPython 9.2.1-43-g2c720d9161-dirty on 2024-12-28; Waveshare ESP32-S3-ETH with ESP32S3

Code/REPL

Various comments and example code using adafruit_httpserver library and manual TLS TCP socket server at: adafruit/Adafruit_CircuitPython_HTTPServer#88

Using latest HTTP Server library from that PR. Safe mode occurs with 9.2.1, or with DEBUG build from recent tip of main.

It is not necessary to have any client attempt to connect.

Behavior

Intermittent safe mode when running HTTPS servers, can be several minutes or a couple hours. DEBUG output shows crashes on Core 0 or on Core 1, with varying error messages. Not a lot of consistency. DEBUG backtraces are either corrupted or yield no useful output ( decode_backtrace.py just prints a number of blank lines).

Changing to a blocking accept() call seems to fix the issue, or at least I have not observed any Safe Mode occurrences when using blocking accept().

Tested predominantly on adafruit_feather_esp32s3_tft and waveshare_esp32_s3_eth (using wifi). @michalpokusa observed the issue on a Memento board.

Example adafruit_feather_esp32s3_tft debug output 
W (13831) CP wifi: got ip
Guru Meditation Error: Core  1 panic'ed (Unhandled debug exception). 
Debug exception reason: BREAK instr 
Core  1 register dump:
PC      : 0x403743c0  PS      : 0x00060536  A0      : 0x4038d768  A1      : 0x3fca4020  
A2      : 0x3c1afb40  A3      : 0x3c1a15b4  A4      : 0x0000003f  A5      : 0x00000030  
A6      : 0x0000000c  A7      : 0x00000df9  A8      : 0x00000001  A9      : 0x3fcac68c  
A10     : 0x00000001  A11     : 0x3fcb7888  A12     : 0x80384cbd  A13     : 0x3fcba5f0  
A14     : 0x00060023  A15     : 0x00000003  SAR     : 0x00000019  EXCCAUSE: 0x00000001  
EXCVADDR: 0x4038d738  LBEG    : 0x40056f5c  LEND    : 0x40056f72  LCOUNT  : 0xffffffff  

Backtrace: 0x403743bd:0x3fca4020 0x4038d765:0x4038d768 |<-CORRUPTED

ELF file SHA256: 30a3826f5

CPU halted.

- - - -        

W (13763) CP wifi: got ip
Guru Meditation Error: Core  0 panic'ed (Double exception). 

Core  0 register dump:
PC      : 0x4038db12  PS      : 0x00040e36  A0      : 0x00000000  A1      : 0x3fcba710  
A2      : 0x00060523  A3      : 0x00000000  A4      : 0x00060520  A5      : 0x00000004  
A6      : 0x3fcac880  A7      : 0x3fcac884  A8      : 0x600c0034  A9      : 0x00000001  
A10     : 0x3fc9e6cc  A11     : 0xffffffff  A12     : 0xb33fffff  A13     : 0x00060523  
A14     : 0x0000001e  A15     : 0x0000abab  SAR     : 0x0000001c  EXCCAUSE: 0x00000002  
EXCVADDR: 0x00060513  LBEG    : 0x420a8e74  LEND    : 0x420a8e87  LCOUNT  : 0x3fcba7b0  

Backtrace: 0x4038db0f:0x3fcba710

ELF file SHA256: 30a3826f5

CPU halted.

- - - -

W (13840) CP wifi: got ip
Guru Meditation Error: Core  1 panic'ed (Unhandled debug exception). 
Debug exception reason: BREAK instr 

Backtrace: 0x403743bd:0x3fcac68c |<-CORRUPTED

Backtrace: 0xfffffffd:0x80379207 |<-CORRUPTED

ELF file SHA256: 30a3826f5

CPU halted.

- - - -

Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Backtrace: 0x42013059:0x3fcba740 0x3fcba77d:0x3fcba780 |<-CORRUPTED
Example waveshare_esp32_s3_eth debug output 
W (15512) CP wifi: got ip
Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x403823ca  PS      : 0x00060e34  A0      : 0x820a9fed  A1      : 0x3fcc15c0  
A2      : 0x00060e23  A3      : 0x00000000  A4      : 0x00060e20  A5      : 0x3fcc7080  
A6      : 0x3fcbcea8  A7      : 0x3fcbce90  A8      : 0x80378eae  A9      : 0x3fcc15a0  
A10     : 0x3fcbcea8  A11     : 0x00000001  A12     : 0x00000000  A13     : 0x00000000  
A14     : 0x00060023  A15     : 0x00000003  SAR     : 0x0000001d  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  

Backtrace: 0x403823c7:0x3fcc15c0 0x420a9fea:0x3fcc15e0 0x420ab3af:0x3fcc1600 0x4038775b:0x3fcc1620

Core  1 register dump:
PC      : 0x42003475  PS      : 0x00060d34  A0      : 0x82003799  A1      : 0x3fcc0820  
A2      : 0x3fca9dc8  A3      : 0x000005e0  A4      : 0x00000001  A5      : 0x00000075  
A6      : 0x3c1e1a10  A7      : 0x000000ac  A8      : 0x3c1e1a20  A9      : 0x3c1e3330  
A10     : 0x0000005f  A11     : 0x00000010  A12     : 0x00000000  A13     : 0x3fca9c30  
A14     : 0x0000003f  A15     : 0x00000003  SAR     : 0x00000006  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x42003d2c  LEND    : 0x42003d80  LCOUNT  : 0x00000000  

Backtrace: 0x42003472:0x3fcc0820 0x42003796:0x3fcc0840 0x4202504a:0x3fcc0860 0x42003eb7:0x3fcc08b0 0x42003322:0x3fcc08f0 0x4201420a:0x3fcc0910 0x4200e5a5:0x3fcc0930 0x42010046:0x3fcc0960 0x420100a9:0x3fcc0980 0x4204ce09:0x3fcc09a0 0x4203ef52:0x3fcc09c0 0x42015026:0x3fcc09f0 0x4200eb8d:0x3fcc0a10 0x4200ec55:0x3fcc0a30 0x4206a8f7:0x3fcc0a50 0x4206affd:0x3fcc0a70 0x42058249:0x3fcc0aa0 0x42015026:0x3fcc0ac0 0x4200eb8d:0x3fcc0ae0 0x4200ec55:0x3fcc0b00 0x420203e9:0x3fcc0b20 0x420151d9:0x3fcc0bc0 0x4200eb8d:0x3fcc0be0 0x4200ec55:0x3fcc0c00 0x420203e9:0x3fcc0c20 0x420151d9:0x3fcc0cc0 0x4200eb8d:0x3fcc0ce0 0x4200eba2:0x3fcc0d00 0x42071c36:0x3fcc0d20 0x4207205e:0x3fcc0dc0 0x420245c1:0x3fcc0de0 0x42024ab9:0x3fcc0e00 0x42025006:0x3fcc0e60 0x42026d2a:0x3fcc0e90 0x42168480:0x3fcc0eb0

ELF file SHA256: cb964ad5b

CPU halted.

- - - - -

W (15452) CP wifi: got ip
Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x403823ca  PS      : 0x00060c34  A0      : 0x820a9fed  A1      : 0x3fcc15c0  
A2      : 0x00060c23  A3      : 0x00000000  A4      : 0x00060c20  A5      : 0x3fcbca20  
A6      : 0x3fcbcea8  A7      : 0x3fcbce90  A8      : 0x80378eae  A9      : 0x3fcc15a0  
A10     : 0x3fcbcea8  A11     : 0x00000001  A12     : 0x00000000  A13     : 0x00000000  
A14     : 0x01ffffff  A15     : 0x00000003  SAR     : 0x0000001d  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  

Backtrace: 0x403823c7:0x3fcc15c0 0x420a9fea:0x3fcc15e0 0x420ab3af:0x3fcc1600 0x4038775b:0x3fcc1620

Core  1 register dump:
PC      : 0x420207fd  PS      : 0x00060334  A0      : 0x820151dc  A1      : 0x3fcc0b20  
A2      : 0x3c1f7642  A3      : 0x3c1f75c8  A4      : 0x3c1e0da4  A5      : 0x3c184b08  
A6      : 0x3c1f7680  A7      : 0x3c1f75ca  A8      : 0x3c1e0d94  A9      : 0x4201fd70  
A10     : 0x3c184b08  A11     : 0x3c1e0de4  A12     : 0x3c182154  A13     : 0x3fcc0bb0  
A14     : 0x3fcc0b48  A15     : 0x00000008  SAR     : 0x00000020  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x400570e8  LEND    : 0x400570f3  LCOUNT  : 0x00000000  

Backtrace: 0x420207fa:0x3fcc0b20 0x420151d9:0x3fcc0bc0 0x4200eb8d:0x3fcc0be0 0x4200ec55:0x3fcc0c00 0x420203e9:0x3fcc0c20 0x420151d9:0x3fcc0cc0 0x4200eb8d:0x3fcc0ce0 0x4200eba2:0x3fcc0d00 0x42071c36:0x3fcc0d20 0x4207205e:0x3fcc0dc0 0x420245c1:0x3fcc0de0 0x42024ab9:0x3fcc0e00 0x42025006:0x3fcc0e60 0x42026d2a:0x3fcc0e90 0x42168480:0x3fcc0eb0

ELF file SHA256: cb964ad5b

CPU halted.

- - - -

I (28429532) wifi:<ba-add>idx:1 (ifx:0, a8:c0:ea:ab:f5:26), tid:6, ssn:2, winSize:64
Guru Meditation Error: Core  1 panic'ed (Interrupt wdt timeout on CPU1). 

Core  1 register dump:
PC      : 0x400570ec  PS      : 0x00060f34  A0      : 0x82003f11  A1      : 0x3fcc0910  
A2      : 0x3c1f5d70  A3      : 0x00000000  A4      : 0x00000030  A5      : 0x3c1f5d80  
A6      : 0x00000002  A7      : 0x00000003  A8      : 0x8200ec58  A9      : 0x3fcc0a10  
A10     : 0x3c19087c  A11     : 0x00000001  A12     : 0x00000000  A13     : 0x3c1efeb8  
A14     : 0x3c1f7d70  A15     : 0x3c1f7c0d  SAR     : 0x0000001c  EXCCAUSE: 0x00000006  
EXCVADDR: 0x00000000  LBEG    : 0x400570e8  LEND    : 0x400570f3  LCOUNT  : 0x00000001  

Backtrace: 0x400570e9:0x3fcc0910 0x42003f0e:0x3fcc0920 0x42003339:0x3fcc0960 0x420113f1:0x3fcc0980 0x4204cded:0x3fcc09a0 0x4203ef52:0x3fcc09c0 0x42015026:0x3fcc09f0 0x4200eb8d:0x3fcc0a10 0x4200ec55:0x3fcc0a30 0x4206a8f7:0x3fcc0a50 0x4206affd:0x3fcc0a70 0x42058249:0x3fcc0aa0 0x42015026:0x3fcc0ac0 0x4200eb8d:0x3fcc0ae0 0x4200ec55:0x3fcc0b00 0x420203e9:0x3fcc0b20 0x420151d9:0x3fcc0bc0 0x4200eb8d:0x3fcc0be0 0x4200ec55:0x3fcc0c00 0x420203e9:0x3fcc0c20 0x420151d9:0x3fcc0cc0 0x4200eb8d:0x3fcc0ce0 0x4200eba2:0x3fcc0d00 0x42071c36:0x3fcc0d20 0x4207205e:0x3fcc0dc0 0x420245c1:0x3fcc0de0 0x42024ab9:0x3fcc0e00 0x42025006:0x3fcc0e60 0x42026d2a:0x3fcc0e90 0x42168480:0x3fcc0eb0

Core  0 register dump:
PC      : 0x403823ca  PS      : 0x00060634  A0      : 0x820a9fed  A1      : 0x3fcc15c0  
A2      : 0x00060623  A3      : 0x00000000  A4      : 0x00060620  A5      : 0x03ca3430  
A6      : 0x3fcbcea8  A7      : 0x3fcbce90  A8      : 0x80378eae  A9      : 0x3fcc15a0  
A10     : 0x3fcbcea8  A11     : 0x00000001  A12     : 0x00000000  A13     : 0x00000000  
A14     : 0x00060023  A15     : 0x00000003  SAR     : 0x0000001d  EXCCAUSE: 0x00000006  
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  

Backtrace: 0x403823c7:0x3fcc15c0 0x420a9fea:0x3fcc15e0 0x420ab3af:0x3fcc1600 0x4038775b:0x3fcc1620

ELF file SHA256: cb964ad5b

CPU halted.

Description

No response

Additional information

No response
@anecdata anecdata added the bug label Jan 4, 2025
@michalpokusa
Copy link

Possibly related #9003

@dhalbert dhalbert added this to the 9.2.x milestone Jan 6, 2025
@anecdata anecdata mentioned this issue Jan 6, 2025
Open
@anecdata
Copy link
Member Author

anecdata commented Jan 6, 2025

Tested ESP32-S2 (Feather TFT) with the fix from "espressif: Allow mbedtls data in psram if available" #9867. The memory fix allows HTTPS server to run on ESP32-S2, but S2 is prone to intermittent Safe Mode like S3.

@anecdata anecdata changed the title ESP32-S3 Safe Mode when using HTTPS / TLS TCP server ESP32-S* Safe Mode when using HTTPS / TLS TCP server Jan 6, 2025
@dhalbert
Copy link
Collaborator

dhalbert commented Jan 6, 2025

This still sounds like there is some stray pointer or use-after-free or something going on. The ability to use PSRAM may make the bug less likely.

@anecdata
Copy link
Member Author

anecdata commented Jan 6, 2025

In continued testing, I've still seen no safe modes with blocking accept(). Safe mode seems to be correlated with non-blocking accept().

@tannewt tannewt added espressif applies to multiple Espressif chips crash labels Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug crash espressif applies to multiple Espressif chips
Projects
None yet
Milestone
9.2.x
Development

No branches or pull requests
4 participants
@tannewt @dhalbert @anecdata @michalpokusa