BG96 - automatic CMUX mode (ESP_MODEM_MODE_CMUX) - how to reinit PPPOS (IDFGH-14662)

[gmorfeus]

Answers checklist.

• I have read the documentation for esp-protocols components and the issue is not addressed there.
• I have updated my esp-protocols branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

I've implemented PPPOS as in the example and all works ok except that, after an hour of being connected, I get IP_EVENT_PPP_LOST_IP. So I tryed to make reconnection. What I tested for now is:

1. detach from GPRS (esp_modem_set_network_attachment_state), check connection to GSM and GPRS network and reattach to GPRS (esp_modem_set_network_attachment_state). All ok, but PPP doesn't connect. BG96 responds to AT commands and reported mode is CMUX mode. Modem status LED blinks as still attached to GPRS network.
2. detach from GPRS, switch to command mode, check connection to network, reattache to GPRS and then switch to MUX mode, I get error -1 to CMUX switch command and BG96 stops responding to AT commands with CME ERROR: 13 (as a response to +CREG while +CREG returning +CREG: 0,0). In this case, modem status led blinks as it is completely disconnected from GSM netwrok.

What to do and how to make PPPOS reconnect?
Thanks.

[david-cermak]

I'd suggest using CMUX manual mode if you need to resume PPP connection on the data channel.
In the default CMUX mode (auto), the only option is to exit the CMUX and enter again. And CMUX exit is not very well supported across devices, although I think it should with BG96.

[gmorfeus]

Hi David,
thanks. It looks like some kind of "lease time" from operator because I see PPPOS drop exactly 1 hour after first connect. In such condition, you propose either:

1. from CMUX go to ESP_MODEM_MODE_COMMAND mode, check connection to GSM/GPRS network and return to ESP_MODEM_MODE_CMUX. I tried this but I get error when trying to go to CMUX mode. The return error is -1 (ESP_FAIL). as you wrote "...exit the CMUX and enter again..." you mean entering COMMAND mode and then CMUX mode?
   or
2. on first start, go to ESP_MODEM_CMUX_MANUAL and, on disconnect, go to ESP_MODEM_CMUX_EXIT and then, after a couple of seconds, enter ESP_MODEM_CMUX_MANUAL again?

In current CMUX mode, I am using PPPOS and also get +CMTI and signal change events. How that changes if I use 2nd method (CMUX_MANUAL)? Or better question, in manual cmux mode, how to use PPPOS and AT at the same time?

Thanks,
Gabrijel

[david-cermak]

Hi Gabrijel,

Yes, that's what I meant:

1. yes -- switch to command mode and back to cmux
2. by using cmux manual manual, you need to first switch to the cmux manual and then use all the submodes to control the devices' terminals separately, by using cmux-manual-data (switching current terminal to PPPoS) and/or cmux-manual-swap (to switch between virtual terminals). I'd suggest playing with these modes in the console example, there're commands like set_mode CMUX1..CMUX5 corresponding to each of the manual aciton.

further than that, you can also try mode detection. This would send some probes to the device in order to detect which mode it's actually in. In case of CMUX it'd go over the virtual terminals and runs detection on these as well.
Can you try to run the autodetect after your PPPoS session drops and share the result here? (just for me to understand which state your device is in -- assuming CMUX manual with both terminals in AT modes)

Thanks,
David

[gmorfeus]

Hi David,

I already tested CMUX auto->COMMAND->CMUX auto but the modem refused to enter CMUX modem returning -1 as error code. It's the same as if I try to enter CMUX mode without exiting to COMMAND mode. What I also tried was just check what mode esp_modem is when PPPOS drops, and it reported mode 0. I'll try DETECT and switch to CMUX to see if that'll do any difference.

If I understand you well, the best way to go is CMUX manual mode. However, I am a little confused about MANUAL_DATA, MANUAL_COMMAND and MANUAL_SWAP. Can you please clarify this ( from: https://docs.espressif.com/projects/esp-protocols/esp_modem/docs/latest/README.html):
"Note that transitioning between “MANUAL_DATA” and “MANUAL_CMD” switches the secondary terminal (dedicated to PPP session) and could be used for recovering data communication if PPP session gets dropped."
What does MANUAL_SWAP mode do? If I understand well, MANUAL_DATA and MANUAL_CMD "knows" what terminal it must work on?
Or better, let me write how I understand CMUX manual mode works: it brings two virtual serial communication to the modem (let's call them T1 and T2). When CMUX_MANUAL is set, it activates T1 and T2 in CMD mode. Executing MANUAL_DATA brings T1 to data mode while T2 is still in CMD mode. Then I call SWAP and continue working with AT commands untill PPPOS drops and when that happens, I have to call SWAP, execute CMUX_DATA and SWAP again to T2 for AT commands. Did I understand well?

For the autodetect results after in CMUX manual mode PPPOS drops...I'll make a log using console example and post here. Do you want full log communication with the modem or just what DETECT mode reports?

Regards,
Gabrijel

[gmorfeus]

Hi David,

using modem_console, PPPOS is now active for almost 3hours! What happened is that my computer went to BSoD so I had to restart it (and had to disconnect serial to ESP32S3, while still having external power on my board). After restart, I connected to serial monitor and that restarted ESP32. Then I tries to execute AT commands but it was an error. So i entered set_mode AUTO and this set all to proper state(s). Then I tried to switch to any other mode but I had no success. See all in log.
log_BG96.txt

Im my code, I've used PPPOS example and implemented all required. What's different from modem_console is that I am using C and not C++. I also configure and bring up AP mode on Wifi of ESP32 before starting up PPPOS. Can something of that be the reason of PPPOS disconnect after 1 hour?

BR,
Gabrijel

[david-cermak]

So i entered set_mode AUTO and this set all to proper state(s). Then I tried to switch to any other mode but I had no success.

If you're in some CMUX mode set_mode AUTO would setup CMUX manual (it can't use AUTO, as the terminals could be in different states). At this point you need to use cmux manual modes (like CMUX3, 4, 5), you can also query the actual mode.

I also configure and bring up AP mode on Wifi of ESP32 before starting up PPPOS. Can something of that be the reason of PPPOS disconnect after 1 hour?

Cannot think of anything that would cause these disconnections, it's really weird that it happens always, exactly after 1 hour. Do you have any client connected to the AP? what's the DHCP lease time? do you see any disconnection/lease renewal on the client side? Are you using NAT to route packets from PPPoS?

[gmorfeus]

Hi,

sorry for the delay. To answer the questions, the disconnect happens regardless if there are client connected to AP or not. Lease time is default and I didn't noticed any renevals on client side (I didn't even look for them). Beside default network settings, I am not using anything special on ESP32 (it's almost that I have used example for PPPOS and AP mode putting all together. I first init network then AP and then PPPOS).

In the end, I did as you proposed and use CMUX manual mode(s). After attaching to GPRS network, I enter CMUX manual mode and then switch to CMUX manual data mode. When I get disconnected event, I switch to CMUX manual command mode, check signal and connection to GPRS network and then enter CMUX data mode. This makes modem connected to PPPOS again and all working fine. There is no need to use CMUX switch. So this seems to solve my problem. I am still not sure what happens when using CMUX (automatic) and DETECT mode but it looks like the esp_modem module gets "lost" in some state-machine "state" and replay to all other modes as "error" when trying to switch modes.

Gabrijel