Replies: 22 comments 3 replies
-
Hi @pat-2010 Can you just confirm that the following:
When you say you've used u-center to record logs successfully, I presume you mean on a separate Windows machine? What happens if you use PyGPSClient's datalogging on that same Windows platform - do you still see 'dropouts'? If not, i'm inclined to think this may simply be an SD write latency issue on the RPi, but I will look into it. Out of interest, have you tried using the gnssstreamer CLI utility to record datalogs? e.g. gnssstreamer --port /dev/tty.usbmodem1201 --baudrate 38400 --timeout 3 --format 16 --verbosity 2 --clioutput 5 --output datalog.log |
Beta Was this translation helpful? Give feedback.
-
FYI I'm away from my usual base at the moment and can't exactly reproduce your configuration, but just tried a quick test using a NEO-M8N with an RPi4 Model B 4MB running Python 3.12 with a SanDisk Ultra (140 MB/s) 64MB SD card and am not seeing the same issue - here's an excerpt from the log covering a 7 minute period:
I'll do further tests when I'm back at base. |
Beta Was this translation helpful? Give feedback.
-
Hi,
1- Logs are in parsed mode. The samples i put in Git-Hub were directly from the log file
2- saving logs to a WD SN740 256gGB SSD on a raspberry pi M.2 hat. Disk latency shouldn't be an issue.
3- 1Hz
4- Log files are about 3 megabytes. The plot below is the duration of the dropout vs time for one of the cases. It's actually the time difference between GNRMC messages when it was longer than 1 second, This was about 15 log files combined. I have seen times where it is a bit more regular.
edit: trying to past in the plot which didn't make it here from the email
![image](https://github.com/user-attachments/assets/20f5352e-d2a6-4b47-b5ce-dc662dfa5f97)
Yes, u-center was run from my desktop. From there I don't have long enough cables to get the gps antenna to a location where I can get RTK fix.
I saw your other email with no dropouts. I do have a laptop with linux. I'll try putting PyGPSClient on it and see if the log works there. I was also running the pi5 headless from my desktop so perhaps something there was interfering with the PI 5 operations.
Thanks for looking at this so quickly. Let me do some experimentation on my end to either find or rule out equipment problems.
Pat
On 10/16/2024 12:29:06 AM, SEMU Admin ***@***.***> wrote:
Hi @pat-2010 [https://github.com/pat-2010]
Can you just confirm that the following:
* You're saving the logs in parsed rather than binary format?
* You're saving the logs to the RPI's SD card?
* What navigation solution interval is your F9P set to? Is it the default 1000ms (i.e. 1Hz)?
* Roughly how big are your data log files, in kB? Do the 'dropouts' occur early in the log file or only when the log reaches a certain size?
When you say you've used u-center to record logs successfully, I presume you mean on a separate Windows machine? What happens if you use PyGPSClient's datalogging on that same Windows platform - do you still see 'dropouts'? If not, i'm inclined to think this may simply be an SD write latency issue on the RPi, but I will look into it.
—
Reply to this email directly, view it on GitHub [#157 (comment)], or unsubscribe [https://github.com/notifications/unsubscribe-auth/BBHNPP5XNHGYSWNOSE5ZHO3Z3YIT7AVCNFSM6AAAAABQADKBPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJVHE2DQOBVGQ].
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
I haven't tried using the gnssstreamer CLI utility. I'm trying to sort out some performance issues with my GPS system, so I want to have the RTCM messages as well as the NMEA messages. Unfortunately ArduSimple (manufacturer of my GPS receiver board) do not have a user's group so I can't see if others have had similar issues. |
Beta Was this translation helpful? Give feedback.
-
Fair enough. As a general observation, unless you particularly need GUI functionality, I'd generally advocate using CLI tools like gnssstreamer (which supports RTCM RTK by the way) on low end platforms like the RPi (particularly if you're running headless), and - where appropriate - logging binary UBX data rather than NMEA (the former is far more concise), but PyGPSClient's datalogging should work regardless - it's just a straight file write. Things may get a bit dicey if you're running a full cohort of NMEA and RTCM messages at very high solution rates (e.g. 20Hz), though the limiting factor in those cases is very often the TX buffer on the receiver itself rather than any downstream data logger. If the 'dropouts' are consistently periodic, that would tend to point to another running process causing contention of some kind - do you have any other daemon processes running on the RPi (
You'll appreciate I'm not in a position to provide direct support for proprietary GNSS gear, but Ardusimple's technical support is normally pretty good, as is the u-blox community forum. Have you tried raising your performance issues with either of them? |
Beta Was this translation helpful? Give feedback.
-
<<If the 'dropouts' are consistently periodic, that would tend to point to another running process causing contention of some kind - do you have any other daemon processes running on the RPi (gpsd is a common candidate)?>> | message type | Time | Delta from last message | time since last dropout <NMEA | GNGGA | time=18:40:55 | 4 | 04:39 <<<You'll appreciate I'm not in a position to provide direct support for proprietary GNSS gear, but Ardusimple's technical support is normally pretty good, as is the u-blox community forum. Have you tried raising your performance issues with either of them?>>> Yes, I've started discussions with Ardusimple tech support. They pointed to the correction service as one of the potential problems, which is why I'm trying to get a complete log of NMEA and RTCM messages covering the period where I am seeing unexpected accuracy variations. Thanks for pointing me to the u-blox community forum. I'll do some research there. |
Beta Was this translation helpful? Give feedback.
-
Are you confident the F9P is actually outputting the 'missing' NMEA messages? Are you seeing them in the PyGPSClient console, or for example via a standard terminal utility like screen -L /dev/ttyACM0 |
Beta Was this translation helpful? Give feedback.
-
On 10/16/2024 11:58:14 AM, SEMU Admin ***@***.***> wrote:
Are you confident the F9P is actually outputting the 'missing' NMEA messages? Are you seeing them in the PyGPSClient console, or for example logging the raw NMEA using a standard terminal utility like screen e.g.
screen -L /dev/ttyACM0
Pat Hascall:
RTCM messages are missing as well.
I have only been using PyGPSClient and it's log files. Given the missing RTCM messages I don't expect it is a problem with F9P output. But I will do a log file for an hour or so and see what happens.The problem is intermittent and can at times there can be many minutes between data dropouts. Makes it hard to look at data that comes once per second and spot it. I will pipe it to a file and examine it there.
—
Reply to this email directly, view it on GitHub [#157 (comment)], or unsubscribe [https://github.com/notifications/unsubscribe-auth/BBHNPP5CXWRT3Y7PWZ6O53TZ32ZMHAVCNFSM6AAAAABQADKBPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJXGY3TSOBXGM].
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Status: I can't get the problem to repeat. I checked my old data and for the 5 setups I did using PyGPSclient I had dropouts in every setup. Today I collected data for 1 1/2 hours without a single dropout. The only difference was that my computer was in my house and not on the back patio and the GPS antenna was in a different location. Also experimented with a different computer (laptop using linux) and didn't see any dropouts there either. So tomorrow I will replicate exactly what I did before with the computer and antenna in the same physical location. I am using wifi to connect to the NTRIP server (but I think I have a good signal on my patio). |
Beta Was this translation helpful? Give feedback.
-
Hmmmm - sounds like a back patio problem 😉 |
Beta Was this translation helpful? Give feedback.
-
I believe the problem is somewhere in my system and not in PyGPSClient. Today I used my pi in a setup matching exactly that when I originally saw the problem. The dropped messages returned. I then replaced the pi with a laptop runningPyGPSClient on linux. No messages were dropped. I'm giving up trying to figure out why the pi drops messages sometimes but not always. I'm collecting enough data even with the drop outs for my purposes. Thanks for your help. |
Beta Was this translation helpful? Give feedback.
-
Hi @pat-2010 Sorry this one remains a puzzler. AFAICS there's nothing in the PyGPSClient datalogging code that would differentiate between different Python platforms or storage media, so I'm inclined to agree it's a local problem in this case. PyGPSClient has been tested extensively on a variety of RPi platforms, but that's not to say there aren't improvements that could be made. Out of interest, which specific M2 hat model are you using? I think I have a few old M2 SSDs lying around back at base - I may get the same M2 hat and see if I can reproduce the problem. I'll keep this issue open for now. Also, when you say "64 bit Debian version 12 (Bookworm)", can I assume you mean the stock version of Raspberry Pi OS 64-bit that's available from e.g. RPi Imager? Are you running the Debian OS on the M2 drive or on a separate SD drive? |
Beta Was this translation helpful? Give feedback.
-
Good morning,
I'm going to try one more experiment. I was using the RPi 5 headless using RealVNC. I'll take a monitor/keyboard out to the setup to see if the RealVNC load on the RPi and wifi could be contributing to the problem.
I'm using the Raspberry Pi M.2 HAT+.
On 10/17/2024 11:38:32 PM, SEMU Admin ***@***.***> wrote:
Hi @pat-2010 [https://github.com/pat-2010]
Sorry this one remains a puzzler. AFAICS there's nothing in the PyGPSClient datalogging code that would differentiate between different Python platforms or storage media, so I'm inclined to agree it's a local problem in this case. PyGPSClient has been tested extensively on a variety of RPi platforms, but that's not to say there aren't improvements that could be made.
Out of interest, which specific M2 hat model are you using? I think I have a few old M2 SSDs lying around back at base - I may get an M2 hat and see if I can reproduce the problem. I'll keep this issue open for now.
—
Reply to this email directly, view it on GitHub [#157 (comment)], or unsubscribe [https://github.com/notifications/unsubscribe-auth/BBHNPP4HOVYQHNAORM77JW3Z4CUGPAVCNFSM6AAAAABQADKBPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRRGU2DIOJVGA].
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Well, I didn't expect that result. I set the RPi up headless with VNC. Ran for 5 minutes, turned off VNC for 5 minutes then turned it back on again. With VNC on there were no dropouts. When I turned VNC off I saw dropouts. These dropouts were periodic, so sound like a background process contention problem like you said earlier. With VNC back on the dropouts stopped. Just the opposite of what I expected. I know that there were some compatibility issues on the RPi 4-5 when the RPi went to Bookworm with a Wayland environment vs the older X-11. I have an older RPi400 that has the older operating system installed. When I have the time I'll break it out and see what happens. Yes the RPi is running the stock version (Bookworm) loaded to SD card using the RPi imager then moved to the M2 drive. The RPi boots from the M2 drive. I realized that when you edit the comment it doesn't show up in my email. So from now on I'll come to github to get the latest comment updates. |
Beta Was this translation helpful? Give feedback.
-
Thanks for the update. I'll endeavour to test using a comparable configuration sometime next week - be interested to see how reproducible the results are. FYI The RPi4 Model B 4MB I briefly tested earlier was also running VNC headless under Bookworm and I was not seeing dropouts. |
Beta Was this translation helpful? Give feedback.
-
My results are not at all reproducible. I set up a RPi 400, headless, using an older OS (buster) and ran for 40 minutes (no NTRIP). Saw no dropouts. Another run was about 35 minutes this morning with NTRIP turned on. and saw dropouts (but with an interval of 8 of the 35 minutes with no dropouts). Restarted the RPi 400. Final run for a bit over 20 minutes using NTRIP. No dropouts. Saw something else that may or may not be yet another symptom of ghosts in my system. The satellite CNo in the bar chart would at times bounce. I pulled up GPGSV data for GPS SVID=10 and saw that signalid 1 (L1 C/A) had a CNo of 24 and for signalid 6 (L2C-L) the CNo was 35. The bar chart looked to be bouncing between those 2 numbers. Is this expected since there can be multiple CNos for a satellite if it is seen in more than one band, but the plot looks to have only one number per satellite? I do have a screen video. It's about 30MB so too big to upload. |
Beta Was this translation helpful? Give feedback.
-
In the interests of brevity the PyGPSClient bar chart currently only outputs a single level per SVID - there's no reason why it couldn't report individual signalIDs (where this data is available) if the bar chart plot was extended - I can look into this. Some context: The original NMEA 0183 protocol predates many modern GNSS developments. If the receiver is only outputting standard NMEA data, the CNo levels are derived from the GSV sentence. GSV didn't originally differentiate between different signal IDs - it simply reported a single value per SVID (originally this would have been the L1 C/A signal). A signalID attribute was added to the GSV sentence in NMEA 4.10 and was further enhanced in NMEA 4.11, but this has not been universally implemented (it's only available in u-blox firmware 27.12 or later). Note also that the standard GSV sentence does not explictly provide the gnssID (aka systemID) - this has to be derived from the SVID range, and the correspondence varies from one NMEA generation to the next as new constellations have been added. Some proprietary NMEA sentences can provide more data, but this is obviously device and protocol specific. The more modern UBX protocol is much more comprehensive and concise. The UBX NAV-SIG message, for example, includes an entry for each individual signalID. Unless you have a particular need to log NMEA data, I would always recommend configuring u-blox receivers like the F9P to suppress NMEA and output only UBX data. A single NAV-SAT message can replace 20 or more GSV sentences. |
Beta Was this translation helpful? Give feedback.
-
Thanks for the detailed explanation on the NEMA protocol. Now that I know why the CNo's are jumping it won't trigger my OCD ;-). Part of my plan is to use the GPS with my phone providing the NTRIP link when I'm too far away from the wifi in my house. I have an iPhone and the only software I can find needs NEMA. So I'm stuck with it. Did some work in Python to process the log files and identify dropouts. Much better than manually transferring to excel. Latest set of tests were not productive. I ran on the RPi 5 first, swapped to the RPi 400 then back to the RPi 5. The RPi 400 did not have dropouts in over 3 hours of running. The RPi 5 did have dropouts when I ran it the first time, but the rate was low for the first 5 minutes. When I switched back to the RPi 5 I still saw dropouts but the rate is about 100 times lower than earlier in the day. I'll leave it to run overnight to see what happens. The earlier runs with higher dropout rates had log files of about 3MiB. The ones with the lower rates were only 1.8 MiB. I believe the reduction is due to spacecraft visibility. So perhaps the issue is loading related. My cpu is at 30% or so, GPU at 80%. Memory usage is less than 25%. Tomorrow I'll try to get the RPi 400 to get dropouts. Do you know how I could get sub second timing for the NMEA messages? The normal sequence is 25 to 30 messages depending on how many satellites are in view. In one run the 75% of the dropouts were within the first 10 messages. I'll look at more data tomorrow to confirm that. If some event in the RPi is interfering and causing the message dropout I would expect it to be random relative to the NMEA sequence. What I'm seeing doesn't look to be random. I believe (and will confirm once I tweak the Python a bit) that the data dropouts are between messages - all messages are complete. And recovery is always with the GNRMC that starts the data for a new second. Luckily, I'm retired, have a bit of spare time, and find this type of problem intriguing. Let me know if the thread is getting too far off topic for a software bug report. |
Beta Was this translation helpful? Give feedback.
-
Again inconsistent results. I stayed on the RPi 5 today and did not use NTRIP. Table of data below. The first run of about 3 hours had dropouts. The rate during the first 45 minutes was lower then went up by a factor of 10 for the last few hours. I switched to gnssstreamer for the second run. That was just over an hour and had no dropouts. Went back to pygpsclient. I started off with a few dropouts but then they went away. The table below shows the dropout counts for each log file (approx 6 minutes each). The log files appear to be fixed size (10000 lines). In the first run as the dropouts increase the time duration for each file also increases. First run 16:21:15 | 14 Last run 20:13:08 | 0 |
Beta Was this translation helpful? Give feedback.
-
Hi @pat-2010 Happy to continue thread but I'll move it into the Q&A Discussions Forum for now. If a clear reproducible bug is identified, I'll raise an issue for that specific bug.
The log files do cycle at a specific size (defined in globals.py as
You may have already worked this out, but with a generation 9+ u-blox device like the F9P you can use either the legacy CFG-RATE command (specifically the Out of interest, what is your ultimate objective here? What is it you're looking to achieve with the F9P? You mentioned concerns about 'accuracy' earlier in the thread. Have you had a look at RTK-TIPS? |
Beta Was this translation helpful? Give feedback.
-
OK so back at base I span up a headless RPi5 8MB running stock 64-bit Debian bookworm with a ZED-F9P running stock firmware HPG 1.50 and default configuration.
Lo and behold on the very first test I did see a couple of random dropouts - here's an excerpt from the Python script for one of the log files - but on subsequent tests I saw no dropouts at all.
PYTHON SCRIPT TO REVIEW LOG FILE: """
Simple python script to parse a text NMEA logfile from PyGPSClient
and identity any non-contiguous timestamps.
"""
from datetime import datetime, timedelta
LOGFILE = "/Users/steve/Downloads/pygpsdata-20241023095703.log"
dropouts = timprev = start = end = i = 0
with open(LOGFILE, "r", encoding="utf-8") as infile:
for line in infile:
timp = line.find("time=")
if timp > 0:
tim = datetime.strptime(line[timp + 5 : timp + 13], "%H:%M:%S")
if i:
delta = tim - timprev
timprev = tim
lbl = ""
if delta > timedelta(seconds=1):
lbl = "DROPOUT?"
dropouts += 1
print(f"{tim} {delta} {lbl}")
end = tim
else:
start = timprev = tim
i += 1
print(
f"logfile duration: {end-start}, records with timestamps: {i}, "
f"dropouts: {dropouts} ({dropouts*100/i:.2f}%)"
) |
Beta Was this translation helpful? Give feedback.
-
Just to close this off, I repeated the tests above with the RPi5 M.2 Hat and Integral 1800MB/s M.2 2230 NVMe SSD - pretty much identical to your original hardware configuration. I repeated the tests 5 times over a continuous period of an hour. Results as follows:
I then enabled PyGPSClient's NTRIP client with a local euref-ip.net NTRIP caster and repeated the tests with an RTK-FLOAT fix status - again, no 'dropouts' seen over a 60 minute period. For yuks, I even ran a concurrent RPi stress test while PyGPSClient was running, but still no 'dropouts' in the logfiles even with the CPU running at 100% and 65° (I'm using the RPi5 active cooler by the way). In summary, I'm seeing no consistent 'dropouts' at all in any of these tests. I'll continue to monitor PyGPSClient performance on RPi platforms - there may well be optimisations I've missed - but I suspect we'll simply have to write your experience off to 'gremlins' on this occasion.
If you're capturing raw navigation data from an F9P for PPK purposes (i.e. UBX RXM-RAWX and/or RXM-SFRBX messages), then significant dropouts may well affect the analysis, but in these circumstances I would strongly recommend switching to UBX rather than NMEA output and using a CLI tool like gnssstreamer to capture the raw binary UBX data. This data would typically then be converted to RINEX format using RTKLIB or similar. |
Beta Was this translation helpful? Give feedback.
-
I am saving a parsed log file. It has NEMA and RTCM messages. Intermittently the data drops out (averages about once every 2 minutes). It typically returns within about 3 seconds. It drops out in the middle of a message cycle and starts up with the GNRMC message that starts the data for a new second. I get around 33 messages per second, depending on the number of satellites in view. They always follow a sequence of GNRMC, GNGGA, multiple GNGSA, multiple GxGSV and a GNGST. The RTCM messages 1004 and 1012 come in at a bit more than 1 per second. I've looked at multiple cases of dropouts and they appear to start randomly in the middle of the 33 message sequence. Both NMEA and RTCM messages drop out. I have not observed any partial messages.
Using pygpsclient 1.4.22.
Here's a clip from the log file. Note the time jumps from 20:49:39 to 20:49:42
<NMEA(GQGSV, numMsg=1, msgNum=1, numSV=1, svid_01=3, elv_01=9.0, az_01=301, cno_01=38, signalID=1)>
<NMEA(GQGSV, numMsg=1, msgNum=1, numSV=1, svid_01=3, elv_01=9.0, az_01=301, cno_01=41, signalID=6)>
<NMEA(GNGST, time=20:49:39, rangeRms=8.9, stdMajor=0.025, stdMinor=0.02, orient=154.0, stdLat=0.01, stdLong=0.01, stdAlt=0.018)>
<NMEA(GNRMC, time=20:49:42, status=A, lat=38.857116665, NS=N, lon=-121.28045184, EW=W, spd=0.011, cog=, date=2024-10-13, mv=, mvEW=, posMode=F, navStatus=V)>
<NMEA(GNGGA, time=20:49:42, lat=38.857116665, NS=N, lon=-121.28045184, EW=W, quality=5, numSV=12, HDOP=0.76, alt=71.605, altUnit=M, sep=-26.73, sepUnit=M, diffAge=1.0, diffStation=372)>
And a second case
<NMEA(GNGST, time=19:49:45, rangeRms=19.0, stdMajor=0.96, stdMinor=0.62, orient=5.7, stdLat=0.39, stdLong=0.26, stdAlt=0.74)>
<NMEA(GNRMC, time=19:49:46, status=A, lat=38.8571101867, NS=N, lon=-121.28045642, EW=W, spd=0.018, cog=, date=2024-10-13, mv=, mvEW=, posMode=D, navStatus=V)>
<NMEA(GNGGA, time=19:49:46, lat=38.8571101867, NS=N, lon=-121.28045642, EW=W, quality=2, numSV=12, HDOP=0.49, alt=73.642, altUnit=M, sep=-26.73, sepUnit=M, diffAge=, diffStation=133)>
<NMEA(GNGSA, opMode=A, navMode=3, svid_01=21, svid_02=31, svid_03=9, svid_04=28, svid_05=2, svid_06=16, svid_07=26, svid_08=4, svid_09=3, svid_10=, svid_11=, svid_12=, PDOP=0.96, HDOP=0.49, VDOP=0.82, systemId=1)>
<NMEA(GNRMC, time=19:49:48, status=A, lat=38.8571102317, NS=N, lon=-121.28045633, EW=W, spd=0.013, cog=, date=2024-10-13, mv=, mvEW=, posMode=D, navStatus=V)>
<NMEA(GNGGA, time=19:49:48, lat=38.8571102317, NS=N, lon=-121.28045633, EW=W, quality=2, numSV=12, HDOP=0.49, alt=73.683, altUnit=M, sep=-26.73, sepUnit=M, diffAge=, diffStation=133)>
Steps to reproduce the behaviour:
I check data logging with the parsed option. Then select the usb/uart option to open the link to the gps receiver.
Expected Behaviour
A log file with all the data.
Desktop:
Running on a Raspberry pi 5. 64 bit Debian version 12 (Bookworm). USB connection to simpleRTK2B budget receiver. Using Python 3.11.2.
GNSS/GPS Device:
<UBX(MON-VER, swVersion=b'EXT CORE 1.00 (0fa0ae)\x00\x00\x00\x00\x00\x00\x00\x00', hwVersion=b'00190000\x00\x00', extension_01=b'ROM BASE 0x118B2060\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_02=b'FWVER=HPG 1.32\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_03=b'PROTVER=27.31\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_04=b'MOD=ZED-F9P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_05=b'GPS;GLO;GAL;BDS\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_06=b'SBAS;QZSS\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')>
Additional context
I have used u-center and recorded the NMEA messages with no dropouts.
Beta Was this translation helpful? Give feedback.
All reactions