Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metro M7: without 32768Hz RTC crystal, RTC timekeeping is not accurate #9908

Open
twrackers opened this issue Dec 22, 2024 · 22 comments
Open
Labels
bug mimxrt10xx iMX RT based boards such as Teensy 4.x
Milestone

Comments

@twrackers
Copy link

I discovered that when a Metro M4 and Metro M7 are running the same code which acts upon a fixed period of time passing (measured by calls to time.monotonic()), my newly purchased M7 claims the time period has completed in only ~85% of the real time required by the M4. Both are sending begin and end messages via serial to a Raspberry Pi 400 which is listening to the serial via another Python code which measures the elapsed times. The Metro M4, as well as a Trinket M0, Gemma M0, and Circuit Playground Express, all come within 1% of the expected time.

Full details can be found in my forum post at https://forums.adafruit.com/viewtopic.php?p=1039811. The test procedure I used, as well as results, are posted there as well.

@dhalbert dhalbert added mimxrt10xx iMX RT based boards such as Teensy 4.x bug labels Dec 22, 2024
@dhalbert dhalbert added this to the 9.2.x milestone Dec 22, 2024
@twrackers
Copy link
Author

twrackers commented Dec 23, 2024

Don't know if this is a clue, but the actual runtime was closer to 83% of what was expected, which is very close to 5/6. Maybe some counter or prescaler is specified incorrectly.

UPDATE: Actually a bit more than 5/6 with further testing. Hopefully I didn't just happen to get a defective M7 somehow. Waiting for further word on this.

@joehopfield
Copy link

Not a bug (except that monotonic() returning a 32bit float should not exist and I bet most naive uses are incorrect)

The test is misusing time.monotonic(), essentially comparing it across machines, and against actual elapsed time.

from the test:

stop_at = time.monotonic() + time_sec
...
while time.monotonic() < stop_at:
pass

docs:

... Only use it to compare against other values from time.monotonic() during the same code run.

On my esp32-s3 qt-py a loop like that only takes about 8 microseconds, and your M7 is much faster - you're exceeding your 22 bits of precision in monotonic's short-float value.

(IMHO time.monotonic should either throw an exception when this happens or should be deprecated. 32bit float is inherently broken for time.monotonic's very vague semantics)

@twrackers
Copy link
Author

So time.monotonic() returns a float which is NOT measured in elapsed seconds? Does time.monotonic_ns() return integer nanoseconds, even if that clock does not have nanosecond resolution? I'm actually trying to pace a loop of code that's running at 10 or 20 Hz while it interacts with code on another Arduino or CircuitPython micro.

I can expect that using too much precision will make the timing not quite periodic, but if monotonic or monotonic_ns are really returning seconds or nanoseconds, the average rate should be about right as long as the master clock oscillator isn't way off.

So what WOULD be a valid way to write a timing loop that forces code to pace itself at a desired rate if I can't trust the monotonic clocks?

@Neradoc
Copy link

Neradoc commented Jan 6, 2025

Your code is correct, it sets a stop time 600 seconds in the future, and waits for that stop time to be reached.
There should be approximately 10 minutes between prints on the receiver side, as you have seen on all other boards.
Since it's only available in the forum post, let me paste it here:

import time

# Test will run for this long, measured by this board's clock.
time_sec = 600

# Allow time for receiving code on Pi to start up.
time.sleep(30)

# Test will stop at this time.
stop_at = time.monotonic() + time_sec

# Change next line to identify board this is saved to.
print("Metro M7")

# Wait for stop time.
while time.monotonic() < stop_at:
    pass

# Identify what the test period was set to.
print(time_sec)

I can use a serial tool like tio with timestamps on to check that the result time does fall within milliseconds of the expected time on all boards I have tried on 9.2.1 (including a Teensy). But I don't have Metro M7 to test.

[00:35:46.584] Pro Micro
[00:45:46.608] 600

@joehopfield
Copy link

The docs (https://docs.circuitpython.org/en/latest/shared-bindings/time/#time.monotonic) suggest time.monotonic_ms() for measuring elapsed time safely. (your test code.py would be easy to change).

docs for the monotonic* funcs are inconsistent (is value "always increasing" or "can never go backward"? both descriptions are used) and has at least one typo/error where #monotonic_ns mentions monotonic.

If the M7 has a bug, a monotonic_ns version of the test will make a clean test case.

(I too wish there were better high resolution time control - squashing to milliseconds on 100+mhz cpus feels restrictive)

@joehopfield
Copy link

Comparing a change in monotonic() to an external clock would seem to violate the first paragraph of its doc, no?

@Neradoc
Copy link

Neradoc commented Jan 7, 2025

They are measuring a 100 seconds difference on a 10 minutes test, the precision of time.monotonic() is not an issue here. It takes over a day for monotonic() to get it's precision to reach a second. Even then that would be enough to measure 10 minutes with 1% accuracy.

This test is not comparing to an external clock, it's comparing to itself to last 600 seconds.
While the starting point of monotonic() is arbitrary, it's still supposed to measure seconds.

@anecdata
Copy link
Member

anecdata commented Jan 7, 2025

There is a calibration here:
https://docs.circuitpython.org/en/latest/shared-bindings/rtc/index.html#rtc.RTC.calibration
But it's relatively small and may not be enough to compensate for this.

@joehopfield
Copy link

Sorry, I think I believed the "always increasing value" in the monotonic() doc. It is monotonic (never decreasing), not strictly increasing. I read "always" as implying it needed to increment by a ns to always increase.
Looking at the code in time/init.c, I see you're correct.
Sorry again - code > docs. :-P

@twrackers
Copy link
Author

twrackers commented Jan 7, 2025

This test is not comparing to an external clock, it's comparing to itself to last 600 seconds.
While the starting point of monotonic() is arbitrary, it's still supposed to measure seconds.

@Neradoc is correct.

Metro M7 (device under test): Hello, Pi400, I'm going to use my clock to mark the start and end of what I consider to be a 10 minute interval. Please use your clock and tell the user how long you measured that time period to be. Minor variations from 10 minutes are to be expected.

Pi400: Wow, M7, I clocked you at way less than 600 seconds.

(Sorry, I get a bit dramatic at times. :) )

@twrackers
Copy link
Author

twrackers commented Jan 7, 2025

From @anecdata :

There is a calibration here: https://docs.circuitpython.org/en/latest/shared-bindings/rtc/index.html#rtc.RTC.calibration But it's relatively small and may not be enough to compensate for this.

I figure if I'm within one or two percent of the expected rate, I'm doing well enough. It's not a precision application I have in mind, nominal rate will probably be 10 Hz.

@twrackers
Copy link
Author

twrackers commented Jan 7, 2025

From @Neradoc again:

[00:35:46.584] Pro Micro
[00:45:46.608] 600

Using Arduino's Serial Monitor with timestamps:
20:19:42.487 -> Metro M7
20:28:17.790 -> 600

Difference: 515.303 sec, 85.9% of expected 600 sec

(My) M7 is definitely running faster than it believes.

@jepler
Copy link
Member

jepler commented Jan 7, 2025

The Metro M7 does not have a 32kHz crystal, which means the timing source is less accurate.

I eventually found this information on the accuracy of the internal 32kHz clock source at https://www.digikey.com/en/htmldatasheets/production/4171371/0/0/1/mimxrt1011dae5a section 4.2.4.2:

In addition, if the clock monitor determines that the OSC32K is not present, then the source of the 32 K will automatically switch to a crude internal ring oscillator. The frequency range of this block is approximately 10–45 kHz. It highly depends on the process, voltage, and temperature.

The value you're seeing corresponds to about 38kHz, which is within the range specified by NXP.

The design of the Metro M7 doesn't have a spot for a 32kHz crystal to be added, and CircuitPython's code doesn't currently allow the RTC to be based on the (more accurate) 24MHz crystal that is used to derive the CPU clock.

@jepler jepler changed the title time.monotonic() runs faster than realtime for Adafruit Metro M7 Express under CP 9.2.1 Metro M7: without 32768Hz RTC crystal, RTC timekeeping is not accurate Jan 7, 2025
@twrackers
Copy link
Author

twrackers commented Jan 7, 2025

The design of the Metro M7 doesn't have a spot for a 32kHz crystal to be added, and CircuitPython's code doesn't currently allow the RTC to be based on the (more accurate) 24MHz crystal that is used to derive the CPU clock.

Well pooh. :(

Guess that M7 will be relegated to tasks that don't require a really accurate timebase. Still got an M4 sitting next to it that does have more precise timing, it actually clocked in at 600.0 seconds on my timing test.

I suppose I could connect an external RTC to the M7 since I'm not looking at very high rate tasks. So far I see as options:

Both have battery backup and QWIIC connectors. The SparkFun part claims to read down to hundredths of a second while the Adafruit one reads down to seconds. And I just noticed I have one of the DS3231 parts sitting on my desk. Maybe I can code the M7 to self-calibrate its onboard RTC, say once every 10 seconds, so it can compensate for the ring oscillator on the fly. Guess I know what I'll be tinkering with tomorrow. ;)

UPDATE: Don't see CircuitPython code to support the RV-8803.

@twrackers
Copy link
Author

twrackers commented Jan 7, 2025

@jepler changed the title time.monotonic() runs faster than realtime for Adafruit Metro M7 Express under CP 9.2.1 Metro M7: without 32768Hz RTC crystal, RTC timekeeping is not accurate

Thanks, much better title for this thread.

@dhalbert
Copy link
Collaborator

dhalbert commented Jan 7, 2025

CircuitPython's code doesn't currently allow the RTC to be based on the (more accurate) 24MHz crystal that is used to derive the CPU clock

@jepler Did you look in detail at this? Could we change the clock source?

@twrackers
Copy link
Author

CircuitPython's code doesn't currently allow the RTC to be based on the (more accurate) 24MHz crystal that is used to derive the CPU clock

@jepler Did you look in detail at this? Could we change the clock source?

That would be cool if it could be done... but I'm still going to spend part of today poking at using my DS3231 to clock the current timing live. I've already seen my test code give fairly different measures on the M7 on different days, probably because of variations in the room temperature. (Ooohh, that cries out for a speed-to-temperature calibration.)

But in the long run, changing the clock source would save some headache for future M7 users.

@jepler
Copy link
Member

jepler commented Jan 7, 2025

I don't think there's an easy way to just make the existing RTC be clocked based on (division of) the 24MHz crystal.

from what I can tell, the RTC peripheral can only be clocked from the 32kHz crystal or the internal ring oscillator. there might be some other peripheral that can be used e.g., as a cycle counter but I don't know the imxrt series in that much detail.

So whatever the solution, for example with the Programmable Interrupt Timer (PIT), it would require changing the whole implementation of ticks in this port.

@jepler
Copy link
Member

jepler commented Jan 7, 2025

image
The RTC is inside the SNVS and can only be clocked from the low speed oscillator (screenshot from i.MX RT1010 Processor Reference Manual, Rev. 0, 09/2019)

@jepler
Copy link
Member

jepler commented Jan 7, 2025

In micropython, ports/mimxrt/ticks.c uses the GPT (general purpose timer) peripheral clocked from the 24MHz crystal (divided by 24, to give 1MHz) as the main timing source. This could be studied & adapted into CircuitPython, possibly.

@twrackers
Copy link
Author

Another Metro M7 oddity:

I'm beginning to think the Metro M7 should have been blue instead of red; I think it's actually a disguised TARDIS (wibbly-wobbly timey-wimey thing).

I connected a DS3231 RTC to the M7's QWIIC port, and ran the following test code.

import adafruit_ds3231
import time
import board

i2c = board.I2C()  # uses board.SCL and board.SDA
rtc = adafruit_ds3231.DS3231(i2c)

while True:
    t = rtc.datetime
    print(t)
    print(f'{t.tm_hour}:{t.tm_min}:{t.tm_sec}')
    time.sleep(1)

Even the time.sleep(1) is running faster than normal, about every seventh pair of lines printed replicates the pair of lines before. (Note: struct_time(...) lines may appear truncated.)

struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=9, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:9
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=10, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:10
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=11, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:11
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=11, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:11
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=12, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:12
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=13, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:13
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=14, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:14
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=15, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:15
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=16, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:16
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=17, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:17
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=17, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:17
struct_time(tm_year=2025, tm_mon=1, tm_mday=7, tm_hour=18, tm_min=9, tm_sec=18, tm_wday=1, tm_yday=-1, tm_isdst=-1)
18:9:18

@twrackers
Copy link
Author

Forgot to mention: the DS3231 has a battery installed and was preset to GMT so it's five hours ahead of my local time (US Eastern Standard Time). Not that it really matters right now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug mimxrt10xx iMX RT based boards such as Teensy 4.x
Projects
None yet
Development

No branches or pull requests

6 participants