cFS freezes when the time source is set to GPS

[ryuhg]

Hi all.

After spending a lot of time with our "lab" setting FSW, I'm now trying to tickle with onboard-environment configurations.
One thing is synchronizing the SC time with its attached GPS receiver's time source. As I understand the CFE_TIME_ExternalGPS() is to be used to send a time tone to the core TIME service and catch up the time difference. And it requires the right platform/mission configuration under the _defs directory.
So this is what I changed from default:

<xxx_platform_cfg.h>

...
CFE_PLATFORM_TIME_CFG_SOURCE = true
...
CFE_PLATFORM_TIME_CFG_SRC_GPS = true
...

<xxx_mission_cfg.h>

...
CFE_MISSION_TIME_CFG_DEFAULT_UTC = true
CFE_MISSION_TIME_CFG_DEFAULT_TAI = false
...
CFE_MISSION_TIME_CFG_FAKE_TONE = false
...

And after building with this configs the software just freezes after initialization. Nothing comes on the screen when it is supposed to!
Doesn't look like my custom Applications are doing this. (I tried deploying only the SCH App but it still freezes) Can anyone help me figure out what is going on?

Thanks.

(Below is the entire log until the freezing)

CFE_PSP: Default Reset SubType = 1
CFE_PSP: Default CPU ID = 1
CFE_PSP: Default Spacecraft ID = 66
CFE_PSP: Default CPU Name: cpu1
OS_Posix_GetSchedulerParams():190:Policy 1: available, min-max: 1-99
OS_Posix_GetSchedulerParams():190:Policy 2: available, min-max: 1-99
OS_Posix_TaskAPI_Impl_Init():376:Selected policy 2 for RT tasks, root task = 99
CFE_PSP: Instantiated software timebase 'cFS-Master' running at 10000 usec
CFE_PSP: Using POSIX monotonic clock as CFE timebase
CFE_PSP: Using MMAP simulated EEPROM implementation
CFE_PSP: Physical RAM access not implemented
CFE_PSP: I/O Port access not implemented
CFE_PSP: EEPROM Range (2) created: Start Address = B6B75000, Size = 00080000 Status = 0
CFE_PSP: Normal exit from previous cFE instance
CFE_PSP: Starting the cFE with a PROCESSOR reset.
2021-001-00:04:00.70352 CFE_ES_Main: CFE_ES_Main in EARLY_INIT state
2021-001-00:04:00.70406 CFE_ES_InitializeFileSystems: Volatile Disk has 100 Percent free space.
CFE_PSP: CFE_PSP_AttachExceptions Called
2021-001-00:04:00.70611 CFE_ES_Main: CFE_ES_Main entering CORE_STARTUP state
2021-001-00:04:00.70625 CFE_ES_CreateObjects: Starting Object Creation calls.
2021-001-00:04:00.70637 CFE_ES_CreateObjects: Calling CFE_ES_CDSEarlyInit
2021-001-00:04:00.70710 CFE_ES_CreateObjects: Calling CFE_EVS_EarlyInit
2021-001-00:04:00.70760 CFE_EVS_EarlyInit: Event Log restored, n=0, c=20, f=1, m=1, o=21
2021-001-00:04:00.70775 CFE_ES_CreateObjects: Calling CFE_SB_EarlyInit
2021-001-00:04:00.71786 CFE_ES_CreateObjects: Calling CFE_TIME_EarlyInit
2021-012-14:02:43.00004 CFE_ES_CreateObjects: Calling CFE_TBL_EarlyInit
2021-012-14:02:43.01185 CFE_ES_CreateObjects: Calling CFE_FS_EarlyInit
EVS Port1 66/1/CFE_EVS 1: cFE EVS Initialized:  cFE DEVELOPMENT BUILD v6.8.0-rc1+dev762 (Codename: Bootes), Last Official Release: cfe v6.7.0
2021-012-14:02:43.17988 CFE_SB_AppInit: Registered 4 events for filtering
EVS Port1 66/1/CFE_SB 1: cFE SB Initialized:  cFE DEVELOPMENT BUILD v6.8.0-rc1+dev762 (Codename: Bootes), Last Official Release: cfe v6.7.0
EVS Port1 66/1/CFE_SB 14: No subscribers for MsgId 0x808,sender CFE_SB
EVS Port1 66/1/CFE_ES 1: cFE ES Initialized:  cFE DEVELOPMENT BUILD v6.8.0-rc1+dev762 (Codename: Bootes), Last Official Release: cfe v6.7.0
EVS Port1 66/1/CFE_SB 14: No subscribers for MsgId 0x808,sender CFE_ES
EVS Port1 66/1/CFE_ES 2: cFS Versions: cfe v6.8.0-rc1+dev762, osal v5.1.0-rc1+dev573, psp v1.5.0-rc1+dev118. cFE chksm 4975
EVS Port1 66/1/CFE_SB 14: No subscribers for MsgId 0x808,sender CFE_ES
EVS Port1 66/1/CFE_ES 91: Version Info: Mission SampleMission, version [unknown]
EVS Port1 66/1/CFE_SB 14: No subscribers for MsgId 0x808,sender CFE_ES
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module core_api, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module es, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module evs, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module fs, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module sb, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module tbl, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module time, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module osal, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module psp, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module msg, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module sbr, version [unknown]
EVS Port1 66/1/CFE_ES 91: Version Info: Core Module resourceid, version [unknown]
EVS Port1 66/1/CFE_ES 92: Build 202111221950 by root@acl-16ZD90P-GX50K, config miman
EVS Port1 66/1/CFE_TIME 1: cFE TIME Initialized:  cFE DEVELOPMENT BUILD v6.8.0-rc1+dev762 (Codename: Bootes), Last Official Release: cfe v6.7.0
EVS Port1 66/1/CFE_TBL 1: cFE TBL Initialized:  cFE DEVELOPMENT BUILD v6.8.0-rc1+dev762 (Codename: Bootes), Last Official Release: cfe v6.7.0
OS_FileOpen_Impl():114:open(/dev/shm/osal:RAM/cfe_es_startup.scr): No such file or directory
2021-012-14:02:43.56244 CFE_ES_CreateObjects: Finished ES CreateObject table entries.
2021-012-14:02:43.56260 CFE_ES_Main: CFE_ES_Main entering CORE_READY state
2021-012-14:02:43.56320 CFE_ES_StartApplications: Cannot Open Volatile Startup file, Trying Nonvolatile.
2021-012-14:02:43.56363 CFE_ES_StartApplications: Opened ES App Startup file: /cf/cfe_es_startup.scr
2021-012-14:02:43.57150 CFE_ES_ParseFileEntry: Loading shared library: /cf/cfe_assert.so
2021-012-14:02:43.57375 [BEGIN] CFE FUNCTIONAL TEST
2021-012-14:02:43.57400 [BEGIN] 01 CFE-STARTUP
2021-012-14:02:43.57879 CFE_ES_ParseFileEntry: Loading shared library: /cf/libcsp.so
2021-012-14:02:43.59252 CFE_ES_ParseFileEntry: Loading shared library: /cf/cfs_lib.so
2021-012-14:02:43.59838 CFE_ES_ParseFileEntry: Loading shared library: /cf/io_lib.so
2021-012-14:02:43.61163 CFE_ES_LoadLibrary: Load Shared Library Init Error = 0xffffffff
2021-012-14:02:43.61630 CFE_ES_ParseFileEntry: Loading shared library: /cf/hw_lib.so
HW Lib Initialized.  Version 1.1.1.0
2021-012-14:02:43.62351 CFE_ES_ParseFileEntry: Loading file: /cf/sch.so, APP: SCH
2021-012-14:02:43.63065 CFE_ES_ParseFileEntry: Loading file: /cf/sc.so, APP: SC
2021-012-14:02:43.63786 CFE_ES_ParseFileEntry: Loading file: /cf/hk.so, APP: HK
2021-012-14:02:43.64459 CFE_ES_ParseFileEntry: Loading file: /cf/ds.so, APP: DS
EVS Port1 66/1/DS 10: Destination file table verify results: desc text = OK, good entries = 3, bad = 0, unused = 13
EVS Port1 66/1/DS 12: Filter table verify results: desc text = OK, good entries = 3, bad = 0, unused = 253
EVS Port1 66/1/DS 1: Application initialized, version 2.5.2.0, data at 0xb5ff81e4
EVS Port1 66/1/HK 1: HK Initialized.  Version 2.4.2.0
EVS Port1 66/1/SC 21: RTS table file load count = 2
EVS Port1 66/1/SC 103: Verify ATS Table: command count = 61, byte count = 946
EVS Port1 66/1/SC 103: Verify ATS Table: command count = 5, byte count = 70
EVS Port1 66/1/SC 21: ATS table file load count = 2
EVS Port1 66/1/SC 9: SC Initialized. Version 2.5.0.0
EVS Port1 66/1/SCH 13: OS Timer Accuracy (10000 > reqd 101 usec) requires Minor Frame MET sync
EVS Port1 66/1/SCH 1: SCH Initialized. Version 2.2.1.0
2021-012-14:02:43.75785 CFE_ES_Main: CFE_ES_Main entering APPS_INIT state
2021-012-14:02:43.75804 CFE_ES_Main: CFE_ES_Main entering OPERATIONAL state

[skliper]

My time services configuration skills are a bit rusty, but I suspect it's due to setting CFE_MISSION_TIME_CFG_FAKE_TONE = false which disables the fake/internal tone used for timekeeping. The core concept is time services uses a time at tone message (either time was or time will be) and a "tone" to synchronize time. If you turn off the fake tone, you need to connect the CFE_TIME_Tone1HzISR to a "real" tone (typically via the PSP through an interrupt.) This is the typical highest accuracy method of time keeping since time services keeps track of when the ISR is called, such that when the time at tone message arrives time can be updated without accuracy dependencies on when the actual time at tone message is received (as long as it's within the appropriate window). Lacking the external tone, there are a few less accurate ways to configure TIME services and whatever your time source is, all of which require the fake tone (basically internally generated by cFE). I think you can register a callback such that you can poll the current time from whatever your time source is when the fake tone occurs, or you could use SCH to send a message whenever the fake tone occurs (both with accuracy penalties associated w/ the callback or message timing.

If you've got a GPS with a 1PPS you can connect to the ISR this is your best bet, but time services is highly configurable (and unfortunately complex), so there's quite a few options.

[ryuhg]

Thanks! I will look into the PSP. But why is the fake tone incompatible with any external sources? In my version defining the CFE_PLATFORM_TIME_CFG_SOURCE true forces the CFE_MISSION_TIME_CFG_FAKE_TONE to be false. Can't the platform tune its time on the fake tones while adjusting STCF by ExternalGPS()?

[skliper]

While having numerous configuration options (some would say too many), time services does have some strange limitations. We've got at least one open issue to address this as it's not ideal (nasa/cFE#302). Out of the box if you set your time source as external (and/or allow it to be configured via the CFE_PALTFORM_TIME_CFG_SOURCE) it assumes you are getting an external tone (for the higher accuracy option I described above). You'll need to remove this restriction (customize) to do the lower accuracy options (fake tone with external source).