[Static route] When a static route present in config_db.json is removed and config reload is done, the route still exists due to presence of kernel entry

[dgsudharsan]

Description

When the below static route present in config_db.json is removed and config reload is performed, the route still exists in system. This is because there is no flow that removes the static route from kernel. When the route is initially added, frr adds it to kernel but when config is removed and config reload is performed, no daemon clears it.

    "STATIC_ROUTE": {
        "default|1000::/120": {
            "blackhole": "true",
            "distance": "0",
            "ifname": "",
            "nexthop": "blackhole",
            "nexthop-vrf": ""
        }
    },

show ipv6 route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

S>*1000::/120 [1/0] (blackhole) 00:01:52

vtysh -c "show run"
Building configuration...

Current configuration:
!
frr version 8.5.4
frr defaults traditional
hostname r-lionfish-07
log syslog informational
log facility local4
fpm address 127.0.0.1
no fpm use-next-hop-groups
agentx
no service integrated-vtysh-config
!
password zebra
enable password zebra
!
ip nht resolve-via-default
!
ipv6 nht resolve-via-default
!
end

ip -6 route show
blackhole 1000::/120 dev lo proto 196 metric 20 pref medium

Steps to reproduce the issue:

1. Have a configuration with static route
2. Remove the route and perform config reload
3. Check show ip route command

Describe the results you received:

Route exists in the show command

Describe the results you expected:

Route shouldn't exist

Output of show version:

Present in 202405 but appears to be a day 1 behavior

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[kalash-nexthop]

@dgsudharsan It's not clear if the route is not just removed from kernel, or if the route is still in rib (and appl_db). Can you please clarify what exact behavior are you seeing? Please also attach show ipv6 route (on sonic and vtysh) output after the config reload is performed and stale route is seen in kernel.

I tried this on my setup, and I indeed see this random behavior. Sometimes the route is deleted from appl_db, and then also from kernel, but at times I see the stale route in appl_db and also in kernel:

rtr1(bash)$ sudo sonic-db-cli CONFIG_DB HGETALL "STATIC_ROUTE|default"
{}


rtr1(bash)$ sonic-db-cli APPL_DB KEYS "ROUTE_TABLE:1000::/120"
ROUTE_TABLE:1000::/120
rtr1(bash)$ sonic-db-cli APPL_DB HGETALL "ROUTE_TABLE:1000::/120"
{'blackhole': 'true'}

rtr1(bash)$ show ipv6 route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

S>*1000::/120 [1/0] (blackhole) 01:00:03

If you are also seeing this, then this seems to be a staticd issue. I restarted staticd at this point, and the stale route got removed from appl_db as well as from kernel. I'm trying to debug this myself.

If you are seeing this stale route only in kernel, then this should be a zebra specific issue.