[Bug]: LAN port not working on Radxa E20C

[ssinyagin]

What happened?

I built a fresh Bookworm image for Radxa E20C, and the WAN port works as expected, but the LAN port (end1) detects the link, but does not send or receive any packets.

Current commit: 1f69720656212154a09fa9a842e323265597a55d

root@radxa-e20c:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: end1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e6:05:1f:e0:0d:f2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e405:1fff:fee0:df2/64 scope link 
       valid_lft forever preferred_lft forever
3: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:e0:4c:0d:1a:f8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.219/24 metric 100 brd 192.168.1.255 scope global dynamic enp1s0
       valid_lft 604790sec preferred_lft 604790sec
    inet6 fe80::2e0:4cff:fe0d:1af8/64 scope link 
       valid_lft forever preferred_lft forever
root@radxa-e20c:~# ip addr add 172.20.0.3/24 dev end1
root@radxa-e20c:~# ip route
default via 192.168.1.1 dev enp1s0 proto dhcp src 192.168.1.219 metric 100 
172.20.0.0/24 dev end1 proto kernel scope link src 172.20.0.3 
192.168.1.0/24 dev enp1s0 proto kernel scope link src 192.168.1.219 metric 100 
192.168.1.1 dev enp1s0 proto dhcp scope link src 192.168.1.219 metric 100 
root@radxa-e20c:~# ping 172.20.0.1
PING 172.20.0.1 (172.20.0.1) 56(84) bytes of data.
From 172.20.0.3 icmp_seq=1 Destination Host Unreachable
From 172.20.0.3 icmp_seq=2 Destination Host Unreachable
From 172.20.0.3 icmp_seq=3 Destination Host Unreachable
^C
--- 172.20.0.1 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4052ms
pipe 4
root@radxa-e20c:~# arp -an
? (192.168.1.31) at 7c:83:34:bd:4a:2e [ether] on enp1s0
? (172.20.0.1) at <incomplete> on end1
? (192.168.1.1) at c8:7f:54:78:d0:c0 [ether] on enp1s0
root@radxa-e20c:~# ethtool end1
Settings for end1:
	Supported ports: [ TP	 MII ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Half 1000baseT/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Half 1000baseT/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  10baseT/Half 10baseT/Full
	                                     100baseT/Half 100baseT/Full
	                                     1000baseT/Full
	Link partner advertised pause frame use: Symmetric Receive-only
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Auto-negotiation: on
	master-slave cfg: preferred slave
	master-slave status: master
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: external
	MDI-X: Unknown
	Supports Wake-on: ug
	Wake-on: d
        Current message level: 0x0000003f (63)
                               drv probe link timer ifdown ifup
	Link detected: yes

also, dmesg messages:

[    2.637813] rk_gmac-dwmac ffbe0000.ethernet: IRQ eth_lpi not found
[    2.638006] rk_gmac-dwmac ffbe0000.ethernet: PTP uses main clock
[    2.638036] rk_gmac-dwmac ffbe0000.ethernet: Looking up phy-supply from device tree
[    2.638052] rk_gmac-dwmac ffbe0000.ethernet: Looking up phy-supply property in node /ethernet@ffbe0000 failed
[    2.638083] rk_gmac-dwmac ffbe0000.ethernet: supply phy not found, using dummy regulator
[    2.638244] rk_gmac-dwmac ffbe0000.ethernet: clock input or output? (output).
[    2.638255] rk_gmac-dwmac ffbe0000.ethernet: TX delay(0x30).
[    2.638265] rk_gmac-dwmac ffbe0000.ethernet: Can not read property: rx_delay.
[    2.638271] rk_gmac-dwmac ffbe0000.ethernet: set rx_delay to 0xffffffff
[    2.638287] rk_gmac-dwmac ffbe0000.ethernet: integrated PHY? (no).
[    2.638298] rk_gmac-dwmac ffbe0000.ethernet: cannot get clock mac_clk_rx
[    2.638308] rk_gmac-dwmac ffbe0000.ethernet: cannot get clock mac_clk_tx
[    2.638332] rk_gmac-dwmac ffbe0000.ethernet: cannot get clock clk_mac_speed
[    2.638588] rk_gmac-dwmac ffbe0000.ethernet: init for RGMII_RXID
[    2.638815] rk_gmac-dwmac ffbe0000.ethernet: User ID: 0x40, Synopsys ID: 0x51
[    2.638830] rk_gmac-dwmac ffbe0000.ethernet:         DWMAC4/5
[    2.638840] rk_gmac-dwmac ffbe0000.ethernet: DMA HW capability register supported
[    2.638848] rk_gmac-dwmac ffbe0000.ethernet: RX Checksum Offload Engine supported
[    2.638855] rk_gmac-dwmac ffbe0000.ethernet: TX Checksum insertion supported
[    2.638862] rk_gmac-dwmac ffbe0000.ethernet: Wake-Up On Lan supported
[    2.638928] rk_gmac-dwmac ffbe0000.ethernet: TSO supported
[    2.638936] rk_gmac-dwmac ffbe0000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    2.638946] rk_gmac-dwmac ffbe0000.ethernet: Enabled RFS Flow TC (entries=10)
[    2.638955] rk_gmac-dwmac ffbe0000.ethernet: TSO feature enabled
[    2.638963] rk_gmac-dwmac ffbe0000.ethernet: Using 40/40 bits DMA host/device width


[    3.515792] r8168 Gigabit Ethernet driver 8.051.02-NAPI loaded
[    3.515888] r8168 0000:01:00.0: enabling device (0000 -> 0003)
[    3.538704] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    3.557429] r8168: This product is covered by one or more of the following patents: US6,570,884, US6,115,776, and US6,327,625.
[    3.559523] rk_gmac-dwmac ffbe0000.ethernet end1: renamed from eth0
[    3.559874] r8168  Copyright (C) 2022 Realtek NIC software team <[email protected]> 
                This program comes with ABSOLUTELY NO WARRANTY; for details, please see <http://www.gnu.org/licenses/>. 
                This is free software, and you are welcome to redistribute it under certain conditions; see <http://www.gnu.org/licenses/>. 
[    3.653189] r8168 0000:01:00.0 enp1s0: renamed from eth1

How to reproduce?

the device does not get the DHCP address on LAN port and also statically assigned IP addresses don't work.

Branch

main (main development branch)

On which host OS are you running the build script and observing this problem?

Debian 12 Bookworm

Are you building on Windows WSL2?

• Yes, my Ubuntu/Debian/OtherOS is running on WSL2

Relevant log URL

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Jira ticket: AR-2613

[ssinyagin]

I checked with tcpdump on both sides of the cable, and the remote side does not see any packets sent by the E20C. tcpdump on E20C shows that packets are being received from the wire.

So, only packet sending is broken.

It's tested on two E20C devices.

[ssinyagin]

I loaded the original IstoreOS from Radxa website, and I was able to ping 192.168.100.1 which is by default bridged through the LAN port, so it's not a hardware failure.

[igorpecovnik]

Hey Stanislav, long time no see!

The problem with this device is that we don't deal with it. There are too many variants out there and there is so little resoures / people - we focus on selected boards only. What is outside "Standard/Platinum support":

https://docs.armbian.com/User-Guide_Board-Support-Rules/#standard-support
https://www.armbian.com/download/?device_support=Standard%20support

we do not have hardware, time and people, but images are still produced as "community supported targets" as most of those still might work well or to some degree. Most of our software stack is developed around mainline kernel, which is why features are missing / are broken. Vendor kernel is (mainly) hw feature complete, but have other problems. We only support a few (flagship) devices with factory kernel.

[ssinyagin]

yeah, I've been busy in completely different areas, but here we go again :)

Radxa informed me that they are an official sponsor of Armbian, so I thought there's a fast track in resolving the issues :)

The LAN port is a built-in RK3528 Ethernet, maybe you know about similar issues with other boards?

[igorpecovnik]

Radxa informed me that they are an official sponsor of Armbian, so I thought there's a fast track in resolving the issues :)

That is correct. That applies for hw under Platinum section. There we have people and know how for fast track operations. Even there sometimes doesn't go as fast we would like.

The LAN port is a built-in RK3528 Ethernet, maybe you know about similar issues with other boards?

We have several boards with this SoC in the build system, but all were added by community or vendors themselves. I am awaiting shipment of https://radxa.com/products/rock2/2f that have 3528A by this or next week. Not sure how things will progress / no plans atm. We are also in the middle of point release cycle with focus on bug fixing on existing target.

I would say, forums https://forum.armbian.com/ are the best way to find some people who know something.

[ssinyagin]

ok, we'll see. I also wrote to Radxa support, probably they have some more information. The bug seems to be something really minor, as everything works except transmitting the packet on the wire.

[ssinyagin]

@mattx433 you opened the PR, probably you have some more input here? #7157

[csrutil]

I encountered the same LAN port issue. Based on my testing, kernel version 6.1.75-vendor-rk35xx works correctly without any LAN connectivity problems. When time permits, I plan to perform a git bisect to identify the specific commit that introduced this regression.

[ssinyagin]

@csrutil thank you. I believe the problem is somewhere around the physical layer chip (YT8531, drivers/net/phy/motorcomm.c).

A similar problem was found in OrangePi: https://www.reddit.com/r/OrangePI/comments/1aojmn3/orange_pi_3b_ethernet_doesnt_work/

Probably, the PHY chip doesn't get enough voltage? The kernel sees the interface as operational and tcpdump shows outgoing packets. Just the other side of the wire doesn't receive anything.

[ssinyagin]

@csrutil where can I find the branch for 6.1.75-vendor-rk35xx ? Searching through github didn't bring anything.

[EvilOlaf]

https://github.com/armbian/linux-rockchip/tree/rk-6.1-rkr3

[ssinyagin]

@EvilOlaf thanks. The changes do involve drivers/net/phy/motorcomm.c, but they only seem to be related to a new chip model introduction.

[CodeChenL]

Hi @sputnik2019 , I noticed you added some rk3528 devices from mangopi and hinlink that have both pcie and gmac enabled.
I was recently investigating this issue and I found that when I enabled pcie overlays on rock 2a and rock 2f it also reproduces the bug of the NIC not being able to get an ip.
Can you please tell me if mangopi and hinlink's NICs and pcie behave normally when using the latest community weekly builds? Thank!

[sputnik2019]

Try ：
ifconfig end1 down
ifconfig end1 up

[CodeChenL]

Try ： ifconfig end1 down ifconfig end1 up

Thanks for the reply, I've tried that and the LAN still doesn't communicate.
It would be nice if you could try run this version image on mangopi and hinlink rk3528 boards with both gmac and pcie enabled when you have time.

[CodeChenL]

I tried to revert pcie to rk-rkr3's state alone: ​​git log --pretty=format:"%H" -- '/home/chen/Documents/GitHub/kernel/drivers/pci/controller/dwc/pci-dw-rockchip.c' | head -n 40 | xargs git revert
This problem was miraculously fixed! ! !
Who would have thought that the network card problem was actually fixed by rollback pcie driver......
I'm excluding which commit is causing it