Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fresh full node start results in spam of 🔍 Discovered new external address for our node #7207

Open
skunert opened this issue Jan 16, 2025 · 5 comments
Open

Fresh full node start results in spam of 🔍 Discovered new external address for our node #7207

skunert opened this issue Jan 16, 2025 · 5 comments
Labels
T0-node This PR/Issue is related to the topic “node”.

Comments

@skunert
Copy link
Contributor

skunert commented Jan 16, 2025 •
edited
Loading

When starting a new node like so:

polkadot-omni-node --chain asset-hub-kusama-genesis.json --tmp

Results in the logs getting flooded with new address discoveries for different ports:

<standard startup message>
2025-01-16 17:10:54 [Parachain] Running JSON-RPC server: addr=127.0.0.1:9944,[::1]:9944    
2025-01-16 17:10:54 [Parachain] 🏁 CPU single core score: 1016.38 MiBs, parallelism score: 882.51 MiBs with expected cores: 8    
2025-01-16 17:10:54 [Parachain] 🏁 Memory score: 12.27 GiBs    
2025-01-16 17:10:54 [Parachain] 🏁 Disk score (seq. writes): 1.01 GiBs    
2025-01-16 17:10:54 [Parachain] 🏁 Disk score (rand. writes): 431.27 MiBs    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/36438/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/52486/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/44950/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/42304/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/36166/ws/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/60362/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/47590/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/42168/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41842/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:55 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/30334/ws/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:55 [Parachain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/30333/ws/p2p/12D3KooWRKHfcHjFxkVMMjV89AF3cUovsRunxpKdbW6mCaPRB3Kz    
2025-01-16 17:10:55 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/57346/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/50818/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/36246/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/37326/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/54562/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/51544/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/59034/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/39896/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/30334/ws/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:57 [Relaychain] Report 12D3KooWMrmrKcBk5PNjzqz3e6QqbPxXumznxxBSekQrWkZhg1p1: -2147483648 to -2147483648. Reason: Unsupported protocol. Banned, disconnecting.    
2025-01-16 17:10:59 [Relaychain] ⚙️  Syncing, target=#26674705 (8 peers), best: #3679 (0x234b…bf17), finalized #3584 (0xcdc0…728c), ⬇ 360.1kiB/s ⬆ 25.1kiB/s    
2025-01-16 17:10:59 [Parachain] 💤 Idle (0 peers), best: #0 (0x9256…0469), finalized #0 (0x9256…0469), ⬇ 0.5kiB/s ⬆ 0.2kiB/s    
2025-01-16 17:11:04 [Relaychain] ⚙️  Syncing 831.8 bps, target=#26674706 (8 peers), best: #7838 (0x814e…7b64), finalized #7680 (0x8862…b1be), ⬇ 265.3kiB/s ⬆ 6.3kiB/s    
2025-01-16 17:11:04 [Parachain] 💤 Idle (0 peers), best: #0 (0x9256…0469), finalized #0 (0x9256…0469), ⬇ 5 B/s ⬆ 25 B/s    
2025-01-16 17:11:04 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/56598/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:04 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/53764/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:04 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/33274/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:06 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/44500/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:09 [Relaychain] ⚙️  Syncing 601.2 bps, target=#26674707 (8 peers), best: #10844 (0x80cf…1e23), finalized #10752 (0x0877…1abf), ⬇ 389.5kiB/s ⬆ 4.5kiB/s    
2025-01-16 17:11:09 [Parachain] 💤 Idle (0 peers), best: #0 (0x9256…0469), finalized #0 (0x9256…0469), ⬇ 5 B/s ⬆ 25 B/s    
2025-01-16 17:11:10 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41300/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:12 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/54882/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:14 [Relaychain] ⚙️  Syncing 785.2 bps, target=#26674708 (8 peers), best: #14770 (0x0be7…9533), finalized #13824 (0xb224…3695), ⬇ 216.4kiB/s ⬆ 5.6kiB/s    
2025-01-16 17:11:14 [Parachain] 💤 Idle (0 peers), best: #0 (0x9256…0469), finalized #0 (0x9256…0469), ⬇ 18 B/s ⬆ 38 B/s    
2025-01-16 17:11:14 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/59388/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/46642/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/56654/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/60186/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/34190/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/45398/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41162/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/50954/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/48586/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/40156/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41818/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/44914/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/60434/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/39046/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/48714/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/35934/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/48054/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/45414/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/46450/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/50894/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41426/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:16 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/56786/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ

This continues for minutes/hours. I encountered this issue on a scaleway dev machine, can easily be reproduced there.

cc @paritytech/networking
@skunert skunert added the T0-node This PR/Issue is related to the topic “node”. label Jan 16, 2025
@lexnv
Copy link
Contributor

lexnv commented Jan 16, 2025

I suspect this is coming from libp2p implementation on substrate side:

  • we confirm all addresses

    polkadot-sdk/substrate/client/network/src/peer_info.rs

    Lines 408 to 419 in be2404c

    FromSwarm::NewExternalAddrCandidate(e @ NewExternalAddrCandidate { addr }) => {
    self.ping.on_swarm_event(FromSwarm::NewExternalAddrCandidate(e));
    self.identify.on_swarm_event(FromSwarm::NewExternalAddrCandidate(e));
    // Manually confirm all external address candidates.
    // TODO: consider adding [AutoNAT protocol](https://docs.rs/libp2p/0.52.3/libp2p/autonat/index.html)
    // (must go through the polkadot protocol spec) or implemeting heuristics for
    // approving external address candidates. This can be done, for example, by
    // approving only addresses reported by multiple peers.
    // See also https://github.com/libp2p/rust-libp2p/pull/4721 introduced
    // in libp2p v0.53 for heuristics approach.
    self.pending_actions.push_back(ToSwarm::ExternalAddrConfirmed(addr.clone()));

  • then we propagate them to discovery
    https://github.com/paritytech/polkadot-sdk/blob/be2404cccd9923c41e2f16bfe655f19574f1ae0e/substrate/client/network/src/discovery.rs#L778-L800C1

In litep2p we wait for 2 confirmations on the same address from different peers before propagating the address

polkadot-sdk/substrate/client/network/src/litep2p/discovery.rs

Line 79 in be2404c

const MIN_ADDRESS_CONFIRMATIONS: usize = 2;

@skunert
Copy link
Contributor Author

skunert commented Jan 17, 2025

Can confirm that it does not happen for litep2p

@skunert
Copy link
Contributor Author

skunert commented Jan 17, 2025

In litep2p we wait for 2 confirmations on the same address from different peers before propagating the address

This sounds reasonable to me.

One question is why we are seeing this just now, you think this broke with the most recent libp2p upgrade?

@lexnv
Copy link
Contributor

lexnv commented Jan 17, 2025 •
edited
Loading

There has been quite a few refactors / changes to the port reuse, I expect they are coming from:

Deprecate port_reuse setting, as this is now decided by the behaviour, not the transport.

Other issue on origin/master

On the side, our origin/master libp2p does not contain the following fix for TCP:

  • Fix the disabling of Nagle's algorithm, which requires setting TCP_NODELAY to true.

This is a bit concerning, as the changelog suggest the TCP_NODELAY issue was unnoticed for ~7months (cc libp2p/rust-libp2p#4916)

Will need to double check this, maybe there was a point release from another branch

@lexnv lexnv closed this as completed Jan 17, 2025
@github-project-automation github-project-automation bot moved this to Blocked ⛔️ in Networking Jan 17, 2025
@lexnv lexnv reopened this Jan 17, 2025
@lexnv lexnv moved this from Blocked ⛔️ to Backlog 🗒 in Networking Jan 17, 2025
@dmitry-markin
Copy link
Contributor

Other issue on origin/master

On the side, our origin/master libp2p does not contain the following fix for TCP:

  • Fix the disabling of Nagle's algorithm, which requires setting TCP_NODELAY to true.

This is a bit concerning, as the changelog suggest the TCP_NODELAY issue was unnoticed for ~7months (cc libp2p/rust-libp2p#4916)

Will need to double check this, maybe there was a point release from another branch

TCP_NODELAY is explicitly set to true in substrate, so this issue does not affect us:

polkadot-sdk/substrate/client/network/src/transport.rs

Line 51 in f5673cf

let tcp_config = tcp::Config::new().nodelay(true);

@dmitry-markin dmitry-markin mentioned this issue Jan 17, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
T0-node This PR/Issue is related to the topic “node”.
Projects
Networking
Status: Backlog 🗒
Milestone
No milestone
Development

No branches or pull requests
3 participants
@skunert @dmitry-markin @lexnv