Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Node discovery via DNS #7129

Merged
merged 42 commits into from
Jun 5, 2024
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
ea7545f
feat: Use besu-dns-discovery
usmansaleem May 22, 2024
5bbdeb6
changelog: Update changelog
usmansaleem May 22, 2024
e75b5de
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 22, 2024
ea62a8a
typo
macfarla May 22, 2024
9a780d8
test: Fix unit test mocking
usmansaleem May 22, 2024
f0684c8
Merge remote-tracking branch 'origin/consensys_dns_discovery' into co…
usmansaleem May 22, 2024
ce46275
test: spotless fix
usmansaleem May 22, 2024
094c976
feat: Add dns discovery classes
usmansaleem May 24, 2024
a778c20
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 24, 2024
1704974
test: Add unit tests
usmansaleem May 24, 2024
486c61c
feat: Use a separate thread to convert vertx dns-client call to sync
usmansaleem May 26, 2024
31a121b
codefix: spotless apply
usmansaleem May 26, 2024
45d8faa
fix unit test and refactor periodic timer task
usmansaleem May 26, 2024
00c519d
changelog: Update changelog
usmansaleem May 26, 2024
2193921
refactor and cleaning up
usmansaleem May 27, 2024
355efee
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 27, 2024
734c786
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 27, 2024
107bf02
add final keyword
usmansaleem May 27, 2024
2ab08bf
changelog: Update changelog
usmansaleem May 27, 2024
0ad16c1
codefix: spotless apply
usmansaleem May 28, 2024
b521d42
revert gradle verification metadata
usmansaleem May 28, 2024
e54cc02
throw illegalstateexception if dnsdaemon is not started as a vertx ve…
usmansaleem May 28, 2024
ab81575
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 28, 2024
e2b8339
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 28, 2024
9e593bc
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 29, 2024
d95011e
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 29, 2024
e77fa1e
codefix: remove unnecessary this prefix
usmansaleem May 30, 2024
400f560
codefix: decode public key in constructor
usmansaleem May 30, 2024
dd2f741
codefix: Remove unnecessary files
usmansaleem May 30, 2024
9b2aed9
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 30, 2024
16507f6
codefix: Fix LOG class reference
usmansaleem May 30, 2024
9750fff
codefix: Reduce initial delay and recurring delay in dnsdaemontest
usmansaleem May 30, 2024
8cf02d7
codefix: Read dns txt entries from a json file in mock dns server
usmansaleem May 30, 2024
c40a643
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 30, 2024
f081e64
Convert Mock Dns Server as Vertx Verticle for easier management
usmansaleem May 31, 2024
574ec28
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 31, 2024
636aedb
increase second lookup time in test
usmansaleem May 31, 2024
b5dfc73
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem May 31, 2024
f91143b
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem Jun 3, 2024
806bf5b
Update NOTICE
usmansaleem Jun 3, 2024
8abc6cc
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem Jun 4, 2024
4ed7736
Merge remote-tracking branch 'upstream/main' into consensys_dns_disco…
usmansaleem Jun 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
- Default bonsai to use full-flat db and code-storage-by-code-hash [#6984](https://github.com/hyperledger/besu/pull/6894)
- New RPC methods miner_setExtraData and miner_getExtraData [#7078](https://github.com/hyperledger/besu/pull/7078)
- Disconnect peers that have multiple discovery ports since they give us bad neighbours [#7089](https://github.com/hyperledger/besu/pull/7089)
- Port Tuweni dns-discovery into Besu. [#7129](https://github.com/hyperledger/besu/pull/7129)

### Known Issues
- [Frequency: occasional < 10%] Chain download halt. Only affects new syncs (new nodes syncing from scratch). Symptom: Block import halts, despite having a full set of peers and world state downloading finishing. Generally restarting besu will resolve the issue. We are tracking this in [#6884](https://github.com/hyperledger/besu/pull/6884)
Expand Down
4 changes: 1 addition & 3 deletions ethereum/p2p/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,6 @@ dependencies {
implementation('io.tmio:tuweni-devp2p') {
exclude group:'ch.qos.logback', module:'logback-classic'
}
implementation('io.tmio:tuweni-dns-discovery'){
exclude group:'ch.qos.logback', module:'logback-classic'
}
implementation 'io.tmio:tuweni-io'
implementation 'io.tmio:tuweni-rlp'
implementation 'io.tmio:tuweni-units'
Expand Down Expand Up @@ -84,6 +81,7 @@ dependencies {
}
testImplementation 'io.vertx:vertx-codegen'
testImplementation 'io.vertx:vertx-unit'
testImplementation 'io.vertx:vertx-junit5'
testImplementation 'org.assertj:assertj-core'
testImplementation 'org.awaitility:awaitility'
testImplementation 'org.junit.jupiter:junit-jupiter'
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
/*
* Copyright contributors to Hyperledger Besu.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
* SPDX-License-Identifier: Apache-2.0
*/
package org.hyperledger.besu.ethereum.p2p.discovery.dns;

import java.util.List;
import java.util.Optional;

import io.vertx.core.AbstractVerticle;
import org.apache.tuweni.devp2p.EthereumNodeRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

// Adapted from https://github.com/tmio/tuweni and licensed under Apache 2.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this sufficient attribution as far as the apache license is concerned?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is. Although we may have to update our NOTICE file with Tuweni's NOTICE file. Our NOTICE file is empty though. Also, we didn't do a verbatim copy of the code, we adapted and converted it into Java language from Kotlin language.

https://opensource.stackexchange.com/questions/8717/how-to-attribute-code-from-a-project-under-apache-v2-in-a-new-project

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we add all contributors info to the notice going forward?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if we are "reproducing" or "including" code from an Apache license project that contains a NOTICE file, it ought to be included in our NOTICE file as well. I am not sure whether we had to include NOTICE attributions from Vertx and other libraries that are included in binary form. @non-fungible-nelson FYI for your input as well.

/**
* Resolves DNS records over time, refreshing records. This is written as a Vertx Verticle which
* allows to run outside the Vertx event loop
*/
public class DNSDaemon extends AbstractVerticle {
private static final Logger LOG = LoggerFactory.getLogger(DNSDaemon.class);
private final String enrLink;
private final long seq;
private final long initialDelay;
private final long delay;
private final Optional<DNSDaemonListener> listener;
private final Optional<String> dnsServer;
private Optional<Long> periodicTaskId = Optional.empty();
private DNSResolver dnsResolver;

/**
* Creates a new DNSDaemon.
*
* @param enrLink the ENR link to start with, of the form enrtree://PUBKEY@domain
* @param listener Listener notified when records are read and whenever they are updated.
* @param seq the sequence number of the root record. If the root record seq is higher, proceed
* with visit.
* @param initialDelay the delay in milliseconds before the first poll of DNS records.
* @param delay the delay in milliseconds at which to poll DNS records. If negative or zero, it
* runs only once.
* @param dnsServer the DNS server to use for DNS query. If null, the default DNS server will be
* used.
*/
public DNSDaemon(
final String enrLink,
final DNSDaemonListener listener,
final long seq,
final long initialDelay,
final long delay,
final String dnsServer) {
this.enrLink = enrLink;
this.listener = Optional.ofNullable(listener);
this.seq = seq;
this.initialDelay = initialDelay;
this.delay = delay;
this.dnsServer = Optional.ofNullable(dnsServer);
}

/** Starts the DNSDaemon. */
@Override
public void start() {
if (vertx == null) {
throw new IllegalStateException("DNSDaemon must be deployed as a vertx verticle.");
}

LOG.info("Starting DNSDaemon for {}, using {} DNS host.", enrLink, dnsServer.orElse("default"));
this.dnsResolver = new DNSResolver(vertx, enrLink, seq, dnsServer);
usmansaleem marked this conversation as resolved.
Show resolved Hide resolved
if (delay > 0) {
periodicTaskId = Optional.of(vertx.setPeriodic(initialDelay, delay, this::refreshENRRecords));
} else {
// do one-shot resolution
refreshENRRecords(0L);
}
}

/** Stops the DNSDaemon. */
@Override
public void stop() {
LOG.info("Stopping DNSDaemon for {}", enrLink);
periodicTaskId.ifPresent(vertx::cancelTimer);
dnsResolver.close();
}

/**
* Refresh enr records by calling dnsResolver and updating the listeners.
*
* @param taskId the task id of the periodic task
*/
void refreshENRRecords(final Long taskId) {
LOG.debug("Refreshing DNS records");
final long startTime = System.nanoTime();
final List<EthereumNodeRecord> ethereumNodeRecords = dnsResolver.collectAll();
final long endTime = System.nanoTime();
LOG.debug("Time taken to DNSResolver.collectAll: {} ms", (endTime - startTime) / 1_000_000);
listener.ifPresent(it -> it.newRecords(dnsResolver.sequence(), ethereumNodeRecords));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
/*
* Copyright contributors to Hyperledger Besu.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
* SPDX-License-Identifier: Apache-2.0
*/
package org.hyperledger.besu.ethereum.p2p.discovery.dns;

import java.util.List;

import org.apache.tuweni.devp2p.EthereumNodeRecord;

// Adapted from https://github.com/tmio/tuweni and licensed under Apache 2.0
/** Callback listening to updates of the DNS records. */
@FunctionalInterface
public interface DNSDaemonListener {
/**
* Callback called when the seq is updated on the DNS server
*
* @param seq the update identifier of the records
* @param records the records stored on the server
*/
void newRecords(long seq, List<EthereumNodeRecord> records);
}
Loading
Loading