Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support throttling block syncing to peers. #1540

Closed
wants to merge 53 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
e68743a
Support throttling block syncing to peers. WIP
jgiszczak Aug 22, 2023
303c3d6
Add exponential backoff to throttle. Fix wretched math.
jgiszczak Aug 23, 2023
3789d17
Merge branch 'main' into p2p-peer-throttle
jgiszczak Aug 23, 2023
70b530b
Experiment: How many tests fail if waitForObj default times out
jgiszczak Aug 24, 2023
b92d84c
Address review comments in net_plugin.
jgiszczak Aug 24, 2023
dc54d46
Further tweak the sync throttle test for machines faster than mine.
jgiszczak Aug 24, 2023
3a50864
Move block sync throttling to the correct layer in the call stack.
jgiszczak Aug 25, 2023
e1c1d42
Move block sync rate limit parsing to plugin initialize.
jgiszczak Aug 25, 2023
92e4e7c
Require IPv6 addresses to be in square bracket format.
jgiszczak Aug 30, 2023
28bb38d
Added throttle exception for configured p2p-peer-addresses.
jgiszczak Aug 30, 2023
1eb1e44
Merge branch 'main' into p2p-peer-throttle
jgiszczak Sep 1, 2023
1983d90
Merge branch 'main' into p2p-peer-throttle
jgiszczak Sep 7, 2023
e998cc6
Debug commit, doesn't build.
jgiszczak Sep 11, 2023
92e4022
Fix build error. Lambda captures by value are const.
jgiszczak Sep 11, 2023
b24f8e3
Fix bare numeric value for peer throttle.
jgiszczak Sep 11, 2023
3f67034
Update supplied_peers only once per configured peer and once per API …
jgiszczak Sep 12, 2023
e59451d
Remove encapsulation violation.
jgiszczak Sep 15, 2023
ec2d36d
Restored connection reconnect method. WIP
jgiszczak Sep 18, 2023
777c8d8
Remove stacktrace additions
heifner Sep 18, 2023
e88c259
Merge branch 'p2p-peer-throttle' of github.com:AntelopeIO/leap into p…
jgiszczak Sep 18, 2023
05be825
Update netApi connect test.
jgiszczak Sep 20, 2023
f8763ae
Merge branch 'main' into p2p-peer-throttle
jgiszczak Sep 20, 2023
24f4aad
GH-1295 Project index to default iterator
heifner Sep 20, 2023
97591fe
Use reconnect method as intended, and avoid threading issue.
jgiszczak Sep 20, 2023
ffee0df
Use std::any_of when finding supplied peers to unlimit a connection.
jgiszczak Sep 20, 2023
fb740ef
Change language in p2p_multiple_listen_test for clarity.
jgiszczak Sep 23, 2023
453bfcf
Tolerate duplicate (empty) peer addresses in connection manager.
jgiszczak Sep 23, 2023
99f02c0
Add rate limit parse unittest.
jgiszczak Sep 27, 2023
b16184a
Tolerate node running with no listen endpoints.
jgiszczak Sep 27, 2023
df6d948
Restore lock of connections mutex when connecting configured peers.
jgiszczak Sep 27, 2023
733849b
Don't pass around iterators that may be invalidated by an erase.
jgiszczak Sep 27, 2023
ed69238
Renamed method.
jgiszczak Sep 27, 2023
2f80663
Break encapsulation less.
jgiszczak Sep 27, 2023
7019b65
Thread safety.
jgiszczak Sep 27, 2023
4d136e3
Revert "Restore lock of connections mutex when connecting configured …
jgiszczak Sep 27, 2023
3708418
Restore lock of connections mutex when connecting configured peers.
jgiszczak Sep 27, 2023
4baec72
Accept suggested refactoring.
jgiszczak Sep 27, 2023
8d2c1c2
Remove some unused machine-generated variables from custom shape file.
jgiszczak Sep 27, 2023
7e37de1
Convert connections mutex to resursive_mutex and update locks.
jgiszczak Sep 28, 2023
669ed0f
Revert mutex and lock type changes.
jgiszczak Sep 29, 2023
a6f7761
Revise connection_monitor for thread safety.
jgiszczak Oct 2, 2023
2ef3a6c
Merge branch 'main' into p2p-peer-throttle
jgiszczak Oct 2, 2023
7acac0c
Misc cleanups
heifner Oct 2, 2023
ff7a8a1
Add block sync bytes received metric and use it in sync throttle test.
jgiszczak Oct 4, 2023
6b2fe63
Add requests module for test.
oschwaldp-oci Oct 4, 2023
a6ed57d
Merge branch 'main' into p2p-peer-throttle
oschwaldp-oci Oct 4, 2023
d1ad2cf
Merge branch 'main' into p2p-peer-throttle
jgiszczak Oct 5, 2023
1e5b427
Add throttling flag to Prometheus peer data and use it in sync test.
jgiszczak Oct 6, 2023
db34bbf
Revise for better repeatability.
jgiszczak Oct 6, 2023
6db4ad8
Customize plugin_config_exception handling in net_plugin.
jgiszczak Oct 6, 2023
d8610c6
Merge branch 'main' into p2p-peer-throttle
jgiszczak Oct 7, 2023
5576604
Merge branch 'main' into p2p-peer-throttle
jgiszczak Oct 9, 2023
b41dd96
Merge branch 'main' into p2p-peer-throttle
jgiszczak Oct 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ class bp_connection_manager {
// Only called from connection strand
std::size_t num_established_clients() const {
uint32_t num_clients = 0;
self()->connections.for_each_connection([&num_clients](auto&& conn) {
self()->connections.for_each_connection([&num_clients](const std::shared_ptr<Connection>& conn) {
if (established_client_connection(conn)) {
++num_clients;
}
Expand All @@ -157,7 +157,7 @@ class bp_connection_manager {
// Only called from connection strand
// This should only be called after the first handshake message is received to check if an incoming connection
// has exceeded the pre-configured max_client_count limit.
bool exceeding_connection_limit(Connection* new_connection) const {
bool exceeding_connection_limit(std::shared_ptr<Connection> new_connection) const {
heifner marked this conversation as resolved.
Show resolved Hide resolved
return auto_bp_peering_enabled() && self()->connections.get_max_client_count() != 0 &&
established_client_connection(new_connection) && num_established_clients() > self()->connections.get_max_client_count();
}
Expand All @@ -182,7 +182,7 @@ class bp_connection_manager {

fc_dlog(self()->get_logger(), "pending_downstream_neighbors: ${pending_downstream_neighbors}",
("pending_downstream_neighbors", to_string(pending_downstream_neighbors)));
for (auto neighbor : pending_downstream_neighbors) { self()->connections.connect(config.bp_peer_addresses[neighbor], *self()->p2p_addresses.begin() ); }
for (auto neighbor : pending_downstream_neighbors) { self()->connections.resolve_and_connect(config.bp_peer_addresses[neighbor], self()->get_first_p2p_address() ); }

pending_neighbors = std::move(pending_downstream_neighbors);
finder.add_upstream_neighbors(pending_neighbors);
Expand Down
3 changes: 3 additions & 0 deletions plugins/net_plugin/include/eosio/net_plugin/net_plugin.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ namespace eosio {
std::chrono::nanoseconds last_bytes_received{0};
size_t bytes_sent{0};
std::chrono::nanoseconds last_bytes_sent{0};
size_t block_sync_bytes_received{0};
size_t block_sync_bytes_sent{0};
bool block_sync_throttling{false};
std::chrono::nanoseconds connection_start_time{0};
std::string log_p2p_address;
};
Expand Down
4 changes: 3 additions & 1 deletion plugins/net_plugin/include/eosio/net_plugin/protocol.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,11 @@ namespace eosio {

// Longest domain name is 253 characters according to wikipedia.
// Addresses include ":port" where max port is 65535, which adds 6 chars.
// Addresses may also include ":bitrate" with suffix and separators, which adds 30 chars,
// for the maximum comma-separated value that fits in a size_t expressed in decimal plus a suffix.
// We also add our own extentions of "[:trx|:blk] - xxxxxxx", which adds 14 chars, total= 273.
// Allow for future extentions as well, hence 384.
constexpr size_t max_p2p_address_length = 253 + 6;
constexpr size_t max_p2p_address_length = 253 + 6 + 30;
constexpr size_t max_handshake_str_length = 384;

struct handshake_message {
Expand Down
Loading
Loading