-
Notifications
You must be signed in to change notification settings - Fork 137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TESTING] Script for benchmarking builders locally #137
Open
kamilchodola
wants to merge
78
commits into
flashbots:main
Choose a base branch
from
NethermindEth:main
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
78 commits
Select commit
Hold shift + click to select a range
2a2511d
Runner script (#1)
OlegJakushkin 00ba47b
Update builder metrics flags
cbermudez97 d032267
fix: update metrics flags
cbermudez97 b5b1d20
feat: update dashboard tooltips
cbermudez97 141ce8e
feat: use ethereum package main branch
cbermudez97 b8d791c
feat: use version 0.6.1 with builder fix
cbermudez97 124cd55
updated build script
OlegJakushkin 4b58a8f
Increase the number of generated transactions. Update custom flood co…
piwonskp a110e8b
Fix additional builder metrics flags (#3)
cbermudez97 6cf4f91
Fix builders not submitting blocks. Use geth as it appear to deliver…
piwonskp 40ac12b
Update issue templates
kamilchodola 4765df9
Revert "Update issue templates"
kamilchodola 35fc869
feat: update old dashboard panels
cbermudez97 b6b0eae
feat: add smaller dashboard for builders comparisson
cbermudez97 d3ec936
Merge pull request #4 from NethermindEth/fix-txs-not-peered
piwonskp 64fc940
Change URL for Kurtosis install (#5)
kamilchodola 30c2139
chore: merge branch 'main' into dashboards-improvements
cbermudez97 67bef29
feat: use ethereum package version 1.0.0
cbermudez97 d9afb54
fix: revert "feat: use ethereum package version 1.0.0"
cbermudez97 25b0784
Merge branch 'main' into dashboards-improvements
cbermudez97 755cce2
feat: add additional prometheus labels
cbermudez97 713bc18
feat: update labels in dashboards
cbermudez97 b91f222
feat: centralize dashboards and apply custom prometheus config
cbermudez97 0e8f484
fix: wrong prometheus config values
cbermudez97 1e0ca5e
Merge remote-tracking branch 'origin/dashboards-improvements' into main
OlegJakushkin 333d014
tested dind
OlegJakushkin 591d8e3
fix: wrong units for slots on blocks build time
cbermudez97 480d3df
feat: update ethereum package branch
cbermudez97 e607941
Use broadcaster instead of peering between builders
piwonskp 5ed3ba5
Merge remote-tracking branch 'origin/broadcaster' into dashboards-imp…
cbermudez97 06a35e2
Use ethereum-package main branch and a newer version of nethermind cl…
piwonskp e985a43
Merge remote-tracking branch 'origin/broadcaster' into dashboards-imp…
cbermudez97 51ade5b
feat: improve dashboard and reduce slot time
cbermudez97 daeb9dd
feat: allow to disable epoch 4 annotation
cbermudez97 029dd54
fix: wrong value mappings
cbermudez97 deb051e
Add slot Time
kamilchodola cd6bff9
Add missing usage
kamilchodola 27fa340
feat: add bids built count
cbermudez97 8a3d7e8
Started finetuning fore public ports on eth package
OlegJakushkin 0a21923
Merge remote-tracking branch 'origin/kch/add_slot_time' into main
OlegJakushkin 5cfcc17
Partial enclave stop and progress monitoring
OlegJakushkin ff4fb5f
Merge pull request #7 from NethermindEth/broadcaster
piwonskp 5f67305
Merge pull request #8 from NethermindEth/dashboards-improvements
kamilchodola f3b8bfe
Merge pull request #9 from NethermindEth/kch/add_slot_time
kamilchodola fd6bfc8
Merge branch 'main' into partial-stop
kamilchodola dbe6a31
Merge pull request #10 from NethermindEth/partial-stop
kamilchodola 78c61e0
Set dev builder image tag as `dev`
piwonskp 720c2e8
Merge pull request #11 from NethermindEth/piwonskp/change-dev-builder…
kamilchodola 69ffbe0
Fix linting
kamilchodola cd2c2f4
Adjust references
kamilchodola cd2b117
Merge pull request #12 from NethermindEth/kch/lint
kamilchodola ded84f9
Fix readme, change -s to 1000 and improve summary
kamilchodola c3f2274
Fix method def
kamilchodola af74e21
Merge pull request #13 from NethermindEth/kch/adjust_default_slot_number
kamilchodola 8f905f3
Pretty print service urls
piwonskp a3498e0
Add third builder - relay validator
piwonskp 1260c63
Print exact url of the dashboard, not the grafana service root
piwonskp 487bae4
Use latest versions of lighthouse and nethermind & fix baseline extra…
piwonskp 58af272
Change the prometheus label so validator is not visible on the dashboard
piwonskp bdeb000
Use default lighthouse image
piwonskp 463fbfd
Merge pull request #14 from NethermindEth/bold-service-urls
piwonskp 354ef10
Merge pull request #15 from NethermindEth/three-builders
kamilchodola c440fa5
Using default Dockerfile for build (#16)
OlegJakushkin c2cf45d
Add architecture diagram
piwonskp b783bfa
Increase fuzzing rate
piwonskp 240b202
Merge pull request #17 from NethermindEth/increase-fuzzing-rate
piwonskp db9578c
Uppercase readme
kamilchodola 9ef02d2
Merge remote-tracking branch 'origin/main' into dind
OlegJakushkin 475e6c7
clean up dind docs
OlegJakushkin b3cb7c0
added wip disclamer
OlegJakushkin 02fea33
fix diff with main
OlegJakushkin cc3f7bb
fix distance to main
OlegJakushkin d32645f
Update scripts/README.md
piwonskp c824177
Update scripts/README.md
piwonskp f488a4f
Merge pull request #18 from NethermindEth/dind
piwonskp 8fc222a
Add dashboard and architecture description to the readme
piwonskp 6e617c2
Fix links to fuzzeers
piwonskp 3cee741
Rename readme -> README
piwonskp File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
# Support setting various labels on the final image | ||
ARG COMMIT="" | ||
ARG VERSION="" | ||
ARG BUILDNUM="" | ||
|
||
# Build Geth in a stock Go builder container | ||
FROM golang:1.20-alpine as builder | ||
|
||
RUN apk add --no-cache gcc musl-dev linux-headers git | ||
|
||
# Get dependencies - will also be cached if we won't change go.mod/go.sum | ||
COPY go.mod /go-ethereum/ | ||
COPY go.sum /go-ethereum/ | ||
RUN cd /go-ethereum && go mod download | ||
|
||
ADD . /go-ethereum | ||
|
||
RUN cd /go-ethereum && go build -o ./build/bin/geth -gcflags=all='-N -l' -v ./cmd/geth | ||
|
||
RUN go install github.com/go-delve/delve/cmd/dlv@latest | ||
# Pull Geth into a second stage deploy alpine container | ||
FROM alpine:latest | ||
|
||
RUN apk add --no-cache ca-certificates | ||
COPY --from=builder /go-ethereum/build/bin/geth /usr/local/bin/ | ||
COPY --from=builder /go/bin/dlv / | ||
EXPOSE 8545 8546 30303 30303/udp 40000 | ||
ENTRYPOINT ["/dlv", "--listen=:40000", "--headless=true", "--continue", "--accept-multiclient", "--api-version=2", "exec", "/usr/local/bin/geth"] | ||
# Add some metadata labels to help programatic image consumption | ||
ARG COMMIT="" | ||
ARG VERSION="" | ||
ARG BUILDNUM="" | ||
|
||
LABEL commit="$COMMIT" version="$VERSION" buildnum="$BUILDNUM" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,125 @@ | ||
# Emulate Network script | ||
|
||
This README accompanies the `emulate_network.go` Go script, which uses Docker, Kurtosis, and the Kurtosis Ethereum-Package enclave to test builder performance. | ||
|
||
## Introduction | ||
|
||
This script streamlines and automates the process of (re-)building the current builder Docker image and interfacing with the Kurtosis platform. It offers a set of commands to assist developers in routine tasks, such as (re-)building Docker images and managing Kurtosis enclaves. | ||
|
||
|
||
## Running in dev environment | ||
|
||
### Prerequisites | ||
|
||
Before using the script, ensure you have the following installed: | ||
|
||
1. **Go**: Ensure you have Go installed. Download and install it from [here](https://golang.org/dl/). | ||
|
||
2. **Docker**: The build process needs Docker. Make sure Docker is installed and running. Check out the [Docker website](https://www.docker.com/get-started) for installation instructions. | ||
|
||
3. **Kurtosis**: The script interfaces with the Kurtosis platform. Ensure `kurtosis` is installed and available in your PATH. Refer to [Kurtosis's official documentation](https://docs.kurtosis.com/install) for installation details. | ||
|
||
4. **ethereum-package**: This script utilizes a modified ethereum-package network configuration, which will be downloaded from [repo](github.com/kurtosis-tech/ethereum-package/) main brunch automatically. | ||
|
||
### How to Run | ||
|
||
To execute the script with Go, navigate to the directory containing the script and run: | ||
|
||
``` | ||
go run emulate_network.go <COMMAND_NAME> [OPTIONS] | ||
``` | ||
|
||
Replace `<COMMAND_NAME>` with one of the available commands and provide the necessary options. | ||
|
||
### Commands and Options | ||
To run script `cd` into this (`./scripts`) folder. | ||
|
||
1. **build**: | ||
- Purpose: Builds a Docker image of the builder. | ||
- Options: | ||
- `-t`: (Optional) Image tag for the Docker build. Defaults to `flashbots/builder:dev`. | ||
- `-d` (Optional) Image Build directory. Defaults to `".."` | ||
- `-f` (Optional) Build dockerfile path. Defaults to `"../Dockerfile"`. Use `"./Dockerfile.debug"` for debug capabilities. | ||
|
||
- Example: | ||
``` | ||
go run emulate_network.go build -t=test-builder -f="../Dockerfile" -d=".." | ||
``` | ||
|
||
|
||
|
||
2. **run**: | ||
- Purpose: Prepares configurations and starts a Kurtosis enclave. | ||
- Options: | ||
- `-t`: (Optional) Image tag. Defaults to `flashbots/builder:dev`. | ||
- `-n`: (Optional) Enclave name. Defaults to `explorer`. | ||
- `-a`: (Optional) Additional builder arguments. | ||
- `-s`: (Optional) Max steps (integer). Defaults to `1000`. Set for `-1` for "unlimited". | ||
- `-k`: (Optional) Path to `kurtosis` executable. Defaults to `kurtosis`. | ||
- `--slotTime`: (Optional) Seconds per slot applied on local devnet. Defaults to 5. | ||
- Example: | ||
``` | ||
go run emulate_network.go run -t=imageTag -a=imageArgs -n=enclaveName -k=/path/to/kurtosis | ||
``` | ||
|
||
3. **stop**: | ||
- Purpose: Stops an active Kurtosis enclave. | ||
- Options: | ||
- `-k`: (Optional) Path to `kurtosis` executable. Defaults to `kurtosis`. | ||
- `-n`: (Required) Enclave name. | ||
- Example: | ||
``` | ||
go run emulate_network.go stop -k=/path/to/kurtosis -n=enclaveName | ||
``` | ||
|
||
4. **help**: | ||
- Purpose: Display a summary of available commands and their options. | ||
- Example: | ||
``` | ||
go run emulate_network.go help | ||
``` | ||
## Dashboard | ||
Grafana dashboard contains three sections: quality comparison, usage of system resources and a section to inspect network behaviour. Each section contain graphs designed to show different properties of the network. | ||
|
||
Quality comparison allows to compare builder implementations by analysing bids produced by each of them. It contains four graphs: | ||
* Bids profit - Profit of bids generated by each builder in time. The most important metric to compare effectiveness of builders. | ||
* Bids creation time - Measures the time spent on creating a single bid. The metric allows to compare performance of builders. | ||
* Bids Txnum - Number of bids submitted per second | ||
* Bids Gas used - Summarized gas of all produced bids per second | ||
|
||
The dashboard also provides graphs showing usage of system resources: CPU, memory, disk and network. It allows to monitor resources and quickly identify regressions in resource utilization of a particular implementation. | ||
|
||
Network inspection gives an overview of the network. It provides information such as the transaction pool, block processing time, transaction propagation and database access. The section may be useful in certain circumstances to get the detailed information and for debugging. | ||
|
||
## Architecture | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Additionally, I think it'll be super useful if there's a "dashboard" section here explaining all the metrics that are being collected and what it is measuring |
||
![Test network architecture](architecture.png) | ||
|
||
The architecture consists of fuzzer layer, broadcaster, network, MEV services and data collection layer. | ||
|
||
Fuzzer layer: | ||
* [mev-flood](https://github.com/flashbots/mev-flood) - Spams the network with uniswap swaps | ||
* [tx-fuzz](https://github.com/MariusVanDerWijden/tx-fuzz/tree/master) - generates random transactions and deploys by utilizing [FuzzyVM](https://github.com/MariusVanDerWijden/FuzzyVM) | ||
* [Custom flood](https://github.com/kurtosis-tech/ethereum-package/blob/c558cb2eab25cc8c3718b1fda6759a0819e6f942/src/mev_custom_flood/sender.py)] - Simple fuzzer sending tokens between two accounts | ||
|
||
Fuzzer layer sends transactions to the broadcaster which replicate those and broadcast them to all nodes in parallel. | ||
|
||
Test network consists of 4 nodes: | ||
* Dev builder - candidate to be merged. | ||
* Baseline builder - current version of builder. | ||
* Validator builder - connected to the relay as validator. Being relay validator affects performance of the builder therefore the third builder is used to keep the competition between dev and baseline fair. | ||
* Proposer - regular execution layer client. The node contains validators that propose and vote for new blocks. It also builds blocks in epochs 0-3. | ||
|
||
Among those nodes there are three builders that send their bids to the relay. Then the best bid is chosen and passed to the proposer node which proposes it as the new block. | ||
|
||
Data collection layer consists of prometheus and grafana. Prometheus gathers the data from all nodes, while grafana displays the data for chosen nodes in form of graphs. | ||
|
||
## Known issues | ||
### Kurtosis errors on network start | ||
In case of system resource related errors or on windows docker restore after sleep Kurtosis may have trouble starting a new enclave. | ||
1. Make sure you have no valuable containers up and running in enclaves or docker | ||
2. To clean up Kurtosis call `kurtosis clean -a` | ||
3. To clean up docker run `docker rm -vf $(docker ps -aq)` | ||
|
||
|
||
## Running in Docker in Docker (WIP) | ||
See [README](./dind#readme). |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
FROM docker:dind | ||
|
||
# Create the /etc/machine-id file | ||
RUN apk add --no-cache dbus && \ | ||
dbus-uuidgen > /etc/machine-id | ||
|
||
# Install required packages | ||
RUN apk add --no-cache git curl tar | ||
|
||
# Download and install Go | ||
RUN curl https://dl.google.com/go/go1.21.3.linux-amd64.tar.gz -o go.tar.gz && \ | ||
tar -C /usr/local -xzf go.tar.gz && \ | ||
rm go.tar.gz | ||
|
||
# Set up Go environment variables | ||
ENV GOPATH /go | ||
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH | ||
RUN mkdir -p "${GOPATH}/src" "${GOPATH}/bin" && chmod -R 777 "${GOPATH}" | ||
|
||
# Download and install Kurtosis CLI | ||
RUN curl -L "https://github.com/kurtosis-tech/kurtosis-cli-release-artifacts/releases/download/0.85.3/kurtosis-cli_0.85.3_linux_amd64.tar.gz" -o kurtosis-cli.tar.gz && \ | ||
tar -xzvf kurtosis-cli.tar.gz -C /usr/local/bin && \ | ||
rm kurtosis-cli.tar.gz | ||
|
||
ADD . /go-ethereum | ||
WORKDIR /go-ethereum/scripts/ | ||
|
||
ENV DOCKER_HOST=unix:///var/run/docker.sock |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
# Emulate Network Using Docker-in-Docker (DinD) | ||
|
||
This documentation details the use of Docker-in-Docker (DinD) to emulate network environments for testing builder performance. This method provides an isolated environment, removing the need for Go or Kurtosis installations on the host system. | ||
|
||
## Introduction | ||
|
||
Utilizing a DinD setup allows developers to create a contained environment where Docker images can be built and Kurtosis enclaves can be managed. This is particularly useful for simulating network conditions in a clean state without affecting the host's setup. | ||
|
||
## Running in a DinD Environment | ||
|
||
### Prerequisites | ||
|
||
- **Docker**: Ensure Docker is installed and running on your system. The DinD process will be running as a Docker container. | ||
|
||
### Setup and Execution | ||
|
||
1. **Create the DinD Environment**: | ||
- Run a DinD container for an isolated build environment named `builder-dind-container` assuming you are currently in this folder: | ||
```shell | ||
docker build -f ./Dockerfile.kurtosis_dind -t builder-dind-image ../../ | ||
docker run -dit --privileged --name builder-dind-container builder-dind-image | ||
``` | ||
***note:*** privileged mode that is not needed when using local hosting in your docker described [here](../) | ||
|
||
2. **Build the Builder Image**: | ||
- Execute the build command within the DinD environment: | ||
```shell | ||
docker exec builder-dind-container go run emulate_network.go build -t=custom-builder-tag | ||
``` | ||
|
||
3. **Start the Enclave**: | ||
- Start an enclave with the specified builder image: | ||
```shell | ||
docker exec builder-dind-container go run emulate_network.go run -n=builder-enclave -t=custom-builder-tag | ||
``` | ||
|
||
4. **Stop the Enclave**: | ||
- To stop a running enclave, use the following command: | ||
```shell | ||
docker exec builder-dind-container go run emulate_network.go stop -n=builder-enclave | ||
``` | ||
|
||
5. **Cleanup**: | ||
- To stop and remove the DinD container, execute: | ||
```shell | ||
docker stop builder-dind-container | ||
docker rm builder-dind-container | ||
``` | ||
|
||
**Note**: Replace `builder-dind-container` with a descriptive name relevant to your project, and `builder-dind-image` with the image name you've prepared for the DinD environment. The `custom-builder-tag` should be replaced with the actual tag name you wish to assign to your builder image. | ||
|
||
## Known Issues and Solutions | ||
Ports control is missing in ethereum-package. | ||
|
||
By following these instructions, developers can leverage a Docker-in-Docker approach to emulate networks and test builder performance in a controlled and isolated manner. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it'll be useful to expand a bit more on the architecture. For example, how the broadcast architecture works (a bit of an explainer of what each of the fuzzers do) and also why a third validation builder is needed (due to performance reasons).