diff --git a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md b/.github/pull_request_template.md similarity index 100% rename from .github/PULL_REQUEST_TEMPLATE/pull_request_template.md rename to .github/pull_request_template.md diff --git a/.github/workflows/go.yml b/.github/workflows/go.yaml similarity index 95% rename from .github/workflows/go.yml rename to .github/workflows/go.yaml index a5b61594e..ffb4b6a45 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yaml @@ -80,6 +80,6 @@ jobs: run: | docker compose up -d node1 node2 node3 db docker compose up --abort-on-container-exit migrate-blocktx migrate-metamorph - docker compose up --exit-code-from tests tests arc-blocktx arc-metamorph arc --scale arc-blocktx=7 --scale arc-metamorph=2 + docker compose up --exit-code-from tests tests arc-blocktx arc-callbacker arc-metamorph arc --scale arc-blocktx=5 --scale arc-metamorph=2 docker compose down working-directory: ./test diff --git a/.github/workflows/image.yml b/.github/workflows/image.yaml similarity index 100% rename from .github/workflows/image.yml rename to .github/workflows/image.yaml diff --git a/.github/workflows/static-analysis.yml b/.github/workflows/static-analysis.yaml similarity index 98% rename from .github/workflows/static-analysis.yml rename to .github/workflows/static-analysis.yaml index 4ac7220cc..2af9fba16 100644 --- a/.github/workflows/static-analysis.yml +++ b/.github/workflows/static-analysis.yaml @@ -71,4 +71,4 @@ jobs: - name: golangci-lint uses: golangci/golangci-lint-action@v3 with: - version: v1.54 + version: v1.60.1 diff --git a/.github/workflows/static.yml b/.github/workflows/static.yaml similarity index 100% rename from .github/workflows/static.yml rename to .github/workflows/static.yaml diff --git a/.gitignore b/.gitignore index 0081a6d90..8dd600295 100644 --- a/.gitignore +++ b/.gitignore @@ -10,4 +10,5 @@ gosec-report.json gotest.out *.env report.xml +broadcaster-cli.yaml coverage-report.html diff --git a/CHANGELOG.md b/CHANGELOG.md index 601e2ac0d..7e9191917 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,9 +4,11 @@ All notable changes to this project will be documented in this file. The format ## Table of Contents - [Unreleased](#unreleased) +- [1.3.0](#130---2024-08-21) +- [1.2.0](#120---2024-08-13) - [1.1.91](#1191---2024-06-26) - [1.1.87](#1187---2024-06-10) -- [1.1.53](#1152---2024-04-11) +- [1.1.53](#1153---2024-04-11) - [1.1.32](#1132---2024-02-21) - [1.1.19](#1119---2024-02-05) - [1.1.16](#1116---2024-01-23) @@ -16,6 +18,21 @@ All notable changes to this project will be documented in this file. The format ## [Unreleased] +### Changed +- Callbacks are sent one by one to the same URL. In the previous implementation, each callback request created a new goroutine to send the callback, which could result in a potential DDoS of the callback receiver. The new approach sends callbacks to the same receiver in a serial manner. Note that URLs are not locked by the `callbacker` instance, so serial sends occur only within a single instance. In other words, the level of parallelism is determined by the number of `callbacker` instances. + +## [1.3.0] - 2024-08-21 + +### Changed +- The functionality for callbacks has been moved from the `metamorph` microservice to the new `callbacker` microservice. + +## [1.2.0] - 2024-08-13 + +### Added +- [Double Spend Detection](https://bitcoin-sv.github.io/arc/#/?id=double-spending) is a feature that introduces `DOUBLE_SPEND_ATTEMPTED` status to transactions that attempt double spend together with `CompetingTxs` field in the API responses and callbacks. +- [Cumulative Fees Validation](https://bitcoin-sv.github.io/arc/#/?id=cumulative-fees-validation) is a feature that checks if a transaction has a sufficient fee not only for itself but also for all unmined ancestors that do not have sufficient fees. +- [Multiple callbacks to single transaction](https://bitcoin-sv.github.io/arc/#/?id=callbacks) Is a feature that adds support for attaching multiple callbacks to a single transaction when submitting an existing transaction with a different data. + ## [1.1.91] - 2024-06-26 ### Changed diff --git a/Makefile b/Makefile index d3975605d..e7e4a344f 100644 --- a/Makefile +++ b/Makefile @@ -14,14 +14,6 @@ deps: build: go build ./... -.PHONY: clean_e2e_tests -clean_e2e_tests: - # Remove containers and images; avoid failure if the image doesn't exist - docker container stop test-tests-1 || true - docker container stop test-arc-1 || true - docker container rm test-tests-1 || true - docker container rm test-arc-1 || true - .PHONY: build_release build_release: mkdir -p build @@ -31,12 +23,19 @@ build_release: build_docker: docker build . -t test-arc --build-arg="APP_COMMIT=$(APP_COMMIT)" --build-arg="APP_VERSION=$(APP_VERSION)" +.PHONY: run +run: + docker compose -f test/docker-compose.yaml down --remove-orphans + docker compose -f test/docker-compose.yaml up --abort-on-container-exit migrate-blocktx migrate-metamorph + docker compose -f test/docker-compose.yaml up --build arc-blocktx arc-callbacker arc-metamorph arc + docker compose -f test/docker-compose.yaml down + .PHONY: run_e2e_tests run_e2e_tests: - docker-compose -f test/docker-compose.yml down - docker-compose -f test/docker-compose.yml up --abort-on-container-exit migrate-blocktx migrate-metamorph - docker-compose -f test/docker-compose.yml up --exit-code-from tests tests arc-blocktx arc-metamorph arc --scale arc-blocktx=7 --scale arc-metamorph=2 - docker-compose -f test/docker-compose.yml down + docker compose -f test/docker-compose.yaml down --remove-orphans + docker compose -f test/docker-compose.yaml up --abort-on-container-exit migrate-blocktx migrate-metamorph + docker compose -f test/docker-compose.yaml up --build --exit-code-from tests tests arc-blocktx arc-callbacker arc-metamorph arc --scale arc-blocktx=4 --scale arc-metamorph=2 + docker compose -f test/docker-compose.yaml down .PHONY: test test: @@ -48,7 +47,7 @@ install_lint: .PHONY: lint lint: - golangci-lint run --config=config/.golangci.yml -v ./... + golangci-lint run --config=config/.golangci.yaml -v ./... staticcheck ./... .PHONY: gen_go @@ -73,10 +72,19 @@ gen: --go-grpc_opt=paths=source_relative \ internal/blocktx/blocktx_api/blocktx_api.proto + protoc \ + --proto_path=. \ + --go_out=. \ + --go_opt=paths=source_relative \ + --go-grpc_out=. \ + --go-grpc_opt=paths=source_relative \ + internal/callbacker/callbacker_api/callbacker_api.proto + .PHONY: clean_gen clean_gen: - rm -f ./pkg/metamorph/metamorph_api/*.pb.go - rm -f ./pkg/blocktx/blocktx_api/*.pb.go + rm -f ./internal/metamorph/metamorph_api/*.pb.go + rm -f ./internal/blocktx/blocktx_api/*.pb.go + rm -f ./internal/callbacker/callbacker_api/*.pb.go .PHONY: coverage coverage: @@ -105,7 +113,7 @@ install_gen: go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2 go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.5.1 go install github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.3.0 - go install github.com/matryer/moq@v0.3.4 + go install github.com/matryer/moq@v0.4.0 .PHONY: docs docs: @@ -116,13 +124,10 @@ gh-pages: .PHONY: api api: - oapi-codegen -config pkg/api/config.yaml pkg/api/arc.yml > pkg/api/arc.go + oapi-codegen -config pkg/api/config.yaml pkg/api/arc.yaml > pkg/api/arc.go .PHONY: compare_config compare_config: rm -f ./config/dumped_config.yaml go run ./cmd/arc/main.go -dump_config "./config/dumped_config.yaml" && go run ./scripts/compare_yamls.go rm ./config/dumped_config.yaml - -.PHONY: clean_restart_e2e_test -clean_restart_e2e_test: clean_e2e_tests build_docker run_e2e_tests diff --git a/README.md b/README.md index 26e875fd6..354eda198 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,7 @@ ARC is a transaction processor for Bitcoin that keeps track of the life cycle of - [ZMQ](#zmq) - [BlockTx](#blocktx) - [BlockTx stores](#blocktx-stores) + - [Callbacker](#callbacker) - [Message Queue](#message-queue) - [K8s-Watcher](#k8s-watcher) - [Broadcaster-cli](#broadcaster-cli) @@ -92,6 +93,9 @@ where options are: -k8s-watcher= whether to start k8s-watcher (default=true) + -callbacker= + whether to start callbacker (default=true) + -config=/location directory to look for config.yaml (default='') @@ -111,9 +115,9 @@ Additionally, ARC relies on a message queue to communicate between Metamorph and docker run -p 4222:4222 nats ``` -The [docker-compose file](./deployments/docker-compose.yml) additionally shows how ARC can be run with the message queue and the Postgres database and db migrations. You can run ARC with all components with the following command +You can run ARC with all components using the [docker-compose.yaml](./test/docker-compose.yaml) file by using the following make command ``` -docker-compose -f deployments/docker-compose.yml up +make run ``` ### Docker @@ -166,21 +170,21 @@ go run main.go -metamorph=true Metamorph keeps track of the lifecycle of a transaction, and assigns it a status, which is returned in the `txStatus` field whenever the transaction is queried. The following statuses are available: -| Code | Status | Description | -|------|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0 | `UNKNOWN` | The transaction has been sent to metamorph, but no processing has taken place. This should never be the case, unless something goes wrong. | -| 10 | `QUEUED` | The transaction has been queued for processing. | -| 20 | `RECEIVED` | The transaction has been properly received by the metamorph processor. | -| 30 | `STORED` | The transaction has been stored in the metamorph store. This should ensure the transaction will be processed and retried if not picked up immediately by a mining node. | -| 40 | `ANNOUNCED_TO_NETWORK` | The transaction has been announced (INV message) to the Bitcoin network. | -| 50 | `REQUESTED_BY_NETWORK` | The transaction has been requested from metamorph by a Bitcoin node. | -| 60 | `SENT_TO_NETWORK` | The transaction has been sent to at least 1 Bitcoin node. | -| 70 | `ACCEPTED_BY_NETWORK` | The transaction has been accepted by a connected Bitcoin node on the ZMQ interface. If metamorph is not connected to ZMQ, this status will never by set. | -| 80 | `SEEN_IN_ORPHAN_MEMPOOL` | The transaction has been sent to at least 1 Bitcoin node but parent transaction was not found. | -| 90 | `SEEN_ON_NETWORK` | The transaction has been seen on the Bitcoin network and propagated to other nodes. This status is set when metamorph receives an INV message for the transaction from another node than it was sent to. | -| 100 | `DOUBLE_SPEND_ATTEMPTED` | The transaction is a double spend attempt. Competing transaction(s) will be returned with this status. | -| 110 | `REJECTED` | The transaction has been rejected by the Bitcoin network. | -| 120 | `MINED` | The transaction has been mined into a block by a mining node. | +| Status | Description | +|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `UNKNOWN` | The transaction has been sent to metamorph, but no processing has taken place. This should never be the case, unless something goes wrong. | +| `QUEUED` | The transaction has been queued for processing. | +| `RECEIVED` | The transaction has been properly received by the metamorph processor. | +| `STORED` | The transaction has been stored in the metamorph store. This should ensure the transaction will be processed and retried if not picked up immediately by a mining node. | +| `ANNOUNCED_TO_NETWORK` | The transaction has been announced (INV message) to the Bitcoin network. | +| `REQUESTED_BY_NETWORK` | The transaction has been requested from metamorph by a Bitcoin node. | +| `SENT_TO_NETWORK` | The transaction has been sent to at least 1 Bitcoin node. | +| `ACCEPTED_BY_NETWORK` | The transaction has been accepted by a connected Bitcoin node on the ZMQ interface. If metamorph is not connected to ZMQ, this status will never by set. | +| `SEEN_IN_ORPHAN_MEMPOOL` | The transaction has been sent to at least 1 Bitcoin node but parent transaction was not found. | +| `SEEN_ON_NETWORK` | The transaction has been seen on the Bitcoin network and propagated to other nodes. This status is set when metamorph receives an INV message for the transaction from another node than it was sent to. | +| `DOUBLE_SPEND_ATTEMPTED` | The transaction is a double spend attempt. Competing transaction(s) will be returned with this status. | +| `REJECTED` | The transaction has been rejected by the Bitcoin network. | +| `MINED` | The transaction has been mined into a block by a mining node. | The statuses have a difference between the codes in order to make it possible to add more statuses in between the existing ones without creating a breaking change. @@ -248,6 +252,17 @@ Metamorph publishes new transactions to the message queue and BlockTx subscribes ![Message Queue](./doc/message_queue.png) +### Callbacker + +Callbacker is a microservice that sends callbacks to a specified URL. + +Callbacker is designed to be horizontally scalable, with each instance operating independently. As a result, they do not communicate with each other and remain unaware of each other's existence. + +You can run callbacker like this: + +```shell +go run main.go -callbacker=true +``` ## K8s-Watcher @@ -283,7 +298,7 @@ The tests can be executed like this: make clean_restart_e2e_test ``` -The [docker-compose](./test/docker-compose.yml) file also shows the minimum setup that is needed for ARC to run. +The [docker-compose](./test/docker-compose.yaml) file also shows the minimum setup that is needed for ARC to run. ## Monitoring @@ -350,7 +365,7 @@ make gen ### Generate REST API -The rest api is defined in a [yaml file](./api/arc.yml) following the OpenAPI 3.0.0 specification. Before the rest API can be generated install the necessary tools by running +The rest api is defined in a [yaml file](./api/arc.yaml) following the OpenAPI 3.0.0 specification. Before the rest API can be generated install the necessary tools by running ``` make install_gen ``` diff --git a/ROADMAP.md b/ROADMAP.md index 17ab1d20b..9fe40ac9f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,17 +1,5 @@ # ROADMAP -## Idempotent transactions - -The capability of ARC to respond with the full block information including block hash, block height and Merkle path in case that a transaction is submitted to it which has been mined previously, but which was not submitted to that instance of ARC. - -## Double spending detection - -Introduction of a new status indicating that a transaction is in an DOUBLE_SPENT_ATTEMPTED state. All competing transactions progress in that status until they either get REJECTED or MINED. Extension of webhooks to notify parties if the transaction state changes due to double spend attempts. - -## Multiple different callbacks per transaction - -Submitting a transaction multiple times with different callback URL and/or callback token will at this pair as subscription to status updates. Callbacks will henceforth be sent to each callback url with specified callback token. - ## Update of transactions in case of block reorgs ARC updates the statuses of transactions in case of block reorgs. Transactions which are not in the block of the longest chain will be updated to `UNKNOWN` status and re-broadcasted. Transactions which are included in the block of the longest chain are updated to `MINED` status. diff --git a/cmd/arc/main.go b/cmd/arc/main.go index 7d23d4217..bb5ce1f99 100644 --- a/cmd/arc/main.go +++ b/cmd/arc/main.go @@ -33,7 +33,7 @@ func main() { } func run() error { - configDir, startApi, startMetamorph, startBlockTx, startK8sWatcher, dumpConfigFile := parseFlags() + configDir, startApi, startMetamorph, startBlockTx, startK8sWatcher, startCallbacker, dumpConfigFile := parseFlags() arcConfig, err := config.Load(configDir) if err != nil { @@ -63,9 +63,10 @@ func run() error { if arcConfig.Tracing != nil { cleanup, err := enableTracing(logger, arcConfig.Tracing.DialAddr) if err != nil { - return err + logger.Error("failed to enable tracing", slog.String("err", err.Error())) + } else { + shutdownFns = append(shutdownFns, cleanup) } - shutdownFns = append(shutdownFns, cleanup) } go func() { @@ -90,11 +91,12 @@ func run() error { } }() - if !isAnyFlagPassed("api", "blocktx", "metamorph", "k8s-watcher") { + if !isAnyFlagPassed("api", "blocktx", "metamorph", "k8s-watcher", "callbacker") { logger.Info("No service selected, starting all") startApi = true startMetamorph = true startBlockTx = true + startCallbacker = true } if startBlockTx { @@ -134,6 +136,14 @@ func run() error { shutdownFns = append(shutdownFns, func() { shutdown() }) } + if startCallbacker { + shutdown, err := cmd.StartCallbacker(logger, arcConfig) + if err != nil { + return fmt.Errorf("failed to start callbacker: %v", err) + } + shutdownFns = append(shutdownFns, shutdown) + } + // setup signal catching signalChan := make(chan os.Signal, 1) signal.Notify(signalChan, syscall.SIGTERM, syscall.SIGINT) @@ -151,14 +161,15 @@ func appCleanup(logger *slog.Logger, shutdownFns []func()) { } } -func parseFlags() (string, bool, bool, bool, bool, string) { +func parseFlags() (string, bool, bool, bool, bool, bool, string) { startApi := flag.Bool("api", false, "start ARC api server") startMetamorph := flag.Bool("metamorph", false, "start metamorph") startBlockTx := flag.Bool("blocktx", false, "start blocktx") startK8sWatcher := flag.Bool("k8s-watcher", false, "start k8s-watcher") + startCallbacker := flag.Bool("callbacker", false, "start callbacker") help := flag.Bool("help", false, "Show help") dumpConfigFile := flag.String("dump_config", "", "dump config to specified file and exit") - configDir := flag.String("config", "", "path to configuration yaml file") + configDir := flag.String("config", "", "path to configuration file") flag.Parse() @@ -178,8 +189,11 @@ func parseFlags() (string, bool, bool, bool, bool, string) { fmt.Println(" -k8s-watcher=") fmt.Println(" whether to start k8s-watcher (default=true)") fmt.Println("") + fmt.Println(" -callbacker=") + fmt.Println(" whether to start callbacker (default=true)") + fmt.Println("") fmt.Println(" -config=/location") - fmt.Println(" directory to look for config.yaml (default='')") + fmt.Println(" directory to look for config (default='')") fmt.Println("") fmt.Println(" -dump_config=/file.yaml") fmt.Println(" dump config to specified file and exit (default='config/dumped_config.yaml')") @@ -187,7 +201,7 @@ func parseFlags() (string, bool, bool, bool, bool, string) { os.Exit(0) } - return *configDir, *startApi, *startMetamorph, *startBlockTx, *startK8sWatcher, *dumpConfigFile + return *configDir, *startApi, *startMetamorph, *startBlockTx, *startK8sWatcher, *startCallbacker, *dumpConfigFile } func isAnyFlagPassed(flags ...string) bool { diff --git a/cmd/arc/services/api.go b/cmd/arc/services/api.go index 859124851..391454e49 100644 --- a/cmd/arc/services/api.go +++ b/cmd/arc/services/api.go @@ -3,9 +3,6 @@ package cmd import ( "context" "fmt" - "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_core" - "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_jetstream" - "github.com/bitcoin-sv/arc/internal/message_queue/nats/nats_connection" "log/slog" "net/http" "net/url" @@ -13,11 +10,15 @@ import ( "github.com/bitcoin-sv/arc/config" "github.com/bitcoin-sv/arc/internal/blocktx/blocktx_api" + "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_core" + "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_jetstream" + "github.com/bitcoin-sv/arc/internal/message_queue/nats/nats_connection" "github.com/bitcoin-sv/arc/internal/metamorph/metamorph_api" "github.com/bitcoin-sv/arc/pkg/api" "github.com/bitcoin-sv/arc/pkg/api/handler" "github.com/bitcoin-sv/arc/pkg/blocktx" "github.com/bitcoin-sv/arc/pkg/metamorph" + "github.com/labstack/echo/v4" echomiddleware "github.com/labstack/echo/v4/middleware" "github.com/ordishs/go-bitcoin" @@ -39,8 +40,27 @@ func StartAPIServer(logger *slog.Logger, arcConfig *config.ArcConfig) (func(), e AllowMethods: []string{http.MethodGet, http.MethodHead, http.MethodPut, http.MethodPatch, http.MethodPost, http.MethodDelete}, })) - // use the standard echo logger - e.Use(echomiddleware.Logger()) + e.Use(echomiddleware.RequestLoggerWithConfig(echomiddleware.RequestLoggerConfig{ + LogStatus: true, + LogURI: true, + LogError: true, + HandleError: true, // forwards error to the global error handler, so it can decide appropriate status code + LogValuesFunc: func(c echo.Context, v echomiddleware.RequestLoggerValues) error { + if v.Error == nil { + logger.LogAttrs(context.Background(), slog.LevelInfo, "REQUEST", + slog.String("uri", v.URI), + slog.Int("status", v.Status), + ) + } else { + logger.LogAttrs(context.Background(), slog.LevelError, "REQUEST_ERROR", + slog.String("uri", v.URI), + slog.Int("status", v.Status), + slog.String("err", v.Error.Error()), + ) + } + return nil + }, + })) // load the ARC handler from config // If you want to customize this for your own server, see examples dir diff --git a/cmd/arc/services/blocktx.go b/cmd/arc/services/blocktx.go index 9a151d64d..7195df3fe 100644 --- a/cmd/arc/services/blocktx.go +++ b/cmd/arc/services/blocktx.go @@ -2,13 +2,14 @@ package cmd import ( "fmt" - "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_core" - "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_jetstream" - "github.com/bitcoin-sv/arc/internal/message_queue/nats/nats_connection" "log/slog" "net" "time" + "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_core" + "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_jetstream" + "github.com/bitcoin-sv/arc/internal/message_queue/nats/nats_connection" + "github.com/bitcoin-sv/arc/config" "github.com/bitcoin-sv/arc/internal/blocktx" "github.com/bitcoin-sv/arc/internal/blocktx/store" @@ -21,7 +22,8 @@ import ( ) const ( - maximumBlockSize = 4294967296 // 4Gb + maximumBlockSize = 4294967296 // 4Gb + blockProcessingBuffer = 100 ) func StartBlockTx(logger *slog.Logger, arcConfig *config.ArcConfig) (func(), error) { @@ -64,7 +66,7 @@ func StartBlockTx(logger *slog.Logger, arcConfig *config.ArcConfig) (func(), err mqClient = nats_core.New(natsConnection, nats_core.WithLogger(logger)) } - peerHandlerOpts := []func(handler *blocktx.PeerHandler){ + processorOpts := []func(handler *blocktx.Processor){ blocktx.WithRetentionDays(btxConfig.RecordRetentionDays), blocktx.WithRegisterTxsChan(registerTxsChan), blocktx.WithRequestTxChan(requestTxChannel), @@ -73,15 +75,18 @@ func StartBlockTx(logger *slog.Logger, arcConfig *config.ArcConfig) (func(), err blocktx.WithFillGapsInterval(btxConfig.FillGapsInterval), } if tracingEnabled { - peerHandlerOpts = append(peerHandlerOpts, blocktx.WithTracer()) + processorOpts = append(processorOpts, blocktx.WithTracer()) } - peerHandler, err := blocktx.NewPeerHandler(logger, blockStore, peerHandlerOpts...) + blockRequestCh := make(chan blocktx.BlockRequest, blockProcessingBuffer) + blockProcessCh := make(chan *p2p.BlockMessage, blockProcessingBuffer) + + processor, err := blocktx.NewProcessor(logger, blockStore, blockRequestCh, blockProcessCh, processorOpts...) if err != nil { return nil, err } - err = peerHandler.Start() + err = processor.Start() if err != nil { return nil, fmt.Errorf("failed to start peer handler: %v", err) } @@ -104,6 +109,8 @@ func StartBlockTx(logger *slog.Logger, arcConfig *config.ArcConfig) (func(), err pm := p2p.NewPeerManager(logger, network, pmOpts...) peers := make([]p2p.PeerI, len(arcConfig.Peers)) + peerHandler := blocktx.NewPeerHandler(logger, blockRequestCh, blockProcessCh) + for i, peerSetting := range arcConfig.Peers { peerURL, err := peerSetting.GetP2PUrl() if err != nil { @@ -121,7 +128,7 @@ func StartBlockTx(logger *slog.Logger, arcConfig *config.ArcConfig) (func(), err peers[i] = peer } - peerHandler.StartFillGaps(peers) + processor.StartFillGaps(peers) server := blocktx.NewServer(blockStore, logger, pm, btxConfig.MaxAllowedBlockHeightMismatch) @@ -138,7 +145,7 @@ func StartBlockTx(logger *slog.Logger, arcConfig *config.ArcConfig) (func(), err return func() { logger.Info("Shutting down blocktx store") - peerHandler.Shutdown() + processor.Shutdown() server.Shutdown() diff --git a/cmd/arc/services/callbacker.go b/cmd/arc/services/callbacker.go new file mode 100644 index 000000000..99ac984c2 --- /dev/null +++ b/cmd/arc/services/callbacker.go @@ -0,0 +1,82 @@ +package cmd + +import ( + "fmt" + "log/slog" + "net" + "net/http" + "time" + + "github.com/bitcoin-sv/arc/config" + "github.com/bitcoin-sv/arc/internal/callbacker" + "google.golang.org/grpc" + "google.golang.org/grpc/health/grpc_health_v1" + "google.golang.org/grpc/reflection" +) + +func StartCallbacker(logger *slog.Logger, appConfig *config.ArcConfig) (func(), error) { + logger = logger.With(slog.String("service", "callbacker")) + logger.Info("Starting") + + config := appConfig.Callbacker + + callbackSender, err := callbacker.NewSender(&http.Client{Timeout: 5 * time.Second}, logger) + if err != nil { + return nil, fmt.Errorf("callbacker failed: %v", err) + } + + callbackDispatcher := callbacker.NewCallbackDispatcher(callbackSender, config.Pause) + + server := callbacker.NewServer(callbackDispatcher, callbacker.WithLogger(logger.With(slog.String("module", "server")))) + err = server.Serve(config.ListenAddr, appConfig.GrpcMessageSize, appConfig.PrometheusEndpoint) + if err != nil { + return nil, fmt.Errorf("GRPCServer failed: %v", err) + } + + healthServer, err := StartHealthServerCallbacker(server, config.Health, logger) + if err != nil { + return nil, fmt.Errorf("failed to start health server: %v", err) + } + + stopFn := func() { + logger.Info("Shutting down callbacker") + + // dispose of dependencies in the correct order: + // 1. server - ensure no new callbacks will be received + // 2. dispatcher - ensure all already accepted callbacks are proccessed + // 3. sender - finally, stop the sender as there are no callbacks left to send. + server.GracefulStop() + callbackDispatcher.GracefulStop() + callbackSender.GracefulStop() + + healthServer.Stop() + + logger.Info("Shutted down") + } + + logger.Info("Ready to work") + return stopFn, nil +} + +func StartHealthServerCallbacker(serv *callbacker.Server, healthConfig *config.HealthConfig, logger *slog.Logger) (*grpc.Server, error) { + gs := grpc.NewServer() + + grpc_health_v1.RegisterHealthServer(gs, serv) // registration + // register your own services + reflection.Register(gs) + + listener, err := net.Listen("tcp", healthConfig.SeverDialAddr) + if err != nil { + return nil, err + } + + go func() { + logger.Info("GRPC health server listening", slog.String("address", healthConfig.SeverDialAddr)) + err = gs.Serve(listener) + if err != nil { + logger.Error("GRPC health server failed to serve", slog.String("err", err.Error())) + } + }() + + return gs, nil +} diff --git a/cmd/arc/services/metamorph.go b/cmd/arc/services/metamorph.go index 51b51bcad..cc18be21f 100644 --- a/cmd/arc/services/metamorph.go +++ b/cmd/arc/services/metamorph.go @@ -3,17 +3,19 @@ package cmd import ( "context" "fmt" - "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_core" - "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_jetstream" - "github.com/bitcoin-sv/arc/internal/message_queue/nats/nats_connection" "log/slog" "net" - "net/http" "net/url" "os" "strconv" "time" + "github.com/bitcoin-sv/arc/internal/callbacker/callbacker_api" + "github.com/bitcoin-sv/arc/internal/grpc_opts" + "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_core" + "github.com/bitcoin-sv/arc/internal/message_queue/nats/client/nats_jetstream" + "github.com/bitcoin-sv/arc/internal/message_queue/nats/nats_connection" + "github.com/bitcoin-sv/arc/config" "github.com/bitcoin-sv/arc/internal/blocktx/blocktx_api" "github.com/bitcoin-sv/arc/internal/metamorph" @@ -75,16 +77,19 @@ func StartMetamorph(logger *slog.Logger, arcConfig *config.ArcConfig) (func(), e mqClient = nats_core.New(natsClient, nats_core.WithLogger(logger)) } - callbacker, err := metamorph.NewCallbacker(&http.Client{Timeout: 5 * time.Second}) + procLogger := logger.With(slog.String("module", "mtm-proc")) + + callbackerConn, err := initGrpcCallbackerConn(arcConfig.Callbacker.DialAddr, arcConfig.PrometheusEndpoint, arcConfig.GrpcMessageSize) if err != nil { - return nil, err + return nil, fmt.Errorf("failed to create callbacker client: %v", err) } + callbacker := metamorph.NewGrpcCallbacker(callbackerConn, procLogger) processorOpts := []metamorph.Option{ metamorph.WithCacheExpiryTime(mtmConfig.ProcessorCacheExpiryTime), metamorph.WithSeenOnNetworkTxTimeUntil(mtmConfig.CheckSeenOnNetworkOlderThan), metamorph.WithSeenOnNetworkTxTime(mtmConfig.CheckSeenOnNetworkPeriod), - metamorph.WithProcessorLogger(logger.With(slog.String("module", "mtm-proc"))), + metamorph.WithProcessorLogger(procLogger), metamorph.WithMessageQueueClient(mqClient), metamorph.WithMinedTxsChan(minedTxsChan), metamorph.WithSubmittedTxsChan(submittedTxsChan), @@ -249,7 +254,7 @@ func initPeerManager(logger *slog.Logger, s store.MetamorphStore, arcConfig *con logger.Info("Assuming bitcoin network", "network", network) - messageCh := make(chan *metamorph.PeerTxMessage, 100) + messageCh := make(chan *metamorph.PeerTxMessage, 10000) var pmOpts []p2p.PeerManagerOptions if arcConfig.Metamorph.MonitorPeers { pmOpts = append(pmOpts, p2p.WithRestartUnhealthyPeers()) @@ -286,3 +291,16 @@ func initPeerManager(logger *slog.Logger, s store.MetamorphStore, arcConfig *con return pm, peerHandler, messageCh, nil } + +func initGrpcCallbackerConn(address, prometheusEndpoint string, grpcMsgSize int) (callbacker_api.CallbackerAPIClient, error) { + dialOpts, err := grpc_opts.GetGRPCClientOpts(prometheusEndpoint, grpcMsgSize) + if err != nil { + return nil, err + } + callbackerConn, err := grpc.NewClient(address, dialOpts...) + if err != nil { + return nil, err + } + + return callbacker_api.NewCallbackerAPIClient(callbackerConn), nil +} diff --git a/cmd/broadcaster-cli/README.md b/cmd/broadcaster-cli/README.md index 8de94143d..3ab4b39f8 100644 --- a/cmd/broadcaster-cli/README.md +++ b/cmd/broadcaster-cli/README.md @@ -19,20 +19,41 @@ go install ./cmd/broadcaster-cli/ `broadcaster-cli` uses flags for adding context needed to run it. The flags and commands available can be shown by running `broadcaster-cli` with the flag `--help`. -As there can be a lot of flags you can also define them in a `yaml` file. The file [broadcaster-cli-example.yaml](./broadcaster-cli-example.yaml) is an example of the configuration. +As there can be a lot of flags you can also define them in a configuration file. The file [broadcaster-cli-example.yaml](./broadcaster-cli-example.yaml) is an example of the configuration. The format of the file can be the following: JSON, TOML, YAML, HCL, INI, envfile or Java properties formats. -If file `broadcaster-cli.yaml` is present in either the folder where `broadcaster-cli` is run or the folder `./cmd/broadcaster-cli/`, then these values will be used as flags (if available to the command). You can still provide the flags, in that case the value provided in the flag will override the value provided in `broadcaster-cli.yaml` +A specific config file can be selected using the `--config` flag. Example: +``` +broadcaster-cli keyset address -- --config ./cmd/broadcaster-cli/broadcaster-cli-example.yaml +``` +Note that the config has to be added as a subcommand with a double dash `--` as shown above. The path to the config file has to be separated by a space. + +If no config file is given using the `--config` flag, `broadcaster-cli` will search for `broadcaster-cli.yaml` in `.` and `./cmd/broadcaster-cli/` folders. + +If a config file was found, then these values will be used as flags (if available to the command). You can still provide the flags, in which case the value provided in the flag will override the value provided in `broadcaster-cli.yaml`. + +Note that a configuration file needs to be given at least for the private keys (see [broadcaster-cli-example.yaml](./broadcaster-cli-example.yaml)) as they cannot be passed as flags. ## How to use broadcaster-cli to send batches of transactions to ARC These instructions will provide the steps needed in order to use `broadcaster-cli` to send transactions to ARC. 1. Create a new key set by running `broadcaster-cli keyset new` - 1. The key set displayed has to be added under to config.yaml under `privateKeys` -2. Add funds to the funding address - 1. Show the funding address by running `broadcaster-cli keyset address` - 2. In case of `testnet` (using the `--testnet` flag) funds can be added using the WoC faucet. For that you can use the command `broadcaster-cli keyset topup --testnet` - 3. You can view the balance of the key set using the command `broadcaster-cli keyset balance` + 1. The key set displayed has to be added to the configuration file under `privateKeys` +2. Add funds to the funding addresses + 1. Show the funding addresses by running `broadcaster-cli keyset address` + 1. In case of `testnet` (using the `--testnet` flag) funds can be added using the WoC faucet. For that you can use the command `broadcaster-cli keyset topup --testnet` + 2. In case of `mainnet` funds could be added to one of the addresses. + 2. The funds can be spread to the other keys using the `utxos split` command. + 1. The following command will split funds of a given UTXO from `key-01` to keys: `key-01`, `key-02`, `key-03`, `key-04` + ``` + broadcaster-cli utxos split --txid=cf111f19bcfb6baab7fc200f0f8fb669dd6c66fd9de212becb0950c92a0b6c40 --satoshis=21953 --vout=0 --from=key-01 --keys=key-01,key-02,key-03,key-04 + ``` + 2. The same command can be used to move all funds from one UTXO from one to another key. The following example shows how to send all funds of the given UTXO from `key-01` to `key-02` + ``` + broadcaster-cli utxos split --txid=cf111f19bcfb6baab7fc200f0f8fb669dd6c66fd9de212becb0950c92a0b6c40 --satoshis=21953 --vout=0 --from=key-01 --keys=key-02 + ``` + 3. In order to just create print the transaction without submitting it, the `--dryrun` flag can be added + 3. You can view the balance of the key set using the command `broadcaster-cli keyset balance` 3. Create UTXO set 1. There must be a certain UTXO set available so that `broadcaster-cli` can broadcast a reasonable number of transactions in batches 2. First look at the existing UTXO set using `broadcaster-cli keyset utxos` diff --git a/cmd/broadcaster-cli/app/keyset/address/address.go b/cmd/broadcaster-cli/app/keyset/address/address.go index 29bb54500..8ad8f88ab 100644 --- a/cmd/broadcaster-cli/app/keyset/address/address.go +++ b/cmd/broadcaster-cli/app/keyset/address/address.go @@ -3,8 +3,9 @@ package address import ( "log/slog" - "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" "github.com/spf13/cobra" + + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" ) var ( @@ -19,14 +20,17 @@ var ( return err } - keySets, err := helper.GetKeySets() + keySetsMap, err := helper.GetSelectedKeySets() if err != nil { return err } - for _, keySet := range keySets { + names := helper.GetOrderedKeys(keySetsMap) + + for _, name := range names { + keySet := keySetsMap[name] - logger.Info("address", slog.String("address", keySet.Address(!isTestnet))) + logger.Info("address", slog.String("name", name), slog.String("address", keySet.Address(!isTestnet)), slog.String("key", keySet.GetMaster().String())) } return nil diff --git a/cmd/broadcaster-cli/app/keyset/balance/balance.go b/cmd/broadcaster-cli/app/keyset/balance/balance.go index 07a9be6ad..a3c06379c 100644 --- a/cmd/broadcaster-cli/app/keyset/balance/balance.go +++ b/cmd/broadcaster-cli/app/keyset/balance/balance.go @@ -3,12 +3,12 @@ package balance import ( "context" "log/slog" - "time" + "github.com/spf13/cobra" + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" "github.com/bitcoin-sv/arc/internal/woc_client" - "github.com/spf13/cobra" ) var Cmd = &cobra.Command{ @@ -27,23 +27,25 @@ var Cmd = &cobra.Command{ logger := helper.GetLogger() - wocClient := woc_client.New(woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) + wocClient := woc_client.New(!isTestnet, woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) - keySets, err := helper.GetKeySets() + keySetsMap, err := helper.GetSelectedKeySets() if err != nil { return err } - for _, keySet := range keySets { + names := helper.GetOrderedKeys(keySetsMap) + for _, name := range names { + keySet := keySetsMap[name] if wocApiKey == "" { time.Sleep(500 * time.Millisecond) } - confirmed, unconfirmed, err := wocClient.GetBalanceWithRetries(context.Background(), !isTestnet, keySet.Address(!isTestnet), 1*time.Second, 5) + confirmed, unconfirmed, err := wocClient.GetBalanceWithRetries(context.Background(), keySet.Address(!isTestnet), 1*time.Second, 5) if err != nil { return err } - logger.Info("balance", slog.String("address", keySet.Address(!isTestnet)), slog.Int64("confirmed", confirmed), slog.Int64("unconfirmed", unconfirmed)) + logger.Info("balance", slog.String("name", name), slog.String("address", keySet.Address(!isTestnet)), slog.Int64("confirmed", confirmed), slog.Int64("unconfirmed", unconfirmed)) } return nil diff --git a/cmd/broadcaster-cli/app/keyset/topup/topup.go b/cmd/broadcaster-cli/app/keyset/topup/topup.go index f57a81ddd..af2e41764 100644 --- a/cmd/broadcaster-cli/app/keyset/topup/topup.go +++ b/cmd/broadcaster-cli/app/keyset/topup/topup.go @@ -3,12 +3,12 @@ package topup import ( "context" "log/slog" - "time" + "github.com/spf13/cobra" + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" "github.com/bitcoin-sv/arc/internal/woc_client" - "github.com/spf13/cobra" ) var Cmd = &cobra.Command{ @@ -26,24 +26,23 @@ var Cmd = &cobra.Command{ logger := helper.GetLogger() - wocClient := woc_client.New(woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) + wocClient := woc_client.New(!isTestnet, woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) - keySets, err := helper.GetKeySets() + keySetsMap, err := helper.GetSelectedKeySets() if err != nil { return err } - for _, keySet := range keySets { - + for keyName, keySet := range keySetsMap { if wocApiKey == "" { time.Sleep(500 * time.Millisecond) } - err = wocClient.TopUp(context.Background(), !isTestnet, keySet.Address(!isTestnet)) + err = wocClient.TopUp(context.Background(), keySet.Address(!isTestnet)) if err != nil { return err } - logger.Info("top up complete", slog.String("address", keySet.Address(!isTestnet))) + logger.Info("top up complete", slog.String("address", keySet.Address(!isTestnet)), slog.String("name", keyName)) } return nil diff --git a/cmd/broadcaster-cli/app/keyset/utxos/table.go b/cmd/broadcaster-cli/app/keyset/utxos/table.go new file mode 100644 index 000000000..cbab5a0ca --- /dev/null +++ b/cmd/broadcaster-cli/app/keyset/utxos/table.go @@ -0,0 +1,145 @@ +package utxos + +import ( + "context" + "errors" + "log/slog" + "sort" + "strconv" + "time" + + "github.com/enescakir/emoji" + "github.com/jedib0t/go-pretty/v6/table" + + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" + "github.com/bitcoin-sv/arc/internal/broadcaster" + "github.com/bitcoin-sv/arc/pkg/keyset" +) + +func getUtxosTable(ctx context.Context, logger *slog.Logger, t table.Writer, keySets map[string]*keyset.KeySet, isTestnet bool, wocClient broadcaster.UtxoClient, maxRows int) table.Writer { + keyTotalOutputs := make([]int, len(keySets)) + keyHeaderRow := make([]interface{}, 0) + headerRow := make([]interface{}, 0) + type row struct { + satoshis string + outputs string + } + columns := make([][]row, len(keySets)) + maxRowNr := 0 + counter := 0 + names := helper.GetOrderedKeys(keySets) + + for _, name := range names { + + ks := keySets[name] + utxos, err := wocClient.GetUTXOsWithRetries(ctx, ks.Script, ks.Address(!isTestnet), 1*time.Second, 5) + if err != nil { + if errors.Is(err, context.Canceled) { + return t + } + logger.Error("failed to get utxos from WoC", slog.String("err", err.Error())) + continue + } + headerRow = append(headerRow, "Sat", "Outputs") + keyHeaderRow = append(keyHeaderRow, name, "") + + outputsMap := map[uint64]int{} + var satoshiSlice []uint64 + var found bool + for _, utxo := range utxos { + _, found = outputsMap[utxo.Satoshis] + if found { + outputsMap[utxo.Satoshis]++ + continue + } + + outputsMap[utxo.Satoshis] = 1 + + satoshiSlice = append(satoshiSlice, utxo.Satoshis) + } + + sort.Slice(satoshiSlice, func(i, j int) bool { + return satoshiSlice[j] < satoshiSlice[i] + }) + + totalOutputs := 0 + for _, satoshi := range satoshiSlice { + satString := strconv.FormatUint(satoshi, 10) + + if satoshi == 1 { + satString = satString + " " + emoji.CrossMark.String() + } + + columns[counter] = append(columns[counter], row{ + satoshis: satString, + outputs: strconv.Itoa(outputsMap[satoshi]), + }) + + // Do not count 1-sat outputs, as they can't be used as utxos for transactions + if satoshi == 1 { + continue + } + + totalOutputs += outputsMap[satoshi] + } + + keyTotalOutputs[counter] = totalOutputs + + if len(columns[counter]) > maxRowNr { + maxRowNr = len(columns[counter]) + } + + counter++ + } + + t.AppendHeader(keyHeaderRow) + t.AppendHeader(headerRow) + + rows := make([][]string, maxRowNr) + + for i := 0; i < maxRowNr; i++ { + for j := range columns { + + if len(columns[j]) < i+1 { + rows[i] = append(rows[i], "") + rows[i] = append(rows[i], "") + continue + } + + rows[i] = append(rows[i], columns[j][i].satoshis) + rows[i] = append(rows[i], columns[j][i].outputs) + } + } + + for i, row := range rows { + tableRow := table.Row{} + + if maxRows != 0 && i == maxRows+1 { + for range row { + tableRow = append(tableRow, "...") + } + + t.AppendRow(tableRow) + + continue + } + + if maxRows != 0 && i > maxRows { + continue + } + + for _, rowVal := range row { + tableRow = append(tableRow, rowVal) + } + + t.AppendRow(tableRow) + } + + totalRow := table.Row{} + for _, total := range keyTotalOutputs { + totalRow = append(totalRow, "Total", total) + } + t.AppendRow(totalRow) + + return t +} diff --git a/cmd/broadcaster-cli/app/keyset/utxos/utxos.go b/cmd/broadcaster-cli/app/keyset/utxos/utxos.go index 4cbfb0bcf..01f7bf71e 100644 --- a/cmd/broadcaster-cli/app/keyset/utxos/utxos.go +++ b/cmd/broadcaster-cli/app/keyset/utxos/utxos.go @@ -2,20 +2,18 @@ package utxos import ( "context" - "fmt" "log" + "os" + "os/signal" - "sort" - "strconv" - - "time" - - "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" - "github.com/bitcoin-sv/arc/internal/woc_client" "github.com/jedib0t/go-pretty/v6/table" "github.com/spf13/cobra" "github.com/spf13/viper" + + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" + "github.com/bitcoin-sv/arc/internal/woc_client" + "github.com/bitcoin-sv/arc/pkg/keyset" ) var Cmd = &cobra.Command{ @@ -34,124 +32,47 @@ var Cmd = &cobra.Command{ } logger := helper.GetLogger() - wocClient := woc_client.New(woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) + wocClient := woc_client.New(!isTestnet, woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) - keySets, err := helper.GetKeySets() + keySetsMap, err := helper.GetSelectedKeySets() if err != nil { return err } - t := table.NewWriter() - - type row struct { - satoshis string - outputs string - } - columns := make([][]row, len(keySets)) - maxRowNr := 0 - - keyTotalOutputs := make([]int, len(keySets)) - keyHeaderRow := make([]interface{}, 0) - headerRow := make([]interface{}, 0) - for i, keyset := range keySets { - - headerRow = append(headerRow, "Sat", "Outputs") - keyHeaderRow = append(keyHeaderRow, "key-"+strconv.Itoa(i), "") - - utxos, err := wocClient.GetUTXOsWithRetries(context.Background(), !isTestnet, keyset.Script, keyset.Address(!isTestnet), 1*time.Second, 5) - if err != nil { - return fmt.Errorf("failed to get utxos from WoC: %v", err) - } - - outputsMap := map[uint64]int{} - satoshiSlice := []uint64{} - var found bool - for _, utxo := range utxos { - _, found = outputsMap[utxo.Satoshis] - if found { - outputsMap[utxo.Satoshis]++ - continue - } - - outputsMap[utxo.Satoshis] = 1 - - satoshiSlice = append(satoshiSlice, utxo.Satoshis) - } - - sort.Slice(satoshiSlice, func(i, j int) bool { - return satoshiSlice[j] < satoshiSlice[i] - }) - - totalOutputs := 0 - - for _, satoshi := range satoshiSlice { - - columns[i] = append(columns[i], row{ - satoshis: strconv.FormatUint(satoshi, 10), - outputs: strconv.Itoa(outputsMap[satoshi]), - }) - - totalOutputs += outputsMap[satoshi] - } - keyTotalOutputs[i] = totalOutputs - - if len(columns[i]) > maxRowNr { - maxRowNr = len(columns[i]) - } - } - - t.AppendHeader(keyHeaderRow) - t.AppendHeader(headerRow) - - rows := make([][]string, maxRowNr) - - for i := 0; i < maxRowNr; i++ { - for j := range columns { - - if len(columns[j]) < i+1 { - rows[i] = append(rows[i], "") - rows[i] = append(rows[i], "") - continue - } - - rows[i] = append(rows[i], columns[j][i].satoshis) - rows[i] = append(rows[i], columns[j][i].outputs) - } - } - - for i, row := range rows { - tableRow := table.Row{} - - if maxRows != 0 && i == maxRows+1 { - for range row { - tableRow = append(tableRow, "...") - } - - t.AppendRow(tableRow) - - continue - } - - if maxRows != 0 && i > maxRows { + signalChan := make(chan os.Signal, 1) + signal.Notify(signalChan, os.Interrupt) // Listen for Ctrl+C + + ctx, cancel := context.WithCancel(context.Background()) + + go func() { + <-signalChan + cancel() + }() + + names := helper.GetOrderedKeys(keySetsMap) + counter := 0 + var t table.Writer + ksRow := map[string]*keyset.KeySet{} + for _, name := range names { + ksRow[name] = keySetsMap[name] + if counter >= 9 { + t = table.NewWriter() + t := getUtxosTable(ctx, logger, t, ksRow, isTestnet, wocClient, maxRows) + t.SetStyle(table.StyleColoredBright) + fmt.Println(t.Render()) + fmt.Println() + ksRow = map[string]*keyset.KeySet{} + counter = 0 continue } - - for _, rowVal := range row { - tableRow = append(tableRow, rowVal) - } - - t.AppendRow(tableRow) + counter++ } - - totalRow := table.Row{} - for _, total := range keyTotalOutputs { - totalRow = append(totalRow, "Total", total) + if len(ksRow) > 0 { + t = table.NewWriter() + t := getUtxosTable(ctx, logger, t, ksRow, isTestnet, wocClient, maxRows) + t.SetStyle(table.StyleColoredBright) + fmt.Println(t.Render()) } - t.AppendRow(totalRow) - - // Todo: Add row with total Satoshis - - fmt.Println(t.Render()) return nil }, diff --git a/cmd/broadcaster-cli/app/root.go b/cmd/broadcaster-cli/app/root.go index d1036a11d..65dc127e9 100644 --- a/cmd/broadcaster-cli/app/root.go +++ b/cmd/broadcaster-cli/app/root.go @@ -1,56 +1,75 @@ package app import ( - "errors" - "log" + "log/slog" + "os" "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/app/keyset" "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/app/utxos" + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" "github.com/spf13/cobra" "github.com/spf13/viper" - "golang.org/x/sys/unix" ) -var RootCmd = &cobra.Command{ - Use: "broadcaster", - Short: "CLI tool to broadcast transactions to ARC", -} +var ( + logger *slog.Logger + RootCmd = &cobra.Command{ + Use: "broadcaster", + Short: "CLI tool to broadcast transactions to ARC", + } +) func init() { + logger = helper.GetLogger() var err error RootCmd.PersistentFlags().Bool("testnet", false, "Use testnet") err = viper.BindPFlag("testnet", RootCmd.PersistentFlags().Lookup("testnet")) if err != nil { - log.Fatal(err) + logger.Error("failed to bind flag testnet", slog.String("err", err.Error())) + os.Exit(1) } RootCmd.PersistentFlags().StringSlice("keys", []string{}, "List of selected private keys") err = viper.BindPFlag("keys", RootCmd.PersistentFlags().Lookup("keys")) if err != nil { - log.Fatal(err) + logger.Error("failed to bind flag keys", slog.String("err", err.Error())) + os.Exit(1) + } RootCmd.PersistentFlags().String("wocAPIKey", "", "Optional WhatsOnChain API key for allowing for higher request rates") err = viper.BindPFlag("wocAPIKey", RootCmd.PersistentFlags().Lookup("wocAPIKey")) if err != nil { - log.Fatal(err) + logger.Error("failed to bind flag wocAPIKey", slog.String("err", err.Error())) + os.Exit(1) + } - viper.AddConfigPath(".") - viper.AddConfigPath("./cmd/broadcaster-cli/") - viper.SetConfigName("broadcaster-cli") - // Todo: Allow setting alternative config file name with flag: e.g. --config=brc-config.yaml + var configFilenameArg string + args := os.Args + for i, arg := range args { + if arg == "-c" || arg == "--config" { + configFilenameArg = args[i+1] + break + } + } + + if configFilenameArg != "" { + viper.SetConfigFile(configFilenameArg) + } else { + viper.AddConfigPath(".") + viper.AddConfigPath("./cmd/broadcaster-cli/") + viper.SetConfigName("broadcaster-cli") + } err = viper.ReadInConfig() if err != nil { - log.Fatalf("failed to read config file: %v", err) + logger.Error("failed to read config file", slog.String("err", err.Error())) + os.Exit(1) } - var viperErr viper.ConfigFileNotFoundError - isConfigFileNotFoundErr := errors.As(err, &viperErr) - - if err != nil && !errors.Is(err, unix.ENOENT) && !isConfigFileNotFoundErr { - log.Fatal(err) + if viper.ConfigFileUsed() != "" { + logger.Info("Config file used", slog.String("filename", viper.ConfigFileUsed())) } RootCmd.AddCommand(keyset.Cmd) diff --git a/cmd/broadcaster-cli/app/utxos/broadcast/broadcast.go b/cmd/broadcaster-cli/app/utxos/broadcast/broadcast.go index 01976cd3b..2e42b1669 100644 --- a/cmd/broadcaster-cli/app/utxos/broadcast/broadcast.go +++ b/cmd/broadcaster-cli/app/utxos/broadcast/broadcast.go @@ -8,12 +8,13 @@ import ( "os" "os/signal" + "github.com/spf13/cobra" + "github.com/spf13/viper" + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" "github.com/bitcoin-sv/arc/internal/broadcaster" "github.com/bitcoin-sv/arc/internal/metamorph/metamorph_api" "github.com/bitcoin-sv/arc/internal/woc_client" - "github.com/spf13/cobra" - "github.com/spf13/viper" ) var Cmd = &cobra.Command{ @@ -72,7 +73,7 @@ var Cmd = &cobra.Command{ return err } - keySets, err := helper.GetKeySets() + keySetsMap, err := helper.GetSelectedKeySets() if err != nil { return err } @@ -108,7 +109,7 @@ var Cmd = &cobra.Command{ return fmt.Errorf("failed to create client: %v", err) } - wocClient := woc_client.New(woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) + wocClient := woc_client.New(!isTestnet, woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) opts := []func(p *broadcaster.Broadcaster){ broadcaster.WithFees(miningFeeSat), @@ -121,18 +122,25 @@ var Cmd = &cobra.Command{ opts = append(opts, broadcaster.WithWaitForStatus(metamorph_api.Status(waitForStatus))) } - rateBroadcaster, err := broadcaster.NewMultiKeyRateBroadcaster(logger, client, keySets, wocClient, isTestnet, opts...) - if err != nil { - return fmt.Errorf("failed to create rate broadcaster: %v", err) + rbs := make([]broadcaster.RateBroadcaster, 0, len(keySetsMap)) + for keyName, ks := range keySetsMap { + rb, err := broadcaster.NewRateBroadcaster(logger.With(slog.String("address", ks.Address(!isTestnet)), slog.String("name", keyName)), client, ks, wocClient, isTestnet, rateTxsPerSecond, limit, opts...) + if err != nil { + return err + } + + rbs = append(rbs, rb) } + rateBroadcaster := broadcaster.NewMultiKeyRateBroadcaster(logger, rbs) + doneChan := make(chan error) // Channel to signal the completion of Start signalChan := make(chan os.Signal, 1) signal.Notify(signalChan, os.Interrupt) // Listen for Ctrl+C go func() { // Start the broadcasting process - err := rateBroadcaster.Start(rateTxsPerSecond, limit) + err := rateBroadcaster.Start() logger.Info("Starting broadcaster", slog.Int("rate [txs/s]", rateTxsPerSecond), slog.Int("batch size", batchSize)) doneChan <- err // Send the completion or error signal }() @@ -140,26 +148,34 @@ var Cmd = &cobra.Command{ select { case <-signalChan: // If an interrupt signal is received - fmt.Println("Shutdown signal received. Shutting down the rate broadcaster.") + logger.Info("Shutdown signal received. Shutting down the rate broadcaster.") case err := <-doneChan: - // Or wait for the normal completion if err != nil { - fmt.Printf("Error during broadcasting: %v\n", err) - } else { - fmt.Println("Broadcasting completed successfully.") + logger.Error("Error during broadcasting", slog.String("err", err.Error())) } } // Shutdown the broadcaster in all cases rateBroadcaster.Shutdown() - fmt.Println("Broadcaster shutdown complete.") + logger.Info("Broadcasting shutdown complete") return nil }, } func init() { - var err error + logger := helper.GetLogger() + Cmd.SetHelpFunc(func(command *cobra.Command, strings []string) { + // Hide unused persistent flags + err := command.Flags().MarkHidden("satoshis") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + + // Call parent help func + command.Parent().HelpFunc()(command, strings) + }) + var err error Cmd.Flags().Int("rate", 10, "Transactions per second to be rate broad casted per key set") err = viper.BindPFlag("rate", Cmd.Flags().Lookup("rate")) if err != nil { @@ -183,4 +199,5 @@ func init() { if err != nil { log.Fatal(err) } + } diff --git a/cmd/broadcaster-cli/app/utxos/consolidate/consolidate.go b/cmd/broadcaster-cli/app/utxos/consolidate/consolidate.go index e63fe8e56..a6556a22b 100644 --- a/cmd/broadcaster-cli/app/utxos/consolidate/consolidate.go +++ b/cmd/broadcaster-cli/app/utxos/consolidate/consolidate.go @@ -3,43 +3,32 @@ package consolidate import ( "errors" "fmt" + "log/slog" + "os" + "os/signal" + + "github.com/spf13/cobra" "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" "github.com/bitcoin-sv/arc/internal/broadcaster" "github.com/bitcoin-sv/arc/internal/woc_client" - "github.com/spf13/cobra" ) var Cmd = &cobra.Command{ Use: "consolidate", Short: "Consolidate UTXO set to 1 output", RunE: func(cmd *cobra.Command, args []string) error { - fullStatusUpdates, err := helper.GetBool("fullStatusUpdates") - if err != nil { - return err - } - isTestnet, err := helper.GetBool("testnet") if err != nil { return err } - callbackURL, err := helper.GetString("callback") - if err != nil { - return err - } - - callbackToken, err := helper.GetString("callbackToken") - if err != nil { - return err - } - authorization, err := helper.GetString("authorization") if err != nil { return err } - keySets, err := helper.GetKeySets() + keySetsMap, err := helper.GetSelectedKeySets() if err != nil { return err } @@ -69,26 +58,59 @@ var Cmd = &cobra.Command{ if err != nil { return fmt.Errorf("failed to create client: %v", err) } - if err != nil { - return fmt.Errorf("failed to create client: %v", err) - } - wocClient := woc_client.New(woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) + names := helper.GetOrderedKeys(keySetsMap) - rateBroadcaster, err := broadcaster.NewUTXOConsolidator(logger, client, keySets, wocClient, isTestnet, - broadcaster.WithFees(miningFeeSat), - broadcaster.WithCallback(callbackURL, callbackToken), - broadcaster.WithFullstatusUpdates(fullStatusUpdates), - ) - if err != nil { - return fmt.Errorf("failed to create broadcaster: %v", err) - } + wocClient := woc_client.New(!isTestnet, woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) + cs := make([]broadcaster.Consolidator, 0, len(keySetsMap)) + for _, keyName := range names { + ks := keySetsMap[keyName] + c, err := broadcaster.NewUTXOConsolidator(logger.With(slog.String("address", ks.Address(!isTestnet)), slog.String("name", keyName)), client, ks, wocClient, isTestnet, broadcaster.WithFees(miningFeeSat)) + if err != nil { + return err + } - err = rateBroadcaster.Consolidate() - if err != nil { - return fmt.Errorf("failed to consolidate utxos: %v", err) + cs = append(cs, c) } + consolidator := broadcaster.NewMultiKeyUtxoConsolidator(logger, cs) + + signalChan := make(chan os.Signal, 1) + signal.Notify(signalChan, os.Interrupt) // Listen for Ctrl+C + + go func() { + <-signalChan + consolidator.Shutdown() + }() + + logger.Info("Starting consolidator") + consolidator.Start() return nil }, } + +func init() { + logger := helper.GetLogger() + + Cmd.SetHelpFunc(func(command *cobra.Command, strings []string) { + // Hide unused persistent flags + err := command.Flags().MarkHidden("fullStatusUpdates") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + err = command.Flags().MarkHidden("callback") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + err = command.Flags().MarkHidden("callbackToken") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + err = command.Flags().MarkHidden("satoshis") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + // Call parent help func + command.Parent().HelpFunc()(command, strings) + }) +} diff --git a/cmd/broadcaster-cli/app/utxos/create/create.go b/cmd/broadcaster-cli/app/utxos/create/create.go index 108d2102e..1c870f8fb 100644 --- a/cmd/broadcaster-cli/app/utxos/create/create.go +++ b/cmd/broadcaster-cli/app/utxos/create/create.go @@ -4,12 +4,16 @@ import ( "errors" "fmt" "log" + "log/slog" + + "github.com/bitcoin-sv/arc/pkg/keyset" + + "github.com/spf13/cobra" + "github.com/spf13/viper" "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" "github.com/bitcoin-sv/arc/internal/broadcaster" "github.com/bitcoin-sv/arc/internal/woc_client" - "github.com/spf13/cobra" - "github.com/spf13/viper" ) var Cmd = &cobra.Command{ @@ -32,32 +36,17 @@ var Cmd = &cobra.Command{ return errors.New("satoshis must be a value greater than 0") } - fullStatusUpdates, err := helper.GetBool("fullStatusUpdates") - if err != nil { - return err - } - isTestnet, err := helper.GetBool("testnet") if err != nil { return err } - callbackURL, err := helper.GetString("callback") - if err != nil { - return err - } - - callbackToken, err := helper.GetString("callbackToken") - if err != nil { - return err - } - authorization, err := helper.GetString("authorization") if err != nil { return err } - keySets, err := helper.GetKeySets() + keySetsMap, err := helper.GetSelectedKeySets() if err != nil { return err } @@ -88,22 +77,23 @@ var Cmd = &cobra.Command{ if err != nil { return fmt.Errorf("failed to create client: %v", err) } - if err != nil { - return fmt.Errorf("failed to create client: %v", err) - } - wocClient := woc_client.New(woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) + wocClient := woc_client.New(!isTestnet, woc_client.WithAuth(wocApiKey), woc_client.WithLogger(logger)) - rateBroadcaster, err := broadcaster.NewUTXOCreator(logger, client, keySets, wocClient, isTestnet, + ks := make([]*keyset.KeySet, len(keySetsMap)) + counter := 0 + for _, keySet := range keySetsMap { + ks[counter] = keySet + counter++ + } + rateBroadcaster, err := broadcaster.NewUTXOCreator(logger, client, ks, wocClient, isTestnet, broadcaster.WithFees(miningFeeSat), - broadcaster.WithCallback(callbackURL, callbackToken), - broadcaster.WithFullstatusUpdates(fullStatusUpdates), ) if err != nil { return fmt.Errorf("failed to create broadcaster: %v", err) } - err = rateBroadcaster.CreateUtxos(outputs, satoshisPerOutput) + err = rateBroadcaster.CreateUtxos(outputs, uint64(satoshisPerOutput)) if err != nil { return fmt.Errorf("failed to create utxos: %v", err) } @@ -113,6 +103,27 @@ var Cmd = &cobra.Command{ } func init() { + + logger := helper.GetLogger() + Cmd.SetHelpFunc(func(command *cobra.Command, strings []string) { + // Hide unused persistent flags + err := command.Flags().MarkHidden("fullStatusUpdates") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + err = command.Flags().MarkHidden("callback") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + err = command.Flags().MarkHidden("callbackToken") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + + // Call parent help func + command.Parent().HelpFunc()(command, strings) + }) + var err error Cmd.Flags().Int("outputs", 0, "Nr of requested outputs") @@ -120,10 +131,4 @@ func init() { if err != nil { log.Fatal(err) } - - Cmd.Flags().Int("satoshis", 0, "Nr of satoshis per output outputs") - err = viper.BindPFlag("satoshis", Cmd.Flags().Lookup("satoshis")) - if err != nil { - log.Fatal(err) - } } diff --git a/cmd/broadcaster-cli/app/utxos/split/split.go b/cmd/broadcaster-cli/app/utxos/split/split.go new file mode 100644 index 000000000..426b820ec --- /dev/null +++ b/cmd/broadcaster-cli/app/utxos/split/split.go @@ -0,0 +1,170 @@ +package split + +import ( + "errors" + "fmt" + "log" + "log/slog" + + "github.com/spf13/cobra" + "github.com/spf13/viper" + + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/helper" + "github.com/bitcoin-sv/arc/internal/broadcaster" + "github.com/bitcoin-sv/arc/pkg/keyset" +) + +var Cmd = &cobra.Command{ + Use: "split", + Short: "Split a UTXO", + RunE: func(cmd *cobra.Command, args []string) error { + + txid, err := helper.GetString("txid") + if err != nil { + return err + } + if txid == "" { + return errors.New("txid is required") + } + + from, err := helper.GetString("from") + if err != nil { + return err + } + if from == "" { + return errors.New("from is required") + } + + satoshis, err := helper.GetUint64("satoshis") + if err != nil { + return err + } + if satoshis == 0 { + return errors.New("satoshis is required") + } + + vout, err := helper.GetUint32("vout") + if err != nil { + return err + } + + dryrun, err := helper.GetBool("dryrun") + if err != nil { + return err + } + + isTestnet, err := helper.GetBool("testnet") + if err != nil { + return err + } + + authorization, err := helper.GetString("authorization") + if err != nil { + return err + } + + miningFeeSat, err := helper.GetInt("miningFeeSatPerKb") + if err != nil { + return err + } + + arcServer, err := helper.GetString("apiURL") + if err != nil { + return err + } + if arcServer == "" { + return errors.New("no api URL was given") + } + + allKeysMap, err := helper.GetAllKeySets() + if err != nil { + return err + } + + keySetsMap, err := helper.GetSelectedKeySets() + if err != nil { + return err + } + + logger := helper.GetLogger() + + client, err := helper.CreateClient(&broadcaster.Auth{ + Authorization: authorization, + }, arcServer) + if err != nil { + return fmt.Errorf("failed to create client: %v", err) + } + + ks := make([]*keyset.KeySet, len(keySetsMap)) + counter := 0 + for _, keySet := range keySetsMap { + ks[counter] = keySet + counter++ + } + fromKs, ok := allKeysMap[from] + if !ok { + return fmt.Errorf("from not found in keySetsMap: %v", from) + } + + splitter, err := broadcaster.NewUTXOSplitter(logger, client, fromKs, ks, isTestnet, + broadcaster.WithFees(miningFeeSat), + ) + if err != nil { + return fmt.Errorf("failed to create broadcaster: %v", err) + } + + err = splitter.SplitUtxo(txid, satoshis, vout, dryrun) + if err != nil { + return fmt.Errorf("failed to create utxos: %v", err) + } + + return nil + }, +} + +func init() { + + logger := helper.GetLogger() + Cmd.SetHelpFunc(func(command *cobra.Command, strings []string) { + // Hide unused persistent flags + err := command.Flags().MarkHidden("fullStatusUpdates") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + err = command.Flags().MarkHidden("callback") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + err = command.Flags().MarkHidden("callbackToken") + if err != nil { + logger.Error("failed to mark flag hidden", slog.String("err", err.Error())) + } + // Call parent help func + command.Parent().HelpFunc()(command, strings) + }) + + var err error + Cmd.Flags().String("from", "", "Key from which to split") + err = viper.BindPFlag("from", Cmd.Flags().Lookup("from")) + if err != nil { + log.Fatal(err) + } + + Cmd.Flags().String("txid", "", "TX ID of UTXO to split") + err = viper.BindPFlag("txid", Cmd.Flags().Lookup("txid")) + if err != nil { + log.Fatal(err) + } + + Cmd.Flags().Uint32("vout", 0, "UTXO position") + err = viper.BindPFlag("vout", Cmd.Flags().Lookup("vout")) + if err != nil { + log.Fatal(err) + } + + Cmd.Flags().Bool("dryrun", false, "Whether or not to submit the splitting tx") + err = viper.BindPFlag("dryrun", Cmd.Flags().Lookup("dryrun")) + if err != nil { + log.Fatal(err) + } +} diff --git a/cmd/broadcaster-cli/app/utxos/utxos.go b/cmd/broadcaster-cli/app/utxos/utxos.go index b61605069..d36f09d1e 100644 --- a/cmd/broadcaster-cli/app/utxos/utxos.go +++ b/cmd/broadcaster-cli/app/utxos/utxos.go @@ -4,12 +4,14 @@ import ( "fmt" "log" + "github.com/spf13/cobra" + "github.com/spf13/viper" + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/app/utxos/broadcast" "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/app/utxos/consolidate" "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/app/utxos/create" + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/app/utxos/split" "github.com/bitcoin-sv/arc/internal/metamorph/metamorph_api" - "github.com/spf13/cobra" - "github.com/spf13/viper" ) var Cmd = &cobra.Command{ @@ -56,7 +58,14 @@ func init() { log.Fatal(err) } + Cmd.PersistentFlags().Int("satoshis", 0, "Nr of satoshis per output outputs") + err = viper.BindPFlag("satoshis", Cmd.PersistentFlags().Lookup("satoshis")) + if err != nil { + log.Fatal(err) + } + Cmd.AddCommand(create.Cmd) Cmd.AddCommand(broadcast.Cmd) Cmd.AddCommand(consolidate.Cmd) + Cmd.AddCommand(split.Cmd) } diff --git a/cmd/broadcaster-cli/helper/functions.go b/cmd/broadcaster-cli/helper/functions.go index 563b69381..350bb3af3 100644 --- a/cmd/broadcaster-cli/helper/functions.go +++ b/cmd/broadcaster-cli/helper/functions.go @@ -1,17 +1,20 @@ package helper import ( + "errors" "fmt" "log/slog" "os" + "sort" "strconv" "strings" "github.com/lmittmann/tint" + "github.com/spf13/viper" + "github.com/bitcoin-sv/arc/internal/broadcaster" "github.com/bitcoin-sv/arc/pkg/keyset" - "github.com/spf13/viper" ) func CreateClient(auth *broadcaster.Auth, arcServer string) (broadcaster.ArcClient, error) { @@ -76,6 +79,16 @@ func GetUint64(settingName string) (uint64, error) { return getSettingFromEnvFile[uint64](settingName) } +func GetUint32(settingName string) (uint32, error) { + + setting := viper.GetUint32(settingName) + if setting != 0 { + return setting, nil + } + + return getSettingFromEnvFile[uint32](settingName) +} + func GetInt64(settingName string) (int64, error) { setting := viper.GetInt64(settingName) @@ -136,11 +149,11 @@ func GetLogger() *slog.Logger { return slog.New(tint.NewHandler(os.Stdout, &tint.Options{Level: slog.LevelDebug})) } -func GetKeySetsFor(keys map[string]string, selectedKeys []string) ([]*keyset.KeySet, error) { - var keySets []*keyset.KeySet +func GetKeySetsFor(keys map[string]string, selectedKeys []string) (map[string]*keyset.KeySet, error) { + keySets := map[string]*keyset.KeySet{} if len(keys) == 0 { - return nil, fmt.Errorf("no keys given in configuration") + return nil, errors.New("no keys given in configuration") } if len(selectedKeys) > 0 { @@ -154,7 +167,7 @@ func GetKeySetsFor(keys map[string]string, selectedKeys []string) ([]*keyset.Key if err != nil { return nil, fmt.Errorf("failed to get selected key set %s: %v", selectedKey, err) } - keySets = append(keySets, fundingKeySet) + keySets[selectedKey] = fundingKeySet } return keySets, nil } @@ -164,7 +177,7 @@ func GetKeySetsFor(keys map[string]string, selectedKeys []string) ([]*keyset.Key if err != nil { return nil, fmt.Errorf("failed to get key set with name %s and value %s: %v", name, key, err) } - keySets = append(keySets, fundingKeySet) + keySets[name] = fundingKeySet } return keySets, nil } @@ -187,7 +200,7 @@ func GetSelectedKeys() ([]string, error) { return keys, nil } -func GetKeySets() ([]*keyset.KeySet, error) { +func GetSelectedKeySets() (map[string]*keyset.KeySet, error) { selectedKeys, err := GetSelectedKeys() if err != nil { return nil, fmt.Errorf("failed to get selected keys: %v", err) @@ -199,8 +212,42 @@ func GetKeySets() ([]*keyset.KeySet, error) { } if len(keys) == 0 { - return nil, fmt.Errorf("no keys given in configuration") + return nil, errors.New("no keys given in configuration") } return GetKeySetsFor(keys, selectedKeys) } + +func GetAllKeySets() (map[string]*keyset.KeySet, error) { + keys, err := GetPrivateKeys() + if err != nil { + return nil, fmt.Errorf("failed to get private keys: %v", err) + } + + if len(keys) == 0 { + return nil, errors.New("no keys given in configuration") + } + keySets := map[string]*keyset.KeySet{} + + for name, key := range keys { + fundingKeySet, _, err := GetKeySetsXpriv(key) + if err != nil { + return nil, fmt.Errorf("failed to get key set with name %s and value %s: %v", name, key, err) + } + keySets[name] = fundingKeySet + } + + return keySets, nil +} + +func GetOrderedKeys[T any](keysMap map[string]T) []string { + + var keys []string + + for key := range keysMap { + keys = append(keys, key) + } + + sort.Strings(keys) + return keys +} diff --git a/cmd/broadcaster-cli/main.go b/cmd/broadcaster-cli/main.go index fb6310b6e..9e353e612 100644 --- a/cmd/broadcaster-cli/main.go +++ b/cmd/broadcaster-cli/main.go @@ -1,9 +1,10 @@ package main import ( - "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/app" "log" "os" + + "github.com/bitcoin-sv/arc/cmd/broadcaster-cli/app" ) func main() { diff --git a/config/.golangci.yml b/config/.golangci.yaml similarity index 100% rename from config/.golangci.yml rename to config/.golangci.yaml diff --git a/config/config.go b/config/config.go index 1e7097a3f..0130c414b 100644 --- a/config/config.go +++ b/config/config.go @@ -22,6 +22,7 @@ type ArcConfig struct { Blocktx *BlocktxConfig `mapstructure:"blocktx"` Api *ApiConfig `mapstructure:"api"` K8sWatcher *K8sWatcherConfig `mapstructure:"k8sWatcher"` + Callbacker *CallbackerConfig `mapstructure:"callbacker"` } type MessageQueueConfig struct { @@ -113,9 +114,17 @@ type StatsConfig struct { type ApiConfig struct { Address string `mapstructure:"address"` WocApiKey string `mapstructure:"wocApiKey"` + WocMainnet bool `mapstructure:"wocMainnet"` DefaultPolicy *bitcoin.Settings `mapstructure:"defaultPolicy"` } type K8sWatcherConfig struct { Namespace string `mapstructure:"namespace"` } + +type CallbackerConfig struct { + ListenAddr string `mapstructure:"listenAddr"` + DialAddr string `mapstructure:"dialAddr"` + Health *HealthConfig `mapstructure:"health"` + Pause time.Duration `mapstructure:"pause"` +} diff --git a/config/defaults.go b/config/defaults.go index 39709c92f..68ba52454 100644 --- a/config/defaults.go +++ b/config/defaults.go @@ -23,6 +23,7 @@ func getDefaultArcConfig() *ArcConfig { Blocktx: getBlocktxConfig(), Api: getApiConfig(), K8sWatcher: nil, // optional + Callbacker: getCallbackerConfig(), } } @@ -161,3 +162,14 @@ func getDbConfig(dbName string) *DbConfig { }, } } + +func getCallbackerConfig() *CallbackerConfig { + return &CallbackerConfig{ + ListenAddr: "localhost:8021", + DialAddr: "localhost:8021", + Health: &HealthConfig{ + SeverDialAddr: "localhost:8025", + }, + Pause: 0, + } +} diff --git a/config/example_config.yaml b/config/example_config.yaml index a56cbce26..00d05734e 100644 --- a/config/example_config.yaml +++ b/config/example_config.yaml @@ -85,6 +85,7 @@ blocktx: api: address: localhost:9090 # address to start api server on wocApiKey: "mainnet_XXXXXXXXXXXXXXXXXXXX" # api key for www.whatsonchain.com + wocMainnet: false # query main or test net on www.whatsonchain.com defaultPolicy: # default policy of bitcoin node excessiveblocksize: 2000000000 blockmaxsize: 512000000 @@ -118,3 +119,10 @@ api: k8sWatcher: # (optional, used only when deploying arc to k8s) namespace: arc-testnet + +callbacker: + listenAddr: localhost:8021 # address space for callbacker to listen on. Can be for example localhost:8021 or :8021 for listening on all addresses + dialAddr: localhost:8021 # address for other services to dial callbacker service + health: + serverDialAddr: localhost:8025 # address at which the grpc health server is exposed + pause: 0s # pause between sending next callback to the same receiver \ No newline at end of file diff --git a/deployments/config/bitcoin.conf b/deployments/config/bitcoin.conf deleted file mode 120000 index 542b3b3e8..000000000 --- a/deployments/config/bitcoin.conf +++ /dev/null @@ -1 +0,0 @@ -test/config/bitcoin.conf \ No newline at end of file diff --git a/deployments/config/config.yaml b/deployments/config/config.yaml deleted file mode 120000 index 562ef2641..000000000 --- a/deployments/config/config.yaml +++ /dev/null @@ -1 +0,0 @@ -test/config/config.yaml \ No newline at end of file diff --git a/deployments/docker-compose.yml b/deployments/docker-compose.yml deleted file mode 100644 index 71b624430..000000000 --- a/deployments/docker-compose.yml +++ /dev/null @@ -1,175 +0,0 @@ -version: '3' -services: - node1: - container_name: node1 - image: bitcoinsv/bitcoin-sv:1.1.0 - ports: - - "18332:18332" - expose: - - "18332" - - "18333" - - "28332" - healthcheck: - test: [ "CMD", "/entrypoint.sh", "bitcoin-cli", "getinfo" ] - volumes: - - ./config/bitcoin.conf:/data/bitcoin.conf - - node1-data:/data - command: [ "/entrypoint.sh", "bitcoind", "-connect=node2:18333", "-connect=node3:18333" ] - - node2: - container_name: node2 - image: bitcoinsv/bitcoin-sv:1.1.0 - ports: - - "48332:18332" - expose: - - "18332" - - "18333" - healthcheck: - test: [ "CMD", "/entrypoint.sh", "bitcoin-cli", "getinfo" ] - volumes: - - ./config/bitcoin.conf:/data/bitcoin.conf - - node2-data:/data - command: [ "/entrypoint.sh", "bitcoind", "-connect=node1:18333", "-connect=node3:18333" ] - - node3: - container_name: node3 - image: bitcoinsv/bitcoin-sv:1.1.0 - ports: - - "58332:18332" - expose: - - "18332" - - "18333" - healthcheck: - test: [ "CMD", "/entrypoint.sh", "bitcoin-cli", "getinfo" ] - volumes: - - ./config/bitcoin.conf:/data/bitcoin.conf - - node3-data:/data - command: [ "/entrypoint.sh", "bitcoind", "-connect=node1:18333", "-connect=node2:18333" ] - - db: - image: postgres:15.4 - restart: always - environment: - - POSTGRES_USER=arcuser - - POSTGRES_PASSWORD=arcpass - - POSTGRES_DB=blocktx - healthcheck: - test: [ "CMD-SHELL", "pg_isready", "-d blocktx", "-U arcuser" ] - interval: 5s - timeout: 5s - retries: 5 - ports: - - '5432:5432' - expose: - - "5432" - - migrate-blocktx: - container_name: migrate-blocktx - image: migrate/migrate:v4.16.2 - entrypoint: - [ - "migrate", - "-path", - "/migrations", - "-database", - "postgres://arcuser:arcpass@db:5432/blocktx?sslmode=disable&x-migrations-table=blocktx", - ] - command: [ "up" ] - volumes: - - ../internal/blocktx/store/postgresql/migrations:/migrations - depends_on: - db: - condition: service_healthy - restart: on-failure - - migrate-metamorph: - container_name: migrate-metamorph - image: migrate/migrate:v4.16.2 - entrypoint: - [ - "migrate", - "-path", - "/migrations", - "-database", - "postgres://arcuser:arcpass@db:5432/blocktx?sslmode=disable&x-migrations-table=metamorph", - ] - command: [ "up" ] - volumes: - - ../internal/metamorph/store/postgresql/migrations:/migrations - depends_on: - db: - condition: service_healthy - restart: on-failure - - nats: - image: nats:2.10.10 - container_name: 'nats-server' - restart: on-failure - expose: - - "4222" - ports: - - "8222:8222" - hostname: nats-server - - arc-blocktx: - build: .. - expose: - - "8011" - volumes: - - ./config/config.yaml:/service/config.yaml - command: [ "./arc", "-blocktx=true" ] - depends_on: - - node1 - - node2 - - node3 - - migrate-blocktx - - migrate-metamorph - - nats - healthcheck: - test: ["CMD", "/bin/grpc_health_probe", "-addr=:8006", "-service=liveness", "-rpc-timeout=5s"] - interval: 10s - timeout: 5s - retries: 3 - deploy: - replicas: 2 - - arc-metamorph: - build: .. - expose: - - "8001" - command: [ "./arc", "-metamorph=true" ] - volumes: - - ./config/config.yaml:/service/config.yaml - depends_on: - arc-blocktx: - condition: service_healthy - healthcheck: - test: ["CMD", "/bin/grpc_health_probe", "-addr=:8005", "-service=liveness", "-rpc-timeout=5s"] - interval: 10s - timeout: 5s - retries: 3 - deploy: - replicas: 3 - - arc: - build: .. - ports: - - "8011:8011" - - "9090:9090" - - "9999:9999" - expose: - - "9090" - command: [ "./arc", "-api=true" ] - volumes: - - ./config/config.yaml:/service/config.yaml - depends_on: - arc-metamorph: - condition: service_healthy - -volumes: - node1-data: - external: false - node2-data: - external: false - node3-data: - external: false diff --git a/doc/README.md b/doc/README.md index f96b751ae..cc8a93935 100644 --- a/doc/README.md +++ b/doc/README.md @@ -11,7 +11,7 @@ possible to connect to a large number of nodes without incurring large bandwidth The ARC design decouples the core functions of a transaction processor and encapsulates them as microservices with the ability to scale horizontally adaptively. Interaction between microservices is decoupled using asynchronous messaging where possible. -ARC consists of 3 core microservices: [API](#API), [Metamorph](#Metamorph) and [BlockTx](#BlockTx), which are all described below. +ARC consists of 4 core microservices: [API](#API), [Metamorph](#Metamorph), [Callbacker](#Callbacker) and [BlockTx](#BlockTx), which are all described below. All the microservices are designed to be horizontally scalable, and can be deployed on a single machine or on multiple machines. Each one has been programmed with a store interface. The default store is postgres, but any database that implements the store interface can be used. @@ -23,7 +23,7 @@ The ARC architecture has been designed to assist in the management of the transa ARC is a transaction processor for Bitcoin that keeps track of the life cycle of a transaction as it is processed by the Bitcoin network. Next to the mining status of a transaction, ARC also keeps track of the various states that a transaction can be in, such as `ANNOUNCED_TO_NETWORK`, `SEEN_IN_ORPHAN_MEMPOOL`, `SENT_TO_NETWORK`, `SEEN_ON_NETWORK`, `MINED`, `REJECTED`, etc. -If a transaction is not `SEEN_ON_NETWORK` within a certain time period (60 seconds by default), ARC will re-send the transaction to the Bitcoin network. ARC also monitors the Bitcoin network for transaction and block messages, and will notify the client when a transaction has been mined, or rejected. +If a transaction is not at least `SEEN_ON_NETWORK` within a certain time period (60 seconds by default), ARC will re-send the transaction to the Bitcoin network. ARC also monitors the Bitcoin network for transaction and block messages, and will notify the client when a transaction has been mined, or rejected. ```mermaid stateDiagram-v2 @@ -52,12 +52,13 @@ stateDiagram-v2 ANNOUNCED_TO_NETWORK --> REQUESTED_BY_NETWORK: Peer has requested the transaction\n with a GETDATA message REQUESTED_BY_NETWORK --> SENT_TO_NETWORK: Transaction has been sent to peer SENT_TO_NETWORK --> ACCEPTED_BY_NETWORK: The transaction has been accepted\n by peer on the ZMQ interface + SENT_TO_NETWORK --> DOUBLE_SPEND_ATTEMPTED: This transaction has competing transactions SENT_TO_NETWORK --> REJECTED: Peer has sent a REJECT message ACCEPTED_BY_NETWORK --> SEEN_ON_NETWORK: ARC has received Transaction ID\n announcement from another peer ACCEPTED_BY_NETWORK --> SEEN_IN_ORPHAN_MEMPOOL: Peer has sent a 'missing inputs' message SEEN_IN_ORPHAN_MEMPOOL --> SEEN_ON_NETWORK: All parent transactions\n have been received by peer SEEN_ON_NETWORK --> MINED: Transaction ID was included in a BLOCK message - SEEN_ON_NETWORK --> DOUBLE_SPEND_ATTEMPTED: This transaction has competing transactions + SEEN_ON_NETWORK --> DOUBLE_SPEND_ATTEMPTED: A competing transactions entered the mempool DOUBLE_SPEND_ATTEMPTED --> MINED: This transaction was accepted and mined DOUBLE_SPEND_ATTEMPTED --> REJECTED: This transaction was rejected in favor\n of one of the competing transactions MINED --> [*] @@ -83,9 +84,9 @@ When possible, the API is responsible for rejecting transactions that would be u Metamorph is a microservice that is responsible for processing transactions sent by the API to the Bitcoin network. It takes care of re-sending transactions if they are not acknowledged by the network within a certain time period (60 seconds by default). -#### Callbacks +### Callbacker -Metamorph also can send callbacks to a specified URL. To register a callback, the client must add the `X-CallbackUrl` header to the request. The callbacker will then send a POST request to the URL specified in the header, with the transaction ID in the body. +Callbacker is a microservice that sends callbacks to a specified URL. To register a callback, the client must add the `X-CallbackUrl` header to the request. The callbacker will then send a POST request to the URL specified in the header, with the transaction ID in the body. The following example shows the format of a callback body @@ -423,13 +424,66 @@ worker -> store: mark txs mined ### Double spending -A transaction `A` is submitted to the network. Shortly later a transaction `B` spending one of the same outputs as transaction `A` (double spend) is submitted to ARC +In a following situation: +> Transaction `A` is submitted to the network. Shortly later, or in exactly the same time, transaction `B` spending one of the same outputs as transaction `A` (double spend) is submitted to ARC. -Expected outcome: -* If transaction `A` was also submitted to ARC it has status `SEEN_ON_NETWORK` -* Transaction `B` has status `REJECTED` +These things will happen: +1. Both transaction `A` and `B` will receive status `DOUBLE_SPEND_ATTEMPTED`. + * A callback will be sent for every double spend transaction (provided that `X-FullStatusUpdates` header is set). + * For each transaction - its competing transactions IDs (hashes) will be returned in the response and/or in the callback. +2. When either transactions `A` or `B` is mined, the other will be rejected. The mined transaction gets status `MINED` and the other gets status `REJECTED`. + * Querying ARC for `MINED` transaction `A` will return an extra information that this transaction was previously a double spend attempt. + * Querying ARC for `REJECTED` transaction `B` will return "double spend attempted" information as rejection reason. + +The same applies to all transactions, if more than two transactions are trying to spend the same UTXO. + +#### Double Spend flow - Examples + +##### Scenario 1 +> Transaction `A` is submitted to the network NOT through Arc. +> A short moment later, transaction `B` spending the same output is submitted to Arc. +> Later, transaction `A` is mined. + +Outcome: +1. A response to submitting transaction `B` will include `DOUBLE_SPEND_ATTEMPTED` status and an ID (hash) of transaction `A` as a competing transaction. +2. After transaction `A` is mined, transaction `B` will be rejected and receive status `REJECTED`. +3. If callback URL is specified - the callback with status `REJECTED` and information about rejection reason (double spend) will be sent for transaction `B`. + + +##### Scenario 2 +> Transaction `A` is submitted to the network through Arc. +> A short moment later, transaction `B` spending the same output is submitted to Arc. +> Later, transaction `A` is mined. + +Outcome: +1. A response for submitting transaction `A` will include `SEEN_ON_NETWORK` status without any information about competing transactions. +2. A response for submitting transaction `B` will include `DOUBLE_SPEND_ATTEMPTED` status and an ID (hash) of transaction `A` as a competing transaction. + * Transaction `A` status will be internally changed to `DOUBLE_SPEND_ATTEMPTED`. + * If callback URL is specified for transaction `A` - the callback with status `DOUBLE_SPEND_ATTEMPTED` and an ID (hash) of transaction `B` as a competing transaction will be sent for transaction `A`. +3. Querying for transaction `A` will now also result in `DOUBLE_SPEND_ATTEMPTED` status and an ID (hash) of transaction `B` as a competing transaction. +4. After transaction `A` is mined, it will receive status `MINED`. + * If callback URL is specified for transaction `A` - a callback with status `DOUBLE_SPEND_ATTEMPTED` and an extra information that this transactions was previously a double spend attempt will be sent. +5. Transaction `B` will be rejected and receive status `REJECTED`. The callback will be sent with an information. + * If callback URL is specified for transaction `B` - a callback with status `REJECTED` and an extra information that this transactions was a double spend attempt will be sent. +6. Querying for transaction `A` will now also result in `MINED` status and an extra information that this transactions was previously a double spend attempt. +7. Querying for transaction `B` will now also result in `REJECTED` status and an extra information that this transactions was a double spend attempt. + + +##### Scenario 3 +> Transaction `A` is submitted outside of Arc to a node that is not directly connected to Arc. +> Transaction `B` spending the same output is submitted to Arc at **exactly** the same moment. + +Outcome: +1. Submitting transaction `B` will initially result in status `SEEN_ON_NETWORK` and no competing transactions. +2. Status for transaction `B` will be changed to `DOUBLE_SPEND_ATTEMPTED` as soon as nodes share transactions `A` and `B` with each other, which usually is a matter of seconds maximum. + * If callback URL is specified for transaction `B` - the callback with status `DOUBLE_SPEND_ATTEMPTED` and an ID (hash) of transaction `A` as a competing transaction will be sent for transaction `B`. +3. If transaction `A` will be mined, transaction `B` will receive status `REJECTED` and a callback will be sent (if callback URL is set). + +#### Edge case +If transactions `A` and `B` are submitted at exactly the same time to different nodes through ARC, they both may initially receive status `SEEN_ON_NETWORK`. The status for both will be updated to `DOUBLE_SPEND_ATTEMPTED` as soon as nodes will share these transactions with each other and therefore realise it's a double spend attempt, which is usually instantaneous. + +The chance of this situation happening is extremely low when submitting transactions through Arc. -The planned feature [Double spending detection](https://github.com/bitcoin-sv/arc/blob/main/ROADMAP.md#double-spending-detection) will ensure that both transaction have status `DOUBLE_SPENT_ATTEMPTED` until one or the other transactions is mined. The mined transaction gets status `MINED` and the other gets status `REJECTED` ### Multiple submissions to the same ARC instance @@ -465,3 +519,68 @@ Expected outcome: * Information and Merkle path of the block received first will be persisted in the transaction record and not overwritten The planned feature [Update of transactions in case of block reorgs](https://github.com/bitcoin-sv/arc/blob/main/ROADMAP.md#update-of-transactions-in-case-of-block-reorgs) will ensure that ARC updates the statuses of transactions. Transactions which are not in the block of the longest chain will be updated to `REJECTED` status and transactions which are included in the block of the longest chain are updated to `MINED` status. + +## Cumulative fees validation + +The "Cumulative Fee Validation" feature is designed to check if the chain of unmined transactions (submitted transaction and its unmined ancestors) has paid a sufficient amount of fees. This validation is carried out based on a specific HTTP header. + +### Usage + +To use the "Cumulative Fee Validation" feature, you need to send the `X-CumulativeFeeValidation` header with the value set to `true`. + +Example usage: +``` +X-CumulativeFeeValidation: true +``` + +#### Special Cases + +If the `X-SkipFeeValidation` header is also sent, the fee validation will be skipped even if `X-CumulativeFeeValidation` is set to `true`. + +Example usage: +``` +X-CumulativeFeeValidation: true +X-SkipFeeValidation: true +``` + +In this case, the fee validation will not be performed. + +### Validation Examples +#### Example 1: Insufficient Fee Paid by One Ancestor +Transaction t0 has two unmined ancestors t1 and t2. + +* t0 has paid a sufficient fee for itself. +* t1 has not paid a sufficient fee. +* t2 has paid a sufficient fee for itself. + +##### Validation Result: +The validation will fail because t1 has not paid a sufficient fee and no other transaction cover it. + +#### Example 2: All Transactions Paid Their Own Fees +Transaction t0 has two unmined ancestors t1 and t2. + +* t0 has paid a sufficient fee. +* t1 has paid a sufficient fee. +* t2 has paid a sufficient fee. + +##### Validation Result: +The validation will pass because both t1 and t2 have paid sufficient fees. + +#### Example 3: Ancestors Did Not Pay, But Transaction Covers All Fees +Transaction t0 has two unmined ancestors t1 and t2. + +* t1 has not paid a sufficient fee. +* t2 has not paid a sufficient fee. +* t0 covers the fees for itself and both t1 and t2. + +##### Validation Result: +The validation will pass because t0 covers the cumulative fees for the entire chain, including t1 and t2. + +The system performs the fee validation and returns a result indicating that the chain of transactions has sufficient fees. + +### Notes + +- The `X-CumulativeFeeValidation` header must be set to `true` for the validation to be performed. +- The `X-SkipFeeValidation` header takes precedence over `X-CumulativeFeeValidation` and causes the fee validation to be skipped. + +This feature is crucial to ensure that the chain of unmined transactions has sufficient fees, which is essential for the effective management of the transaction network. diff --git a/doc/api.md b/doc/api.md index f6b3723c4..6982aaf3c 100644 --- a/doc/api.md +++ b/doc/api.md @@ -472,7 +472,8 @@ This endpoint is used to get the current status of a previously submitted transa "txid": "7927233d10dacd5606cee5bf0b28668fc191e730029ace4c7fc40ede59a2825e", "merklePath": "string", "txStatus": "MINED", - "extraInfo": null + "extraInfo": null, + "competingTxs": null } ``` @@ -507,6 +508,7 @@ X-MaxTimeout: 0 X-SkipFeeValidation: true X-SkipScriptValidation: true X-SkipTxValidation: true +X-CumulativeFeeValidation: true X-CallbackToken: string X-WaitForStatus: 0 X-WaitFor: string @@ -524,6 +526,7 @@ const headers = { 'X-SkipFeeValidation':'true', 'X-SkipScriptValidation':'true', 'X-SkipTxValidation':'true', + 'X-CumulativeFeeValidation':'true', 'X-CallbackToken':'string', 'X-WaitForStatus':'0', 'X-WaitFor':'string', @@ -580,6 +583,7 @@ func main() { "X-SkipFeeValidation": []string{"true"}, "X-SkipScriptValidation": []string{"true"}, "X-SkipTxValidation": []string{"true"}, + "X-CumulativeFeeValidation": []string{"true"}, "X-CallbackToken": []string{"string"}, "X-WaitForStatus": []string{"0"}, "X-WaitFor": []string{"string"}, @@ -610,6 +614,7 @@ headers = { 'X-SkipFeeValidation' => 'true', 'X-SkipScriptValidation' => 'true', 'X-SkipTxValidation' => 'true', + 'X-CumulativeFeeValidation' => 'true', 'X-CallbackToken' => 'string', 'X-WaitForStatus' => '0', 'X-WaitFor' => 'string', @@ -635,6 +640,7 @@ headers = { 'X-SkipFeeValidation': 'true', 'X-SkipScriptValidation': 'true', 'X-SkipTxValidation': 'true', + 'X-CumulativeFeeValidation': 'true', 'X-CallbackToken': 'string', 'X-WaitForStatus': '0', 'X-WaitFor': 'string', @@ -658,6 +664,7 @@ curl -X POST https://tapi.taal.com/arc/v1/tx \ -H 'X-SkipFeeValidation: true' \ -H 'X-SkipScriptValidation: true' \ -H 'X-SkipTxValidation: true' \ + -H 'X-CumulativeFeeValidation: true' \ -H 'X-CallbackToken: string' \ -H 'X-WaitForStatus: 0' \ -H 'X-WaitFor: string' \ @@ -695,6 +702,7 @@ This endpoint is used to send a raw transaction to a miner for inclusion in the |X-SkipFeeValidation|header|boolean|false|Whether we should skip fee validation or not.| |X-SkipScriptValidation|header|boolean|false|Whether we should skip script validation or not.| |X-SkipTxValidation|header|boolean|false|Whether we should skip overall tx validation or not.| +|X-CumulativeFeeValidation|header|boolean|false|Whether we should perform cumulative fee validation for fee consolidation txs or not.| |X-CallbackToken|header|string|false|Access token for notification callback endpoint. It will be used as a Authorization header for the http callback| |X-WaitForStatus|header|integer|false|DEPRECATED, soon will become unsupported, please use 'X-WaitFor' header. Which status to wait for from the server before returning (2 = RECEIVED, 3 = STORED, 4 = ANNOUNCED_TO_NETWORK, 5 = REQUESTED_BY_NETWORK, 6 = SENT_TO_NETWORK, 7 = ACCEPTED_BY_NETWORK, 8 = SEEN_ON_NETWORK)| |X-WaitFor|header|string|false|Which status to wait for from the server before returning ('QUEUED', 'RECEIVED', 'STORED', 'ANNOUNCED_TO_NETWORK', 'REQUESTED_BY_NETWORK', 'SENT_TO_NETWORK', 'ACCEPTED_BY_NETWORK', 'SEEN_ON_NETWORK')| @@ -776,6 +784,7 @@ This endpoint is used to send a raw transaction to a miner for inclusion in the |467|Unknown|Mined ancestors not found in BEEF|[ErrorMinedAncestorsNotFound](#schemaerrorminedancestorsnotfound)| |468|Unknown|Invalid BUMPs in BEEF|[ErrorCalculatingMerkleRoots](#schemaerrorcalculatingmerkleroots)| |469|Unknown|Invalid Merkle Roots|[ErrorValidatingMerkleRoots](#schemaerrorvalidatingmerkleroots)| +|473|Unknown|Cumulative Fee validation failed|[ErrorCumulativeFees](#schemaerrorcumulativefees)|