Skip to content

Commit

Permalink
resolving merge conflicts
Browse files Browse the repository at this point in the history
  • Loading branch information
@kameshraj23
kameshraj23 committed Aug 29, 2024
2 parents 0006a3b + 7bd1617 commit e18ec0f
Show file tree
Hide file tree
Showing 208 changed files with 9,925 additions and 5,243 deletions.
0 ...REQUEST_TEMPLATE/pull_request_template.md → .github/pull_request_template.md
View file
File renamed without changes.
2 changes: 1 addition & 1 deletion .github/workflows/go.yml → .github/workflows/go.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,6 @@ jobs:
run: |
docker compose up -d node1 node2 node3 db
docker compose up --abort-on-container-exit migrate-blocktx migrate-metamorph
docker compose up --exit-code-from tests tests arc-blocktx arc-metamorph arc --scale arc-blocktx=7 --scale arc-metamorph=2
docker compose up --exit-code-from tests tests arc-blocktx arc-callbacker arc-metamorph arc --scale arc-blocktx=5 --scale arc-metamorph=2
docker compose down
working-directory: ./test
0 .github/workflows/image.yml → .github/workflows/image.yaml
View file
File renamed without changes.
2 changes: 1 addition & 1 deletion .github/workflows/static-analysis.yml → .github/workflows/static-analysis.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,4 +71,4 @@ jobs:
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
version: v1.54
version: v1.60.1
0 .github/workflows/static.yml → .github/workflows/static.yaml
View file
File renamed without changes.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ gosec-report.json
gotest.out
*.env
report.xml
broadcaster-cli.yaml
coverage-report.html
19 changes: 18 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ All notable changes to this project will be documented in this file. The format

## Table of Contents
- [Unreleased](#unreleased)
- [1.3.0](#130---2024-08-21)
- [1.2.0](#120---2024-08-13)
- [1.1.91](#1191---2024-06-26)
- [1.1.87](#1187---2024-06-10)
- [1.1.53](#1152---2024-04-11)
- [1.1.53](#1153---2024-04-11)
- [1.1.32](#1132---2024-02-21)
- [1.1.19](#1119---2024-02-05)
- [1.1.16](#1116---2024-01-23)
Expand All @@ -16,6 +18,21 @@ All notable changes to this project will be documented in this file. The format

## [Unreleased]

### Changed
- Callbacks are sent one by one to the same URL. In the previous implementation, each callback request created a new goroutine to send the callback, which could result in a potential DDoS of the callback receiver. The new approach sends callbacks to the same receiver in a serial manner. Note that URLs are not locked by the `callbacker` instance, so serial sends occur only within a single instance. In other words, the level of parallelism is determined by the number of `callbacker` instances.

## [1.3.0] - 2024-08-21

### Changed
- The functionality for callbacks has been moved from the `metamorph` microservice to the new `callbacker` microservice.

## [1.2.0] - 2024-08-13

### Added
- [Double Spend Detection](https://bitcoin-sv.github.io/arc/#/?id=double-spending) is a feature that introduces `DOUBLE_SPEND_ATTEMPTED` status to transactions that attempt double spend together with `CompetingTxs` field in the API responses and callbacks.
- [Cumulative Fees Validation](https://bitcoin-sv.github.io/arc/#/?id=cumulative-fees-validation) is a feature that checks if a transaction has a sufficient fee not only for itself but also for all unmined ancestors that do not have sufficient fees.
- [Multiple callbacks to single transaction](https://bitcoin-sv.github.io/arc/#/?id=callbacks) Is a feature that adds support for attaching multiple callbacks to a single transaction when submitting an existing transaction with a different data.

## [1.1.91] - 2024-06-26

### Changed
Expand Down
45 changes: 25 additions & 20 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,6 @@ deps:
build:
go build ./...

.PHONY: clean_e2e_tests
clean_e2e_tests:
# Remove containers and images; avoid failure if the image doesn't exist
docker container stop test-tests-1 || true
docker container stop test-arc-1 || true
docker container rm test-tests-1 || true
docker container rm test-arc-1 || true

.PHONY: build_release
build_release:
mkdir -p build
Expand All @@ -31,12 +23,19 @@ build_release:
build_docker:
docker build . -t test-arc --build-arg="APP_COMMIT=$(APP_COMMIT)" --build-arg="APP_VERSION=$(APP_VERSION)"

.PHONY: run
run:
docker compose -f test/docker-compose.yaml down --remove-orphans
docker compose -f test/docker-compose.yaml up --abort-on-container-exit migrate-blocktx migrate-metamorph
docker compose -f test/docker-compose.yaml up --build arc-blocktx arc-callbacker arc-metamorph arc
docker compose -f test/docker-compose.yaml down

.PHONY: run_e2e_tests
run_e2e_tests:
docker-compose -f test/docker-compose.yml down
docker-compose -f test/docker-compose.yml up --abort-on-container-exit migrate-blocktx migrate-metamorph
docker-compose -f test/docker-compose.yml up --exit-code-from tests tests arc-blocktx arc-metamorph arc --scale arc-blocktx=7 --scale arc-metamorph=2
docker-compose -f test/docker-compose.yml down
docker compose -f test/docker-compose.yaml down --remove-orphans
docker compose -f test/docker-compose.yaml up --abort-on-container-exit migrate-blocktx migrate-metamorph
docker compose -f test/docker-compose.yaml up --build --exit-code-from tests tests arc-blocktx arc-callbacker arc-metamorph arc --scale arc-blocktx=4 --scale arc-metamorph=2
docker compose -f test/docker-compose.yaml down

.PHONY: test
test:
Expand All @@ -48,7 +47,7 @@ install_lint:

.PHONY: lint
lint:
golangci-lint run --config=config/.golangci.yml -v ./...
golangci-lint run --config=config/.golangci.yaml -v ./...
staticcheck ./...

.PHONY: gen_go
Expand All @@ -73,10 +72,19 @@ gen:
--go-grpc_opt=paths=source_relative \
internal/blocktx/blocktx_api/blocktx_api.proto

protoc \
--proto_path=. \
--go_out=. \
--go_opt=paths=source_relative \
--go-grpc_out=. \
--go-grpc_opt=paths=source_relative \
internal/callbacker/callbacker_api/callbacker_api.proto

.PHONY: clean_gen
clean_gen:
rm -f ./pkg/metamorph/metamorph_api/*.pb.go
rm -f ./pkg/blocktx/blocktx_api/*.pb.go
rm -f ./internal/metamorph/metamorph_api/*.pb.go
rm -f ./internal/blocktx/blocktx_api/*.pb.go
rm -f ./internal/callbacker/callbacker_api/*.pb.go

.PHONY: coverage
coverage:
Expand Down Expand Up @@ -105,7 +113,7 @@ install_gen:
go install google.golang.org/protobuf/cmd/[email protected]
go install google.golang.org/grpc/cmd/[email protected]
go install github.com/oapi-codegen/oapi-codegen/v2/cmd/[email protected]
go install github.com/matryer/moq@v0.3.4
go install github.com/matryer/moq@v0.4.0

.PHONY: docs
docs:
Expand All @@ -116,13 +124,10 @@ gh-pages:

.PHONY: api
api:
oapi-codegen -config pkg/api/config.yaml pkg/api/arc.yml > pkg/api/arc.go
oapi-codegen -config pkg/api/config.yaml pkg/api/arc.yaml > pkg/api/arc.go

.PHONY: compare_config
compare_config:
rm -f ./config/dumped_config.yaml
go run ./cmd/arc/main.go -dump_config "./config/dumped_config.yaml" && go run ./scripts/compare_yamls.go
rm ./config/dumped_config.yaml

.PHONY: clean_restart_e2e_test
clean_restart_e2e_test: clean_e2e_tests build_docker run_e2e_tests
53 changes: 34 additions & 19 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ ARC is a transaction processor for Bitcoin that keeps track of the life cycle of
- [ZMQ](#zmq)
- [BlockTx](#blocktx)
- [BlockTx stores](#blocktx-stores)
- [Callbacker](#callbacker)
- [Message Queue](#message-queue)
- [K8s-Watcher](#k8s-watcher)
- [Broadcaster-cli](#broadcaster-cli)
Expand Down Expand Up @@ -92,6 +93,9 @@ where options are:
-k8s-watcher=<true|false>
whether to start k8s-watcher (default=true)
-callbacker=<true|false>
whether to start callbacker (default=true)
-config=/location
directory to look for config.yaml (default='')
Expand All @@ -111,9 +115,9 @@ Additionally, ARC relies on a message queue to communicate between Metamorph and
docker run -p 4222:4222 nats
```
The [docker-compose file](./deployments/docker-compose.yml) additionally shows how ARC can be run with the message queue and the Postgres database and db migrations. You can run ARC with all components with the following command
You can run ARC with all components using the [docker-compose.yaml](./test/docker-compose.yaml) file by using the following make command
```
docker-compose -f deployments/docker-compose.yml up
make run
```
### Docker
Expand Down Expand Up @@ -166,21 +170,21 @@ go run main.go -metamorph=true
Metamorph keeps track of the lifecycle of a transaction, and assigns it a status, which is returned in the `txStatus` field whenever the transaction is queried.
The following statuses are available:

| Code | Status | Description |
|------|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | `UNKNOWN` | The transaction has been sent to metamorph, but no processing has taken place. This should never be the case, unless something goes wrong. |
| 10 | `QUEUED` | The transaction has been queued for processing. |
| 20 | `RECEIVED` | The transaction has been properly received by the metamorph processor. |
| 30 | `STORED` | The transaction has been stored in the metamorph store. This should ensure the transaction will be processed and retried if not picked up immediately by a mining node. |
| 40 | `ANNOUNCED_TO_NETWORK` | The transaction has been announced (INV message) to the Bitcoin network. |
| 50 | `REQUESTED_BY_NETWORK` | The transaction has been requested from metamorph by a Bitcoin node. |
| 60 | `SENT_TO_NETWORK` | The transaction has been sent to at least 1 Bitcoin node. |
| 70 | `ACCEPTED_BY_NETWORK` | The transaction has been accepted by a connected Bitcoin node on the ZMQ interface. If metamorph is not connected to ZMQ, this status will never by set. |
| 80 | `SEEN_IN_ORPHAN_MEMPOOL` | The transaction has been sent to at least 1 Bitcoin node but parent transaction was not found. |
| 90 | `SEEN_ON_NETWORK` | The transaction has been seen on the Bitcoin network and propagated to other nodes. This status is set when metamorph receives an INV message for the transaction from another node than it was sent to. |
| 100 | `DOUBLE_SPEND_ATTEMPTED` | The transaction is a double spend attempt. Competing transaction(s) will be returned with this status. |
| 110 | `REJECTED` | The transaction has been rejected by the Bitcoin network. |
| 120 | `MINED` | The transaction has been mined into a block by a mining node. |
| Status | Description |
|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `UNKNOWN` | The transaction has been sent to metamorph, but no processing has taken place. This should never be the case, unless something goes wrong. |
| `QUEUED` | The transaction has been queued for processing. |
| `RECEIVED` | The transaction has been properly received by the metamorph processor. |
| `STORED` | The transaction has been stored in the metamorph store. This should ensure the transaction will be processed and retried if not picked up immediately by a mining node. |
| `ANNOUNCED_TO_NETWORK` | The transaction has been announced (INV message) to the Bitcoin network. |
| `REQUESTED_BY_NETWORK` | The transaction has been requested from metamorph by a Bitcoin node. |
| `SENT_TO_NETWORK` | The transaction has been sent to at least 1 Bitcoin node. |
| `ACCEPTED_BY_NETWORK` | The transaction has been accepted by a connected Bitcoin node on the ZMQ interface. If metamorph is not connected to ZMQ, this status will never by set. |
| `SEEN_IN_ORPHAN_MEMPOOL` | The transaction has been sent to at least 1 Bitcoin node but parent transaction was not found. |
| `SEEN_ON_NETWORK` | The transaction has been seen on the Bitcoin network and propagated to other nodes. This status is set when metamorph receives an INV message for the transaction from another node than it was sent to. |
| `DOUBLE_SPEND_ATTEMPTED` | The transaction is a double spend attempt. Competing transaction(s) will be returned with this status. |
| `REJECTED` | The transaction has been rejected by the Bitcoin network. |
| `MINED` | The transaction has been mined into a block by a mining node. |

The statuses have a difference between the codes in order to make it possible to add more statuses in between the existing ones without creating a breaking change.

Expand Down Expand Up @@ -248,6 +252,17 @@ Metamorph publishes new transactions to the message queue and BlockTx subscribes

![Message Queue](./doc/message_queue.png)

### Callbacker

Callbacker is a microservice that sends callbacks to a specified URL.

Callbacker is designed to be horizontally scalable, with each instance operating independently. As a result, they do not communicate with each other and remain unaware of each other's existence.

You can run callbacker like this:

```shell
go run main.go -callbacker=true
```

## K8s-Watcher

Expand Down Expand Up @@ -283,7 +298,7 @@ The tests can be executed like this:
make clean_restart_e2e_test
```

The [docker-compose](./test/docker-compose.yml) file also shows the minimum setup that is needed for ARC to run.
The [docker-compose](./test/docker-compose.yaml) file also shows the minimum setup that is needed for ARC to run.


## Monitoring
Expand Down Expand Up @@ -350,7 +365,7 @@ make gen
### Generate REST API
The rest api is defined in a [yaml file](./api/arc.yml) following the OpenAPI 3.0.0 specification. Before the rest API can be generated install the necessary tools by running
The rest api is defined in a [yaml file](./api/arc.yaml) following the OpenAPI 3.0.0 specification. Before the rest API can be generated install the necessary tools by running
```
make install_gen
```
Expand Down
12 changes: 0 additions & 12 deletions ROADMAP.md
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,5 @@
# ROADMAP

## Idempotent transactions

The capability of ARC to respond with the full block information including block hash, block height and Merkle path in case that a transaction is submitted to it which has been mined previously, but which was not submitted to that instance of ARC.

## Double spending detection

Introduction of a new status indicating that a transaction is in an DOUBLE_SPENT_ATTEMPTED state. All competing transactions progress in that status until they either get REJECTED or MINED. Extension of webhooks to notify parties if the transaction state changes due to double spend attempts.

## Multiple different callbacks per transaction

Submitting a transaction multiple times with different callback URL and/or callback token will at this pair as subscription to status updates. Callbacks will henceforth be sent to each callback url with specified callback token.

## Update of transactions in case of block reorgs

ARC updates the statuses of transactions in case of block reorgs. Transactions which are not in the block of the longest chain will be updated to `UNKNOWN` status and re-broadcasted. Transactions which are included in the block of the longest chain are updated to `MINED` status.
Loading

0 comments on commit e18ec0f

Please sign in to comment.