diff --git a/docs/developing/octez-client.md b/docs/developing/octez-client.md new file mode 100644 index 000000000..c97a29095 --- /dev/null +++ b/docs/developing/octez-client.md @@ -0,0 +1,17 @@ +--- +title: The Octez client +authors: 'Thomas Zoughebi, Aymeric Bethencourt, Maxime Fernandez, Tim McMackin' +lastUpdated: 27th October 2023 +--- + +The Octez command-line client is part of the Octez suite of tools for interacting with Tezos and running nodes. +Developers use the Octez client for many tasks, including: + +- Working with accounts +- Sending transactions and interacting with contracts +- Originating smart contracts and smart rollups + +Other parts of Octez allow you to host, manage, and monitor nodes, bake blocks, and host RPC nodes. + +This documentation is an overview of the Octez client for Tezos smart contract and dApp developers. +For more detailed information, see the [Octez documentation](https://tezos.gitlab.io/) and opentezos.com. diff --git a/docs/developing/octez-client/accounts.md b/docs/developing/octez-client/accounts.md new file mode 100644 index 000000000..26ca837d0 --- /dev/null +++ b/docs/developing/octez-client/accounts.md @@ -0,0 +1,49 @@ +--- +title: Creating accounts +--- + +You can create or import accounts into the Octez client just like you do so in wallet applications. + +Octez keeps a local list of aliases for addresses, including user accounts, smart contracts, and smart rollups. +You can list the aliases that the Octez client is aware of by running the command `octez-client list known addresses`. +When you run transactions with Octez, you can use the alias in place of the account address. + +## Creating accounts + +To create an account, run this command, replacing `local_account` with a local name for the new account: + +```bash +octez-client gen keys local_account +``` + +You can see the address of the account by running this command: + +```bash +octez-client show address local_account +``` + +The account address (technically the hash of the public key) starts with `tz1`, `tz2`, or `tz3`. +You can use this address to send tez to this account, such as from a faucet if you are using a testnet. +See [Using sandboxes and testnets](../testnets). + + + +## Importing pre-existing accounts + +If you already have a Tezos account, you can import your private key to use the account in Octez: + +1. Export the private key from your wallet application, being careful not to expose it to anyone. + +1. Run this command, replacing `$ALIAS` with a local alias for the account and `$PRIVATE_KEY` with the private key: + + ```bash + octez-client import secret key $ALIAS unencrypted:$PRIVATE_KEY + ``` + +Now you can use the alias in place of the address when you send transactions with Octez. diff --git a/docs/developing/octez-client/installing.md b/docs/developing/octez-client/installing.md new file mode 100644 index 000000000..0e47ff7a3 --- /dev/null +++ b/docs/developing/octez-client/installing.md @@ -0,0 +1,98 @@ +--- +title: Installing the Octez client +lastUpdated: 27th October 2023 +--- + +You can install the Octez client directly on your computer or use a Docker image that has the most recent version of the Octez tools installed. + +## Installing the Octez client locally + +You can install the Octez client on your computer by using your package manager. +Then, initialize it to use the RPC node of your choice: + +1. Install the client: + + - For MacOS, run these commands: + + ```bash + brew tap serokell/tezos-packaging-stable https://github.com/serokell/tezos-packaging-stable.git + brew install tezos-client + ``` + + - For Ubuntu, Windows WSL, and Linux distributions that use `apt`, run these commands: + + ```bash + REPO="ppa:serokell/tezos" + sudo add-apt-repository -y $REPO && sudo apt-get update + sudo apt-get install -y tezos-client + ``` + + - For Fedora and Linux distributions that use Copr, run these commands: + + ```bash + REPO="@Serokell/Tezos" + dnf copr enable -y $REPO && dnf update -y + dnf install -y tezos-client + ``` + + For more local installation options, see [How to get Tezos](https://tezos.gitlab.io/introduction/howtoget.html) in the Octez documentation. + +1. Verify that the Octez client is installed by running this command: + + ```bash + octez-client --version + ``` + + If you see a message with the version of Octez that you have installed, the Octez client is installed correctly. + For help on Octez, run `octez-client --help` or see the [Octez documentation](http://tezos.gitlab.io/index.html). + +1. Initialize the client's configuration file by running this command: + + ```bash + octez-client config init + ``` + + This command creates a default configuration file in the location `$HOME/.tezos-client/config`. + +1. Set the RPC node to use: + + 1. Get the URL of a public RPC node or a private node that you have access to. + For example, you can get the URL of a testnet node from https://teztnets.xyz/, such as `https://rpc.ghostnet.teztnets.xyz` for Ghostnet. + + 1. Set your Octez client to use this node by running this command on the command line, replacing the Ghostnet URL with the URL that you copied: + + ```bash + octez-client --endpoint https://rpc.ghostnet.teztnets.xyz config update + ``` + + If you are using a testnet, Octez shows a warning that you are not using Mainnet. + You can hide this message by setting the `TEZOS_CLIENT_UNSAFE_DISABLE_DISCLAIMER` environment variable to "YES". + + 1. Verify that you are using the correct RPC URL by running this command: + + ```bash + octez-client config show + ``` + + The response from Octez includes the URL of the RPC node that you are using. + +For a full list of Octez client commands, run `octez-client man`. + +## Using the Octez client in the Tezos Docker image + +The Tezos Docker image contains the latest version of the Octez client and the other Octez tools. +To start a container from this image, run this command: + +```bash +docker run -it --rm --entrypoint /bin/sh --name octez-container tezos/tezos:latest +``` + +You can verify that the Octez client is available in the container by running this command: + +```bash +octez-client --version +``` + +Then set the RPC node for the client as described above. + +For more information about using the Docker image, see [Using Docker Images And Docker-Compose](https://tezos.gitlab.io/introduction/howtoget.html#using-docker-images-and-docker-compose) in the Octez documentation. diff --git a/docs/developing/octez-client/transactions.md b/docs/developing/octez-client/transactions.md new file mode 100644 index 000000000..954553e23 --- /dev/null +++ b/docs/developing/octez-client/transactions.md @@ -0,0 +1,73 @@ +--- +title: Interacting with contracts +authors: Tim McMackin +lastUpdated: 27 October 2023 +--- + +Before sending transactions to contracts with the Octez client, make sure that your client is configured to use an RPC node for the network that you want to use. +To verify the RPC node that you are using, run `octez-client config show` and look at the value in the `entrypoint` field. + +## Sending tez + +To send tez from a source account to a target account, use the `octez-client transfer` command, as in this example: + +```bash +octez-client transfer 42 from account1 to account2 --fee-cap 0.9 +``` + +You can use addresses or local aliases for the source and target accounts, but Octez must have the private key for the source account in order to sign the transaction. + +To check the balance of an account after a transaction, use the `octez-client get balance for` command and pass the alias or address of an account, as in this example: + +```bash +octez-client get balance for account1 +``` + +## Calling smart contracts + +To call a smart contract, use the `octez-client transfer` command, as in this example from the [Deploy a smart contract](../../tutorials/smart-contract) tutorial: + +```bash +octez-client --wait none transfer 0 \ + from $MY_TZ_ADDRESS to my-counter \ + --entrypoint 'increment' --arg '5' --burn-cap 0.1 +``` + +This command calls the `increment` entrypoint of the `my-counter` contract and includes 0 tez and up to 0.1 tez in fees with the transaction. +It passes the parameter "5" to the endpoint. +You can use a local alias or the full address for the smart contract. + +Because entrypoints accept parameters in Michelson code, you must encode the values that you send as Michelson values. +The high-level languages have tools to help you encode values in Michelson. +For example, if you are using LIGO to create an entrypoint that accepts an integer, address, and string as a parameter, you can pass the parameter values as a LIGO expression and get the Michelson version with this command: + +```bash +ligo compile parameter MyContract.jsligo -e "myentrypoint" \ + '[5, "tz1QCVQinE8iVj1H2fckqx6oiM85CNJSK9Sx" as address, "hello"]' +``` + +The compiled parameter value looks like this: + +``` +(Pair 5 "tz1QCVQinE8iVj1H2fckqx6oiM85CNJSK9Sx" "hello") +``` + +Then you can use this value for the value of the `--arg` argument in the `octez-client transfer` command. + +## Originating (deploying) smart contracts + +To deploy (originate) a smart contract to the current network, use the `octez-client originate contract` command, as in this example from the [Deploy a smart contract](../../tutorials/smart-contract) tutorial: + +```bash +octez-client originate contract my-counter \ + transferring 0 from $MY_TZ_ADDRESS \ + running increment.tz \ + --init 10 --burn-cap 0.1 --force +``` + +This command accepts the compiled version of the contract as a Michelson `.tz` file. +See the documentation for the high-level language you are using for how to compile a contract. +Like the command to call a smart contract, this command accepts the initial value of the contract storage as a Michelson-encoded value. + +After you originate the contract, you can use the local alias from this command to send transactions to its entrypoints. +You can see the aliases of contracts by running the command `octez-client list known contracts`. diff --git a/docs/reference/octez.md b/docs/reference/octez.md deleted file mode 100644 index 26f7267c8..000000000 --- a/docs/reference/octez.md +++ /dev/null @@ -1,302 +0,0 @@ ---- -title: Getting Started with Octez -authors: 'Thomas Zoughebi, Aymeric Bethencourt, and Maxime Fernandez' -lastUpdated: 5th June 2023 ---- - -## Octez - The Tezos Client - -_Octez_ is the official client to interact with a Tezos node via RPC (remote procedural calls). In theory, there can be multiple implementations of the Tezos protocol, but currently, there is only Octez. Octez consists of several binaries (i.e., executable files), including a client, a node, and a baker. - -### How to install the *octez-client* - -#### On Mac OS with [Homebrew](https://brew.sh/) - -```bash -brew tap serokell/tezos-packaging-stable https://github.com/serokell/tezos-packaging-stable.git -brew install tezos-client -``` - -#### On Ubuntu with binaries - -```bash -sudo add-apt-repository ppa:serokell/tezos && sudo apt-get update -sudo apt install tezos-client -y -``` - -#### On Fedora with binaries - -```bash -dnf copr enable -y @Serokell/Tezos && dnf update -y -dnf install -y tezos-client -``` - -#### From [sources with OPAM](https://tezos.gitlab.io/introduction/howtoget.html#building-from-sources-via-opam) - -### Connecting to a node - -Below we'll connect to a community node (https://ghostnet.tezos.marigold.dev) on the Ghostnet **testnet**. We'll use the `--endpoint` parameter to update the configuration of the Octez Client on a Ubuntu system: - -`octez-client --endpoint https://ghostnet.tezos.marigold.dev config update` - - -The result should look like this: - -```bash -Warning: - - This is NOT the Tezos Mainnet. - - Do NOT use your fundraiser keys on this network. -``` - -On Ubuntu, the config file should be written in "/.octez-client/config" under your "home" folder. For example, after the last command, the new "config" file looks like this (with Nano): - -```bash -{ "base_dir": "/home/userName/.octez-client", - "endpoint": "https://ghostnet.tezos.marigold.dev", "web_port": 8080, - "confirmations": 0 } -``` - -### Account funding - -To get free tez on testnet, you must first generate a Tezos account. If you don't already have one, you can generate one with the following command: - -`octez-client gen keys my_account` - -You can now retrieve the generated address (the hash of the public key, it starts with tz1, tz2 or tz3) with the command: - -`octez-client show address my_account` - -The result should look like this: - -```bash -Warning: - - This is NOT the Tezos Mainnet. - - Do NOT use your fundraiser keys on this network. - -Hash: tz1VvyNvPUdypHaTgznTLSkumj9YMZxpmB9p -Public Key: edpkufA6kH6hw4ckZWWmYuLZpfwfXc9abiaEDLqH2iviFXnK9N4oct -``` - -You can now go to [the testnets website](https://teztnets.xyz/), select your testnet faucet and request tez! - -### Octez Client user manual and version - -:::note -The full command list for *Octez client* is available [here](https://tezos.gitlab.io/shell/cli-commands.html). -::: - -## Octez Client examples - -### Get balance - -Get the balance of an account: - -```bash -octez-client get balance for [account_id] -``` - -Example with our previously registered user "my_account": - -```bash -octez-client get balance for my_account -``` - -The response: - -```bash -Warning: - - This is NOT the Tezos Mainnet. - - Do NOT use your fundraiser keys on this network. - -1 ꜩ -``` - -The previous response is also a way of checking if the account is activated on the testnet (first transfer). - -### Get timestamp - -Get the UTC of the latest downloaded block. Keep in mind, timezones may differ from your local time: - -```bash -octez-client get timestamp -``` - -Result example: - -```bash -Warning: - - This is NOT the Tezos Mainnet. - - Do NOT use your fundraiser keys on this network. - -2022-10-04T13:34:00Z -``` - -### List known addresses - -This call only lists registered **implicit accounts** in your Octez client. - -```bash -octez-client list known addresses -``` - -Example response: - -```bash -Warning: - - This is NOT the Tezos Mainnet. - - Do NOT use your fundraiser keys on this network. - -my_account: tz1VvyNvPUdypHaTgznTLSkumj9YMZxpmB9p (unencrypted sk known) -``` - -### List known contracts - -This call lists **all registered accounts**, *implicit* **and** *originated* in your Octez client. - -```bash -octez-client list known contracts -``` - -Our example: - -```bash -Warning: - - This is NOT the Tezos Mainnet. - - Do NOT use your fundraiser keys on this network. - -my_account: tz1VvyNvPUdypHaTgznTLSkumj9YMZxpmB9p -``` - -Everything is correct: We don't have any *originated* account and only one *implicit* account! - -### Transfers and receipts - -The command below transfers 42 ꜩ from the account *user1* to *user2* (you can use Tezos addresses directly): - -```bash -octez-client transfer 42 from user1 to user2 -``` - -Notice that you can specify the maximum fee for this operation appending, "`--fee-cap`" (defaults to 1.0). The network would determine the actual fee, lower than this cap: - -```bash -octez-client transfer 42 from user1 to user2 --fee-cap 0.9 -``` - -You can also add "`--dry-run`" or "`-D`" if you want to test and display the transaction without finalizing it. - -For our example, let's generate another account and feed it with tez from [the faucet website](https://faucet.ghostnet.teztnets.xyz/): - -```bash -$ octez-client gen keys my_account_2 -$ octez-client show address my_account_2 -Hash: tz1M9Snt3Sdcv9YkTrergj3ar6FuQ2g4T9y3 -Public Key: edpktfqbZHfRRSRcJ86hqxQZvgfFMLwR6zMZAXA5UgE81L7WqHt579 -``` - -Let's verify the balance (and activation): - -```bash -octez-client get balance for my_account_2 -``` - -Response (without the warning message): - -```bash -2 ꜩ -``` - -Let's finally try a transfer of 1.5 ꜩ from my_account_2 to my_account with a 0.5 ꜩ fee cap: - -```bash -octez-client transfer 1.5 from my_account_2 to my_account --fee-cap 0.5 -``` - -Response (without the warning message) should be: - -```bash -Node is bootstrapped. -Estimated storage: no bytes added -Estimated gas: 1000.040 units (will add 0 for safety) -Estimated storage: no bytes added -Operation successfully injected in the node. -Operation hash is 'oocBxnYDf8qiT3EmRFZxR9axGKqEJMxWnFAnzN5Dwz8QFxUe89Z' -Waiting for the operation to be included... -Operation found in block: BLzxBbdk1yq61Z3U36652RBxMM6PMfitA53Rk3xFiZpMX9yQZVa (pass: 3, offset: 0) -This sequence of operations was run: - Manager signed operations: - From: tz1M9Snt3Sdcv9YkTrergj3ar6FuQ2g4T9y3 - Fee to the baker: ꜩ0.00036 - Expected counter: 12102064 - Gas limit: 1000 - Storage limit: 0 bytes - Balance updates: - tz1M9Snt3Sdcv9YkTrergj3ar6FuQ2g4T9y3 ... -ꜩ0.00036 - payload fees(the block proposer) ....... +ꜩ0.00036 - Revelation of manager public key: - Contract: tz1M9Snt3Sdcv9YkTrergj3ar6FuQ2g4T9y3 - Key: edpktfqbZHfRRSRcJ86hqxQZvgfFMLwR6zMZAXA5UgE81L7WqHt579 - This revelation was successfully applied - Consumed gas: 1000 - Manager signed operations: - From: tz1M9Snt3Sdcv9YkTrergj3ar6FuQ2g4T9y3 - Fee to the baker: ꜩ0.000258 - Expected counter: 12102065 - Gas limit: 1001 - Storage limit: 0 bytes - Balance updates: - tz1M9Snt3Sdcv9YkTrergj3ar6FuQ2g4T9y3 ... -ꜩ0.000258 - payload fees(the block proposer) ....... +ꜩ0.000258 - Transaction: - Amount: ꜩ1.5 - From: tz1M9Snt3Sdcv9YkTrergj3ar6FuQ2g4T9y3 - To: tz1VvyNvPUdypHaTgznTLSkumj9YMZxpmB9p - This transaction was successfully applied - Consumed gas: 1000.040 - Balance updates: - tz1M9Snt3Sdcv9YkTrergj3ar6FuQ2g4T9y3 ... -ꜩ1.5 - tz1VvyNvPUdypHaTgznTLSkumj9YMZxpmB9p ... +ꜩ1.5 - -The operation has only been included 0 blocks ago. -We recommend to wait more. -Use command - octez-client wait for oocBxnYDf8qiT3EmRFZxR9axGKqEJMxWnFAnzN5Dwz8QFxUe89Z to be included --confirmations 1 --branch BLngFFgWJ1TJUgPTp4CLvnyGFzwhKrpcazRFysptBmhZKKH4w94 -and/or an external block explorer. -``` - -Let's check both balances (testnet warning messages removed): - -```bash -$ octez-client get balance for my_account -2.5 ꜩ - -$ octez-client get balance for my_account_2 -0.499382 ꜩ -``` - -Everything is fine. - -You can observe your actions on explorers like *tzkt* or *tzstats*: - -* Mainnet: [TzKT](https://tzkt.io/), [TzStats](https://tzstats.com) -* Ghostnet: [TzKT](https://ghostnet.tzkt.io/), [TzStats](https://edo.tzstats.com) - -OpenTezos has a guide on [how to use an explorer](https://opentezos.com/explorer). - -:::note -For a more comprehensive guide on how to install Octez, including via Docker images, see the [docs](https://tezos.gitlab.io/introduction/howtoget.html) by Nomadic Labs. - -The above is an abridged version of the [Octez guide](https://opentezos.france-ioi.org/en/a/1332828229711949831;p=1,3556096645042535892;pa=0) found on OpenTezos. -::: diff --git a/docs/reference/octez/best-practices.md b/docs/reference/octez/best-practices.md deleted file mode 100644 index 491555844..000000000 --- a/docs/reference/octez/best-practices.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -title: Octez best practices -authors: Nomadic Labs -lastUpdated: 27th June 2023 ---- - -An alternative to using the services of a public node is to run your own node. - -This document presents a series of recommended practices for running Tezos nodes. It chiefly concerns the configuration of nodes for baking, but more largely applies to running the nodes in other contexts. Conversely, some other practices are signaled as being potentially dangerous; it is advisable to avoid them unless you really know what you are doing. - -The discussion is centered on the Octez implementation of the Tezos node, although some aspects may apply to Tezos nodes in general. - -## Context - -Public blockchains are security-sensitive platforms. Indeed, given the high economic values at stake, such platforms are frequently targets of various kinds of attacks, such as denial of service (DOS), unforeseen and malicious uses of APIs, etc. - -The Tezos mainnet is no exception to this general situation. This is why the Octez implementation of the Tezos blockchain comes with a number of built-in protection mechanisms. For flexibility reasons, these mechanisms can be disabled or reconfigured by advanced users, but this implies mastering the associated risks. - -For instance, the RPC interface offered by the Octez "octez-node" binary is disabled by default, as a first protection mechanism, and must be explicitly enabled for the clients to communicate with the node. However, it is dangerous to make the whole node interface publicly accessible on the internet, for example, because many RPC calls are costly, and can thus be used for DOS attacks. For this reason, a second protection mechanism allows configuring the RPC interface using access control lists to enable only specific RPC calls. Thus, advanced users may disable the first protection mechanism and configure the RPC interface at a fine grain, according to their own expertise, and at their own risks. Still, the recommended practice is to open the RPC interface only on localhost. - -Besides protecting against attacks, the configuration of a node is also important for: preventing resource exhaustion (disk, RAM, ...), optimizing the performance of each node or of the network as a whole, optimizing rewards for bakers (e.g., by avoiding missed endorsements), etc. - -The rest of this document is structured around different aspects of the Octez node configuration, and for each such aspect recommends best practices or warns about dangerous practices. - -## RPC interface - -The Tezos node offers an [RPC programming interface](https://tezos.gitlab.io/shell/rpc.html), that is used by clients such as a wallet or a baker. This is a rich API providing many features. - -Some of these RPC are sensitive in terms of security: - -- either because they perform sensitive operations that shouldn't be available to untrusted peers, such as banning/trusting a peer, configuring the operations filtered out by the node mempool, etc. - -- or because they perform costly operations that could serve for denial of service (DOS) attacks. - -As a concrete example (in the second category), there are two RPC calls to estimate baking and endorsing rights for the next cycles (GET endorsing_rights and GET baking_rights). Calling these RPCs repeatedly with no limitation by remote nodes can stall a node, preventing it to do any useful work. - -For this reasons, the Octez implementation of the Tezos nodes contains three built-in protection mechanisms: - -- When running a node, the RPC interface is disabled by default. It has to be explicitly enabled using option `--rpc-addr`. - -- The RPC interface can be configured at a fine grain in the node configuration file using an ACL (access control list) mechanism, by listing either the calls that are enabled (whitelist) or disabled (blacklist). - -- A contextual policy is provided for configuring the RPC interface in the form of a default ACL. For the localhost interface, all RPCs are enabled by default. For other network interfaces of the current host, a subset of RPCs considered usually safe are enabled, while RPCs considered sensitive and costly are disabled. However, the default list for remote hosts should only be considered as a base for customizing an ACL under your own responsibility. - -See [here](http://tezos.gitlab.io/user/node-configuration.html#default-acl-for-rpc) for more details on the default ACLs and some examples. - -Advanced users may also establish more complex configurations using the [proxy server](https://tezos.gitlab.io/user/proxy-server.html), under their responsibility. - -Based on these protection mechanisms, the recommended practice is to enable the RPC interface only on the localhost interface, thus only serving local binaries such as octez-client and octez-baker. - -:::warning -Advanced users choosing to open the RPC interface for remote hosts do this at their own risks, and should carefully use the ACL mechanism for fine-grain configuration. -::: - - -## History modes -The Octez node is configurable to keep more or less of the blockchain history. Different trade-offs between the required storage space and the possible queries on the history, are available in the form of three "history modes" called: Archive, Full, and Rolling (see https://tezos.gitlab.io/user/history_modes.html). These modes are suitable for different applications. For instance, the Archive mode stores all the blockchain history of balances since the genesis, and thus has the heaviest storage needs. It is suitable for applications such as indexers and block explorers. The lightest mode in storage consumption is the Rolling mode. The Full mode is an intermediate tradeoff (where everything is recomputable but only recent history is stored), generally suitable for bakers. - -The Rolling mode is also compatible with baking, when used with caution. Specifically, as bakers are responsible, among others, of revealing nonces of blocks they produced during the previous cycle, they must proceed carefully when importing a snapshot. Indeed, a baker importing a snapshot in Rolling mode between the production of some block B and the revelation of the corresponding nonce, may lose information about block B, and fail to reveal its nonce, if the cycles reconstituted after the snapshot was imported are not including block B. (As a consequence, the baker would also lose the endorsing rewards for the cycle when they failed to reveal the nonce). For example, if the baker must reveal a nonce for block B at level L, the imported snapshot must record level L or lower. - -These history modes also exhibit different CPU performance characteristics, which are worth taking into account. Specifically, nodes in Full mode are quicker in validating blocks than nodes in Archive mode (because in the latter case, data are searched in a larger file). This is especially true when nodes in Full mode have been restarted recently from a snapshot, and keep thus a limited history. - -It is a recommended practice to use the Full mode for baking, and to regularly restart the baker from a recent snapshot, in order to ensure quick block validation. - -It is not advised to set bakers in Archive mode, unless you have specific reasons to do so, as this typically slows down block validation. It is not advised to set bakers in Rolling mode either (even when it is not required to help other nodes to bootstrap), to always be able to reveal the nonce of the blocks they produced. - -## Connections - -In order to participate to the P2P network underlying the blockchain, the Tezos node establishes connections to different peers. The number of such connections is highly configurable, using options such as `--connections`, `--min-connections`, `--max-connections`, etc. - -Being connected to more peers increases the chances of rapidly having up-to-date information about proposed operations and blocks, which is a very important consideration for bakers. On the other hand, having too many connections (e.g. hundreds) may incur an unacceptable overhead of communicating and synchronizing with all these peers. Therefore, it is advisable to configure a node to establish a reasonable number of connections. By default, the Tezos node currently targets 50 connections. - -It is advised not to configure a node to target too many connections (hundreds or so), as this could incur an unreasonable communication and synchronization overhead for the node. The default number of connections provides a reasonable base for defining an appropriate value for most cases. - -## Disk space - -Running a Tezos node requires providing enough disk space for its storage requirements. Indeed, if a node runs out of disk space, it shuts down and thus discontinues its service. - -The disk space consumed by a node depends on its history mode, as explained above. But it is also important to know that, independendently of the archive mode, the disk space consumption varies in time. Thus, at regular intervals (about every 8 hours), disk space consumption increases temporarily, as the node re-orgranizes its storage. During this operation, the node keeps a copy of the old storage index, which explains why this temporary increase can typically add around 20G of disk space that are freed soon afterwards. More precisely, the extra space corresponds to the temporary copy of the `/context/index` subdirectory, where `` is the node's data directory. - -Prior to running a node, make sure that it has not only enough disk space for its regular functioning (which may depend on its history mode and other options), but also extra available space for a copy of its index (typically around 20G extra space). - -:::warning Disk space -Not allocating enough disk space for a node, including for its dynamic variations, incurs the risk of seeing the node regularly stopped. -::: - - -## Memory space - -Running a Tezos node requires providing enough memory space (RAM) for its functioning. Indeed, if a node runs out of RAM, it shuts down and thus discontinues its service. - -The memory consumption of the node varies in time for the same reason as its disk space consumption: the node regularly reorganizes its storage (about every 8 hours). During normal operation, a node can typically work with 4G of RAM. During the storage reorganization operation, the node can consume twice as memory (up to 8G), but as each such operation is rather short-lived, it is safe to allocate the extra space as swap space on the disk. - -:::warning RAM -Prior to running a node, make sure that enough RAM is available (typically 4GB), and also enough swap space on disk (typically 4GB). -::: - - -## Performance - -Ensuring a good performance level for a node requires allocating enough resources of different kinds (RAM, disk, CPUs) for its typical and peek needs. Currently, the central performance bottleneck of a Tezos node is constituted by disk bandwidth for read/write operations. For this reason, when dimensioning a machine for running a node, it is important to ensure sufficient disk bandwidth. This implies that increasing the number of CPUs machine and the RAM size of a machine to run several nodes on it does not scale well: the disk bandwidth will be the limiting factor. Rather, it is more resource-effective to run single nodes on moderately powerful machines. - -When dimensioning machines for running Tezos nodes, it is advised to ensure a good disk bandwidth, such as those offered by SSD disks (and ideally, SSD/NVE disks). Running several nodes on the same machine should be considered with caution: pay attention not to saturate the disk bandwidth. - -:::warning multi-CPU machine -Running several nodes on a multi-CPU machine will likely lead to suboptimal performance, because the disk bandwith will probably constitute the main bottleneck. -::: - - -## Delayed endorsement - -When a block is created and propagated on the network, nodes that have endorsing rights at that time slot can verify the validity of the block and emit an endorsement operation, which can be seen as a vote for including that block. Missing endorsement opportunities can result in missed rewards, but also has negative consequences on the network, such as slowing down the network and making the chain more susceptible to forks. -One possible reason for missing an endorsement opportunity is when the endorsement operation arrives too late, after the block has been created. Thus, it is important for bakers to emit endorsement operations as soon as possible. - -Make sure that your baker process is not patched to delay endorsements, in order to avoid missed endorsements. So, if you are running an endorser with `--endorsement-delay `, we suggest that you restart your endorser without it. - -:::warning Delaying endorsements -Delaying endorsements, for historical reasons or for any reasons, incurs the risk of the endorsements arriving too late to be considered by block creators, hence resulting in missed reward opportunities. -::: - diff --git a/docs/reference/octez/cli-rpc.md b/docs/reference/octez/cli-rpc.md deleted file mode 100644 index b9cda9920..000000000 --- a/docs/reference/octez/cli-rpc.md +++ /dev/null @@ -1,291 +0,0 @@ ---- -title: CLI and RPC -lastUpdated: 30th June 2023 ---- -## Connecting to the network - -The _tezos-client_ and Tezos RPC need to be connected to a Tezos node. You can connect to your own [Tezos node](https://opentezos.com/%20) or you can use a community node on the Tezos mainnet or testnet. - -You can find a list of community nodes [here](https://tezostaquito.io/docs/rpc_nodes/). - -If you use a testnet, you can get test XTZ [from this faucet](https://faucet.marigold.dev/). - -## Tezos RPC (Remote Procedure Call) - -[RPC](https://tezos.gitlab.io/developer/rpc.html) is a client-server protocol where the requesting program is the client and the program providing the service is the server. - -Tezos nodes provide a JSON/RPC interface to interact with the Tezos network. Note that although it uses RPC and is JSON based, it does not follow the `JSON-RPC` protocol. - -A complete list of RPC calls is available [here](http://tezos.gitlab.io/shell/rpc.html#rpc-index-shell). Make sure to check the protocol version before using these calls. The calls available for the current protocol are available [here](https://tezos.gitlab.io/active/rpc.html). - -### RPC call examples - -**Get block** - -This call is used to find all the information about a block. The associated metadata may not be present depending on the history mode and block distance from the head. - -```bash -GET NodeUrl/chains/[chain_id]/blocks/[blocks_id] -``` - -Example: To get the block number _1400114_ from the mainnet using the ECAD Labs node, do as follows: - -```bash -GET https://mainnet.api.tez.ie/chains/main/blocks/1400114 -``` - -**Get contract storage** - -This call is used to access the storage of the contract. - -```bash -GET NodeUrl/chains/[chain_id]/blocks/[blocks_id]/context/contracts/[contract_id]/storage -``` - -Example: To get the storage of contract _KT1Hkg5qeNhfwpKW4fXvq7HGZB9z2EnmCCA9_ from block number _1400114_ on the mainnet using the ECAD Labs node, do as follows: - -```bash -GET https://mainnet.api.tez.ie/chains/main/blocks/1400114/context/contracts/KT1Hkg5qeNhfwpKW4fXvq7HGZB9z2EnmCCA9/storage -``` - -### Tezos-client (CLI) - -_Tezos-client_ is the official client when interacting with a Tezos node via RPC. Let's take a look at some examples and how to use it. - -#### How to install - -* On Mac OS with [Homebrew](https://brew.sh/): - -```bash -$ brew install tezos-client -``` - -* On Ubuntu with binaries: - -```bash -$ sudo add-apt-repository ppa:serokell/tezos && sudo apt-get update -$ sudo apt-get install -y tezos-client -``` - -* On Fedora with binaries: - -```bash -$ dnf copr enable -y @Serokell/Tezos && dnf update -y -$ dnf install -y tezos-client -``` - -* [From sources with OPAM](https://tezos.gitlab.io/introduction/howtoget.html#building-from-sources-via-opam). - -#### Account Activation - -Activate your account by replacing `#accountName` and `#faucet` below with an account name and the path to a faucet file. - -```bash -$ tezos-client activate account #accountName with ~/Downloads/#faucet.json -``` - -Example : - -```bash -$ tezos-client activate account user1 with ~/Downloads/tz1VH3sHQ5SNby95S9EtPQBqZrhgv2DqjPvy.json -``` - -#### Client manual - -* For global options - -```bash -$ tezos-client --help -``` - -* For command options - -```bash -$ tezos-client [global options] command --help -``` - -* For version information - -```bash -$ tezos-client --version -``` - -The full documentation on _Tezos client_ can be found [here](https://tezos.gitlab.io/active/cli-commands.html). - -#### Client examples - -This is a non-exhaustive list of possible commands with _tezos-client_. To discover more commands, please refer to the [CLI manual](https://tezos.gitlab.io/active/cli-commands.html). - -**Get balance** - -To get the balance of your account you can execute the following command: - -```bash -$ tezos-client get balance for #accountName -``` - -**Get timestamp** - -This call is useful to check if the node is synchronized. It returns the UTC time of the latest downloaded block, timezones may differ from your local time. - -```bash -$ tezos-client get timestamp -``` - -**List known addresses** - -This call lists implicit accounts registered in your _tezos-client_. - -```bash -$ tezos-client list known addresses -``` - -**List known contracts** - -This call lists all accounts (implicit and smart contract) registered in your _tezos-client_. - -```bash -$ tezos-client list known contracts -``` - -**Transfers and receipts** - -The command line below makes a transaction of 42ꜩ from the account _user1_ to _user2_ (you can also just use the tezos addresses directly): - -```bash -$ tezos-client transfer 30 from user1 to user2 --burn-cap 0.5 -``` - -Notice that `--burn-cap` specifies the maximum fee the user is willing to pay for this operation. The actual fee is determined by the network. - -You can also add `--dry-run` if you just want to practice by running a transaction simulation instead of a real transaction. - -The receipt for this command is as follows: - -`Current head: BM3smBpBVtHD (timestamp: 2021-03-12T09:42:28.000-00:00, validation: 2021-03-12T09:42:38.372-00:00)Node is bootstrapped.Estimated gas: 1000 units (will add 100 for safety)Operation successfully injected in the node.Operation hash is 'oo4745Q5mq8snHYAxUYWedBCVb7yQJ7jvFhKECPN9xqgwE4Ni8A'Waiting for the operation to be included...Operation found in block: BKnKoaYqCz3dTWr66x4X1mvXC95kuozRkdd23LDuM5ZA1ayF5mi (pass: 3, offset: 0)This sequence of operations was run: Manager signed operations: From: tz1gWQz5iTP6UDkWjm1jnKsCq1HHG4hgEmJn Fee to the baker: ꜩ0.000369 Expected counter: 1412932 Gas limit: 1100 Storage limit: 0 bytes Balance updates: tz1gWQz5iTP6UDkWjm1jnKsCq1HHG4hgEmJn ............. -ꜩ0.000369 fees(tz1aWXP237BLwNHJcCD4b3DutCevhqq2T1Z9,248) ... +ꜩ0.000369 Revelation of manager public key: Contract: tz1gWQz5iTP6UDkWjm1jnKsCq1HHG4hgEmJn Key: edpktpMjpmsK6fLKGrXzhp67tRdu2m5HyzGrp1tuS8ZJBc7dGqBeAq This revelation was successfully applied Consumed gas: 1000` - -You can observe your actions on [https://tzstats.com](https://tzstats.com/) - -### tezos-admin-client - -The admin client allows you to interact with the peer-to-peer layer to: - -* check the status of the connections -* force connections to known peers -* ban/un-ban peers - -#### How to install - -* On Mac OS with [Homebrew](https://brew.sh/): - -```bash -$ brew install tezos-admin-client -``` - -* On Ubuntu with binaries: - -```bash -$ sudo add-apt-repository ppa:serokell/tezos && sudo apt-get update$ sudo apt-get install -y tezos-admin-client -``` - -* On Fedora with binaries: - -```bash -$ dnf copr enable -y @Serokell/Tezos && dnf update -y$ dnf install -y tezos-admin-client -``` - -* [From sources with OPAM](https://tezos.gitlab.io/introduction/howtoget.html#building-from-sources-via-opam). - -#### Admin-client manual - -* Global options: - -```bash -$ tezos-admin-client --help -``` - -* Command options: - -```bash -$ tezos-admin-client [global options] command --help -``` - -* Version information: - -```bash -$ tezos-admin-client --version -``` - -The full documentation about the tezos-admin-client can be found [here](https://tezos.gitlab.io/shell/cli-commands.html). - -#### Admin-client examples - -This is a non-exhaustive list of possible commands with the tezos-admin-client. To discover more commands please refer to the [CLI manual](https://tezos.gitlab.io/shell/cli-commands.html). - -**Commands for the low-level RPC layer** - -* List RPCs under a given URL prefix: - -```bash -$ tezos-admin-client rpc list [URL] -``` - -* Get the input and the output JSON schemas of an RPC: - -```bash -$ tezos-admin-client rpc schema [HTTP method] [url] -``` - -* Get the readable input and output formats of an RPC: - -```bash -$ tezos-admin-client rpc get [url] -``` - -* Call an RPC with the POST method and input params: - -```bash -$ tezos-admin-client list protocols -``` - -**Commands for managing protocols** - -* List protocols known by the node: - -```bash -$ tezos-admin-client rpc post [url] with [input] -``` - -* Inject a new protocol into the node: (*given_dir* is the directory containing the sources of a protocol) - -```bash -$ tezos-admin-client inject protocol [given_dir] -``` - -**Commands to report the node's status:** - -* The last heads that have been considered by the node: - -```bash -$ tezos-admin-client list heads [-o --output [path]] -``` - -**Commands for editing and viewing the client's config file:** - -* Show the current config file content and command line arguments: - -```bash -$ tezos-admin-client config show -``` - -* Update the config based on the current CLI values: - -```bash -$ tezos-admin-client config update -``` - -* A useful command to debug a node that is not syncing: - -```bash -$ tezos-admin-client tezos-admin-client p2p stat -``` diff --git a/docs/reference/octez/installing.md b/docs/reference/octez/installing.md deleted file mode 100644 index 402af7aac..000000000 --- a/docs/reference/octez/installing.md +++ /dev/null @@ -1,298 +0,0 @@ ---- -title: Installing Octez -lastUpdated: 30th June 2023 ---- - -To start, we'll download and install tezos-client and create a couple of test wallets. We'll use [tezos-client](https://tezos.gitlab.io/api/cli-commands.html) - a command line interface to Tezos. - -## Installation - -### Linux (64-bit) - -A quick and easy way to get tezos-client running on Linux is to download the latest `tezos-client` binary, make it executable, and put it somewhere in your path. Alternatively, you can add a package repository for your distribution, and install it from there. Using a package is a good idea for production systems as it automates the installation and allows easy updates. - -#### Option 1: Install the binary - -```sh -$ wget https://github.com/serokell/tezos-packaging/releases/latest/download/tezos-client -$ chmod +x tezos-client -$ mkdir -p $HOME/.local/bin -$ mv tezos-client $HOME/.local/bin -$ echo 'export PATH="$HOME/.local/bin:$PATH"' >> $HOME/.bashrc -$ source $HOME/.bashrc -``` -#### Option 2: Install from source - -This information is based on documentation from: [Get Tezos](https://tezos.gitlab.io/introduction/howtoget.html%20) - -Tips: - -* Use ```tab```to autocomplete partial commands previously entered -* Use the up and down arrows to cycle through previously entered commands - -**1.** Make sure your system is up to date. - -```sh -sudo apt-get update -``` - -```sh -sudo apt-get upgrade -``` - -* If asked if you want to use a certain amount of space for the update, enter: ```y``` - - -**2.** Install rust. - -```sh -sudo apt install -y rsync git m4 build-essential patch unzip wget pkg-config libgmp-dev libev-dev libhidapi-dev libffi-dev opam jq zlib1g-dev -``` - -```sh -wget https://sh.rustup.rs/rustup-init.sh -``` - -```sh -chmod +x rustup-init.sh -``` - -```sh -./rustup-init.sh --profile minimal --default-toolchain 1.52.1 -y -``` - -**3.** Loading the cargo environment variable - -```sh -source $HOME/.cargo/env -``` - -**4.** Get the sources - -```sh -git clone https://gitlab.com/tezos/tezos.git -``` - -```sh -cd tezos -``` - -```sh -git checkout latest-release -``` -**5.** Install the Tezos dependencies. - -```sh -opam init --bare -``` - -* You will be asked: ```do you want to modify ~/.profile? [N/y/f]```press ```n```. -* You will then be asked: ```A hook can be added to opam's init scripts to ensure that the shell remains in sync with the opam environment when they are loaded. Set that up? [y/N]```press ```n```. - -```sh -make build-deps -``` - -**6.** Compile sources ( This step can take a long time depending on your hardware ) - -```sh -eval $(opam env) -``` - -```sh -make -``` - -**7.** Get rolling snapshot - -```sh -wget https://mainnet.xtz-shots.io/rolling -O tezos-mainnet.rolling -``` - -**8.** Import the snapshot (This step can take a long time depending on your hardware) - -```sh -./tezos-node snapshot import tezos-mainnet.rolling -``` - -**9.** Run the node - -```sh -./tezos-node run --allow-all-rpc localhost:8732 --rpc-addr localhost:8732 --history-mode experimental-rolling -``` - -**Q&A** - -* How do I stop the node? - - * Press ```Ctrl-c```from within the terminal tab it is running in. Give it a minute or two, if it still does not work press it again. - -* Where is Tezos installed? - - * ```~/tezos``` - -* Where is the blockchain data stored? - - * ```~/.tezos-node``` - - * Note that the period in front of the directory denotes a hidden directory. Use ```ls -a```to show all files and directories including hidden directories. -* What do I do if my node data becomes corrupted and I need to re-import a new snapshot? - - * You will need to first remove the old data. To do that run ```rm -rf .tezos-node```from your home directory. - * Then from the ```tezos```directory use the same commands above to re-import new snapshots. - -**Updating the node** - -**1.** Pull updates from the git repo - -```sh -git pull -``` - -* This must be done in the tezos directory - -**2.** Install the latest dependencies and compile the sources - -```sh -make build-deps -``` - -```sh -eval $(opam env) -``` - -```sh -make -``` - -#### Option 3: Using packages on Ubuntu or Fedora - -```sh -sudo add-apt-repository ppa:serokell/tezos && sudo apt-get update -sudo apt-get install -y tezos-client -sudo apt-get install -y tezos-node -sudo apt-get install -y tezos-baker-010-ptgranad -sudo apt-get install -y tezos-endorser-010-ptgranad -sudo apt-get install -y tezos-accuser-010-ptgranad -``` - -### Windows - -Install one of Linux distributions using [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/about) (e.g. Ubuntu 18.04 LTS) and follow instructions for Linux. - -### Mac OS - -With [Homebrew](https://brew.sh/): - -```sh -$ brew tap serokell/tezos-packaging https://github.com/serokell/tezos-packaging.git -$ brew install tezos-client -``` - -`tezos-packaging` also provides prebuilt brew bottles for some macOS versions. - -## Configure - -We'll configure `tezos-client` to use a public test network Tezos node: - -```sh -$ tezos-client --endpoint https://rpcalpha.tzbeta.net config update -``` - -`--endpoint` parameter specifies the address of the server, `config update` writes it to `tezos-client`'s configuration filed at `$HOME/.tezos-client/config`. - -Alternatively, one can use an isolated sandboxed network instead of using a public test-network, which we'll do in the [“Sandbox”](./sandbox.md) section. - -### Try it out - -Verify that you can run tezos-client and that it points to test network: - -```sh -$ tezos-client -Warning: - - This is NOT the Tezos Mainnet. - - Do NOT use your fundraiser keys on this network. - -Usage: - tezos-client [global options] command [command options] - tezos-client --help (for global options) - tezos-client [global options] command --help (for command options) - -To browse the documentation: - tezos-client [global options] man (for a list of commands) - tezos-client [global options] man -v 3 (for the full manual) - -Global options (must come before the command): - -d --base-dir : client data directory - -c --config-file : configuration file - -t --timings: show RPC request times - --chain : chain on which to apply contextual commands (possible tags are 'main' and 'test') - -b --block : block on which to apply contextual commands (possible tags are 'head' and 'genesis') - -w --wait >: how many confirmation blocks before to consider an operation as included - -p --protocol : use commands of a specific protocol - -l --log-requests: log all requests to the node - -A --addr : [DEPRECATED: use --endpoint instead] IP address of the node - -P --port : [DEPRECATED: use --endpoint instead] RPC port of the node - -S --tls: [DEPRECATED: use --endpoint instead] use TLS to connect to node. - -E --endpoint : HTTP(S) endpoint of the node RPC interface; e.g. 'http://localhost:8732' - -R --remote-signer : URI of the remote signer - -f --password-filename : path to the password filename - -M --mode : how to interact with the node -``` - -Now that we know we are on a test network we can temporarily disable this warning so that we don't see it with each command. - -```sh -$ export TEZOS_CLIENT_UNSAFE_DISABLE_DISCLAIMER=yes -``` - -## Create Test Wallets - -> With tezos-client installed we can now make a couple of test wallets. - -* Go to the [faucet](https://faucet.tzalpha.net/). -* Complete the captcha and download the wallet in the form of a .json file. -* The file name is the wallet address (also known as the public key hash, or PKH) with a .json extension. -* If you look inside the file you will see a property called *pkh* which contains the address. -* The public key hash is used to identify the account on the Tezos blockchain and can be thought of as an address or account number. -* Next, we'll activate the account, passing it the path to the `.json` wallet we just downloaded -* We'll first create an account for Alice, then Bob, so we can perform some test transactions. -* Don't be alarmed by the blank `Error:` we'll explain why shortly. - -```sh -$ tezos-client activate account alice with ~/Downloads/tz1QLne6uZFxPRdRfJG8msx5RouENpJoRsfP.json -Node is bootstrapped, ready for injecting operations. -Operation successfully injected in the node. -Operation hash is 'oog2gMSBNWWTgHujoKViJaCed4wq27gPnLpHKQ27C5savX9ewAq' -Waiting for the operation to be included... -Error: - -``` - -> Note empty `Error:` message at the end of the output. `tezos-client` attempts to wait for operation inclusion but the public Tezos node we are using disallows access (as a security measure) to the part of node api that is necessary for this functionality. We use `--wait none` throughout these tutorials to avoid this error. If you use your own local Tezos node you can omit `--wait none`, the error won't happen. - -Make a variable for Alice's account address (PKH) (notice that the address is the same as the name of your faucet `.json` file): - -```sh -$ ALICE_ADDRESS="tz1QLne6uZFxPRdRfJG8msx5RouENpJoRsfP" -``` - -Ensure that the activation was successful: - -```sh -$ tezos-client get balance for $ALICE_ADDRESS -56828.546322 ꜩ -``` - -Now, we'll create a new wallet for Bob so we can do some test transactions in the next section. - -* Go back to the [faucet](https://faucet.tzalpha.net/) and download a second wallet - -```sh -$ tezos-client activate account bob with tz1ZQYMDETodNBAc2XVbhZFGme8KniuPqrSw.json -$ BOB_ADDRESS="tz1ZQYMDETodNBAc2XVbhZFGme8KniuPqrSw" -$ tezos-client get balance for $BOB_ADDRESS -``` - diff --git a/docs/reference/octez/monitoring.md b/docs/reference/octez/monitoring.md deleted file mode 100644 index 51fd64a92..000000000 --- a/docs/reference/octez/monitoring.md +++ /dev/null @@ -1,463 +0,0 @@ ---- -title: Monitoring nodes -authors: Jean-Baptiste Col -lastUpdated: 29th June 2023 ---- - -This section will guide you through the steps to set up a monitoring tool that will allow you to track various metrics of an Octez node. - -Until now, the only tool developers had to monitor the behavior of their Tezos node was to look at the logs, [adjust the log verbosity](https://tezos.gitlab.io/user/logging.html?highlight=log#node-specific-configuration), and reconstruct all relevant information from this stream. But getting more insight into a node's performance was tedious and difficult. For instance, the number of connected peers, the number of pending operations, or the number of times the validator switched branches, were not easy to observe continuously. - -After a few iterations of different methods to gather node information and statistics that could be easily analyzed, we have recently chosen to include metrics within the node. With Octez Metrics it's simple to get a myriad of statistics about your node -- and quite efficiently so. You can also attach a [Grafana](https://grafana.com/) dashboard to get a visual representation of how your node is performing. And with [Grafazos](https://gitlab.com/nomadic-labs/grafazos), you can get customized ready-to-use dashboards for monitoring your Tezos node. - -What metrics can we monitor ? - -There are two types of metrics on an Octez node that can be monitored: - -- hardware metrics -- node metrics - -[Here](https://tezos.gitlab.io/developer/openmetrics.html) is an exhaustive list of Octez node metrics. Hardware metrics depend on your machine characteristics, but we will focus on performance metrics like the CPU usage, RAM, memory, etc. - -## Overview of monitoring tools - -### [Netdata](https://www.netdata.cloud/) - -Netdata is a light and open-source software that collects and exposes hardware metrics of a physical machine, and is capable of collecting fresh data every second. Netdata is a complete tool that provides various graphics visualization (histogram, tables, gauge, points), alerting + triggering tools, and many other features. - -### [Grafana](https://grafana.com/) - -Grafana is software that takes JSON in input and makes dashboards that you can display using your browser (the high-level web interface that displays all the metrics). In our case, JSONs are provided by Grafazos. - -### [Grafazos](https://gitlab.com/nomadic-labs/grafazos) -Grafazos is a jsonnet library written by Nomadic Labs, which uses itself [grafonet-lib](https://github.com/grafana/grafonnet-lib), which is a jsonnet library to write Grafana dashboards as code. - -### [jsonnet](https://jsonnet.org/) - -Jsonnet is a programming language that allows to create JSONs easily. - -### [Prometheus](https://prometheus.io/docs/introduction/overview/#what-is-prometheus) - -Prometheus server is a toolkit that scrapes and stores time series data by making requests to the nodes. Basically, prometheus is fed by both netdata and tezos-metrics (which is deprecated and not used), and then, grafana displays the data gathered by prometheus. - - -## Light monitoring of an Octez node with Netdata - -### Set up the monitoring - -#### Step 1: Install Netdata on your node host - -We recommend to look at [Install on Linux with one-line installer](https://learn.netdata.cloud/docs/get-started#install-on-linux-with-one-line-installer) to make a personalised installation. Else, copy the following command: - -```bash -wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh -``` - -#### Step 2: Add the collector plugin for Octez metrics - -This plugin scrapes the metrics of your octez node to display the metrics with Netdata. You have the choice between bash, python or go plugins. The best performing plugin is the plugin coded in go. However, if you want to modify a plugin to your liking, choose the language that suits you best. The plugin must be placed in a specific directory as described below: - -**Bash plugin:** - -Create a file named `tezosMetrics.chart.sh` in `/usr/libexec/netdata/charts.d` and paste the following bash code into the file. - -```bash -#!/bin/bash - -url=http://127.0.0.1:9091/metrics #Octez metrics' are avaible on port 9091 by default. You can changer the url variable if you haven't the same port. -tezosMetrics_update_every=1 -tezosMetrics_priority=1 -lastTimestamp= - - -tezosMetrics_get() { -response=$(curl $url 2>/dev/null) -#With this following command you get all metrics of your node except octez_version{...} -eval "$(echo "$response" | grep -v \# | grep -v octez_version | sed -e 's/[.].*$//g' | sed -e 's/", /_/g'| sed -e 's/{/_/g' | sed -e 's/="/_/g' | sed -e 's/"}//g'| sed -e 's/ /=/g'| sed -e 's///g'| sed -e 's/\//_/g')" -#this command allows to catch "octez_version" metric -eval "$(echo "$response" | grep -v \# | grep octez_version | sed -e 's/, / \n/g' | sed -e 's/{/\n/g'| sed -e 's/}//g' | sed -e 's/ 0.*//g' |tail -n 6 |sed -e 's/^/octez_/')" -return 0 -} - - - -#The following function is called once at netdata startup to check the availability of metrics -tezosMetrics_check() { - status=$(curl -o /dev/null -s -w "%{http_code}\n" $url) - [ $status -eq 200 ] || return 1 - return 0 -} - - - -#The following function is called once at netdata startup to create the charts -tezosMetrics_create() { -response=$(curl $url 2>/dev/null) -metrics=$(echo "$response" | grep -v \# | grep -v octez_version | sed -e 's/", /_/g'| sed -e 's/{/_/g' | sed -e 's/="/_/g' | sed -e 's/"}//g'| sed -e 's/[ ].*$//g'| sed -e 's///g'| sed -e 's/\//_/g') -#CHART type.id name title units family context charttype priority update_every options plugin module -#DIMENSION id name algorithm multiplier divisor options -#More details here: https://learn.netdata.cloud/docs/agent/collectors/plugins.d -for metric in $metrics #this loop creates the charts and the dimensions of all metrics except octez_version metric (because of its format is a bit different) -do -cat << EOF -CHART tezosMetrics.$metric '' "$metric" "" tezosMetricsFamily tezosMetricsContext line $((tezosMetrics_priority)) $tezosMetrics_update_every '' '' 'tezosMetrics' -DIMENSION $metric absolute 1 1 -EOF -done -#The code below creates the chart for octez_version -eval "$(echo "$response" | grep -v \# | grep octez_version | sed -e 's/, / \n/g' | sed -e 's/{/\n/g'| sed -e 's/}//g' | sed -e 's/ 0.*//g' |tail -n 6 |sed -e 's/^/octez_/')" -cat << EOF -CHART tezosMetrics.octez_version '' 'node_version: $octez_version | octez_chain_name: $octez_chain_name | octez_distributed_db_version: $octez_distributed_db_version | octez_p2p_version: $octez_p2p_version | octez_commit_hash: $octez_commit_hash | octez_commit_date: $octez_commit_date' '' tezosMetricsFamily tezosMetricsContext line $((tezosMetrics_priority)) $tezosMetrics_update_every '' '' 'tezosMetrics' -DIMENSION 'octez_version' 'node_version: $octez_version | octez_chain_name: $octez_chain_name | octez_distributed_db_version: $octez_distributed_db_version | octez_p2p_version: $octez_p2p_version | octez_commit_hash: $octez_commit_hash | octez_commit_date: $octez_commit_date' absolute 1 1 -EOF -return 0 -} - - - -#The following function update the values of the charts -tezosMetrics_update() { -tezosMetrics_get || return 1 -for metric in $metrics -do -cat << VALUESOF -BEGIN tezosMetrics.$metric $1 -SET $metric = $[$metric] -END -VALUESOF -done -#The code below updates the octez_version metric chart -cat << EOF -BEGIN tezosMetrics.octez_version $1 -SET octez_version = 0 -END -EOF - -return 0 -} -``` - -#### Step 3: Open the metrics port of your node - -When starting your node, add the `--metrics-addr=:9091` option to open a port for the Octez metrics. (By default the port is 9091, but you can arbitrarily choose any other one, at your convenience. Just make sure to change the port in the plugin code accordingly.). - -Here is an example, when launching your Tezos node: - -```bash -octez-node run --rpc-addr 127.0.0.1:8732 --log-output tezos.log --metrics-addr=:9091 -``` - -#### Step 4: Restart the Netdata server - -Once you've gotten the plugin on your machine, restart the Netdata server using: - -```bash -sudo systemctl restart netdata -``` - -#### Step 5: Open the dashboard - -You have in fact two ways to [open your dashboard](https://learn.netdata.cloud/docs/dashboard/how-dashboard-works#open-the-dashboard): - -- You can access Netdata's dashboard by navigating to http://NODE:19999 in your browser (replace NODE by either localhost or the hostname/IP address of a remote node) - -- You can also use [Netdata Cloud](https://app.netdata.cloud/) to create custom dashboards, monitor several nodes, and invite users to watch your dashboard... It's free of charges and your data are never stored in a remote cloud server, but rather in the local disk of your machines. - -We recommend to use Netdata Cloud for a more user-friendly usage. The rest of this tutorial is based on Netdata Cloud. - -#### Step 6: Create a custom dashboard with Netdata Cloud! - -Netdata provides a really intuitive tool to create custom dashboards. If you wants more details, read [Netdata documentation](https://learn.netdata.cloud/docs/cloud/visualize/dashboards). - -[Here](https://tezos.gitlab.io/developer/openmetrics.html) is an exhaustive list of Octez node metrics. With Netdata, your node metrics will have this format: "tezosMetrics.name_of_your_metric". - -For instance you can monitor the following relevant metrics: -- `octez_version` to get the chain name, version of your node, the Octez commit date, etc. -- `octez_validator_chain_is_bootsrtapped` to see if your node is bootstrapped (it returns 1 if the node is bootstrapped, 0 otherwise). -- `octez_p2p_connections_outgoing` which represents the number of outgoing connections. -- `octez_validator_chain_last_finished_request_completion_timestamp` which is the timestamp at which the latest request handled by the worker was completed. -- `octez_p2p_peers_accepted` is the number of accepted connections. -- `octez_p2p_connections_active` is the number of active connections. -- `octez_store_invalid_blocks` is the number of blocks known to be invalid stored on the disk. -- `octez_validator_chain_head_round` is the round (in the consensus protocol) of the current node’s head. -- `ocaml_gc_allocated_bytes` is the total number of bytes allocated since the program was started. -- `octez_mempool_pending_applied` is the number of pending operations in the mempool, waiting to be applied. - -You can also monitor relevant hardware data (metrics names may differ depending on your hardware configuration): - -- `disk_space._` to get the remaining free space on your disks. -- `cpu.cpu_freq` to get the frequency of each core of your machine. -- `system.ram` to get the remaining free RAM and it's overall usage. - -Plenty of others metrics are available, depending on your machine and needs. - -This is an example of a Demo Dashboard with the following metrics: -- `octez_version` -- `octez_validator_chain_is_bootstrapped` -- `Disk Space Usage` -- `CPU Usage` -- `RAM Usage` -- `octez_p2p_connections_active` -- `octez_p2p_peers_running` -- `octez_store_last_written_block_size` - -![Example of a light monitoring dashboard](/img/reference/netdata_dashboard.png) - -### Monitoring several nodes - -#### Step1. Install Netdata on each of the machines hosting your nodes as described in #### Step1 of "Set up the monitoring" part. - -#### Step2. On Netdata Cloud click on the button `Connect Nodes`, and follow the instructions. - -### Managing alerts - -Netdata sets up alerts by default for some predefined metrics which may not fit your needs. You can set up personalized ones to be received automatically by email, Slack, or SMS. - -#### How to reach config files? - -Go to `/etc/netdata` and use the command `edit-config` to edit config files. - -For example, to edit the cpu.conf health configuration file, run: - -```bash -cd /etc/netdata -sudo ./edit_config health.d/cpu.conf -``` - -To list config files available just use ./edit-config command. Alert file names have this format: `health.d/file_name.conf` - -(Configuration files are stored in `/usr/lib/netdata/conf.d` and alerts in `health.d` folder) - -#### Manage existing alerts: Edit health configuration files - -You can manage existing alerts by [editing health configuration file](https://learn.netdata.cloud/docs/monitor/configure-alarms#edit-health-configuration-files). - -For example, here is the first health entity in `health.d/cpu.conf`: - -``` -template: 10min_cpu_usage #Name of the alarm/template.(required) - on: system.cpu #The chart this alarm should attach to.(required) - os: linux #Which operating systems to run this chart. (optional) - hosts: * #Which hostnames will run this alarm.(optional) - lookup: average -10m unaligned of user,system,softirq,irq,guest #The database lookup to find and process metrics for the chart specified through on.(required) - units: % - every: 1m #The frequency of the alarm. - #"warn" and "crit" expressions evaluating to true or false, and when true, will trigger the alarm. - warn: $this > (($status >= $WARNING) ? (75) : (85)) - crit: $this > (($status == $CRITICAL) ? (85) : (95)) - delay: down 15m multiplier 1.5 max 1h #Optional hysteresis settings to prevent floods of notifications. - info: average cpu utilization for the last 10 minutes (excluding iowait, nice and steal) - to: sysadmin -``` - -[Here](https://learn.netdata.cloud/docs/agent/health/reference#entity-format) is an exhaustive liste of alerts parameters. - -#### Create personnalised alerts: Write a new health entity - -To create personalised alerts, create a `.conf` file in `/usr/lib/netdata/conf.d/health.d` and edit this new file using the template shown previously: - -As an example, here is a warning alert if the metric octez_p2p_connections_active is below 3 connections, and a critical alert if there are no connections at all (equal to 0): - -``` -alarm: tezos_node_p2p_connections_active (cannot be chart name, dimension name, family name, or chart variables names.) - on: octez_p2p_connections_active - hosts: * - lookup: average -10m unaligned of user,system,softirq,irq,guest - warn: $this<3 - crit: $this==0 - info: "tezos_node_p2p_connections_actives" represents the current number of active p2p connections with your node. - to: sysadmin -``` - -(You can access official Netdata documentation [here](https://learn.netdata.cloud/docs/monitor/configure-alarms#write-a-new-health-entity)) - - -### Documentation on managing alerts - -- [Basic documentation for configuring health alarms](https://learn.netdata.cloud/docs/monitor/configure-alarms) -- [Health configuration reference (detailed documentation)](https://learn.netdata.cloud/docs/agent/health/reference) -- [Supported notification endpoints](https://learn.netdata.cloud/docs/monitor/enable-notifications#supported-notification-endpoints) - - -### Manage data retention - -The metrics collected every second by Netdata can be stored in the local memory of your machine for a variable duration of time, depending on the desired history. This allows you to keep track of the activity of your node and to watch its evolution over time. - -#### Get netdata.conf file - -You can choose how long data will be stored on your local machine. To do so, you will need the `netdata.conf` file in the `/etc/netdata/` repository. If you haven't this file yet, you can download it using the following command: - -```bash -wget -O /etc/netdata/netdata.conf http://localhost:19999/netdata.conf -``` - -You can view your current configuration at: `http://localhost:19999/netdata.conf` - -#### Calculate the data base space needed - -To calculate the space needed to store your metrics you will need: - -- The time in seconds of how long time you want to keep the data -- The number of metrics (you can find it in your localhost dashboard as shown in the image below) -- The Tier of Netdata (by default 0). - -For 2000 metrics, collected every second and retained for a month, Tier 0 needs: 1 byte x 2,000 metrics x 3,600 secs per hour x 24 hours per day x 30 days per month = 5,184MB. - -![Number of metrics location in the dashboard](/img/reference/number-of-metrics.jpeg) - -#### Modify netdata.conf file - -Now you just have to change the value of "dbengine multihost disk space" in [db] section in netdata.conf file by the value calculated before (in MB). - -``` -[db] - dbengine multihost disk space MB =5184 - dbengine page cache size MB = 32 - # update every = 1 - # mode = dbengine - # dbengine page cache with malloc = no - # dbengine disk space MB = 256 - # memory deduplication (ksm) = yes - # cleanup obsolete charts after secs = 3600 - # gap when lost iterations above = 1 - # storage tiers = 1 - # dbengine page fetch timeout secs = 3 - # dbengine page fetch retries = 3 - # dbengine page descriptors in file mapped memory = no - # cleanup orphan hosts after secs = 3600 - # delete obsolete charts files = yes - # delete orphan hosts files = yes - # enable zero metrics = no - # dbengine pages per extent = 64 -``` - -#### Documentation about data retention - -- [Netdata parameters of metrics retention / Tiering](https://learn.netdata.cloud/guides/longer-metrics-storage/) -- [Database overview](https://learn.netdata.cloud/docs/agent/database) -- [The database engine / Tiering](https://learn.netdata.cloud/docs/agent/database/engine) -- [Change how long Netdata stores metrics](https://learn.netdata.cloud/docs/store/change-metrics-storage) -- [Daemon configuration](https://learn.netdata.cloud/docs/agent/daemon/config#global-section-options) - -## Full monitoring with Octez Metrics - -Here is the global picture of a monitoring system, connecting all these tools together: - -![](/img/reference/all-in-all.png) - -A Grafazos dashboard looks like this: - -![](/img/reference/octez-metrics-dashboard.gif) - -Table 1: Grafana dashboard of a Tezos node - -As you can immediately see at the top, the dashboard will tell you your node's bootstrap status and whether it's synchronized, followed by tables and graphs of other data points. - - -### Setting up Octez Metrics - -To use Octez Metrics, you just start your node with the metrics server enabled. The node integrates a server that registers the implemented metrics and outputs them for each ``/metrics`` HTTP request. - -When you start your node you add the ``--metrics-addr`` option which takes as a parameter ```` or ```` or ``:``. This option can be used either when starting your node, or in the [configuration file](https://tezos.gitlab.io/user/node-configuration.html). - -Your node is now ready to have metrics scraped with requests to the metrics server. For instance, if the node server is configured to expose metrics on port 9932 (the default), then you can scrape the metrics with the request ``http://localhost:9932/metrics``. -The result of the request is the list the node metrics described as: - -``` -#HELP metric description -#TYPE metric type -octez_metric_name{label_name=label_value} x.x -``` - -Note the metrics are implemented to have the lowest possible impact on the node performance, and most of the metrics are only computed when scraping it. So starting the node with the metrics server shouldn't be a cause for concern. More details on Octez Metrics can be found in the Tezos Developer Documentation: [see here](https://tezos.gitlab.io/user/node-monitoring.html) for further detail on how to setup your monitoring; and [here](https://tezos.gitlab.io/developer/openmetrics), for the complete list of the metrics scrapped by the Octez node. - -### Types of metrics - -The available metrics give a full overview of your node, including its characteristics, status, and health. In addition, they can give insight into whether an issue is local to your node, or it is affecting the network at large -- or both. - -The metric `octez_version` delivers the node's main properties through label-value pairs. It provides the node version or the network it is connected to. - -Other primary metrics you likely want to see are the *chain validator*[^1] ones, which describe the status of your node: `octez_validator_chain_is_bootstrapped` and `octez_validator_chain_synchronisation_status`. A healthy node should always have these values set to 1. You can also see information about the *head* and *requests* from the chain validator. - -There are two other validators, the *block validator*[^2] and the *peer validator*[^3], which give you insight on how your node is handling the progression of the chain. You can learn more about the validators [here](https://tezos.gitlab.io/shell/validation.html) - -To keep track of pending operations, you can check the `octez_mempool` metric. - -You can get a view of your node's connections with the *p2p layer* metrics (prefixed with `octez_p2p`). These metrics allows you to keep track of the *connections*, *peers* and *points* of your node. - -The [store](https://tezos.gitlab.io/shell/storage.html) can also be monitored with metrics on the save-point, checkpoint and caboose level, including the number of invalid blocks stored, the last written block size, and the last store merge time. - -Finally, if you use the RPC server of your node, it is likely decisive in the operation of your node. -For each RPC called, two metrics are associated: `octez_rpc_calls_sum{endpoint="...";method="..."}` and `octez_rpc_calls_count{endpoint="...";method="..."}` (with appropriate label values). *call_sum* is the sum of the execution times, and *call_count* is the number of executions. - -1. The chain validator is responsible for handling valid blocks and selecting the best head for its chain. -2. The block validator validates blocks and notifies the corresponding chain validator. -3. Each peer validator treats new head proposals from its associated peer, retrieving all the operations, and if valid, triggers a validation of the new head. - -:::note Octez version -Note that the metrics described here are those available with Octez v14--it is likely to evolve with future Octez versions. - -### Dashboards - -While scraping metrics with server requests does give access to node metrics, it is, unfortunately, not enough for useful node monitoring. Since it only gives a single slice into the node's health, you don't really see what's happening over time. Therefore, a more useful way of monitoring your node is to create a time series of metrics. - -Indeed, if you liked the poster, why not see the whole movie? - -The [Prometheus](https://prometheus.io/) tool is designed for this purpose which collects metric data over time. - -In addition, in order to get the most out of your metrics, it should be associated with a visual dashboard. -A [Grafana](https://grafana.com/) dashboard, generated by [Grafazos](https://gitlab.com/nomadic-labs/grafazos) gives a greater view into your node. Once your node is launched, you can provide extracted time series of metrics to Grafana dashboards. - -The Grafazos version for Octez v14 provides the following four ready-to-use dashboards: - -- `octez-compact`: A compact dashboard that gives a brief overview of the various node metrics on a single page. -- `octez-basic`: A basic dashboard with all the node metrics. -- `octez-with-logs`: Same as basic but also displays the node's logs, with [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) Promtail (for exporting the logs). -- `octez-full`: A full dashboard with the logs and hardware data. This dashboard should be used with [Netdata](https://www.netdata.cloud/) (for supporting hardware data) in addition to Promtail. - -Note that the last two dashboards require the use of additional (though standard) tools for hardware metrics and logs (Netdata, Loki, and Promatail). - -Let's look at the *basic* dashboard in more detail. The dashboard is divided into several panels. The first one is the *node panel*, which can be considered the main part of the dashboard. This panel lays out the core information on the node such as its status, characteristics, and statistics on the node's evolution (head level, validation time, operations, invalid blocks, etc.). - -The others panels are specific to different parts of the node: - -- the p2p layer; -- the workers; -- the RPC server; - -along with a miscellaneous section. - -Some metrics are self-explanatory, such as *P2P total connections*, which shows both the connections your node initiated and the number of connections initiated by peers. Another metric you may want to keep an eye on is *Invalid blocks history*, which should always be 0 -- any other value would indicate something unusual or malicious is going on. - -Another useful metric is the *Block validation time*, which measures the time between when a request is registered in the worker till the worker pops the request and marks it complete. This should generally be under 1 second. If it's persistently longer, that could indicate trouble too. - -![](/img/reference/metrics-block-validation-time.png) - -Graph 2: Block validation time - - -The *P2P connections* graph will show you immediately if your node is having trouble connecting to peers, or if there's a drop-off in the number of connections. -A healthy node should typically have a few dozen peer connections (depending on how it was configured). - - -![](/img/reference/metrics-p2p-connections.png) - -Graph 3: P2P connections - -The *Peer validator* graph shows a number of different metrics including *unavailable protocols*. An up-to-date, healthy node should see this as a low number. If not it can indicate that your node is running an old version of Octez, or that your node is being fed bad data from peers. - -*Note again these dashboards are built for Octez v14 and are likely to evolve with the Octez versions*. - -### Working with Grafazos - -Grafazos allows you to set different options when generating the ready-to-use dashboards described above. For instance, you can specify the ``node instance label``, which is useful for a dashboard that aims to monitor several nodes. - -Furthermore, you can manually explore the metrics from the Prometheus data source with Grafana and design your own dashboards. -Or you can also use Grafazos to import ready-to-use dashboards for your node monitoring. You can find the packages stored [here](https://gitlab.com/nomadic-labs/grafazos/-/packages). There is a package for each version of Octez. - -Grafana is a relatively user-friendly tool, so play with creating a custom one as you like. You may also want to use the "explore" section of Grafana. Grafazos is also particularly useful in automatic deployment of Tezos nodes via provisioning tools such as Puppet or Ansible. - -## Conclusion - -Octez Metrics gives Tezos users insight into how their node is performing, and ability to observe the overall network health. The best way to keep your node healthy and keep the entire Tezos network healthy is with monitoring tools to gain insite into network health. - diff --git a/docs/reference/octez/node-cluster.md b/docs/reference/octez/node-cluster.md deleted file mode 100644 index 62e51110d..000000000 --- a/docs/reference/octez/node-cluster.md +++ /dev/null @@ -1,339 +0,0 @@ ---- -title: Deploying a cluster of nodes using Pulumi -authors: 'Oxhead Alpha, Daniel, Nomadic Labs' -lastUpdated: 14th June 2023 ---- - -**(Thank you to [Oxhead Alpha](https://www.oxheadalpha.com/) for coding this great tool)** - -Deploying a Tezos node is a good way to address certain needs when working with the Tezos blockchain. It can be used to broadcast operations or to query specific information such as the latest block of the chain or the storage of a particular smart contract. - -But when working with decentralized applications (dApps), the node can become a bottleneck. - -The more people interact with a decentralized application, the more queries the node has to handle, and **a single node is not suited to answer hundreds or thousands of requests per second**. - -In those situations, it becomes necessary to set up an infrastructure that can handle the load – by deploying a **cluster of nodes**. - -This tutorial, inspired by [Oxhead Alpha's medium article](https://medium.com/the-aleph/deploy-scalable-tezos-nodes-in-the-cloud-bbe4f4f4ddcc), aims to be a practical guide to deploying your own cluster of Tezos nodes using AWS cloud’s managed Kubernetes offering (EKS). For further technical explanations, we refer you to the above-mentioned article. - -For our setup, we will be using [Pulumi](https://www.pulumi.com/), an amazingly flexible infrastructure-as-code (IaC) tool. At the end of the tutorial, we will expose the nodes’ RPC endpoint to the internet via a load balancing system. - -The resulting infrastructure is highly available, scalable, customizable, and supports rolling upgrades (a must-have, given Tezos' amendment pace). The tool enables us to scale the number of Tezos nodes as desired (0 to n). - -To consult the associated code on Github, please check [this link](https://github.com/oxheadalpha/tezos-pulumi) out. - -# Prerequisites - -- [AWS account and CLI tool](https://aws.amazon.com/cli/?nc1=h_ls) -- [Pulumi account and CLI tool](https://www.pulumi.com/docs/get-started/install/) -- [Nodejs](https://nodejs.org/en/download/package-manager/) -- [kubectl](https://kubernetes.io/docs/tasks/tools/) -- [k9s](https://github.com/derailed/k9s) -- [Lens](https://k8slens.dev/) (used for the monitoring) - -Follow [the Pulumi tutorial](https://www.pulumi.com/docs/get-started/aws/begin/) to create an AWS account if you don’t already have one, and install Pulumi. *Your AWS IAM user must have the ability to create a VPC and EKS cluster*. - -# 1 Create a Pulumi project - -Create a Pulumi project using the following lines: - -```bash -mkdir tezos-aws && cd tezos-aws -pulumi new aws-typescript -``` - -:::info -You may have to execute the following command, before executing any Pulumi interaction: -```sh -export PATH="$PATH:~/.pulumi/bin" -``` -::: - -# 2 Tezos Pulumi installation - -Install the tezos-pulumi node module: - -```bash -npm install @oxheadalpha/tezos-pulumi --save-exact -``` - -Below are the most important files, which we need to take a careful look at: - -- **values.yaml**: file that specifies what we’d like the tezos-k8s Helm chart to deploy inside k8s. You may see [here](https://github.com/oxheadalpha/tezos-k8s/blob/master/charts/tezos/values.yaml) for the full tezos-k8s values spec but what we provide here is sufficient for the tutorial. - -- **index.ts**: Typescript file which will deploy the Tezos infrastructure. It is very customizable as we can easily write IaC (infrastructure as code) in languages like Typescript. - -## 2.1 Populate the *value.yaml* file - -Create a values.yaml file with the code provided below: - -```yml -# Define the types of Tezos nodes we want to deploy in our cluster -nodes: - rolling-node: - storage_size: 100Gi - # We create a new k8s storage class in the Pulumi index.ts file that allows - # us to expand an EBS volume's size. The default gp2 storage class created - # by EKS does not allow for volume expansion. We tell our `rolling-nodes` to - # use this storage class. - storageClassName: "gp2-volume-expansion" - # Run the Octez implementation of Tezos node, as opposed to Tezedge's - # implementation. - runs: - - octez_node - # Create 2 Tezos rolling nodes that will be distributed across the 2 cluster - # EC2 nodes we will be deploying. - instances: - - config: - shell: - history_mode: rolling - - config: - shell: - history_mode: rolling - -# Have the nodes download and use a tarball of a mainnet rolling node -# filesystem. This allows the nodes to sync much faster with the head of the -# chain than if they started from the genesis block. -rolling_tarball_url: https://mainnet.xtz-shots.io/rolling-tarball -# The default tezos-k9s Helm chart values.yaml uses this field by default. So we -# set it to null here to disable it and use the tarball url instead. Syncing a -# node from a tarball is much faster than a Tezos snapshot. -rolling_snapshot_url: null -``` - -## 2.2 Populate the *index.ts* file - -Paste the following code in the index.ts file generated by Pulumi, overwriting the Pulumi generated code. Read the comments to get an understanding of what is happening: - -```ts -import * as awsx from "@pulumi/awsx" -import * as eks from "@pulumi/eks" -import * as k8s from "@pulumi/kubernetes" -import * as pulumi from "@pulumi/pulumi" -import * as tezos from "@oxheadalpha/tezos-pulumi" - -/** https://www.pulumi.com/docs/intro/concepts/project/ */ -const project = pulumi.getProject() -/** https://www.pulumi.com/docs/intro/concepts/stack/ */ -const stack = pulumi.getStack() - -const projectStack = `${project}-${stack}` - -/** Create a vpc to deploy your k8s cluster into. By default the vpc will use - * the first 2 availability zones in the region. Public and private subnets will - * be created in each zone. Private, for cluster nodes, and public for - * internet-facing load balancers. - */ -const vpc = new awsx.ec2.Vpc( - projectStack, - { - subnets: [ - // Tag subnets for specific load-balancer usage. - // Any non-null tag value is valid. - // See: - // - https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html - // - https://github.com/pulumi/pulumi-eks/issues/196 - // - https://github.com/pulumi/pulumi-eks/issues/415 - { type: "public", tags: { "kubernetes.io/role/elb": "1" } }, - { type: "private", tags: { "kubernetes.io/role/internal-elb": "1" } }, - ], - }, - { - // Inform pulumi to ignore tag changes to the VPCs or subnets, so that - // tags auto-added by AWS EKS do not get removed during future - // refreshes and updates, as they are added outside of pulumi's management - // and would be removed otherwise. - // See: https://github.com/pulumi/pulumi-eks/issues/271#issuecomment-548452554 - transformations: [ - (args: any) => { - if (["aws:ec2/vpc:Vpc", "aws:ec2/subnet:Subnet"].includes(args.type)) { - return { - props: args.props, - opts: pulumi.mergeOptions(args.opts, { ignoreChanges: ["tags"] }), - } - } - return - }, - ], - } -) - -/** Stack outputs: https://www.pulumi.com/learn/building-with-pulumi/stack-outputs/ */ -export const vpcId = vpc.id -export const vpcPublicSubnetIds = vpc.publicSubnetIds -export const vpcPrivateSubnetIds = vpc.privateSubnetIds - -/** Create the EKS cluster. The cluster will be created in the new vpc. The - * autoscaling group will spin up 2 cluster nodes (EC2 instances) where they - * will be distributed across our 2 private subnets. Each subnet is in 1 of 2 - * vpc zones. - */ -const cluster = new eks.Cluster(projectStack, { - vpcId: vpc.id, - publicSubnetIds: vpc.publicSubnetIds, - privateSubnetIds: vpc.privateSubnetIds, - // At time of writing we found this instance type to be adequate - instanceType: "t3.large", - // Set `minSize` and `desiredCapacity` to 0 if you ever want to pause your - // cluster's workload. - minSize: 2, - desiredCapacity: 2, -}) - -/** Stack outputs: https://www.pulumi.com/learn/building-with-pulumi/stack-outputs/ */ -export const clusterName = cluster.eksCluster.name -export const clusterId = cluster.eksCluster.id -export const clusterVersion = cluster.eksCluster.version -export const clusterStatus = cluster.eksCluster.status -export const kubeconfig = pulumi.secret(cluster.kubeconfig) - -/** - * The default gp2 storage class on EKS doesn't allow for volumes to be - * expanded. Create a storage class here that allows for expansion. - * - * https://www.jeffgeerling.com/blog/2019/expanding-k8s-pvs-eks-on-aws - */ -const gp2ExpansionStorageClass = new k8s.storage.v1.StorageClass( - "gp2-volume-expansion", - { - provisioner: "kubernetes.io/aws-ebs", - allowVolumeExpansion: true, - parameters: { - type: "gp2", - fsType: "ext4", - }, - volumeBindingMode: "WaitForFirstConsumer", - reclaimPolicy: "Delete", - metadata: { - name: "gp2-volume-expansion", - }, - }, - { provider: cluster.provider, parent: cluster } -) - -/** We will use the cluster instance role as the default role to attach policies - * to. In our tutorial, the only policy will be the alb controller policy. */ -const clusterInstanceRoles = cluster.instanceRoles.apply((roles) => roles) -const defaultIamRole = clusterInstanceRoles[0] - -/** - * Deploy the AWS loadbalancer controller to manage the creation of the load - * balancers that expose your Tezos node. An application load balancer will be - * created for the RPC ingress. The IAM policy created for the controller is - * attached to the default cluster node role. - * - * https://github.com/kubernetes-sigs/aws-load-balancer-controller - */ -const albController = new tezos.aws.AlbIngressController( - { - clusterName: cluster.eksCluster.name, - iamRole: defaultIamRole, - }, - { provider: cluster.provider, parent: cluster } -) - -const namespace = "mainnet" -/** Create the k8s namespace to deploy resources into */ -const mainnetNamespace = new k8s.core.v1.Namespace( - namespace, - { metadata: { name: namespace } }, - { provider: cluster.provider, parent: cluster } -) - -/** Deploy the tezos-k8s Helm chart into the mainnet namespace. This will create - * the Tezos rolling node amongst other things. */ -const helmChart = new tezos.TezosK8sHelmChart( - `${namespace}-tezos-aws`, - { - namespace, - // The path to a Helm values.yaml file - valuesFiles: "./values.yaml", - // The latest tezos-k8s version as of the time of this writing. - version: "6.0.1", - }, - { - provider: cluster.provider, - parent: mainnetNamespace, - } -) - -/** Create the RPC ingress to expose your node's RPC endpoint. The alb - * controller will create an application load balancer. */ -const rpcIngress = new tezos.aws.RpcIngress( - `${namespace}-rpc-ingress`, - { metadata: { name: `${namespace}-rpc-ingress`, namespace } }, - { - provider: cluster.provider, - dependsOn: albController.chart.ready, - parent: mainnetNamespace, - } -) -``` - -# 3 Deployment - -Now it's time to deploy our Tezos infrastructure! - -Run the following command to launch your cluster: - -```bash -pulumi up -``` - -:::caution -You may probably have to execute the following command, before launching the cluster. Chose your prefered AWS region (`us-east-2` - for Ohio, `eu-central-1` for Frankfurt, etc.): -```sh -pulumi config set region -``` -::: -Pulumi will display a url that you can visit to view the status of your deployment on the Pulumi web console. - -# 4 Monitoring - -In order to monitor the cluster liveness, we have chosen [Lens](https://k8slens.dev/) (An IDE for Kubernetes) as a monitoring tool. - -Execute the following command to provide the kubeconfig file to Lens: - -```sh -pulumi stack output kubeconfig --show-secrets --json > kubeconfig.json -``` - -Now, you can graphically monitor your cluster liveness (in our case, our 3 rolling nodes): - -

- -![lens](/img/reference/lens.png) -Lens monitoring -

- -Note that by clicking on the load balancer, you will find its url. Let’s curl it in a terminal to get the Mainnet chain id: - -```sh -curl http://
/chains/main/chain_id -"NetXdQprcVkpaWU" -``` - -:::info -The `
` RPC endpoint is a URL generated by AWS. It is possible to deploy the RPC behind a specific domain name. This would require buying or importing a domain and configuring it with [Route53](https://aws.amazon.com/route53/). -::: - -:::info -[k9s](https://github.com/derailed/k9s) can also be used, instead of Lens, to monitor your cluster via CLI. -As above, the following lines allow to provide the config and launch k9s: -``` -# Get the cluster's kubeconfig and save it to a file - -pulumi stack output kubeconfig --show-secrets --json > kubeconfig.json - -# Set the KUBECONFIG environment variable to point k9s to our kubeconfig. - -export KUBECONFIG=./kubeconfig.json - -# Let's get into our cluster! - -k9s -``` -::: - -Congratulations! Now you have your own cluster of Tezos nodes, running on AWS cloud, and exposed to the internet! 🎉 diff --git a/docs/reference/octez/sandbox.md b/docs/reference/octez/sandbox.md deleted file mode 100644 index 3b5781893..000000000 --- a/docs/reference/octez/sandbox.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -title: Run a Sandbox -hide_title: true -lastUpdated: 30th June 2023 ---- - -## Use Docker and Flextesa to Run an Independent Tezos Sandbox - -Running ephemeral and isolated sandboxes can be useful to experiment with faster networks or to automate reproducible tests. - -Here we use [Flextesa](https://gitlab.com/tezos/flextesa) to run one or more Tezos nodes, bakers, and endorsers contained in a sandbox environment. The default sandbox is configured to be compatible with the `tezos-client` installed in the [“Client-setup”](https://assets.tqtezos.com/docs/setup/1-tezos-client) section. - -### Dependencies - -This example requires Docker, available for Linux, Mac, or Windows at [https://www.docker.com](https://www.docker.com/). - -### Starting and Using a Sandbox - -Start the sandbox _in the background_ (will run with baking enabled): - -```sh -docker run --rm --name my-sandbox --detach -p 20000:20000 \ - oxheadalpha/flextesa:20230502 mumbaibox start -``` - -After a few seconds, this should succeed: - -```sh -tezos-client config reset # Cleans-up left-over configuration. -tezos-client --endpoint http://localhost:20000 bootstrapped -``` - -Configure the client to communicate with the sandbox: - -```sh -tezos-client --endpoint http://localhost:20000 config update -``` - -Then, instead of using a public faucet, you can just use tez by importing accounts already existing in the sandbox. They are visible with: - -```sh - $ docker run --rm oxheadalpha/flextesa:20230502 mumbaibox info - -Usable accounts: - -- alice - * edpkvGfYw3LyB1UcCahKQk4rF2tvbMUk8GFiTuMjL75uGXrpvKXhjn - * tz1VSUr8wwNhLAzempoch5d6hLRiTh8Cjcjb - * unencrypted:edsk3QoqBuvdamxouPhin7swCvkQNgq4jP5KZPbwWNnwdZpSpJiEbq -- bob - * edpkurPsQ8eUApnLUJ9ZPDvu98E8VNj4KtJa1aZr16Cr5ow5VHKnz4 - * tz1aSkwEot3L2kmUvcoxzjMomb9mvBNuzFK6 - * unencrypted:edsk3RFfvaFaxbHx8BMtEW1rKQcPtDML3LXjNqMNLCzC3wLC1bWbAt - -Root path (logs, chain data, etc.): /tmp/mini-box (inside container). -``` - -You may then just import them: - -```sh -tezos-client import secret key alice unencrypted:edsk3QoqBuvdamxouPhin7swCvkQNgq4jP5KZPbwWNnwdZpSpJiEbq --force -tezos-client import secret key bob unencrypted:edsk3RFfvaFaxbHx8BMtEW1rKQcPtDML3LXjNqMNLCzC3wLC1bWbAt --force -``` - -Check their balances: - -```sh -tezos-client get balance for alice -``` - -### Using The Sandbox - -See also the [Tezos Client](https://assets.tqtezos.com/docs/setup/1-tezos-client) section or the [Wallet-usage](https://tezos.gitlab.io/introduction/howtouse.html#transfers-and-receipts) tutorial of the Tezos manual. - -For instance, you can originate the most minimalistic [contract](https://gitlab.com/tezos/tezos/blob/mainnet/src/bin_client/test/contracts/attic/id.tz): - -```sh -# Download the contract: -wget https://gitlab.com/tezos/tezos/raw/mainnet/src/bin_client/test/contracts/attic/id.tz -# Run origination: -tezos-client originate contract hello-id transferring 0 from bob running id.tz --init "\"hello world\"" --burn-cap 1 --force -``` - -### Shutting Down The Sandbox - -When you're done playing, just destroy the container: - -```sh -docker kill my-sandbox -``` - -### Advanced Usage - -#### Tweak Protocol Constants - -One can see the configuration of the protocol running in the sandbox with: - -```sh -tezos-client rpc get /chains/main/blocks/head/context/constants -``` - -One important field is `"time_between_blocks": [ "5" ],` which means that blocks are baked every 5 seconds (as opposed to 60 seconds on mainnet). - -This constant can be configured with the `block_time` environment variable, see the example below: - -```sh -docker run --rm --name my-sandbox -e block_time=2 --detach -p 20000:20000 \ - oxheadalpha/flextesa:20230502 mumbaibox start -``` - -The above command runs a full sandbox with the Florence protocol and a faster time-between-blocks of 2 seconds. - -Many other parameters are set by the `mumbaibox` [script](https://gitlab.com/tezos/flextesa/-/blob/master/src/scripts/tutorial-box.sh). All the configuration options available can be seen with the command: - -```sh -docker run --rm -it oxheadalpha/flextesa:20230502 flextesarl mini-net --help -``` - -#### Try The Nairobi Protocol - -The Docker image also contains a `nairobibox` script: - -```sh -docker run --rm --name my-sandbox --detach -p 20000:20000 \ - oxheadalpha/flextesa:20230502 nairobibox start -``` - -You can then check that the protocol hash is `PtNairobiyssHuh87hEhfVBGCVrK3WnS8Z2FT4ymB5tAa4r1nQf`: - -```sh - $ tezos-client rpc get /chains/main/blocks/head/metadata | grep protocol - -{ "protocol": "PtNairobiyssHuh87hEhfVBGCVrK3WnS8Z2FT4ymB5tAa4r1nQf", - "next_protocol": "PtNairobiyssHuh87hEhfVBGCVrK3WnS8Z2FT4ymB5tAa4r1nQf", -``` - -or that there are new constants like the one related to liquidity baking: - -```sh - $ tezos-client rpc get /chains/main/blocks/head/context/constants | grep liquidity - "liquidity_baking_subsidy": "2500000", - "liquidity_baking_sunset_level": 525600, - "liquidity_baking_escape_ema_threshold": 1000000 } -``` - -### Further Reading - -For more issues or questions, see the [Flextesa](https://gitlab.com/tezos/flextesa) repository, and for even more advanced usage, see the [documentation](https://tezos.gitlab.io/flextesa/) (esp. for the [`mini-net` command](https://tezos.gitlab.io/flextesa/mini-net.html)). - diff --git a/docs/reference/octez/upgrading.md b/docs/reference/octez/upgrading.md deleted file mode 100644 index 9a3bd8783..000000000 --- a/docs/reference/octez/upgrading.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -title: Upgrading nodes -authors: Nomadic Labs -lastUpdated: 29th June 2023 ---- - -Here we cover the CLI commands to install the necessary dependencies to upgrade an Octez node. - -## Upgrade an Octez node - -As you may know, Tezos is an evolving blockchain. Through its on-chain governance mechanism, Tezos smoothly evolves to become more secure and scalable -over time. - -The following commands help to upgrade your node to the latest Octez version. - -### Docker and docker-compose - -#### Upgrade the docker image - -To upgrade your node to the latest Octez version, run the lastest image. Note that if you run the latest image, restarting your container is sufficient. - -See the [list of releases](https://tezos.gitlab.io/releases/releases.html) for the latest version number, and replace v17 below with that. - -To upgrade to version 17 for instance, run: - -```bash -docker run tezos/tezos:v17 ... -``` - -### Serokell PPA with Tezos packages - -#### Upgrade tezos packages - -To fetch the latest node version, run the following command: - -```bash -sudo apt-get update -sudo apt-get upgrade -``` - -### From source - -#### Upgrade from scratch - -Execute the following commands in your tezos repository: - -```bash -git fetch -git checkout v<:CURRENT_OCTEZ_VERSION:> -opam switch remove . -rm -rf _opam _build -make build-deps -eval $(opam env) -make -``` - -:::warning Opam switch remove -`opam switch remove .` is only needed if you are updating an already compiled repository, not if you are compiling from a freshly cloned repository. - -#### Upgrade using opam packages - -Run the following commands: - -```bash -opam update -opam depext -opam upgrade -``` - -:::warning Closing terminal -Be careful when closing terminal windows because this stops the node. - -:::note Using screen -Use [screen](https://doc.ubuntu-fr.org/screen), or [nohup](https://www.computerhope.com/unix/unohup.htm) to keep the node running in the background. diff --git a/docs/smart-contracts/multisig-specialized.md b/docs/smart-contracts/multisig-specialized.md index aa4a77ada..aed6650f5 100644 --- a/docs/smart-contracts/multisig-specialized.md +++ b/docs/smart-contracts/multisig-specialized.md @@ -33,7 +33,7 @@ Clone the [`lorentz-contract-multisig` repository](https://github.com/tqtezos/lo https://github.com/tqtezos/lorentz-contract-multisig/blob/master/README.md) to install the dependencies required to use the `lorentz-contract-multisig` -Don't forget to set up an `ALICE_ADDRESS` and `BOB_ADDRESS` on the test network as described in [Installing Octez](../reference/octez/installing); we will be using these variables throughout this tutorial. +Set up an `ALICE_ADDRESS` and `BOB_ADDRESS` on the test network for the Alice and Bob accounts; we will be using these variables throughout this tutorial. ## Getting your public key diff --git a/sidebars.js b/sidebars.js index 1f0cd8204..57e5e5ae8 100644 --- a/sidebars.js +++ b/sidebars.js @@ -73,6 +73,19 @@ const sidebars = { 'developing/dev-environments', 'developing/testnets', // 'developing/sandbox', // TODO + { + type: 'category', + label: 'The Octez client', + link: { + id: 'developing/octez-client', + type: 'doc', + }, + items: [ + 'developing/octez-client/installing', + 'developing/octez-client/accounts', + 'developing/octez-client/transactions', + ], + }, { type: 'category', label: 'Getting information about the blockchain', @@ -187,22 +200,6 @@ const sidebars = { type: 'category', label: 'Reference', items: [ - { - type: 'category', - label: 'The Octez client', - link: { - id: 'reference/octez', - type: 'doc', - }, - items: [ - 'reference/octez/cli-rpc', - 'reference/octez/installing', - 'reference/octez/upgrading', - 'reference/octez/monitoring', - 'reference/octez/sandbox', - 'reference/octez/best-practices', - ], - }, // 'reference/rpc', // TODO // 'reference/encoding', // TODO // 'reference/merkle-formats', // TODO diff --git a/static/img/reference/all-in-all.png b/static/img/reference/all-in-all.png deleted file mode 100644 index f34cffc65..000000000 Binary files a/static/img/reference/all-in-all.png and /dev/null differ diff --git a/static/img/reference/lens.png b/static/img/reference/lens.png deleted file mode 100644 index f2b45e5e1..000000000 Binary files a/static/img/reference/lens.png and /dev/null differ diff --git a/static/img/reference/metrics-block-validation-time.png b/static/img/reference/metrics-block-validation-time.png deleted file mode 100644 index 073ce7bb6..000000000 Binary files a/static/img/reference/metrics-block-validation-time.png and /dev/null differ diff --git a/static/img/reference/metrics-p2p-connections.png b/static/img/reference/metrics-p2p-connections.png deleted file mode 100644 index 9a0ff9097..000000000 Binary files a/static/img/reference/metrics-p2p-connections.png and /dev/null differ diff --git a/static/img/reference/netdata_dashboard.png b/static/img/reference/netdata_dashboard.png deleted file mode 100644 index cf040e2d2..000000000 Binary files a/static/img/reference/netdata_dashboard.png and /dev/null differ diff --git a/static/img/reference/number-of-metrics.jpeg b/static/img/reference/number-of-metrics.jpeg deleted file mode 100644 index 06bcde868..000000000 Binary files a/static/img/reference/number-of-metrics.jpeg and /dev/null differ diff --git a/static/img/reference/octez-metrics-dashboard.gif b/static/img/reference/octez-metrics-dashboard.gif deleted file mode 100644 index a51a1e83e..000000000 Binary files a/static/img/reference/octez-metrics-dashboard.gif and /dev/null differ