Merge branch 'master' of github.com:ergoplatform/ergo into v5.0.21

FAQ.md

@@ -51,7 +51,7 @@ The details of the Ergo emission schedule and monetary supply can be found in th
 * Telegram: https://t.me/ergoplatform
 
 * Ecosystem: https://sigmaverse.io
-* 
+
 * Github: https://github.com/ergoplatform/ergo
 
 * Documents: https://ergoplatform.org/en/documents/

README.md

@@ -76,7 +76,7 @@ Ergo utilizes three types of tests:
 
 1) Unit and property tests: These can be run using the `sbt test` command.
 2) Integration tests: These tests require Docker to be installed. Run them with the `sudo sbt it:test` command.
-3) Bootstrapping tests: These tests are time-consuming as they verify that the node is syncing with the main network in various regimes. Docker is also required for these tests. Run them with the `sudo sbt it2:test` command.
+3) Bootstrapping tests: These tests are time-consuming as they verify that the node is syncing with the main network in various modes. Docker is also required for these tests. Run them with the `sudo sbt it2:test` command.
 
 ## Setting up the Project in an IDE
 
@@ -86,9 +86,17 @@ Ensure that the project can be built with sbt before opening it in an IDE. You m
 
 To open the project in IntelliJ IDEA, select File / Open and navigate to the project folder. This will initiate the Project Import Wizard, which uses the SBT configuration (build.sbt file) to generate the project configuration files for IDEA. You can view the project configuration in the `File / Project Structure...` dialog. If the import is successful, you should be able to compile the project in the IDE. 
 
+## Modules
+
+This repository has modular structure, so only parts which are needed for an application could be used:
+
+* [avldb](avldb/README.md) - implementation of authenticated AVL+ tree used in Ergo, with persistence
+* [ergo-core](ergo-core/README.md) - functionality needed for an SPV client (P2P messages, block section stuctures, PoW, NiPoPoW)
+* ergo-wallet - Java and Scala functionalities to sign and verify transactions
+
 ## Contributing to Ergo
 
-Ergo is an open-source project and we welcome contributions from developers and testers! Join the discussion on [Ergo Discord](https://discord.gg/kj7s7nb) in the #development channel and check out our [Contributing documentation](https://docs.ergoplatform.com/contribute/).
+Ergo is an open-source project and we welcome contributions from developers and testers! Join the discussion over [Ergo Discord](https://discord.gg/kj7s7nb) in #development channel, or Telegram: https://t.me/ErgoDevelopers. Please also check out our [Contributing documentation](https://docs.ergoplatform.com/contribute/).
 
 ## Frequently Asked Questions
 

ci/release-binaries.py

@@ -25,7 +25,8 @@
 MAIN_JRE = os.environ.get("ERGO_RELEASE_PLATFORM") 
 VERSION = os.environ.get("ERGO_RELEASE_TAG")
 
-JAR_FILENAME = f"ergo-{VERSION}.jar"
+# filename is ergo-X.XXX.jar but tag version is vX.XXX
+JAR_FILENAME = f"ergo-{VERSION[1:]}.jar"
 SCRIPT_LOGO = f"""
 
              .-*%@#+-.                                                    

ergo-core/README.md

@@ -0,0 +1,179 @@
+# ergo-core
+
+Toy working example client code available [here](https://github.com/ccellado/ergo-test-client/tree/main).
+
+## Establishing connections to the node
+
+```mermaid
+sequenceDiagram
+    Client-->>Node: Establish TCP connection
+    Node->>Client: Handshake
+    Client->>Node: Handhake
+    Note over Client,Node: Message exchange started
+```
+
+### Connect to peer 📞
+First connect to peer node and get `Handshake` message.
+
+For the rest of the guide assume the message body as `byteBuffer`.
+```scala
+import org.ergoplatform.network.HandshakeSerializer
+
+{ byteBuffer =>
+  val handshake = HandshakeSerializer.parseBytesTry(byteBuffer)
+}
+```
+
+### Handshake 🤝
+After getting and successfully reading `Handshake` message from peer, the peer expects to receive the `Handshake` back
+
+Create PeerSpec and all parameters (including features)
+[PeerSpec](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala)'s doc:
+```
+* Declared information about peer
+  *
+  * @param agentName       - Network agent name. May contain information about client code
+  *                        stack, starting from core code-base up to the end graphical interface.
+  *                        Basic format is `/Name:Version(comments)/Name:Version/.../`,
+  *                        e.g. `/Ergo-Scala-client:2.0.0(iPad; U; CPU OS 3_2_1)/AndroidBuild:0.8/`
+  * @param protocolVersion - Identifies protocol version being used by the node
+  * @param nodeName        - Custom node name
+  * @param declaredAddress - Public network address of the node if any
+  * @param features        - Set of node capabilities
+```
+```scala
+import org.ergoplatform.network.PeerSpec
+import org.ergoplatform.network.Version
+import java.net.InetSocketAddress
+
+val mySpec = PeerSpec(
+  agentName = "morphicus",
+  protocolVersion = Version("version of the ergo-core library"),
+  nodeName = "ToTheMoon",
+  // required for non-local communication
+  declaredAddress = Some(InetSocketAddress("tothemoon.ergo", "5016")),
+  // note [1]
+  features = Seq.empty
+)
+```
+
+[1] All the available peer features [PeerFeatureDescriptors](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala)
+
+
+Create Handshake message with peer spec and UNIX time of the message
+```scala
+import org.ergoplatform.network.Handshake
+import org.ergoplatform.network.PeerSpec
+import org.ergoplatform.network.HandshakeSerializer
+
+val handshakeMessage = Handshake(mySpec, System.currentTimeMillis())
+val handshakeMessageSerialized = HandshakeSerializer.toBytes(handshakeMessage)
+```
+Serialize the message and send it.
+If the message arrived successfully, start communicating with the peer node.
+
+All communication is wrapped with Message headers, format described [here](https://docs.ergoplatform.com/dev/p2p/network/#message-format).
+
+## Syncing with the node
+
+```mermaid
+sequenceDiagram
+    Client-->>Node: SyncInfo( empty )
+    Node->>Client: InvData
+    Client-->Client: Checking local db
+    Client-->>Node: RequestModifiers( InvData )
+    Node->>Client: ModifiersData
+    Client-->Client: Checking PoW
+    Note over Client,Node: Client verified N headers
+    Client-->>Node: SyncInfo( List of N headers )
+    Node->>Client: InvData( with succeeding headers )
+    Note over Client,Node: Repeat
+```
+
+The peer node will start sending `SyncInfo` messages to us, since it is checking for new block information.
+Our client is syncing instead.
+
+### Send [ErgoSyncInfoV2](src/main/scala/org/ergoplatform/nodeView/history/ErgoSyncInfo.scala) ♲
+With empty `lastHeaders` so the node knows client is just beginning to sync.
+
+```scala
+import org.ergoplatform.nodeView.history.ErgoSyncInfoV2
+import org.ergoplatform.nodeView.history.ErgoSyncInfoMessageSpec
+
+val syncMessage = ErgoSyncInfoV2(Seq())
+val syncMessageSerialized = ErgoSyncInfoMessageSpec.toBytes(syncMessage)
+```
+Node replies with [InvData](src/main/scala/org/ergoplatform/network/message/InvData.scala) message, containing youngest headers id's the node has.
+Here the client checks if it has the headers already verified in its local db. The semantics of this is out of the tutorial scope.
+
+Reply with `RequestModifier` message containing `InvData` the peer node sent previous.
+
+### Send [RequestModifiers](src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala) 📥
+
+```scala
+import org.ergoplatform.network.message.InvSpec
+import org.ergoplatform.network.message.RequestModifierSpec
+
+{ byteBuffer =>
+  val invData = InvSpec.parseBytesTry(byteBuffer)
+  val message = RequestModifierSpec.toBytes(invData)
+}
+```
+Now received [ModifiersData](src/main/scala/org/ergoplatform/network/message/ModifiersData.scala) with block data.
+
+### Checking the PoW headers 🕵️
+
+Before adding those blocks to local database, check the headers.
+
+```scala
+import org.ergoplatform.network.message.ModifiersSpec
+import org.ergoplatform.modifiers.history.header.HeaderSerializer
+
+{ byteBuffer =>
+  val data = ModifiersSpec.parseBytes(byteBuffer)
+  val blockDataVerified = data.modifiers.map((id, bytes) =>
+    (id, HeaderSerializer.parseBytes(bytes))
+  )
+}
+```
+If successful, `blockDataVerified` contains the map with `Headers` of block data. Cheers 🙌
+
+## Checking NiPoPoW proofs
+
+[What is NiPoPoW](https://docs.ergoplatform.com/dev/protocol/nipopows/)
+
+### Request NiPoPoW proof 🛸
+
+```scala
+import org.ergoplatform.network.message.{GetNipopowProofSpec, NipopowProofData}
+import org.ergoplatform.nodeView.history.ErgoHistoryUtils.{P2PNipopowProofM, P2PNipopowProofK}
+
+val nipopowRequest = GetNipopowProofSpec.toBytes(
+  NipopowProofData(
+    m = P2PNipopowProofM,
+    k = P2PNipopowProofK,
+    None
+  )
+)
+```
+
+Received the [NipopowProofSpec](src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala) message.
+
+### Verify NiPoPoW proof 🦾
+
+Need to have [chainSettings](src/main/scala/org/ergoplatform/settings/ChainSettings.scala) in order to make a `nipopoSerializer` instance.
+
+```scala
+import org.ergoplatform.modifiers.history.popow.{NipopowAlgos, NipopowProofSerializer}
+import org.ergoplatform.network.message.NipopowProofSpec
+
+lazy val nipopowAlgos: NipopowAlgos = new NipopowAlgos(chainSettings)
+lazy val nipopowSerializer = new NipopowProofSerializer(nipopowAlgos)
+
+{ byteBuffer =>
+  val data = NipopowProofSpec.parseBytes(byteBuffer)
+  val proofData = nipopowSerializer.parseBytes(data)
+}
+```
+
+If successful, `proofData.isValid` will be true. Cheers 👽 

ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala

@@ -0,0 +1,37 @@
+package org.ergoplatform.network
+
+import org.ergoplatform.network.message.MessageConstants.MessageCode
+import org.ergoplatform.network.message.MessageSpecV1
+import scorex.util.serialization.{Reader, Writer}
+
+/**
+ * The `Handshake` message provides information about the transmitting node
+ * to the receiving node at the beginning of a connection. Until both peers
+ * have exchanged `Handshake` messages, no other messages will be accepted.
+ */
+object HandshakeSerializer extends MessageSpecV1[Handshake] {
+  override val messageCode: MessageCode = 75: Byte
+  override val messageName: String = "Handshake"
+
+  val maxHandshakeSize: Int = 8096
+
+  /**
+   * Serializing handshake into a byte writer.
+   *
+   * @param hs - handshake instance
+   * @param w  - writer to write bytes to
+   */
+  override def serialize(hs: Handshake, w: Writer): Unit = {
+    // first writes down handshake time, then peer specification of our node
+    w.putULong(hs.time)
+    PeerSpecSerializer.serialize(hs.peerSpec, w)
+  }
+
+  override def parse(r: Reader): Handshake = {
+    require(r.remaining <= maxHandshakeSize, s"Too big handshake. Size ${r.remaining} exceeds $maxHandshakeSize limit")
+    val time = r.getULong()
+    val data = PeerSpecSerializer.parse(r)
+    Handshake(data, time)
+  }
+
+}

ergo-core/src/main/scala/org/ergoplatform/network/message/GetNipopowProofSpec.scala

@@ -0,0 +1,50 @@
+package org.ergoplatform.network.message
+
+import org.ergoplatform.network.message.MessageConstants.MessageCode
+import org.ergoplatform.sdk.wallet.Constants.ModifierIdLength
+import org.ergoplatform.settings.Algos
+import scorex.util.ModifierId
+import scorex.util.serialization.{Reader, Writer}
+
+/**
+ * The `GetNipopowProof` message requests a `NipopowProof` message from the receiving node
+ */
+object GetNipopowProofSpec extends MessageSpecV1[NipopowProofData] {
+
+  val SizeLimit = 1000
+
+  val messageCode: MessageCode = 90: Byte
+  val messageName: String = "GetNipopowProof"
+
+  override def serialize(data: NipopowProofData, w: Writer): Unit = {
+    w.putInt(data.m)
+    w.putInt(data.k)
+    data.headerIdBytesOpt match {
+      case Some(idBytes) =>
+        w.put(1)
+        w.putBytes(idBytes)
+      case None =>
+        w.put(0)
+    }
+    w.putUShort(0) // to allow adding new data in future, we are adding possible pad length
+  }
+
+  override def parse(r: Reader): NipopowProofData = {
+    require(r.remaining <= SizeLimit, s"Too big GetNipopowProofSpec message(size: ${r.remaining})")
+
+    val m = r.getInt()
+    val k = r.getInt()
+
+    val headerIdPresents = r.getByte() == 1
+    val headerIdOpt = if (headerIdPresents) {
+      Some(ModifierId @@ Algos.encode(r.getBytes(ModifierIdLength)))
+    } else {
+      None
+    }
+    val remainingBytes = r.getUShort()
+    if (remainingBytes > 0 && remainingBytes < SizeLimit) {
+      r.getBytes(remainingBytes) // current version of reader just skips possible additional bytes
+    }
+    NipopowProofData(m, k, headerIdOpt)
+  }
+}

ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala

@@ -0,0 +1,50 @@
+package org.ergoplatform.network.message
+
+import org.ergoplatform.modifiers.ErgoNodeViewModifier
+import org.ergoplatform.modifiers.NetworkObjectTypeId
+import org.ergoplatform.network.message.MessageConstants.MessageCode
+import scorex.util.Extensions.LongOps
+import scorex.util.{bytesToId, idToBytes}
+import scorex.util.serialization.{Reader, Writer}
+
+/**
+ * The `Inv` message (inventory message) transmits one or more inventories of
+ * objects known to the transmitting peer.
+ * It can be sent unsolicited to announce new transactions or blocks,
+ * or it can be sent in reply to a `SyncInfo` message (or application-specific messages like `GetMempool`).
+ *
+ */
+object InvSpec extends MessageSpecV1[InvData] {
+
+  val maxInvObjects: Int = 400
+
+  override val messageCode: MessageCode = 55: Byte
+  override val messageName: String = "Inv"
+
+  override def serialize(data: InvData, w: Writer): Unit = {
+    val typeId = data.typeId
+    val elems = data.ids
+    require(elems.nonEmpty, "empty inv list")
+    require(elems.lengthCompare(maxInvObjects) <= 0, s"more invs than $maxInvObjects in a message")
+    w.put(typeId)
+    w.putUInt(elems.size)
+    elems.foreach { id =>
+      val bytes = idToBytes(id)
+      assert(bytes.length == ErgoNodeViewModifier.ModifierIdSize)
+      w.putBytes(bytes)
+    }
+  }
+
+  override def parse(r: Reader): InvData = {
+    val typeId = NetworkObjectTypeId.fromByte(r.getByte())
+    val count = r.getUInt().toIntExact
+    require(count > 0, "empty inv list")
+    require(count <= maxInvObjects, s"$count elements in a message while limit is $maxInvObjects")
+    val elems = (0 until count).map { _ =>
+      bytesToId(r.getBytes(ErgoNodeViewModifier.ModifierIdSize))
+    }
+
+    InvData(typeId, elems)
+  }
+
+}

ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersData.scala

@@ -0,0 +1,9 @@
+package org.ergoplatform.network.message
+
+import org.ergoplatform.modifiers.NetworkObjectTypeId
+import scorex.util.ModifierId
+
+/**
+ * Wrapper for block sections of the same type. Used to send multiple block sections at once ove the wire.
+ */
+case class ModifiersData(typeId: NetworkObjectTypeId.Value, modifiers: Map[ModifierId, Array[Byte]])