diff --git a/FAQ.md b/FAQ.md index a041b2a8dc..b8e6b45259 100644 --- a/FAQ.md +++ b/FAQ.md @@ -51,7 +51,7 @@ The details of the Ergo emission schedule and monetary supply can be found in th * Telegram: https://t.me/ergoplatform * Ecosystem: https://sigmaverse.io -* + * Github: https://github.com/ergoplatform/ergo * Documents: https://ergoplatform.org/en/documents/ diff --git a/README.md b/README.md index a296bdf8b2..53d2e5a64e 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ Ergo utilizes three types of tests: 1) Unit and property tests: These can be run using the `sbt test` command. 2) Integration tests: These tests require Docker to be installed. Run them with the `sudo sbt it:test` command. -3) Bootstrapping tests: These tests are time-consuming as they verify that the node is syncing with the main network in various regimes. Docker is also required for these tests. Run them with the `sudo sbt it2:test` command. +3) Bootstrapping tests: These tests are time-consuming as they verify that the node is syncing with the main network in various modes. Docker is also required for these tests. Run them with the `sudo sbt it2:test` command. ## Setting up the Project in an IDE @@ -86,9 +86,17 @@ Ensure that the project can be built with sbt before opening it in an IDE. You m To open the project in IntelliJ IDEA, select File / Open and navigate to the project folder. This will initiate the Project Import Wizard, which uses the SBT configuration (build.sbt file) to generate the project configuration files for IDEA. You can view the project configuration in the `File / Project Structure...` dialog. If the import is successful, you should be able to compile the project in the IDE. +## Modules + +This repository has modular structure, so only parts which are needed for an application could be used: + +* [avldb](avldb/README.md) - implementation of authenticated AVL+ tree used in Ergo, with persistence +* [ergo-core](ergo-core/README.md) - functionality needed for an SPV client (P2P messages, block section stuctures, PoW, NiPoPoW) +* ergo-wallet - Java and Scala functionalities to sign and verify transactions + ## Contributing to Ergo -Ergo is an open-source project and we welcome contributions from developers and testers! Join the discussion on [Ergo Discord](https://discord.gg/kj7s7nb) in the #development channel and check out our [Contributing documentation](https://docs.ergoplatform.com/contribute/). +Ergo is an open-source project and we welcome contributions from developers and testers! Join the discussion over [Ergo Discord](https://discord.gg/kj7s7nb) in #development channel, or Telegram: https://t.me/ErgoDevelopers. Please also check out our [Contributing documentation](https://docs.ergoplatform.com/contribute/). ## Frequently Asked Questions diff --git a/ci/release-binaries.py b/ci/release-binaries.py index 6c79be04c0..0182c0413e 100644 --- a/ci/release-binaries.py +++ b/ci/release-binaries.py @@ -25,7 +25,8 @@ MAIN_JRE = os.environ.get("ERGO_RELEASE_PLATFORM") VERSION = os.environ.get("ERGO_RELEASE_TAG") -JAR_FILENAME = f"ergo-{VERSION}.jar" +# filename is ergo-X.XXX.jar but tag version is vX.XXX +JAR_FILENAME = f"ergo-{VERSION[1:]}.jar" SCRIPT_LOGO = f""" .-*%@#+-. diff --git a/ergo-core/README.md b/ergo-core/README.md new file mode 100644 index 0000000000..b07bce00ef --- /dev/null +++ b/ergo-core/README.md @@ -0,0 +1,179 @@ +# ergo-core + +Toy working example client code available [here](https://github.com/ccellado/ergo-test-client/tree/main). + +## Establishing connections to the node + +```mermaid +sequenceDiagram + Client-->>Node: Establish TCP connection + Node->>Client: Handshake + Client->>Node: Handhake + Note over Client,Node: Message exchange started +``` + +### Connect to peer 📞 +First connect to peer node and get `Handshake` message. + +For the rest of the guide assume the message body as `byteBuffer`. +```scala +import org.ergoplatform.network.HandshakeSerializer + +{ byteBuffer => + val handshake = HandshakeSerializer.parseBytesTry(byteBuffer) +} +``` + +### Handshake 🤝 +After getting and successfully reading `Handshake` message from peer, the peer expects to receive the `Handshake` back + +Create PeerSpec and all parameters (including features) +[PeerSpec](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala)'s doc: +``` +* Declared information about peer + * + * @param agentName - Network agent name. May contain information about client code + * stack, starting from core code-base up to the end graphical interface. + * Basic format is `/Name:Version(comments)/Name:Version/.../`, + * e.g. `/Ergo-Scala-client:2.0.0(iPad; U; CPU OS 3_2_1)/AndroidBuild:0.8/` + * @param protocolVersion - Identifies protocol version being used by the node + * @param nodeName - Custom node name + * @param declaredAddress - Public network address of the node if any + * @param features - Set of node capabilities +``` +```scala +import org.ergoplatform.network.PeerSpec +import org.ergoplatform.network.Version +import java.net.InetSocketAddress + +val mySpec = PeerSpec( + agentName = "morphicus", + protocolVersion = Version("version of the ergo-core library"), + nodeName = "ToTheMoon", + // required for non-local communication + declaredAddress = Some(InetSocketAddress("tothemoon.ergo", "5016")), + // note [1] + features = Seq.empty +) +``` + +[1] All the available peer features [PeerFeatureDescriptors](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala) + + +Create Handshake message with peer spec and UNIX time of the message +```scala +import org.ergoplatform.network.Handshake +import org.ergoplatform.network.PeerSpec +import org.ergoplatform.network.HandshakeSerializer + +val handshakeMessage = Handshake(mySpec, System.currentTimeMillis()) +val handshakeMessageSerialized = HandshakeSerializer.toBytes(handshakeMessage) +``` +Serialize the message and send it. +If the message arrived successfully, start communicating with the peer node. + +All communication is wrapped with Message headers, format described [here](https://docs.ergoplatform.com/dev/p2p/network/#message-format). + +## Syncing with the node + +```mermaid +sequenceDiagram + Client-->>Node: SyncInfo( empty ) + Node->>Client: InvData + Client-->Client: Checking local db + Client-->>Node: RequestModifiers( InvData ) + Node->>Client: ModifiersData + Client-->Client: Checking PoW + Note over Client,Node: Client verified N headers + Client-->>Node: SyncInfo( List of N headers ) + Node->>Client: InvData( with succeeding headers ) + Note over Client,Node: Repeat +``` + +The peer node will start sending `SyncInfo` messages to us, since it is checking for new block information. +Our client is syncing instead. + +### Send [ErgoSyncInfoV2](src/main/scala/org/ergoplatform/nodeView/history/ErgoSyncInfo.scala) ♲ +With empty `lastHeaders` so the node knows client is just beginning to sync. + +```scala +import org.ergoplatform.nodeView.history.ErgoSyncInfoV2 +import org.ergoplatform.nodeView.history.ErgoSyncInfoMessageSpec + +val syncMessage = ErgoSyncInfoV2(Seq()) +val syncMessageSerialized = ErgoSyncInfoMessageSpec.toBytes(syncMessage) +``` +Node replies with [InvData](src/main/scala/org/ergoplatform/network/message/InvData.scala) message, containing youngest headers id's the node has. +Here the client checks if it has the headers already verified in its local db. The semantics of this is out of the tutorial scope. + +Reply with `RequestModifier` message containing `InvData` the peer node sent previous. + +### Send [RequestModifiers](src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala) 📥 + +```scala +import org.ergoplatform.network.message.InvSpec +import org.ergoplatform.network.message.RequestModifierSpec + +{ byteBuffer => + val invData = InvSpec.parseBytesTry(byteBuffer) + val message = RequestModifierSpec.toBytes(invData) +} +``` +Now received [ModifiersData](src/main/scala/org/ergoplatform/network/message/ModifiersData.scala) with block data. + +### Checking the PoW headers 🕵️ + +Before adding those blocks to local database, check the headers. + +```scala +import org.ergoplatform.network.message.ModifiersSpec +import org.ergoplatform.modifiers.history.header.HeaderSerializer + +{ byteBuffer => + val data = ModifiersSpec.parseBytes(byteBuffer) + val blockDataVerified = data.modifiers.map((id, bytes) => + (id, HeaderSerializer.parseBytes(bytes)) + ) +} +``` +If successful, `blockDataVerified` contains the map with `Headers` of block data. Cheers 🙌 + +## Checking NiPoPoW proofs + +[What is NiPoPoW](https://docs.ergoplatform.com/dev/protocol/nipopows/) + +### Request NiPoPoW proof 🛸 + +```scala +import org.ergoplatform.network.message.{GetNipopowProofSpec, NipopowProofData} +import org.ergoplatform.nodeView.history.ErgoHistoryUtils.{P2PNipopowProofM, P2PNipopowProofK} + +val nipopowRequest = GetNipopowProofSpec.toBytes( + NipopowProofData( + m = P2PNipopowProofM, + k = P2PNipopowProofK, + None + ) +) +``` + +Received the [NipopowProofSpec](src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala) message. + +### Verify NiPoPoW proof 🦾 + +Need to have [chainSettings](src/main/scala/org/ergoplatform/settings/ChainSettings.scala) in order to make a `nipopoSerializer` instance. + +```scala +import org.ergoplatform.modifiers.history.popow.{NipopowAlgos, NipopowProofSerializer} +import org.ergoplatform.network.message.NipopowProofSpec + +lazy val nipopowAlgos: NipopowAlgos = new NipopowAlgos(chainSettings) +lazy val nipopowSerializer = new NipopowProofSerializer(nipopowAlgos) + +{ byteBuffer => + val data = NipopowProofSpec.parseBytes(byteBuffer) + val proofData = nipopowSerializer.parseBytes(data) +} +``` + +If successful, `proofData.isValid` will be true. Cheers 👽 \ No newline at end of file diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala b/ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala new file mode 100644 index 0000000000..f2e6d3db97 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala @@ -0,0 +1,37 @@ +package org.ergoplatform.network + +import org.ergoplatform.network.message.MessageConstants.MessageCode +import org.ergoplatform.network.message.MessageSpecV1 +import scorex.util.serialization.{Reader, Writer} + +/** + * The `Handshake` message provides information about the transmitting node + * to the receiving node at the beginning of a connection. Until both peers + * have exchanged `Handshake` messages, no other messages will be accepted. + */ +object HandshakeSerializer extends MessageSpecV1[Handshake] { + override val messageCode: MessageCode = 75: Byte + override val messageName: String = "Handshake" + + val maxHandshakeSize: Int = 8096 + + /** + * Serializing handshake into a byte writer. + * + * @param hs - handshake instance + * @param w - writer to write bytes to + */ + override def serialize(hs: Handshake, w: Writer): Unit = { + // first writes down handshake time, then peer specification of our node + w.putULong(hs.time) + PeerSpecSerializer.serialize(hs.peerSpec, w) + } + + override def parse(r: Reader): Handshake = { + require(r.remaining <= maxHandshakeSize, s"Too big handshake. Size ${r.remaining} exceeds $maxHandshakeSize limit") + val time = r.getULong() + val data = PeerSpecSerializer.parse(r) + Handshake(data, time) + } + +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/GetNipopowProofSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/GetNipopowProofSpec.scala new file mode 100644 index 0000000000..a28dd40e7c --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/GetNipopowProofSpec.scala @@ -0,0 +1,50 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.network.message.MessageConstants.MessageCode +import org.ergoplatform.sdk.wallet.Constants.ModifierIdLength +import org.ergoplatform.settings.Algos +import scorex.util.ModifierId +import scorex.util.serialization.{Reader, Writer} + +/** + * The `GetNipopowProof` message requests a `NipopowProof` message from the receiving node + */ +object GetNipopowProofSpec extends MessageSpecV1[NipopowProofData] { + + val SizeLimit = 1000 + + val messageCode: MessageCode = 90: Byte + val messageName: String = "GetNipopowProof" + + override def serialize(data: NipopowProofData, w: Writer): Unit = { + w.putInt(data.m) + w.putInt(data.k) + data.headerIdBytesOpt match { + case Some(idBytes) => + w.put(1) + w.putBytes(idBytes) + case None => + w.put(0) + } + w.putUShort(0) // to allow adding new data in future, we are adding possible pad length + } + + override def parse(r: Reader): NipopowProofData = { + require(r.remaining <= SizeLimit, s"Too big GetNipopowProofSpec message(size: ${r.remaining})") + + val m = r.getInt() + val k = r.getInt() + + val headerIdPresents = r.getByte() == 1 + val headerIdOpt = if (headerIdPresents) { + Some(ModifierId @@ Algos.encode(r.getBytes(ModifierIdLength))) + } else { + None + } + val remainingBytes = r.getUShort() + if (remainingBytes > 0 && remainingBytes < SizeLimit) { + r.getBytes(remainingBytes) // current version of reader just skips possible additional bytes + } + NipopowProofData(m, k, headerIdOpt) + } +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala new file mode 100644 index 0000000000..dec5a76902 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala @@ -0,0 +1,50 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.modifiers.ErgoNodeViewModifier +import org.ergoplatform.modifiers.NetworkObjectTypeId +import org.ergoplatform.network.message.MessageConstants.MessageCode +import scorex.util.Extensions.LongOps +import scorex.util.{bytesToId, idToBytes} +import scorex.util.serialization.{Reader, Writer} + +/** + * The `Inv` message (inventory message) transmits one or more inventories of + * objects known to the transmitting peer. + * It can be sent unsolicited to announce new transactions or blocks, + * or it can be sent in reply to a `SyncInfo` message (or application-specific messages like `GetMempool`). + * + */ +object InvSpec extends MessageSpecV1[InvData] { + + val maxInvObjects: Int = 400 + + override val messageCode: MessageCode = 55: Byte + override val messageName: String = "Inv" + + override def serialize(data: InvData, w: Writer): Unit = { + val typeId = data.typeId + val elems = data.ids + require(elems.nonEmpty, "empty inv list") + require(elems.lengthCompare(maxInvObjects) <= 0, s"more invs than $maxInvObjects in a message") + w.put(typeId) + w.putUInt(elems.size) + elems.foreach { id => + val bytes = idToBytes(id) + assert(bytes.length == ErgoNodeViewModifier.ModifierIdSize) + w.putBytes(bytes) + } + } + + override def parse(r: Reader): InvData = { + val typeId = NetworkObjectTypeId.fromByte(r.getByte()) + val count = r.getUInt().toIntExact + require(count > 0, "empty inv list") + require(count <= maxInvObjects, s"$count elements in a message while limit is $maxInvObjects") + val elems = (0 until count).map { _ => + bytesToId(r.getBytes(ErgoNodeViewModifier.ModifierIdSize)) + } + + InvData(typeId, elems) + } + +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersData.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersData.scala new file mode 100644 index 0000000000..ad01035252 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersData.scala @@ -0,0 +1,9 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.modifiers.NetworkObjectTypeId +import scorex.util.ModifierId + +/** + * Wrapper for block sections of the same type. Used to send multiple block sections at once ove the wire. + */ +case class ModifiersData(typeId: NetworkObjectTypeId.Value, modifiers: Map[ModifierId, Array[Byte]]) diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersSpec.scala new file mode 100644 index 0000000000..c1d1118cd5 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersSpec.scala @@ -0,0 +1,68 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.modifiers.{ErgoNodeViewModifier, NetworkObjectTypeId} +import org.ergoplatform.network.message.MessageConstants.MessageCode +import scorex.util.{ModifierId, ScorexLogging, bytesToId, idToBytes} +import scorex.util.serialization.{Reader, Writer} +import scorex.util.Extensions._ +import scala.collection.immutable + +/** + * The `Modifier` message is a reply to a `RequestModifier` message which requested these modifiers. + */ +object ModifiersSpec extends MessageSpecV1[ModifiersData] with ScorexLogging { + + val maxMessageSize: Int = 2048576 + + private val maxMsgSizeWithReserve = maxMessageSize * 4 // due to big ADProofs + + override val messageCode: MessageCode = 33: Byte + override val messageName: String = "Modifier" + + private val HeaderLength = 5 // msg type Id + modifiersCount + + override def serialize(data: ModifiersData, w: Writer): Unit = { + + val typeId = data.typeId + val modifiers = data.modifiers + require(modifiers.nonEmpty, "empty modifiers list") + + val (msgCount, msgSize) = modifiers.foldLeft((0, HeaderLength)) { case ((c, s), (_, modifier)) => + val size = s + ErgoNodeViewModifier.ModifierIdSize + 4 + modifier.length + val count = if (size <= maxMsgSizeWithReserve) c + 1 else c + count -> size + } + + w.put(typeId) + w.putUInt(msgCount) + + modifiers.take(msgCount).foreach { case (id, modifier) => + w.putBytes(idToBytes(id)) + w.putUInt(modifier.length) + w.putBytes(modifier) + } + + if (msgSize > maxMsgSizeWithReserve) { + log.warn(s"Message with modifiers ${modifiers.keySet} has size $msgSize exceeding limit $maxMsgSizeWithReserve.") + } + } + + override def parse(r: Reader): ModifiersData = { + val typeId = NetworkObjectTypeId.fromByte(r.getByte()) // 1 byte + val count = r.getUInt().toIntExact // 8 bytes + require(count > 0, s"Illegal message with 0 modifiers of type $typeId") + val resMap = immutable.Map.newBuilder[ModifierId, Array[Byte]] + (0 until count).foldLeft(HeaderLength) { case (msgSize, _) => + val id = bytesToId(r.getBytes(ErgoNodeViewModifier.ModifierIdSize)) + val objBytesCnt = r.getUInt().toIntExact + val newMsgSize = msgSize + ErgoNodeViewModifier.ModifierIdSize + objBytesCnt + if (newMsgSize > maxMsgSizeWithReserve) { // buffer for safety + throw new Exception("Too big message with modifiers, size: " + maxMsgSizeWithReserve) + } + val obj = r.getBytes(objBytesCnt) + resMap += (id -> obj) + newMsgSize + } + ModifiersData(typeId, resMap.result()) + } +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofData.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofData.scala new file mode 100644 index 0000000000..1d7f0b8ed4 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofData.scala @@ -0,0 +1,8 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.settings.Algos +import scorex.util.ModifierId + +case class NipopowProofData(m: Int, k: Int, headerId: Option[ModifierId]) { + def headerIdBytesOpt: Option[Array[Byte]] = headerId.map(Algos.decode).flatMap(_.toOption) +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala new file mode 100644 index 0000000000..9806961126 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala @@ -0,0 +1,32 @@ +package org.ergoplatform.network.message + +import scorex.util.Extensions._ +import scorex.util.serialization.{Reader, Writer} + +/** + * The `NipopowProof` message is a reply to a `GetNipopowProof` message. + */ +object NipopowProofSpec extends MessageSpecV1[Array[Byte]] { + + val SizeLimit = 2000000 + override val messageCode: Byte = 91 + override val messageName: String = "NipopowProof" + + override def serialize(proof: Array[Byte], w: Writer): Unit = { + w.putUInt(proof.length) + w.putBytes(proof) + w.putUShort(0) // to allow adding new data in future, we are adding possible pad length + } + + override def parse(r: Reader): Array[Byte] = { + require(r.remaining <= SizeLimit, s"Too big NipopowProofSpec message(size: ${r.remaining})") + val proofSize = r.getUInt().toIntExact + require(proofSize > 0 && proofSize < SizeLimit) + val proof = r.getBytes(proofSize) + val remainingBytes = r.getUShort() + if (remainingBytes > 0 && remainingBytes < SizeLimit) { + r.getBytes(remainingBytes) // current version of reader just skips possible additional bytes + } + proof + } +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala new file mode 100644 index 0000000000..1cf0e4d0d6 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala @@ -0,0 +1,29 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.network.message.MessageConstants.MessageCode +import scorex.util.serialization.{Reader, Writer} + +/** + * The `RequestModifier` message requests one or more modifiers from another node. + * The objects are requested by an inventory, which the requesting node + * typically received previously by way of an `Inv` message. + * + * This message cannot be used to request arbitrary data, such as historic transactions no + * longer in the memory pool. Full nodes may not even be able to provide older blocks if + * they’ve pruned old transactions from their block database. + * For this reason, the `RequestModifier` message should usually only be used to request + * data from a node which previously advertised it had that data by sending an `Inv` message. + * + */ +object RequestModifierSpec extends MessageSpecV1[InvData] { + override val messageCode: MessageCode = 22: Byte + override val messageName: String = "RequestModifier" + + override def serialize(data: InvData, w: Writer): Unit = { + InvSpec.serialize(data, w) + } + + override def parse(r: Reader): InvData = { + InvSpec.parse(r) + } +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/nodeView/history/ErgoHistoryUtils.scala b/ergo-core/src/main/scala/org/ergoplatform/nodeView/history/ErgoHistoryUtils.scala index 3f1f3801fd..27f8a55774 100644 --- a/ergo-core/src/main/scala/org/ergoplatform/nodeView/history/ErgoHistoryUtils.scala +++ b/ergo-core/src/main/scala/org/ergoplatform/nodeView/history/ErgoHistoryUtils.scala @@ -23,6 +23,14 @@ object ErgoHistoryUtils { val EmptyHistoryHeight: Int = 0 val GenesisHeight: Int = EmptyHistoryHeight + 1 // first block has height == 1 + /** + * Minimal superchain length ('m' in KMZ17 paper) value used in NiPoPoW proofs for bootstrapping + */ + val P2PNipopowProofM = 6 + /** + * Suffix length ('k' in KMZ17 paper) value used in NiPoPoW proofs for bootstrapping + */ + val P2PNipopowProofK = 10 def heightOf(headerOpt: Option[Header]): Int = headerOpt.map(_.height).getOrElse(EmptyHistoryHeight) } diff --git a/scalastyle-config.xml b/scalastyle-config.xml index aba7f24c94..0df840521c 100644 --- a/scalastyle-config.xml +++ b/scalastyle-config.xml @@ -66,7 +66,7 @@ - + @@ -94,7 +94,7 @@ - + diff --git a/src/main/scala/org/ergoplatform/http/api/BlocksApiRoute.scala b/src/main/scala/org/ergoplatform/http/api/BlocksApiRoute.scala index 64a98e6044..41fc7d8ea9 100644 --- a/src/main/scala/org/ergoplatform/http/api/BlocksApiRoute.scala +++ b/src/main/scala/org/ergoplatform/http/api/BlocksApiRoute.scala @@ -44,8 +44,6 @@ case class BlocksApiRoute(viewHolderRef: ActorRef, readersHolder: ActorRef, ergo getModifierByIdR } - private val maxHeadersInOneQuery = ergoSettings.chainSettings.epochLength * 2 - private def getHistory: Future[ErgoHistoryReader] = (readersHolder ? GetDataFromHistory[ErgoHistoryReader](r => r)).mapTo[ErgoHistoryReader] @@ -105,7 +103,7 @@ case class BlocksApiRoute(viewHolderRef: ActorRef, readersHolder: ActorRef, ergo val headers = maxHeaderOpt .toIndexedSeq .flatMap { maxHeader => - history.headerChainBack(maxHeadersInOneQuery, maxHeader, _.height <= fromHeight + 1).headers + history.headerChainBack(MaxHeaders, maxHeader, _.height <= fromHeight + 1).headers } headers.toList.asJson } diff --git a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala index 11508302e7..b7b5842154 100644 --- a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala +++ b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala @@ -1,157 +1,14 @@ package org.ergoplatform.network.message -import org.ergoplatform.modifiers.{ErgoNodeViewModifier, NetworkObjectTypeId} -import org.ergoplatform.network.{Handshake, PeerSpec, PeerSpecSerializer} +import org.ergoplatform.network.{PeerSpec, PeerSpecSerializer} import org.ergoplatform.nodeView.state.SnapshotsInfo import org.ergoplatform.nodeView.state.UtxoState.{ManifestId, SubtreeId} -import org.ergoplatform.settings.Algos import org.ergoplatform.network.message.MessageConstants.MessageCode import scorex.crypto.hash.Digest32 import scorex.util.Extensions._ import scorex.util.serialization.{Reader, Writer} -import scorex.util.{ModifierId, ScorexLogging, bytesToId, idToBytes} import org.ergoplatform.sdk.wallet.Constants.ModifierIdLength -import scala.collection.immutable - -/** - * Wrapper for block sections of the same type. Used to send multiple block sections at once ove the wire. - */ -case class ModifiersData(typeId: NetworkObjectTypeId.Value, modifiers: Map[ModifierId, Array[Byte]]) - -case class NipopowProofData(m: Int, k: Int, headerId: Option[ModifierId]) { - def headerIdBytesOpt: Option[Array[Byte]] = headerId.map(Algos.decode).flatMap(_.toOption) -} - - - -/** - * The `Inv` message (inventory message) transmits one or more inventories of - * objects known to the transmitting peer. - * It can be sent unsolicited to announce new transactions or blocks, - * or it can be sent in reply to a `SyncInfo` message (or application-specific messages like `GetMempool`). - * - */ -object InvSpec extends MessageSpecV1[InvData] { - - val maxInvObjects: Int = 400 - - override val messageCode: MessageCode = 55: Byte - override val messageName: String = "Inv" - - override def serialize(data: InvData, w: Writer): Unit = { - val typeId = data.typeId - val elems = data.ids - require(elems.nonEmpty, "empty inv list") - require(elems.lengthCompare(maxInvObjects) <= 0, s"more invs than $maxInvObjects in a message") - w.put(typeId) - w.putUInt(elems.size) - elems.foreach { id => - val bytes = idToBytes(id) - assert(bytes.length == ErgoNodeViewModifier.ModifierIdSize) - w.putBytes(bytes) - } - } - - override def parse(r: Reader): InvData = { - val typeId = NetworkObjectTypeId.fromByte(r.getByte()) - val count = r.getUInt().toIntExact - require(count > 0, "empty inv list") - require(count <= maxInvObjects, s"$count elements in a message while limit is $maxInvObjects") - val elems = (0 until count).map { _ => - bytesToId(r.getBytes(ErgoNodeViewModifier.ModifierIdSize)) - } - - InvData(typeId, elems) - } - -} - -/** - * The `RequestModifier` message requests one or more modifiers from another node. - * The objects are requested by an inventory, which the requesting node - * typically received previously by way of an `Inv` message. - * - * This message cannot be used to request arbitrary data, such as historic transactions no - * longer in the memory pool. Full nodes may not even be able to provide older blocks if - * they’ve pruned old transactions from their block database. - * For this reason, the `RequestModifier` message should usually only be used to request - * data from a node which previously advertised it had that data by sending an `Inv` message. - * - */ -object RequestModifierSpec extends MessageSpecV1[InvData] { - override val messageCode: MessageCode = 22: Byte - override val messageName: String = "RequestModifier" - - override def serialize(data: InvData, w: Writer): Unit = { - InvSpec.serialize(data, w) - } - - override def parse(r: Reader): InvData = { - InvSpec.parse(r) - } -} - -/** - * The `Modifier` message is a reply to a `RequestModifier` message which requested these modifiers. - */ -object ModifiersSpec extends MessageSpecV1[ModifiersData] with ScorexLogging { - - val maxMessageSize: Int = 2048576 - - private val maxMsgSizeWithReserve = maxMessageSize * 4 // due to big ADProofs - - override val messageCode: MessageCode = 33: Byte - override val messageName: String = "Modifier" - - private val HeaderLength = 5 // msg type Id + modifiersCount - - override def serialize(data: ModifiersData, w: Writer): Unit = { - - val typeId = data.typeId - val modifiers = data.modifiers - require(modifiers.nonEmpty, "empty modifiers list") - - val (msgCount, msgSize) = modifiers.foldLeft((0, HeaderLength)) { case ((c, s), (_, modifier)) => - val size = s + ErgoNodeViewModifier.ModifierIdSize + 4 + modifier.length - val count = if (size <= maxMsgSizeWithReserve) c + 1 else c - count -> size - } - - w.put(typeId) - w.putUInt(msgCount) - - modifiers.take(msgCount).foreach { case (id, modifier) => - w.putBytes(idToBytes(id)) - w.putUInt(modifier.length) - w.putBytes(modifier) - } - - if (msgSize > maxMsgSizeWithReserve) { - log.warn(s"Message with modifiers ${modifiers.keySet} has size $msgSize exceeding limit $maxMsgSizeWithReserve.") - } - } - - override def parse(r: Reader): ModifiersData = { - val typeId = NetworkObjectTypeId.fromByte(r.getByte()) // 1 byte - val count = r.getUInt().toIntExact // 8 bytes - require(count > 0, s"Illegal message with 0 modifiers of type $typeId") - val resMap = immutable.Map.newBuilder[ModifierId, Array[Byte]] - (0 until count).foldLeft(HeaderLength) { case (msgSize, _) => - val id = bytesToId(r.getBytes(ErgoNodeViewModifier.ModifierIdSize)) - val objBytesCnt = r.getUInt().toIntExact - val newMsgSize = msgSize + ErgoNodeViewModifier.ModifierIdSize + objBytesCnt - if (newMsgSize > maxMsgSizeWithReserve) { // buffer for safety - throw new Exception("Too big message with modifiers, size: " + maxMsgSizeWithReserve) - } - val obj = r.getBytes(objBytesCnt) - resMap += (id -> obj) - newMsgSize - } - ModifiersData(typeId, resMap.result()) - } -} - /** * The `GetPeer` message requests an `Peers` message from the receiving node, * preferably one with lots of `PeerSpec` of other receiving nodes. @@ -204,39 +61,6 @@ class PeersSpec(peersLimit: Int) extends MessageSpecV1[Seq[PeerSpec]] { } } -/** - * The `Handshake` message provides information about the transmitting node - * to the receiving node at the beginning of a connection. Until both peers - * have exchanged `Handshake` messages, no other messages will be accepted. - */ -object HandshakeSerializer extends MessageSpecV1[Handshake] { - override val messageCode: MessageCode = 75: Byte - override val messageName: String = "Handshake" - - val maxHandshakeSize: Int = 8096 - - /** - * Serializing handshake into a byte writer. - * - * @param hs - handshake instance - * @param w - writer to write bytes to - */ - override def serialize(hs: Handshake, w: Writer): Unit = { - // first writes down handshake time, then peer specification of our node - w.putULong(hs.time) - PeerSpecSerializer.serialize(hs.peerSpec, w) - } - - override def parse(r: Reader): Handshake = { - require(r.remaining <= maxHandshakeSize, s"Too big handshake. Size ${r.remaining} exceeds $maxHandshakeSize limit") - val time = r.getULong() - val data = PeerSpecSerializer.parse(r) - Handshake(data, time) - } - -} - - /** * The `GetSnapshotsInfo` message requests an `SnapshotsInfo` message from the receiving node */ @@ -377,76 +201,3 @@ object UtxoSnapshotChunkSpec extends MessageSpecV1[Array[Byte]] { } } - -/** - * The `GetNipopowProof` message requests a `NipopowProof` message from the receiving node - */ -object GetNipopowProofSpec extends MessageSpecV1[NipopowProofData] { - - val SizeLimit = 1000 - - val messageCode: MessageCode = 90: Byte - val messageName: String = "GetNipopowProof" - - override def serialize(data: NipopowProofData, w: Writer): Unit = { - w.putInt(data.m) - w.putInt(data.k) - data.headerIdBytesOpt match { - case Some(idBytes) => - w.put(1) - w.putBytes(idBytes) - case None => - w.put(0) - } - w.putUShort(0) // to allow adding new data in future, we are adding possible pad length - } - - override def parse(r: Reader): NipopowProofData = { - require(r.remaining <= SizeLimit, s"Too big GetNipopowProofSpec message(size: ${r.remaining})") - - val m = r.getInt() - val k = r.getInt() - - val headerIdPresents = r.getByte() == 1 - val headerIdOpt = if (headerIdPresents) { - Some(ModifierId @@ Algos.encode(r.getBytes(ModifierIdLength))) - } else { - None - } - val remainingBytes = r.getUShort() - if (remainingBytes > 0 && remainingBytes < SizeLimit) { - r.getBytes(remainingBytes) // current version of reader just skips possible additional bytes - } - NipopowProofData(m, k, headerIdOpt) - } - -} - -/** - * The `NipopowProof` message is a reply to a `GetNipopowProof` message. - */ -object NipopowProofSpec extends MessageSpecV1[Array[Byte]] { - - val SizeLimit = 2000000 - override val messageCode: Byte = 91 - override val messageName: String = "NipopowProof" - - override def serialize(proof: Array[Byte], w: Writer): Unit = { - w.putUInt(proof.length) - w.putBytes(proof) - w.putUShort(0) // to allow adding new data in future, we are adding possible pad length - } - - override def parse(r: Reader): Array[Byte] = { - require(r.remaining <= SizeLimit, s"Too big NipopowProofSpec message(size: ${r.remaining})") - val proofSize = r.getUInt().toIntExact - require(proofSize > 0 && proofSize < SizeLimit) - val proof = r.getBytes(proofSize) - val remainingBytes = r.getUShort() - if (remainingBytes > 0 && remainingBytes < SizeLimit) { - r.getBytes(remainingBytes) // current version of reader just skips possible additional bytes - } - proof - } - -} diff --git a/src/main/scala/org/ergoplatform/nodeView/history/storage/modifierprocessors/PopowProcessor.scala b/src/main/scala/org/ergoplatform/nodeView/history/storage/modifierprocessors/PopowProcessor.scala index 8baa167e40..59922347a3 100644 --- a/src/main/scala/org/ergoplatform/nodeView/history/storage/modifierprocessors/PopowProcessor.scala +++ b/src/main/scala/org/ergoplatform/nodeView/history/storage/modifierprocessors/PopowProcessor.scala @@ -6,7 +6,7 @@ import org.ergoplatform.modifiers.BlockSection import org.ergoplatform.modifiers.history.extension.Extension import org.ergoplatform.modifiers.history.header.Header import org.ergoplatform.modifiers.history.popow.{NipopowAlgos, NipopowProverWithDbAlgs, NipopowProof, NipopowProofSerializer, PoPowHeader, PoPowParams} -import org.ergoplatform.nodeView.history.ErgoHistoryUtils.GenesisHeight +import org.ergoplatform.nodeView.history.ErgoHistoryUtils import org.ergoplatform.nodeView.history.ErgoHistoryReader import org.ergoplatform.settings.{ChainSettings, NipopowSettings} import org.ergoplatform.settings.Constants.HashLength @@ -41,19 +41,19 @@ trait PopowProcessor extends BasicReaders with ScorexLogging { lazy val nipopowSerializer = new NipopowProofSerializer(nipopowAlgos) private lazy val nipopowVerifier = - new NipopowVerifier(chainSettings.genesisId.orElse(bestHeaderIdAtHeight(GenesisHeight))) + new NipopowVerifier(chainSettings.genesisId.orElse(bestHeaderIdAtHeight(ErgoHistoryUtils.GenesisHeight))) protected val NipopowSnapshotHeightKey: ByteArrayWrapper = ByteArrayWrapper(Array.fill(HashLength)(50: Byte)) /** * Minimal superchain length ('m' in KMZ17 paper) value used in NiPoPoW proofs for bootstrapping */ - val P2PNipopowProofM = 6 + val P2PNipopowProofM = ErgoHistoryUtils.P2PNipopowProofM /** * Suffix length ('k' in KMZ17 paper) value used in NiPoPoW proofs for bootstrapping */ - val P2PNipopowProofK = 10 + val P2PNipopowProofK = ErgoHistoryUtils.P2PNipopowProofK /** * Checks and appends new header to history diff --git a/src/main/scala/org/ergoplatform/nodeView/state/UtxoStateReader.scala b/src/main/scala/org/ergoplatform/nodeView/state/UtxoStateReader.scala index a2c10bd248..d891a6e30e 100644 --- a/src/main/scala/org/ergoplatform/nodeView/state/UtxoStateReader.scala +++ b/src/main/scala/org/ergoplatform/nodeView/state/UtxoStateReader.scala @@ -142,7 +142,7 @@ trait UtxoStateReader extends ErgoStateReader with UtxoSetSnapshotPersistence { * @param txs - transactions to generate proofs * @return proof for specified transactions and new state digest */ - def proofsForTransactions(txs: Seq[ErgoTransaction]): Try[(SerializedAdProof, ADDigest)] = persistentProver.synchronized { + def proofsForTransactions(txs: Seq[ErgoTransaction]): Try[(SerializedAdProof, ADDigest)] = synchronized { val rootHash = persistentProver.digest log.trace(s"Going to create proof for ${txs.length} transactions at root ${Algos.encode(rootHash)}") if (txs.isEmpty) { diff --git a/src/main/scala/scorex/core/network/PeerConnectionHandler.scala b/src/main/scala/scorex/core/network/PeerConnectionHandler.scala index 32dc1e25a9..8c32487397 100644 --- a/src/main/scala/scorex/core/network/PeerConnectionHandler.scala +++ b/src/main/scala/scorex/core/network/PeerConnectionHandler.scala @@ -4,12 +4,12 @@ import akka.actor.{Actor, ActorRef, Cancellable, Props, SupervisorStrategy} import akka.io.Tcp import akka.io.Tcp._ import akka.util.{ByteString, CompactByteString} -import org.ergoplatform.network.{Handshake, PeerSpec, Version} +import org.ergoplatform.network.{Handshake, HandshakeSerializer, PeerSpec, Version} import org.ergoplatform.network.Version.Eip37ForkVersion import scorex.core.app.ScorexContext import scorex.core.network.NetworkController.ReceivableMessages.{Handshaked, PenalizePeer} import scorex.core.network.PeerConnectionHandler.ReceivableMessages -import org.ergoplatform.network.message.{HandshakeSerializer, MessageSerializer} +import org.ergoplatform.network.message.MessageSerializer import org.ergoplatform.network.peer.{PeerInfo, PenaltyType} import org.ergoplatform.settings.ScorexSettings import scorex.util.ScorexLogging diff --git a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala index 371b356384..893f05c723 100644 --- a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala +++ b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala @@ -104,9 +104,9 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { }.get val inputs: IndexedSeq[Input] = IndexedSeq( new Input(ADKey @@ Base16.decode("c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742").get, - new ProverResult(Base16.decode("b4a04b4201da0578be3dac11067b567a73831f35b024a2e623c1f8da230407f63bab62c62ed9b93808b106b5a7e8b1751fa656f4c5de4674").get, new ContextExtension(Map()))), + new ProverResult(Base16.decode("b4a04b4201da0578be3dac11067b567a73831f35b024a2e623c1f8da230407f63bab62c62ed9b93808b106b5a7e8b1751fa656f4c5de4674").get, ContextExtension.empty)), new Input(ADKey @@ Base16.decode("ca796a4fc9c0d746a69702a77bd78b1a80a5ef5bf5713bbd95d93a4f23b27ead").get, - new ProverResult(Base16.decode("5aea4d78a234c35accacdf8996b0af5b51e26fee29ea5c05468f23707d31c0df39400127391cd57a70eb856710db48bb9833606e0bf90340").get, new ContextExtension(Map()))), + new ProverResult(Base16.decode("5aea4d78a234c35accacdf8996b0af5b51e26fee29ea5c05468f23707d31c0df39400127391cd57a70eb856710db48bb9833606e0bf90340").get, ContextExtension.empty)), ) val outputCandidates: IndexedSeq[ErgoBoxCandidate] = IndexedSeq( new ErgoBoxCandidate(1000000000L, minerPk, height, Colls.emptyColl, Map()), @@ -543,4 +543,46 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { } } + property("context extension with neg id") { + val negId: Byte = -10 + + val b = new ErgoBox(1000000000L, Constants.TrueLeaf, Colls.emptyColl, + Map.empty, ModifierId @@ "c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742", + 0, 0) + val input = Input(b.id, ProverResult(Array.emptyByteArray, ContextExtension(Map(negId -> IntConstant(0))))) + + val oc = new ErgoBoxCandidate(b.value, b.ergoTree, b.creationHeight) + + val utx = UnsignedErgoTransaction(IndexedSeq(input), IndexedSeq(oc)) + + val txTry = defaultProver.sign(utx, IndexedSeq(b), IndexedSeq.empty, emptyStateContext) + .map(ErgoTransaction.apply) + + txTry.isSuccess shouldBe false + + txTry.toEither.left.get.isInstanceOf[NegativeArraySizeException] shouldBe true + } + + + property("context extension with neg and pos ids") { + val negId: Byte = -20 + + val b = new ErgoBox(1000000000L, Constants.TrueLeaf, Colls.emptyColl, + Map.empty, ModifierId @@ "c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742", + 0, 0) + val ce = ContextExtension(Map(negId -> IntConstant(0), (-negId).toByte -> IntConstant(1))) + val input = Input(b.id, ProverResult(Array.emptyByteArray, ce)) + + val oc = new ErgoBoxCandidate(b.value, b.ergoTree, b.creationHeight) + + val utx = UnsignedErgoTransaction(IndexedSeq(input), IndexedSeq(oc)) + + val txTry = defaultProver.sign(utx, IndexedSeq(b), IndexedSeq.empty, emptyStateContext) + .map(ErgoTransaction.apply) + + txTry.isSuccess shouldBe false + + txTry.toEither.left.get.isInstanceOf[ArrayIndexOutOfBoundsException] shouldBe true + } + } diff --git a/src/test/scala/org/ergoplatform/network/HandshakeSpecification.scala b/src/test/scala/org/ergoplatform/network/HandshakeSpecification.scala index 44f71edebb..3c3a22c960 100644 --- a/src/test/scala/org/ergoplatform/network/HandshakeSpecification.scala +++ b/src/test/scala/org/ergoplatform/network/HandshakeSpecification.scala @@ -3,7 +3,6 @@ package org.ergoplatform.network import java.nio.ByteBuffer import org.ergoplatform.nodeView.state.StateType import org.ergoplatform.utils.ErgoPropertyTest -import org.ergoplatform.network.message.HandshakeSerializer import scorex.util.encode.Base16 class HandshakeSpecification extends ErgoPropertyTest with DecodingUtils {