From 058c5915d5cdd4f28dbea71bb9608971eff40ace Mon Sep 17 00:00:00 2001 From: ccellado Date: Thu, 21 Dec 2023 15:32:54 +0300 Subject: [PATCH 01/23] Add examples readme to ergo-core --- ergo-core/README.md | 64 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 ergo-core/README.md diff --git a/ergo-core/README.md b/ergo-core/README.md new file mode 100644 index 0000000000..e148110d67 --- /dev/null +++ b/ergo-core/README.md @@ -0,0 +1,64 @@ +# Ergo Core library + +## Establishing connections to the node + +1. We need to send a Handshake message + +Create PeerSpec and all parameters (including features) + +[PeerSpec](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala)'s doc: +``` +* Declared information about peer + * + * @param agentName - Network agent name. May contain information about client code + * stack, starting from core code-base up to the end graphical interface. + * Basic format is `/Name:Version(comments)/Name:Version/.../`, + * e.g. `/Ergo-Scala-client:2.0.0(iPad; U; CPU OS 3_2_1)/AndroidBuild:0.8/` + * @param protocolVersion - Identifies protocol version being used by the node + * @param nodeName - Custom node name + * @param declaredAddress - Public network address of the node if any + * @param features - Set of node capabilities +``` + +All the available peer features are stored inside [PeerFeatureDescriptors](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala) + + +```scala +import org.ergoplatform.network.PeerSpec +import org.ergoplatform.network.Version +import java.net.InetSocketAddress + +val mySpec = PeerSpec( + agentName = "MyCoolErgoClient:0.1", + protocolVersion = Version("version of the ergo-core library"), + nodeName = "MyNodeName", + declaredAddress = Some(InetSocketAddress("https://ergocoolclient.xyz", "5016")), + features = Seq() +) +``` + +Then, we are ready to create our Handshake message with peer spec and UNIX time of the message +```scala +import org.ergoplatform.network.Handshake +import org.ergoplatform.network.PeerSpec + +val handshakeMessage = Handshake(mySpec, System.currentTimeMillis()) +``` +Now we can serialize the message and send it +If the message arrived successfully, we'll receive Handshake message back, so we can start to exchange messages with the node + +2. Now we can start syncing with the node + +``` +/** + * @param lastHeaders - some recent headers (including last one) known to a peer + */ +``` +Send [ErgoSyncInfoV2](src/main/scala/org/ergoplatform/nodeView/history/ErgoSyncInfo.scala) with empty `lastHeaders`, so the node knows we're just beginning to sync +```scala +import org.ergoplatform.nodeView.history.ErgoSyncInfoV2 +import org.ergoplatform.nodeView.history.ErgoSyncInfoMessageSpec + +val syncMessage = ErgoSyncInfoV2(Seq()) +val messageSerializedToBytes = ErgoSyncInfoMessageSpec.toBytes(emptySync) +``` \ No newline at end of file From 1c454291ef266c560880f74a2ae9fe34b89cdaa5 Mon Sep 17 00:00:00 2001 From: ccellado Date: Thu, 21 Dec 2023 15:58:58 +0300 Subject: [PATCH 02/23] Add examples readme to ergo-core Move appropriate specs to core --- ergo-core/README.md | 20 ++++- .../network/HandshakeSerializer.scala | 37 ++++++++++ .../network/message/InvSpec.scala | 50 +++++++++++++ scalastyle-config.xml | 4 +- .../network/message/BasicMessagesRepo.scala | 73 ------------------- 5 files changed, 108 insertions(+), 76 deletions(-) create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala diff --git a/ergo-core/README.md b/ergo-core/README.md index e148110d67..c7ea82005a 100644 --- a/ergo-core/README.md +++ b/ergo-core/README.md @@ -41,8 +41,10 @@ Then, we are ready to create our Handshake message with peer spec and UNIX time ```scala import org.ergoplatform.network.Handshake import org.ergoplatform.network.PeerSpec +import org.ergoplatform.network.HandshakeSerializer val handshakeMessage = Handshake(mySpec, System.currentTimeMillis()) +val messageSerializedToBytes = HandshakeSerializer.toBytes(handshakeMessage) ``` Now we can serialize the message and send it If the message arrived successfully, we'll receive Handshake message back, so we can start to exchange messages with the node @@ -61,4 +63,20 @@ import org.ergoplatform.nodeView.history.ErgoSyncInfoMessageSpec val syncMessage = ErgoSyncInfoV2(Seq()) val messageSerializedToBytes = ErgoSyncInfoMessageSpec.toBytes(emptySync) -``` \ No newline at end of file +``` +The client will start receiving [InvData](src/main/scala/org/ergoplatform/network/message/InvData.scala) messages with +``` +/** + * P2P network message which is encoding "inventory", transactions or block sections the node has + * + * @param typeId + * @param ids + */ +``` +```scala +import org.ergoplatform.network.message.InvSpec + +{ invDataMessage => + val invData = InvSpec.parseBytesTry(invDataMessage) +} +``` diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala b/ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala new file mode 100644 index 0000000000..f2e6d3db97 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala @@ -0,0 +1,37 @@ +package org.ergoplatform.network + +import org.ergoplatform.network.message.MessageConstants.MessageCode +import org.ergoplatform.network.message.MessageSpecV1 +import scorex.util.serialization.{Reader, Writer} + +/** + * The `Handshake` message provides information about the transmitting node + * to the receiving node at the beginning of a connection. Until both peers + * have exchanged `Handshake` messages, no other messages will be accepted. + */ +object HandshakeSerializer extends MessageSpecV1[Handshake] { + override val messageCode: MessageCode = 75: Byte + override val messageName: String = "Handshake" + + val maxHandshakeSize: Int = 8096 + + /** + * Serializing handshake into a byte writer. + * + * @param hs - handshake instance + * @param w - writer to write bytes to + */ + override def serialize(hs: Handshake, w: Writer): Unit = { + // first writes down handshake time, then peer specification of our node + w.putULong(hs.time) + PeerSpecSerializer.serialize(hs.peerSpec, w) + } + + override def parse(r: Reader): Handshake = { + require(r.remaining <= maxHandshakeSize, s"Too big handshake. Size ${r.remaining} exceeds $maxHandshakeSize limit") + val time = r.getULong() + val data = PeerSpecSerializer.parse(r) + Handshake(data, time) + } + +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala new file mode 100644 index 0000000000..cf318f06a2 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala @@ -0,0 +1,50 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.NodeViewModifier +import org.ergoplatform.modifiers.NetworkObjectTypeId +import org.ergoplatform.network.message.MessageConstants.MessageCode +import scorex.util.Extensions.LongOps +import scorex.util.{bytesToId, idToBytes} +import scorex.util.serialization.{Reader, Writer} + +/** + * The `Inv` message (inventory message) transmits one or more inventories of + * objects known to the transmitting peer. + * It can be sent unsolicited to announce new transactions or blocks, + * or it can be sent in reply to a `SyncInfo` message (or application-specific messages like `GetMempool`). + * + */ +object InvSpec extends MessageSpecV1[InvData] { + + val maxInvObjects: Int = 400 + + override val messageCode: MessageCode = 55: Byte + override val messageName: String = "Inv" + + override def serialize(data: InvData, w: Writer): Unit = { + val typeId = data.typeId + val elems = data.ids + require(elems.nonEmpty, "empty inv list") + require(elems.lengthCompare(maxInvObjects) <= 0, s"more invs than $maxInvObjects in a message") + w.put(typeId) + w.putUInt(elems.size) + elems.foreach { id => + val bytes = idToBytes(id) + assert(bytes.length == NodeViewModifier.ModifierIdSize) + w.putBytes(bytes) + } + } + + override def parse(r: Reader): InvData = { + val typeId = NetworkObjectTypeId.fromByte(r.getByte()) + val count = r.getUInt().toIntExact + require(count > 0, "empty inv list") + require(count <= maxInvObjects, s"$count elements in a message while limit is $maxInvObjects") + val elems = (0 until count).map { _ => + bytesToId(r.getBytes(NodeViewModifier.ModifierIdSize)) + } + + InvData(typeId, elems) + } + +} diff --git a/scalastyle-config.xml b/scalastyle-config.xml index aba7f24c94..0df840521c 100644 --- a/scalastyle-config.xml +++ b/scalastyle-config.xml @@ -66,7 +66,7 @@ - + @@ -94,7 +94,7 @@ - + diff --git a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala index 41ba8a43a5..3cb95bf098 100644 --- a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala +++ b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala @@ -26,47 +26,7 @@ case class NipopowProofData(m: Int, k: Int, headerId: Option[ModifierId]) { -/** - * The `Inv` message (inventory message) transmits one or more inventories of - * objects known to the transmitting peer. - * It can be sent unsolicited to announce new transactions or blocks, - * or it can be sent in reply to a `SyncInfo` message (or application-specific messages like `GetMempool`). - * - */ -object InvSpec extends MessageSpecV1[InvData] { - - val maxInvObjects: Int = 400 - - override val messageCode: MessageCode = 55: Byte - override val messageName: String = "Inv" - - override def serialize(data: InvData, w: Writer): Unit = { - val typeId = data.typeId - val elems = data.ids - require(elems.nonEmpty, "empty inv list") - require(elems.lengthCompare(maxInvObjects) <= 0, s"more invs than $maxInvObjects in a message") - w.put(typeId) - w.putUInt(elems.size) - elems.foreach { id => - val bytes = idToBytes(id) - assert(bytes.length == NodeViewModifier.ModifierIdSize) - w.putBytes(bytes) - } - } - - override def parse(r: Reader): InvData = { - val typeId = NetworkObjectTypeId.fromByte(r.getByte()) - val count = r.getUInt().toIntExact - require(count > 0, "empty inv list") - require(count <= maxInvObjects, s"$count elements in a message while limit is $maxInvObjects") - val elems = (0 until count).map { _ => - bytesToId(r.getBytes(NodeViewModifier.ModifierIdSize)) - } - - InvData(typeId, elems) - } -} /** * The `RequestModifier` message requests one or more modifiers from another node. @@ -205,39 +165,6 @@ class PeersSpec(peersLimit: Int) extends MessageSpecV1[Seq[PeerSpec]] { } } -/** - * The `Handshake` message provides information about the transmitting node - * to the receiving node at the beginning of a connection. Until both peers - * have exchanged `Handshake` messages, no other messages will be accepted. - */ -object HandshakeSerializer extends MessageSpecV1[Handshake] { - override val messageCode: MessageCode = 75: Byte - override val messageName: String = "Handshake" - - val maxHandshakeSize: Int = 8096 - - /** - * Serializing handshake into a byte writer. - * - * @param hs - handshake instance - * @param w - writer to write bytes to - */ - override def serialize(hs: Handshake, w: Writer): Unit = { - // first writes down handshake time, then peer specification of our node - w.putULong(hs.time) - PeerSpecSerializer.serialize(hs.peerSpec, w) - } - - override def parse(r: Reader): Handshake = { - require(r.remaining <= maxHandshakeSize, s"Too big handshake. Size ${r.remaining} exceeds $maxHandshakeSize limit") - val time = r.getULong() - val data = PeerSpecSerializer.parse(r) - Handshake(data, time) - } - -} - - /** * The `GetSnapshotsInfo` message requests an `SnapshotsInfo` message from the receiving node */ From 9acf7440e4ed03a328f73c0cea48c7d07fa7024a Mon Sep 17 00:00:00 2001 From: ccellado Date: Thu, 21 Dec 2023 15:32:54 +0300 Subject: [PATCH 03/23] Add examples readme to ergo-core --- ergo-core/README.md | 64 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 ergo-core/README.md diff --git a/ergo-core/README.md b/ergo-core/README.md new file mode 100644 index 0000000000..e148110d67 --- /dev/null +++ b/ergo-core/README.md @@ -0,0 +1,64 @@ +# Ergo Core library + +## Establishing connections to the node + +1. We need to send a Handshake message + +Create PeerSpec and all parameters (including features) + +[PeerSpec](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala)'s doc: +``` +* Declared information about peer + * + * @param agentName - Network agent name. May contain information about client code + * stack, starting from core code-base up to the end graphical interface. + * Basic format is `/Name:Version(comments)/Name:Version/.../`, + * e.g. `/Ergo-Scala-client:2.0.0(iPad; U; CPU OS 3_2_1)/AndroidBuild:0.8/` + * @param protocolVersion - Identifies protocol version being used by the node + * @param nodeName - Custom node name + * @param declaredAddress - Public network address of the node if any + * @param features - Set of node capabilities +``` + +All the available peer features are stored inside [PeerFeatureDescriptors](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala) + + +```scala +import org.ergoplatform.network.PeerSpec +import org.ergoplatform.network.Version +import java.net.InetSocketAddress + +val mySpec = PeerSpec( + agentName = "MyCoolErgoClient:0.1", + protocolVersion = Version("version of the ergo-core library"), + nodeName = "MyNodeName", + declaredAddress = Some(InetSocketAddress("https://ergocoolclient.xyz", "5016")), + features = Seq() +) +``` + +Then, we are ready to create our Handshake message with peer spec and UNIX time of the message +```scala +import org.ergoplatform.network.Handshake +import org.ergoplatform.network.PeerSpec + +val handshakeMessage = Handshake(mySpec, System.currentTimeMillis()) +``` +Now we can serialize the message and send it +If the message arrived successfully, we'll receive Handshake message back, so we can start to exchange messages with the node + +2. Now we can start syncing with the node + +``` +/** + * @param lastHeaders - some recent headers (including last one) known to a peer + */ +``` +Send [ErgoSyncInfoV2](src/main/scala/org/ergoplatform/nodeView/history/ErgoSyncInfo.scala) with empty `lastHeaders`, so the node knows we're just beginning to sync +```scala +import org.ergoplatform.nodeView.history.ErgoSyncInfoV2 +import org.ergoplatform.nodeView.history.ErgoSyncInfoMessageSpec + +val syncMessage = ErgoSyncInfoV2(Seq()) +val messageSerializedToBytes = ErgoSyncInfoMessageSpec.toBytes(emptySync) +``` \ No newline at end of file From c364817cc9a9c1761019c83782d321c90acdbdcd Mon Sep 17 00:00:00 2001 From: ccellado Date: Thu, 21 Dec 2023 15:58:58 +0300 Subject: [PATCH 04/23] Add examples readme to ergo-core Move appropriate specs to core --- ergo-core/README.md | 20 ++++- .../network/HandshakeSerializer.scala | 37 ++++++++++ .../network/message/InvSpec.scala | 50 +++++++++++++ scalastyle-config.xml | 4 +- .../network/message/BasicMessagesRepo.scala | 73 ------------------- 5 files changed, 108 insertions(+), 76 deletions(-) create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala diff --git a/ergo-core/README.md b/ergo-core/README.md index e148110d67..c7ea82005a 100644 --- a/ergo-core/README.md +++ b/ergo-core/README.md @@ -41,8 +41,10 @@ Then, we are ready to create our Handshake message with peer spec and UNIX time ```scala import org.ergoplatform.network.Handshake import org.ergoplatform.network.PeerSpec +import org.ergoplatform.network.HandshakeSerializer val handshakeMessage = Handshake(mySpec, System.currentTimeMillis()) +val messageSerializedToBytes = HandshakeSerializer.toBytes(handshakeMessage) ``` Now we can serialize the message and send it If the message arrived successfully, we'll receive Handshake message back, so we can start to exchange messages with the node @@ -61,4 +63,20 @@ import org.ergoplatform.nodeView.history.ErgoSyncInfoMessageSpec val syncMessage = ErgoSyncInfoV2(Seq()) val messageSerializedToBytes = ErgoSyncInfoMessageSpec.toBytes(emptySync) -``` \ No newline at end of file +``` +The client will start receiving [InvData](src/main/scala/org/ergoplatform/network/message/InvData.scala) messages with +``` +/** + * P2P network message which is encoding "inventory", transactions or block sections the node has + * + * @param typeId + * @param ids + */ +``` +```scala +import org.ergoplatform.network.message.InvSpec + +{ invDataMessage => + val invData = InvSpec.parseBytesTry(invDataMessage) +} +``` diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala b/ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala new file mode 100644 index 0000000000..f2e6d3db97 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/HandshakeSerializer.scala @@ -0,0 +1,37 @@ +package org.ergoplatform.network + +import org.ergoplatform.network.message.MessageConstants.MessageCode +import org.ergoplatform.network.message.MessageSpecV1 +import scorex.util.serialization.{Reader, Writer} + +/** + * The `Handshake` message provides information about the transmitting node + * to the receiving node at the beginning of a connection. Until both peers + * have exchanged `Handshake` messages, no other messages will be accepted. + */ +object HandshakeSerializer extends MessageSpecV1[Handshake] { + override val messageCode: MessageCode = 75: Byte + override val messageName: String = "Handshake" + + val maxHandshakeSize: Int = 8096 + + /** + * Serializing handshake into a byte writer. + * + * @param hs - handshake instance + * @param w - writer to write bytes to + */ + override def serialize(hs: Handshake, w: Writer): Unit = { + // first writes down handshake time, then peer specification of our node + w.putULong(hs.time) + PeerSpecSerializer.serialize(hs.peerSpec, w) + } + + override def parse(r: Reader): Handshake = { + require(r.remaining <= maxHandshakeSize, s"Too big handshake. Size ${r.remaining} exceeds $maxHandshakeSize limit") + val time = r.getULong() + val data = PeerSpecSerializer.parse(r) + Handshake(data, time) + } + +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala new file mode 100644 index 0000000000..cf318f06a2 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala @@ -0,0 +1,50 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.NodeViewModifier +import org.ergoplatform.modifiers.NetworkObjectTypeId +import org.ergoplatform.network.message.MessageConstants.MessageCode +import scorex.util.Extensions.LongOps +import scorex.util.{bytesToId, idToBytes} +import scorex.util.serialization.{Reader, Writer} + +/** + * The `Inv` message (inventory message) transmits one or more inventories of + * objects known to the transmitting peer. + * It can be sent unsolicited to announce new transactions or blocks, + * or it can be sent in reply to a `SyncInfo` message (or application-specific messages like `GetMempool`). + * + */ +object InvSpec extends MessageSpecV1[InvData] { + + val maxInvObjects: Int = 400 + + override val messageCode: MessageCode = 55: Byte + override val messageName: String = "Inv" + + override def serialize(data: InvData, w: Writer): Unit = { + val typeId = data.typeId + val elems = data.ids + require(elems.nonEmpty, "empty inv list") + require(elems.lengthCompare(maxInvObjects) <= 0, s"more invs than $maxInvObjects in a message") + w.put(typeId) + w.putUInt(elems.size) + elems.foreach { id => + val bytes = idToBytes(id) + assert(bytes.length == NodeViewModifier.ModifierIdSize) + w.putBytes(bytes) + } + } + + override def parse(r: Reader): InvData = { + val typeId = NetworkObjectTypeId.fromByte(r.getByte()) + val count = r.getUInt().toIntExact + require(count > 0, "empty inv list") + require(count <= maxInvObjects, s"$count elements in a message while limit is $maxInvObjects") + val elems = (0 until count).map { _ => + bytesToId(r.getBytes(NodeViewModifier.ModifierIdSize)) + } + + InvData(typeId, elems) + } + +} diff --git a/scalastyle-config.xml b/scalastyle-config.xml index aba7f24c94..0df840521c 100644 --- a/scalastyle-config.xml +++ b/scalastyle-config.xml @@ -66,7 +66,7 @@ - + @@ -94,7 +94,7 @@ - + diff --git a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala index 11508302e7..4390744a7e 100644 --- a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala +++ b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala @@ -25,47 +25,7 @@ case class NipopowProofData(m: Int, k: Int, headerId: Option[ModifierId]) { -/** - * The `Inv` message (inventory message) transmits one or more inventories of - * objects known to the transmitting peer. - * It can be sent unsolicited to announce new transactions or blocks, - * or it can be sent in reply to a `SyncInfo` message (or application-specific messages like `GetMempool`). - * - */ -object InvSpec extends MessageSpecV1[InvData] { - - val maxInvObjects: Int = 400 - - override val messageCode: MessageCode = 55: Byte - override val messageName: String = "Inv" - - override def serialize(data: InvData, w: Writer): Unit = { - val typeId = data.typeId - val elems = data.ids - require(elems.nonEmpty, "empty inv list") - require(elems.lengthCompare(maxInvObjects) <= 0, s"more invs than $maxInvObjects in a message") - w.put(typeId) - w.putUInt(elems.size) - elems.foreach { id => - val bytes = idToBytes(id) - assert(bytes.length == ErgoNodeViewModifier.ModifierIdSize) - w.putBytes(bytes) - } - } - - override def parse(r: Reader): InvData = { - val typeId = NetworkObjectTypeId.fromByte(r.getByte()) - val count = r.getUInt().toIntExact - require(count > 0, "empty inv list") - require(count <= maxInvObjects, s"$count elements in a message while limit is $maxInvObjects") - val elems = (0 until count).map { _ => - bytesToId(r.getBytes(ErgoNodeViewModifier.ModifierIdSize)) - } - - InvData(typeId, elems) - } -} /** * The `RequestModifier` message requests one or more modifiers from another node. @@ -204,39 +164,6 @@ class PeersSpec(peersLimit: Int) extends MessageSpecV1[Seq[PeerSpec]] { } } -/** - * The `Handshake` message provides information about the transmitting node - * to the receiving node at the beginning of a connection. Until both peers - * have exchanged `Handshake` messages, no other messages will be accepted. - */ -object HandshakeSerializer extends MessageSpecV1[Handshake] { - override val messageCode: MessageCode = 75: Byte - override val messageName: String = "Handshake" - - val maxHandshakeSize: Int = 8096 - - /** - * Serializing handshake into a byte writer. - * - * @param hs - handshake instance - * @param w - writer to write bytes to - */ - override def serialize(hs: Handshake, w: Writer): Unit = { - // first writes down handshake time, then peer specification of our node - w.putULong(hs.time) - PeerSpecSerializer.serialize(hs.peerSpec, w) - } - - override def parse(r: Reader): Handshake = { - require(r.remaining <= maxHandshakeSize, s"Too big handshake. Size ${r.remaining} exceeds $maxHandshakeSize limit") - val time = r.getULong() - val data = PeerSpecSerializer.parse(r) - Handshake(data, time) - } - -} - - /** * The `GetSnapshotsInfo` message requests an `SnapshotsInfo` message from the receiving node */ From 1a02dc157495043fe93fedde9ec2fa42937c08f0 Mon Sep 17 00:00:00 2001 From: ccellado Date: Thu, 18 Jan 2024 10:05:22 +0400 Subject: [PATCH 05/23] Move classes required for communication & some fixes --- ergo-core/README.md | 4 +-- .../network/message/InvSpec.scala | 6 ++-- .../network/message/RequestModifierSpec.scala | 29 +++++++++++++++++ .../network/message/BasicMessagesRepo.scala | 31 +------------------ .../core/network/PeerConnectionHandler.scala | 4 +-- .../network/HandshakeSpecification.scala | 1 - 6 files changed, 37 insertions(+), 38 deletions(-) create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala diff --git a/ergo-core/README.md b/ergo-core/README.md index c7ea82005a..cde6653d35 100644 --- a/ergo-core/README.md +++ b/ergo-core/README.md @@ -44,7 +44,7 @@ import org.ergoplatform.network.PeerSpec import org.ergoplatform.network.HandshakeSerializer val handshakeMessage = Handshake(mySpec, System.currentTimeMillis()) -val messageSerializedToBytes = HandshakeSerializer.toBytes(handshakeMessage) +val handshakeMessageSerialized = HandshakeSerializer.toBytes(handshakeMessage) ``` Now we can serialize the message and send it If the message arrived successfully, we'll receive Handshake message back, so we can start to exchange messages with the node @@ -62,7 +62,7 @@ import org.ergoplatform.nodeView.history.ErgoSyncInfoV2 import org.ergoplatform.nodeView.history.ErgoSyncInfoMessageSpec val syncMessage = ErgoSyncInfoV2(Seq()) -val messageSerializedToBytes = ErgoSyncInfoMessageSpec.toBytes(emptySync) +val syncMessageSerialized = ErgoSyncInfoMessageSpec.toBytes(syncMessage) ``` The client will start receiving [InvData](src/main/scala/org/ergoplatform/network/message/InvData.scala) messages with ``` diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala index cf318f06a2..dec5a76902 100644 --- a/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/InvSpec.scala @@ -1,6 +1,6 @@ package org.ergoplatform.network.message -import org.ergoplatform.NodeViewModifier +import org.ergoplatform.modifiers.ErgoNodeViewModifier import org.ergoplatform.modifiers.NetworkObjectTypeId import org.ergoplatform.network.message.MessageConstants.MessageCode import scorex.util.Extensions.LongOps @@ -30,7 +30,7 @@ object InvSpec extends MessageSpecV1[InvData] { w.putUInt(elems.size) elems.foreach { id => val bytes = idToBytes(id) - assert(bytes.length == NodeViewModifier.ModifierIdSize) + assert(bytes.length == ErgoNodeViewModifier.ModifierIdSize) w.putBytes(bytes) } } @@ -41,7 +41,7 @@ object InvSpec extends MessageSpecV1[InvData] { require(count > 0, "empty inv list") require(count <= maxInvObjects, s"$count elements in a message while limit is $maxInvObjects") val elems = (0 until count).map { _ => - bytesToId(r.getBytes(NodeViewModifier.ModifierIdSize)) + bytesToId(r.getBytes(ErgoNodeViewModifier.ModifierIdSize)) } InvData(typeId, elems) diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala new file mode 100644 index 0000000000..1cf0e4d0d6 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala @@ -0,0 +1,29 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.network.message.MessageConstants.MessageCode +import scorex.util.serialization.{Reader, Writer} + +/** + * The `RequestModifier` message requests one or more modifiers from another node. + * The objects are requested by an inventory, which the requesting node + * typically received previously by way of an `Inv` message. + * + * This message cannot be used to request arbitrary data, such as historic transactions no + * longer in the memory pool. Full nodes may not even be able to provide older blocks if + * they’ve pruned old transactions from their block database. + * For this reason, the `RequestModifier` message should usually only be used to request + * data from a node which previously advertised it had that data by sending an `Inv` message. + * + */ +object RequestModifierSpec extends MessageSpecV1[InvData] { + override val messageCode: MessageCode = 22: Byte + override val messageName: String = "RequestModifier" + + override def serialize(data: InvData, w: Writer): Unit = { + InvSpec.serialize(data, w) + } + + override def parse(r: Reader): InvData = { + InvSpec.parse(r) + } +} diff --git a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala index 4390744a7e..f7cdea8c55 100644 --- a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala +++ b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala @@ -1,7 +1,7 @@ package org.ergoplatform.network.message import org.ergoplatform.modifiers.{ErgoNodeViewModifier, NetworkObjectTypeId} -import org.ergoplatform.network.{Handshake, PeerSpec, PeerSpecSerializer} +import org.ergoplatform.network.{PeerSpec, PeerSpecSerializer} import org.ergoplatform.nodeView.state.SnapshotsInfo import org.ergoplatform.nodeView.state.UtxoState.{ManifestId, SubtreeId} import org.ergoplatform.settings.Algos @@ -23,35 +23,6 @@ case class NipopowProofData(m: Int, k: Int, headerId: Option[ModifierId]) { def headerIdBytesOpt: Option[Array[Byte]] = headerId.map(Algos.decode).flatMap(_.toOption) } - - - - -/** - * The `RequestModifier` message requests one or more modifiers from another node. - * The objects are requested by an inventory, which the requesting node - * typically received previously by way of an `Inv` message. - * - * This message cannot be used to request arbitrary data, such as historic transactions no - * longer in the memory pool. Full nodes may not even be able to provide older blocks if - * they’ve pruned old transactions from their block database. - * For this reason, the `RequestModifier` message should usually only be used to request - * data from a node which previously advertised it had that data by sending an `Inv` message. - * - */ -object RequestModifierSpec extends MessageSpecV1[InvData] { - override val messageCode: MessageCode = 22: Byte - override val messageName: String = "RequestModifier" - - override def serialize(data: InvData, w: Writer): Unit = { - InvSpec.serialize(data, w) - } - - override def parse(r: Reader): InvData = { - InvSpec.parse(r) - } -} - /** * The `Modifier` message is a reply to a `RequestModifier` message which requested these modifiers. */ diff --git a/src/main/scala/scorex/core/network/PeerConnectionHandler.scala b/src/main/scala/scorex/core/network/PeerConnectionHandler.scala index 32dc1e25a9..8c32487397 100644 --- a/src/main/scala/scorex/core/network/PeerConnectionHandler.scala +++ b/src/main/scala/scorex/core/network/PeerConnectionHandler.scala @@ -4,12 +4,12 @@ import akka.actor.{Actor, ActorRef, Cancellable, Props, SupervisorStrategy} import akka.io.Tcp import akka.io.Tcp._ import akka.util.{ByteString, CompactByteString} -import org.ergoplatform.network.{Handshake, PeerSpec, Version} +import org.ergoplatform.network.{Handshake, HandshakeSerializer, PeerSpec, Version} import org.ergoplatform.network.Version.Eip37ForkVersion import scorex.core.app.ScorexContext import scorex.core.network.NetworkController.ReceivableMessages.{Handshaked, PenalizePeer} import scorex.core.network.PeerConnectionHandler.ReceivableMessages -import org.ergoplatform.network.message.{HandshakeSerializer, MessageSerializer} +import org.ergoplatform.network.message.MessageSerializer import org.ergoplatform.network.peer.{PeerInfo, PenaltyType} import org.ergoplatform.settings.ScorexSettings import scorex.util.ScorexLogging diff --git a/src/test/scala/org/ergoplatform/network/HandshakeSpecification.scala b/src/test/scala/org/ergoplatform/network/HandshakeSpecification.scala index 44f71edebb..3c3a22c960 100644 --- a/src/test/scala/org/ergoplatform/network/HandshakeSpecification.scala +++ b/src/test/scala/org/ergoplatform/network/HandshakeSpecification.scala @@ -3,7 +3,6 @@ package org.ergoplatform.network import java.nio.ByteBuffer import org.ergoplatform.nodeView.state.StateType import org.ergoplatform.utils.ErgoPropertyTest -import org.ergoplatform.network.message.HandshakeSerializer import scorex.util.encode.Base16 class HandshakeSpecification extends ErgoPropertyTest with DecodingUtils { From 0a2b89354bfa6a189997ace8d84ff0369a9ead42 Mon Sep 17 00:00:00 2001 From: ccellado Date: Thu, 18 Jan 2024 13:04:36 +0400 Subject: [PATCH 06/23] Move classes required for communication & some fixes --- .../network/message/ModifiersData.scala | 9 +++ .../network/message/ModifiersSpec.scala | 68 ++++++++++++++++++ .../network/message/BasicMessagesRepo.scala | 70 +------------------ 3 files changed, 78 insertions(+), 69 deletions(-) create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersData.scala create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersSpec.scala diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersData.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersData.scala new file mode 100644 index 0000000000..ad01035252 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersData.scala @@ -0,0 +1,9 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.modifiers.NetworkObjectTypeId +import scorex.util.ModifierId + +/** + * Wrapper for block sections of the same type. Used to send multiple block sections at once ove the wire. + */ +case class ModifiersData(typeId: NetworkObjectTypeId.Value, modifiers: Map[ModifierId, Array[Byte]]) diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersSpec.scala new file mode 100644 index 0000000000..c1d1118cd5 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/ModifiersSpec.scala @@ -0,0 +1,68 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.modifiers.{ErgoNodeViewModifier, NetworkObjectTypeId} +import org.ergoplatform.network.message.MessageConstants.MessageCode +import scorex.util.{ModifierId, ScorexLogging, bytesToId, idToBytes} +import scorex.util.serialization.{Reader, Writer} +import scorex.util.Extensions._ +import scala.collection.immutable + +/** + * The `Modifier` message is a reply to a `RequestModifier` message which requested these modifiers. + */ +object ModifiersSpec extends MessageSpecV1[ModifiersData] with ScorexLogging { + + val maxMessageSize: Int = 2048576 + + private val maxMsgSizeWithReserve = maxMessageSize * 4 // due to big ADProofs + + override val messageCode: MessageCode = 33: Byte + override val messageName: String = "Modifier" + + private val HeaderLength = 5 // msg type Id + modifiersCount + + override def serialize(data: ModifiersData, w: Writer): Unit = { + + val typeId = data.typeId + val modifiers = data.modifiers + require(modifiers.nonEmpty, "empty modifiers list") + + val (msgCount, msgSize) = modifiers.foldLeft((0, HeaderLength)) { case ((c, s), (_, modifier)) => + val size = s + ErgoNodeViewModifier.ModifierIdSize + 4 + modifier.length + val count = if (size <= maxMsgSizeWithReserve) c + 1 else c + count -> size + } + + w.put(typeId) + w.putUInt(msgCount) + + modifiers.take(msgCount).foreach { case (id, modifier) => + w.putBytes(idToBytes(id)) + w.putUInt(modifier.length) + w.putBytes(modifier) + } + + if (msgSize > maxMsgSizeWithReserve) { + log.warn(s"Message with modifiers ${modifiers.keySet} has size $msgSize exceeding limit $maxMsgSizeWithReserve.") + } + } + + override def parse(r: Reader): ModifiersData = { + val typeId = NetworkObjectTypeId.fromByte(r.getByte()) // 1 byte + val count = r.getUInt().toIntExact // 8 bytes + require(count > 0, s"Illegal message with 0 modifiers of type $typeId") + val resMap = immutable.Map.newBuilder[ModifierId, Array[Byte]] + (0 until count).foldLeft(HeaderLength) { case (msgSize, _) => + val id = bytesToId(r.getBytes(ErgoNodeViewModifier.ModifierIdSize)) + val objBytesCnt = r.getUInt().toIntExact + val newMsgSize = msgSize + ErgoNodeViewModifier.ModifierIdSize + objBytesCnt + if (newMsgSize > maxMsgSizeWithReserve) { // buffer for safety + throw new Exception("Too big message with modifiers, size: " + maxMsgSizeWithReserve) + } + val obj = r.getBytes(objBytesCnt) + resMap += (id -> obj) + newMsgSize + } + ModifiersData(typeId, resMap.result()) + } +} diff --git a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala index f7cdea8c55..6c31d4c588 100644 --- a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala +++ b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala @@ -1,6 +1,5 @@ package org.ergoplatform.network.message -import org.ergoplatform.modifiers.{ErgoNodeViewModifier, NetworkObjectTypeId} import org.ergoplatform.network.{PeerSpec, PeerSpecSerializer} import org.ergoplatform.nodeView.state.SnapshotsInfo import org.ergoplatform.nodeView.state.UtxoState.{ManifestId, SubtreeId} @@ -9,80 +8,13 @@ import org.ergoplatform.network.message.MessageConstants.MessageCode import scorex.crypto.hash.Digest32 import scorex.util.Extensions._ import scorex.util.serialization.{Reader, Writer} -import scorex.util.{ModifierId, ScorexLogging, bytesToId, idToBytes} +import scorex.util.ModifierId import org.ergoplatform.sdk.wallet.Constants.ModifierIdLength -import scala.collection.immutable - -/** - * Wrapper for block sections of the same type. Used to send multiple block sections at once ove the wire. - */ -case class ModifiersData(typeId: NetworkObjectTypeId.Value, modifiers: Map[ModifierId, Array[Byte]]) - case class NipopowProofData(m: Int, k: Int, headerId: Option[ModifierId]) { def headerIdBytesOpt: Option[Array[Byte]] = headerId.map(Algos.decode).flatMap(_.toOption) } -/** - * The `Modifier` message is a reply to a `RequestModifier` message which requested these modifiers. - */ -object ModifiersSpec extends MessageSpecV1[ModifiersData] with ScorexLogging { - - val maxMessageSize: Int = 2048576 - - private val maxMsgSizeWithReserve = maxMessageSize * 4 // due to big ADProofs - - override val messageCode: MessageCode = 33: Byte - override val messageName: String = "Modifier" - - private val HeaderLength = 5 // msg type Id + modifiersCount - - override def serialize(data: ModifiersData, w: Writer): Unit = { - - val typeId = data.typeId - val modifiers = data.modifiers - require(modifiers.nonEmpty, "empty modifiers list") - - val (msgCount, msgSize) = modifiers.foldLeft((0, HeaderLength)) { case ((c, s), (_, modifier)) => - val size = s + ErgoNodeViewModifier.ModifierIdSize + 4 + modifier.length - val count = if (size <= maxMsgSizeWithReserve) c + 1 else c - count -> size - } - - w.put(typeId) - w.putUInt(msgCount) - - modifiers.take(msgCount).foreach { case (id, modifier) => - w.putBytes(idToBytes(id)) - w.putUInt(modifier.length) - w.putBytes(modifier) - } - - if (msgSize > maxMsgSizeWithReserve) { - log.warn(s"Message with modifiers ${modifiers.keySet} has size $msgSize exceeding limit $maxMsgSizeWithReserve.") - } - } - - override def parse(r: Reader): ModifiersData = { - val typeId = NetworkObjectTypeId.fromByte(r.getByte()) // 1 byte - val count = r.getUInt().toIntExact // 8 bytes - require(count > 0, s"Illegal message with 0 modifiers of type $typeId") - val resMap = immutable.Map.newBuilder[ModifierId, Array[Byte]] - (0 until count).foldLeft(HeaderLength) { case (msgSize, _) => - val id = bytesToId(r.getBytes(ErgoNodeViewModifier.ModifierIdSize)) - val objBytesCnt = r.getUInt().toIntExact - val newMsgSize = msgSize + ErgoNodeViewModifier.ModifierIdSize + objBytesCnt - if (newMsgSize > maxMsgSizeWithReserve) { // buffer for safety - throw new Exception("Too big message with modifiers, size: " + maxMsgSizeWithReserve) - } - val obj = r.getBytes(objBytesCnt) - resMap += (id -> obj) - newMsgSize - } - ModifiersData(typeId, resMap.result()) - } -} - /** * The `GetPeer` message requests an `Peers` message from the receiving node, * preferably one with lots of `PeerSpec` of other receiving nodes. From 1063b50ec2394f42afeeb34950d9260dce693557 Mon Sep 17 00:00:00 2001 From: Colby Cellador <44584960+ccellado@users.noreply.github.com> Date: Fri, 19 Jan 2024 01:15:49 +0400 Subject: [PATCH 07/23] Update release-binaries.py Fix tag filename --- ci/release-binaries.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ci/release-binaries.py b/ci/release-binaries.py index 6c79be04c0..0182c0413e 100644 --- a/ci/release-binaries.py +++ b/ci/release-binaries.py @@ -25,7 +25,8 @@ MAIN_JRE = os.environ.get("ERGO_RELEASE_PLATFORM") VERSION = os.environ.get("ERGO_RELEASE_TAG") -JAR_FILENAME = f"ergo-{VERSION}.jar" +# filename is ergo-X.XXX.jar but tag version is vX.XXX +JAR_FILENAME = f"ergo-{VERSION[1:]}.jar" SCRIPT_LOGO = f""" .-*%@#+-. From cac704444802bd74a1f144d53ffbff9161486482 Mon Sep 17 00:00:00 2001 From: ccellado Date: Fri, 19 Jan 2024 09:13:35 +0400 Subject: [PATCH 08/23] Move classes required for communication & some fixes --- .../network/message/GetNipopowProofSpec.scala | 50 ++++++++++++ .../network/message/NipopowProofData.scala | 8 ++ .../network/message/NipopowProofSpec.scala | 32 ++++++++ .../network/message/BasicMessagesRepo.scala | 77 ------------------- 4 files changed, 90 insertions(+), 77 deletions(-) create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/message/GetNipopowProofSpec.scala create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofData.scala create mode 100644 ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/GetNipopowProofSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/GetNipopowProofSpec.scala new file mode 100644 index 0000000000..a28dd40e7c --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/GetNipopowProofSpec.scala @@ -0,0 +1,50 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.network.message.MessageConstants.MessageCode +import org.ergoplatform.sdk.wallet.Constants.ModifierIdLength +import org.ergoplatform.settings.Algos +import scorex.util.ModifierId +import scorex.util.serialization.{Reader, Writer} + +/** + * The `GetNipopowProof` message requests a `NipopowProof` message from the receiving node + */ +object GetNipopowProofSpec extends MessageSpecV1[NipopowProofData] { + + val SizeLimit = 1000 + + val messageCode: MessageCode = 90: Byte + val messageName: String = "GetNipopowProof" + + override def serialize(data: NipopowProofData, w: Writer): Unit = { + w.putInt(data.m) + w.putInt(data.k) + data.headerIdBytesOpt match { + case Some(idBytes) => + w.put(1) + w.putBytes(idBytes) + case None => + w.put(0) + } + w.putUShort(0) // to allow adding new data in future, we are adding possible pad length + } + + override def parse(r: Reader): NipopowProofData = { + require(r.remaining <= SizeLimit, s"Too big GetNipopowProofSpec message(size: ${r.remaining})") + + val m = r.getInt() + val k = r.getInt() + + val headerIdPresents = r.getByte() == 1 + val headerIdOpt = if (headerIdPresents) { + Some(ModifierId @@ Algos.encode(r.getBytes(ModifierIdLength))) + } else { + None + } + val remainingBytes = r.getUShort() + if (remainingBytes > 0 && remainingBytes < SizeLimit) { + r.getBytes(remainingBytes) // current version of reader just skips possible additional bytes + } + NipopowProofData(m, k, headerIdOpt) + } +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofData.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofData.scala new file mode 100644 index 0000000000..1d7f0b8ed4 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofData.scala @@ -0,0 +1,8 @@ +package org.ergoplatform.network.message + +import org.ergoplatform.settings.Algos +import scorex.util.ModifierId + +case class NipopowProofData(m: Int, k: Int, headerId: Option[ModifierId]) { + def headerIdBytesOpt: Option[Array[Byte]] = headerId.map(Algos.decode).flatMap(_.toOption) +} diff --git a/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala b/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala new file mode 100644 index 0000000000..9806961126 --- /dev/null +++ b/ergo-core/src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala @@ -0,0 +1,32 @@ +package org.ergoplatform.network.message + +import scorex.util.Extensions._ +import scorex.util.serialization.{Reader, Writer} + +/** + * The `NipopowProof` message is a reply to a `GetNipopowProof` message. + */ +object NipopowProofSpec extends MessageSpecV1[Array[Byte]] { + + val SizeLimit = 2000000 + override val messageCode: Byte = 91 + override val messageName: String = "NipopowProof" + + override def serialize(proof: Array[Byte], w: Writer): Unit = { + w.putUInt(proof.length) + w.putBytes(proof) + w.putUShort(0) // to allow adding new data in future, we are adding possible pad length + } + + override def parse(r: Reader): Array[Byte] = { + require(r.remaining <= SizeLimit, s"Too big NipopowProofSpec message(size: ${r.remaining})") + val proofSize = r.getUInt().toIntExact + require(proofSize > 0 && proofSize < SizeLimit) + val proof = r.getBytes(proofSize) + val remainingBytes = r.getUShort() + if (remainingBytes > 0 && remainingBytes < SizeLimit) { + r.getBytes(remainingBytes) // current version of reader just skips possible additional bytes + } + proof + } +} diff --git a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala index 6c31d4c588..82ac60dde5 100644 --- a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala +++ b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala @@ -11,10 +11,6 @@ import scorex.util.serialization.{Reader, Writer} import scorex.util.ModifierId import org.ergoplatform.sdk.wallet.Constants.ModifierIdLength -case class NipopowProofData(m: Int, k: Int, headerId: Option[ModifierId]) { - def headerIdBytesOpt: Option[Array[Byte]] = headerId.map(Algos.decode).flatMap(_.toOption) -} - /** * The `GetPeer` message requests an `Peers` message from the receiving node, * preferably one with lots of `PeerSpec` of other receiving nodes. @@ -207,76 +203,3 @@ object UtxoSnapshotChunkSpec extends MessageSpecV1[Array[Byte]] { } } - -/** - * The `GetNipopowProof` message requests a `NipopowProof` message from the receiving node - */ -object GetNipopowProofSpec extends MessageSpecV1[NipopowProofData] { - - val SizeLimit = 1000 - - val messageCode: MessageCode = 90: Byte - val messageName: String = "GetNipopowProof" - - override def serialize(data: NipopowProofData, w: Writer): Unit = { - w.putInt(data.m) - w.putInt(data.k) - data.headerIdBytesOpt match { - case Some(idBytes) => - w.put(1) - w.putBytes(idBytes) - case None => - w.put(0) - } - w.putUShort(0) // to allow adding new data in future, we are adding possible pad length - } - - override def parse(r: Reader): NipopowProofData = { - require(r.remaining <= SizeLimit, s"Too big GetNipopowProofSpec message(size: ${r.remaining})") - - val m = r.getInt() - val k = r.getInt() - - val headerIdPresents = r.getByte() == 1 - val headerIdOpt = if (headerIdPresents) { - Some(ModifierId @@ Algos.encode(r.getBytes(ModifierIdLength))) - } else { - None - } - val remainingBytes = r.getUShort() - if (remainingBytes > 0 && remainingBytes < SizeLimit) { - r.getBytes(remainingBytes) // current version of reader just skips possible additional bytes - } - NipopowProofData(m, k, headerIdOpt) - } - -} - -/** - * The `NipopowProof` message is a reply to a `GetNipopowProof` message. - */ -object NipopowProofSpec extends MessageSpecV1[Array[Byte]] { - - val SizeLimit = 2000000 - override val messageCode: Byte = 91 - override val messageName: String = "NipopowProof" - - override def serialize(proof: Array[Byte], w: Writer): Unit = { - w.putUInt(proof.length) - w.putBytes(proof) - w.putUShort(0) // to allow adding new data in future, we are adding possible pad length - } - - override def parse(r: Reader): Array[Byte] = { - require(r.remaining <= SizeLimit, s"Too big NipopowProofSpec message(size: ${r.remaining})") - val proofSize = r.getUInt().toIntExact - require(proofSize > 0 && proofSize < SizeLimit) - val proof = r.getBytes(proofSize) - val remainingBytes = r.getUShort() - if (remainingBytes > 0 && remainingBytes < SizeLimit) { - r.getBytes(remainingBytes) // current version of reader just skips possible additional bytes - } - proof - } - -} From c83aa33a2fa13beb3f9b140e35c2cc60a8344450 Mon Sep 17 00:00:00 2001 From: ccellado Date: Tue, 23 Jan 2024 16:31:07 +0400 Subject: [PATCH 09/23] Move classes required for communication & some fixes --- ergo-core/README.md | 155 ++++++++++++++---- .../nodeView/history/ErgoHistoryUtils.scala | 8 + .../modifierprocessors/PopowProcessor.scala | 8 +- 3 files changed, 137 insertions(+), 34 deletions(-) diff --git a/ergo-core/README.md b/ergo-core/README.md index cde6653d35..3dc70112db 100644 --- a/ergo-core/README.md +++ b/ergo-core/README.md @@ -1,11 +1,31 @@ -# Ergo Core library +# ergo-core ## Establishing connections to the node -1. We need to send a Handshake message +```mermaid +sequenceDiagram + Client-->>+Node: Establish TCP connection + Node->>+Client: Handshake + Client->>Node: Handhake + Note over Client,Node: Message exchange started +``` -Create PeerSpec and all parameters (including features) +### Connect to peer 📞 +First connect to peer node and get `Handshake` message. +For the rest of the guide assume the `Message body` as `byteBuffer`. +```scala +import org.ergoplatform.network.HandshakeSerializer + +{ byteBuffer => + val handshake = HandshakeSerializer.parseBytesTry(byteBuffer) +} +``` + +### Handshake 🤝 +After getting and successfully reading `Handshake` message from peer, the peer expects to receive the `Handshake` back + +Create PeerSpec and all parameters (including features) [PeerSpec](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala)'s doc: ``` * Declared information about peer @@ -19,25 +39,26 @@ Create PeerSpec and all parameters (including features) * @param declaredAddress - Public network address of the node if any * @param features - Set of node capabilities ``` - -All the available peer features are stored inside [PeerFeatureDescriptors](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala) - - ```scala import org.ergoplatform.network.PeerSpec import org.ergoplatform.network.Version import java.net.InetSocketAddress val mySpec = PeerSpec( - agentName = "MyCoolErgoClient:0.1", + agentName = "morphicus", protocolVersion = Version("version of the ergo-core library"), - nodeName = "MyNodeName", - declaredAddress = Some(InetSocketAddress("https://ergocoolclient.xyz", "5016")), - features = Seq() + nodeName = "ToTheMoon", + // required for non-local communication + declaredAddress = Some(InetSocketAddress("tothemoon.ergo", "5016")), + // note [1] + features = Seq.empty ) ``` -Then, we are ready to create our Handshake message with peer spec and UNIX time of the message +[1] All the available peer features [PeerFeatureDescriptors](src/main/scala/org/ergoplatform/settings/PeerFeatureDescriptors.scala) + + +Create Handshake message with peer spec and UNIX time of the message ```scala import org.ergoplatform.network.Handshake import org.ergoplatform.network.PeerSpec @@ -46,17 +67,33 @@ import org.ergoplatform.network.HandshakeSerializer val handshakeMessage = Handshake(mySpec, System.currentTimeMillis()) val handshakeMessageSerialized = HandshakeSerializer.toBytes(handshakeMessage) ``` -Now we can serialize the message and send it -If the message arrived successfully, we'll receive Handshake message back, so we can start to exchange messages with the node +Serialize the message and send it. +If the message arrived successfully, start communicating with the peer node. -2. Now we can start syncing with the node +All communication is wrapped with Message headers, format described [here](https://docs.ergoplatform.com/dev/p2p/network/#message-format). +## Syncing with the node + +```mermaid +sequenceDiagram + Client-->>Node: SyncInfo( empty ) + Node->>Client: InvData + Client-->Client: Checking local db + Client-->>Node: RequestModifiers( InvData ) + Node->>Client: ModifiersData + Client-->Client: Checking PoW + Note over Client,Node: Client verified N headers + Client-->>Node: SyncInfo( List of N headers ) + Node->>Client: InvData( with succeeding headers ) + Note over Client,Node: Repeat ``` -/** - * @param lastHeaders - some recent headers (including last one) known to a peer - */ -``` -Send [ErgoSyncInfoV2](src/main/scala/org/ergoplatform/nodeView/history/ErgoSyncInfo.scala) with empty `lastHeaders`, so the node knows we're just beginning to sync + +The peer node will start sending `SyncInfo` messages to us, since it is checking for new block information. +Our client is syncing instead. + +### Send [ErgoSyncInfoV2](src/main/scala/org/ergoplatform/nodeView/history/ErgoSyncInfo.scala) ♲ +with empty `lastHeaders`, so the node knows client is just beginning to sync + ```scala import org.ergoplatform.nodeView.history.ErgoSyncInfoV2 import org.ergoplatform.nodeView.history.ErgoSyncInfoMessageSpec @@ -64,19 +101,77 @@ import org.ergoplatform.nodeView.history.ErgoSyncInfoMessageSpec val syncMessage = ErgoSyncInfoV2(Seq()) val syncMessageSerialized = ErgoSyncInfoMessageSpec.toBytes(syncMessage) ``` -The client will start receiving [InvData](src/main/scala/org/ergoplatform/network/message/InvData.scala) messages with +Node replies with [InvData](src/main/scala/org/ergoplatform/network/message/InvData.scala) message, containing youngest headers id's the node has. +Here the client checks if it has the headers already verified in its local db. The semantics of this is out of the tutorial scope. + +Reply with `RequestModifier` message containing `InvData` the peer node sent previous. + +### Send [RequestModifiers](src/main/scala/org/ergoplatform/network/message/RequestModifierSpec.scala) 📥 + +```scala +import org.ergoplatform.network.message.InvSpec +import org.ergoplatform.network.message.RequestModifierSpec + +{ byteBuffer => + val invData = InvSpec.parseBytesTry(byteBuffer) + val message = RequestModifierSpec.toBytes(invData) +} ``` -/** - * P2P network message which is encoding "inventory", transactions or block sections the node has - * - * @param typeId - * @param ids - */ +Now received [ModifiersData](src/main/scala/org/ergoplatform/network/message/ModifiersData.scala) with block data. + +### Checking the PoW headers 🕵️ + +Before adding those blocks to local database, check the headers. + +```scala +import org.ergoplatform.network.message.ModifiersSpec +import org.ergoplatform.modifiers.history.header.HeaderSerializer + +{ byteBuffer => + val data = ModifiersSpec.parseBytes(byteBuffer) + val blockDataVerified = data.modifiers.map((id, bytes) => + (id, HeaderSerializer.parseBytes(bytes)) + ) +} +``` +If successful, `blockDataVerified` contains the map with `Headers` of block data. Cheers 🙌 + +## Checking NiPoPoW proofs + +[What is NiPoPoW](https://docs.ergoplatform.com/dev/protocol/nipopows/) + +### Request NiPoPoW proof 🛸 + +```scala +import org.ergoplatform.network.message.{GetNipopowProofSpec, NipopowProofData} +import org.ergoplatform.nodeView.history.ErgoHistoryUtils.{P2PNipopowProofM, P2PNipopowProofK} + +val nipopowRequest = GetNipopowProofSpec.toBytes( + NipopowProofData( + m = P2PNipopowProofM, + k = P2PNipopowProofK, + None + ) +) ``` + +Received the [NipopowProofSpec](src/main/scala/org/ergoplatform/network/message/NipopowProofSpec.scala) message. + +### Verify NiPoPoW proof 🦾 + +Need to have [chainSettings](src/main/scala/org/ergoplatform/settings/ChainSettings.scala) in order to make a `nipopoSerializer` instance. + ```scala -import org.ergoplatform.network.message.InvSpec +import org.ergoplatform.modifiers.history.popow.{NipopowAlgos, NipopowProofSerializer} +import org.ergoplatform.network.message.NipopowProofSpec + +lazy val nipopowAlgos: NipopowAlgos = new NipopowAlgos(chainSettings) +lazy val nipopowSerializer = new NipopowProofSerializer(nipopowAlgos) -{ invDataMessage => - val invData = InvSpec.parseBytesTry(invDataMessage) +{ byteBuffer => + val data = NipopowProofSpec.parseBytes(byteBuffer) + val proofData = nipopowSerializer.parseBytes(data) } ``` + +If successful, `proofData.isValid` will be true. Cheers 👽 \ No newline at end of file diff --git a/ergo-core/src/main/scala/org/ergoplatform/nodeView/history/ErgoHistoryUtils.scala b/ergo-core/src/main/scala/org/ergoplatform/nodeView/history/ErgoHistoryUtils.scala index 3f1f3801fd..27f8a55774 100644 --- a/ergo-core/src/main/scala/org/ergoplatform/nodeView/history/ErgoHistoryUtils.scala +++ b/ergo-core/src/main/scala/org/ergoplatform/nodeView/history/ErgoHistoryUtils.scala @@ -23,6 +23,14 @@ object ErgoHistoryUtils { val EmptyHistoryHeight: Int = 0 val GenesisHeight: Int = EmptyHistoryHeight + 1 // first block has height == 1 + /** + * Minimal superchain length ('m' in KMZ17 paper) value used in NiPoPoW proofs for bootstrapping + */ + val P2PNipopowProofM = 6 + /** + * Suffix length ('k' in KMZ17 paper) value used in NiPoPoW proofs for bootstrapping + */ + val P2PNipopowProofK = 10 def heightOf(headerOpt: Option[Header]): Int = headerOpt.map(_.height).getOrElse(EmptyHistoryHeight) } diff --git a/src/main/scala/org/ergoplatform/nodeView/history/storage/modifierprocessors/PopowProcessor.scala b/src/main/scala/org/ergoplatform/nodeView/history/storage/modifierprocessors/PopowProcessor.scala index 8baa167e40..59922347a3 100644 --- a/src/main/scala/org/ergoplatform/nodeView/history/storage/modifierprocessors/PopowProcessor.scala +++ b/src/main/scala/org/ergoplatform/nodeView/history/storage/modifierprocessors/PopowProcessor.scala @@ -6,7 +6,7 @@ import org.ergoplatform.modifiers.BlockSection import org.ergoplatform.modifiers.history.extension.Extension import org.ergoplatform.modifiers.history.header.Header import org.ergoplatform.modifiers.history.popow.{NipopowAlgos, NipopowProverWithDbAlgs, NipopowProof, NipopowProofSerializer, PoPowHeader, PoPowParams} -import org.ergoplatform.nodeView.history.ErgoHistoryUtils.GenesisHeight +import org.ergoplatform.nodeView.history.ErgoHistoryUtils import org.ergoplatform.nodeView.history.ErgoHistoryReader import org.ergoplatform.settings.{ChainSettings, NipopowSettings} import org.ergoplatform.settings.Constants.HashLength @@ -41,19 +41,19 @@ trait PopowProcessor extends BasicReaders with ScorexLogging { lazy val nipopowSerializer = new NipopowProofSerializer(nipopowAlgos) private lazy val nipopowVerifier = - new NipopowVerifier(chainSettings.genesisId.orElse(bestHeaderIdAtHeight(GenesisHeight))) + new NipopowVerifier(chainSettings.genesisId.orElse(bestHeaderIdAtHeight(ErgoHistoryUtils.GenesisHeight))) protected val NipopowSnapshotHeightKey: ByteArrayWrapper = ByteArrayWrapper(Array.fill(HashLength)(50: Byte)) /** * Minimal superchain length ('m' in KMZ17 paper) value used in NiPoPoW proofs for bootstrapping */ - val P2PNipopowProofM = 6 + val P2PNipopowProofM = ErgoHistoryUtils.P2PNipopowProofM /** * Suffix length ('k' in KMZ17 paper) value used in NiPoPoW proofs for bootstrapping */ - val P2PNipopowProofK = 10 + val P2PNipopowProofK = ErgoHistoryUtils.P2PNipopowProofK /** * Checks and appends new header to history From 387925241a0bb98eee21f36aa70ad531263ef88b Mon Sep 17 00:00:00 2001 From: ccellado Date: Tue, 23 Jan 2024 16:33:18 +0400 Subject: [PATCH 10/23] Move classes required for communication & some fixes --- ergo-core/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ergo-core/README.md b/ergo-core/README.md index 3dc70112db..3290f36643 100644 --- a/ergo-core/README.md +++ b/ergo-core/README.md @@ -1,5 +1,7 @@ # ergo-core +Toy working example client code available [here](https://github.com/ccellado/ergo-test-client/tree/main). + ## Establishing connections to the node ```mermaid From a50fdbe5d4c9e8af29fa4cca0b9e5e3cb1b20a2d Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Tue, 23 Jan 2024 16:46:07 +0300 Subject: [PATCH 11/23] more ContextExtension tests --- .../mempool/ErgoTransactionSpec.scala | 59 ++++++++++++++++++- 1 file changed, 57 insertions(+), 2 deletions(-) diff --git a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala index 371b356384..176382ae18 100644 --- a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala +++ b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala @@ -104,9 +104,9 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { }.get val inputs: IndexedSeq[Input] = IndexedSeq( new Input(ADKey @@ Base16.decode("c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742").get, - new ProverResult(Base16.decode("b4a04b4201da0578be3dac11067b567a73831f35b024a2e623c1f8da230407f63bab62c62ed9b93808b106b5a7e8b1751fa656f4c5de4674").get, new ContextExtension(Map()))), + new ProverResult(Base16.decode("b4a04b4201da0578be3dac11067b567a73831f35b024a2e623c1f8da230407f63bab62c62ed9b93808b106b5a7e8b1751fa656f4c5de4674").get, ContextExtension.empty)), new Input(ADKey @@ Base16.decode("ca796a4fc9c0d746a69702a77bd78b1a80a5ef5bf5713bbd95d93a4f23b27ead").get, - new ProverResult(Base16.decode("5aea4d78a234c35accacdf8996b0af5b51e26fee29ea5c05468f23707d31c0df39400127391cd57a70eb856710db48bb9833606e0bf90340").get, new ContextExtension(Map()))), + new ProverResult(Base16.decode("5aea4d78a234c35accacdf8996b0af5b51e26fee29ea5c05468f23707d31c0df39400127391cd57a70eb856710db48bb9833606e0bf90340").get, ContextExtension.empty)), ) val outputCandidates: IndexedSeq[ErgoBoxCandidate] = IndexedSeq( new ErgoBoxCandidate(1000000000L, minerPk, height, Colls.emptyColl, Map()), @@ -543,4 +543,59 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { } } + property("context extension with neg id") { + val negId: Byte = -20 + + ADKey @@ Base16.decode("c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742").get + + val b = new ErgoBox(1000000000L, Constants.TrueLeaf, Colls.emptyColl, + Map.empty, ModifierId @@ "c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742", + 0, 0) + val input = Input(b.id, ProverResult(Array.emptyByteArray, ContextExtension(Map(negId -> IntConstant(0))))) + + val oc = new ErgoBoxCandidate(b.value, b.ergoTree, b.creationHeight) + + val utx = UnsignedErgoTransaction(IndexedSeq(input), IndexedSeq(oc)) + + val tx = defaultProver.sign(utx, IndexedSeq(b), IndexedSeq.empty, emptyStateContext) + .map(ErgoTransaction.apply) + .get + + tx.statefulValidity(IndexedSeq(b), IndexedSeq.empty, emptyStateContext) shouldBe true + } + + /* + import sigma.{AnyValue, Coll} + import sigmastate.SType + import sigmastate.SType.AnyOps + import sigmastate.Values.{ByteArrayConstant, EvaluatedValue} + import sigmastate.eval.CostingSigmaDslBuilder.Colls + import sigmastate.eval.Evaluation.stypeToRType + import sigmastate.eval.Extensions.toAnyValue + import sigmastate.interpreter.ContextExtension + + val ce = ContextExtension(Map(-6.toByte -> ByteArrayConstant(Array(0: Byte)))) + val ce2 = ContextExtension.serializer.fromBytes(ContextExtension.serializer.toBytes(ce)) + + def contextVars(m: Map[Byte, AnyValue]): Coll[AnyValue] = { + val maxKey = if (m.keys.isEmpty) 0 else m.keys.max // TODO optimize: max takes 90% of this method + val res = new Array[AnyValue](maxKey + 1) + for ((id, v) <- m) { + res(id) = v + } + Colls.fromArray(res) + } + + if (false) { + val varMap = ce2.values.map { case (k, v: EvaluatedValue[_]) => + val tVal = stypeToRType[SType](v.tpe) + k -> toAnyValue(v.value.asWrappedType)(tVal) + }.toMap + contextVars(varMap) + } */ + + + // val arr = new Array[Int](-5) + + } From 82fc3e5ddfc8d12f933ffc93bfde9af809a3d22a Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Tue, 23 Jan 2024 17:12:20 +0300 Subject: [PATCH 12/23] neg and pos ids test --- .../mempool/ErgoTransactionSpec.scala | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala index 176382ae18..68cd3d37a5 100644 --- a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala +++ b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala @@ -564,6 +564,29 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { tx.statefulValidity(IndexedSeq(b), IndexedSeq.empty, emptyStateContext) shouldBe true } + + property("context extension with neg and pos ids") { + val negId: Byte = -20 + + ADKey @@ Base16.decode("c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742").get + + val b = new ErgoBox(1000000000L, Constants.TrueLeaf, Colls.emptyColl, + Map.empty, ModifierId @@ "c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742", + 0, 0) + val ce = ContextExtension(Map(negId -> IntConstant(0), (-negId) -> IntConstant(1))) + val input = Input(b.id, ProverResult(Array.emptyByteArray, ce)) + + val oc = new ErgoBoxCandidate(b.value, b.ergoTree, b.creationHeight) + + val utx = UnsignedErgoTransaction(IndexedSeq(input), IndexedSeq(oc)) + + val tx = defaultProver.sign(utx, IndexedSeq(b), IndexedSeq.empty, emptyStateContext) + .map(ErgoTransaction.apply) + .get + + tx.statefulValidity(IndexedSeq(b), IndexedSeq.empty, emptyStateContext) shouldBe true + } + /* import sigma.{AnyValue, Coll} import sigmastate.SType From 8863126a802694c3f78fd5ade019190ab44f2cf0 Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Tue, 23 Jan 2024 17:23:20 +0300 Subject: [PATCH 13/23] comp fix --- .../ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala index 68cd3d37a5..1bbd4f8f4e 100644 --- a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala +++ b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala @@ -573,7 +573,7 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { val b = new ErgoBox(1000000000L, Constants.TrueLeaf, Colls.emptyColl, Map.empty, ModifierId @@ "c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742", 0, 0) - val ce = ContextExtension(Map(negId -> IntConstant(0), (-negId) -> IntConstant(1))) + val ce = ContextExtension(Map(negId -> IntConstant(0), (-negId).toByte -> IntConstant(1))) val input = Input(b.id, ProverResult(Array.emptyByteArray, ce)) val oc = new ErgoBoxCandidate(b.value, b.ergoTree, b.creationHeight) From b2dba013a529614874d67241bf1ed27111d5a70a Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Wed, 24 Jan 2024 12:32:45 +0300 Subject: [PATCH 14/23] finalizing checks, removing unused code in new context ext tests --- .../mempool/ErgoTransactionSpec.scala | 46 ++----------------- 1 file changed, 5 insertions(+), 41 deletions(-) diff --git a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala index 1bbd4f8f4e..ca902a62d7 100644 --- a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala +++ b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala @@ -544,7 +544,7 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { } property("context extension with neg id") { - val negId: Byte = -20 + val negId: Byte = -1 ADKey @@ Base16.decode("c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742").get @@ -557,11 +557,10 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { val utx = UnsignedErgoTransaction(IndexedSeq(input), IndexedSeq(oc)) - val tx = defaultProver.sign(utx, IndexedSeq(b), IndexedSeq.empty, emptyStateContext) + val txTry = defaultProver.sign(utx, IndexedSeq(b), IndexedSeq.empty, emptyStateContext) .map(ErgoTransaction.apply) - .get - tx.statefulValidity(IndexedSeq(b), IndexedSeq.empty, emptyStateContext) shouldBe true + txTry.isSuccess shouldBe false } @@ -580,45 +579,10 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { val utx = UnsignedErgoTransaction(IndexedSeq(input), IndexedSeq(oc)) - val tx = defaultProver.sign(utx, IndexedSeq(b), IndexedSeq.empty, emptyStateContext) + val txTry = defaultProver.sign(utx, IndexedSeq(b), IndexedSeq.empty, emptyStateContext) .map(ErgoTransaction.apply) - .get - tx.statefulValidity(IndexedSeq(b), IndexedSeq.empty, emptyStateContext) shouldBe true + txTry.isSuccess shouldBe false } - /* - import sigma.{AnyValue, Coll} - import sigmastate.SType - import sigmastate.SType.AnyOps - import sigmastate.Values.{ByteArrayConstant, EvaluatedValue} - import sigmastate.eval.CostingSigmaDslBuilder.Colls - import sigmastate.eval.Evaluation.stypeToRType - import sigmastate.eval.Extensions.toAnyValue - import sigmastate.interpreter.ContextExtension - - val ce = ContextExtension(Map(-6.toByte -> ByteArrayConstant(Array(0: Byte)))) - val ce2 = ContextExtension.serializer.fromBytes(ContextExtension.serializer.toBytes(ce)) - - def contextVars(m: Map[Byte, AnyValue]): Coll[AnyValue] = { - val maxKey = if (m.keys.isEmpty) 0 else m.keys.max // TODO optimize: max takes 90% of this method - val res = new Array[AnyValue](maxKey + 1) - for ((id, v) <- m) { - res(id) = v - } - Colls.fromArray(res) - } - - if (false) { - val varMap = ce2.values.map { case (k, v: EvaluatedValue[_]) => - val tVal = stypeToRType[SType](v.tpe) - k -> toAnyValue(v.value.asWrappedType)(tVal) - }.toMap - contextVars(varMap) - } */ - - - // val arr = new Array[Int](-5) - - } From 01692461ac1fe014f9ca40224a0a05d2d56d76e2 Mon Sep 17 00:00:00 2001 From: ccellado Date: Wed, 24 Jan 2024 15:33:04 +0400 Subject: [PATCH 15/23] Minor edits --- ergo-core/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/ergo-core/README.md b/ergo-core/README.md index 3290f36643..b07bce00ef 100644 --- a/ergo-core/README.md +++ b/ergo-core/README.md @@ -6,8 +6,8 @@ Toy working example client code available [here](https://github.com/ccellado/erg ```mermaid sequenceDiagram - Client-->>+Node: Establish TCP connection - Node->>+Client: Handshake + Client-->>Node: Establish TCP connection + Node->>Client: Handshake Client->>Node: Handhake Note over Client,Node: Message exchange started ``` @@ -15,7 +15,7 @@ sequenceDiagram ### Connect to peer 📞 First connect to peer node and get `Handshake` message. -For the rest of the guide assume the `Message body` as `byteBuffer`. +For the rest of the guide assume the message body as `byteBuffer`. ```scala import org.ergoplatform.network.HandshakeSerializer @@ -94,7 +94,7 @@ The peer node will start sending `SyncInfo` messages to us, since it is checking Our client is syncing instead. ### Send [ErgoSyncInfoV2](src/main/scala/org/ergoplatform/nodeView/history/ErgoSyncInfo.scala) ♲ -with empty `lastHeaders`, so the node knows client is just beginning to sync +With empty `lastHeaders` so the node knows client is just beginning to sync. ```scala import org.ergoplatform.nodeView.history.ErgoSyncInfoV2 From 82de0b20ca6403ed817992235de4d1c6c339e4b9 Mon Sep 17 00:00:00 2001 From: ccellado Date: Fri, 26 Jan 2024 17:55:08 +0400 Subject: [PATCH 16/23] Minor fix --- .../org/ergoplatform/network/message/BasicMessagesRepo.scala | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala index 82ac60dde5..b7b5842154 100644 --- a/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala +++ b/src/main/scala/org/ergoplatform/network/message/BasicMessagesRepo.scala @@ -3,12 +3,10 @@ package org.ergoplatform.network.message import org.ergoplatform.network.{PeerSpec, PeerSpecSerializer} import org.ergoplatform.nodeView.state.SnapshotsInfo import org.ergoplatform.nodeView.state.UtxoState.{ManifestId, SubtreeId} -import org.ergoplatform.settings.Algos import org.ergoplatform.network.message.MessageConstants.MessageCode import scorex.crypto.hash.Digest32 import scorex.util.Extensions._ import scorex.util.serialization.{Reader, Writer} -import scorex.util.ModifierId import org.ergoplatform.sdk.wallet.Constants.ModifierIdLength /** From d66962c7bbb168c3590ccca62c1e0b7917611ccb Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Mon, 29 Jan 2024 10:42:49 +0300 Subject: [PATCH 17/23] code improvs --- .../modifiers/mempool/ErgoTransactionSpec.scala | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala index ca902a62d7..893f05c723 100644 --- a/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala +++ b/src/test/scala/org/ergoplatform/modifiers/mempool/ErgoTransactionSpec.scala @@ -544,9 +544,7 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { } property("context extension with neg id") { - val negId: Byte = -1 - - ADKey @@ Base16.decode("c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742").get + val negId: Byte = -10 val b = new ErgoBox(1000000000L, Constants.TrueLeaf, Colls.emptyColl, Map.empty, ModifierId @@ "c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742", @@ -561,14 +559,14 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { .map(ErgoTransaction.apply) txTry.isSuccess shouldBe false + + txTry.toEither.left.get.isInstanceOf[NegativeArraySizeException] shouldBe true } property("context extension with neg and pos ids") { val negId: Byte = -20 - ADKey @@ Base16.decode("c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742").get - val b = new ErgoBox(1000000000L, Constants.TrueLeaf, Colls.emptyColl, Map.empty, ModifierId @@ "c95c2ccf55e03cac6659f71ca4df832d28e2375569cec178dcb17f3e2e5f7742", 0, 0) @@ -583,6 +581,8 @@ class ErgoTransactionSpec extends ErgoPropertyTest with ErgoTestConstants { .map(ErgoTransaction.apply) txTry.isSuccess shouldBe false + + txTry.toEither.left.get.isInstanceOf[ArrayIndexOutOfBoundsException] shouldBe true } } From 2f418c7da2defa3ac5f6982a81190c385ca1998c Mon Sep 17 00:00:00 2001 From: stenolog <158459002+stenolog@users.noreply.github.com> Date: Sat, 3 Feb 2024 14:01:58 +0000 Subject: [PATCH 18/23] fix proofHash related sporadic concurrency bug - fixes https://github.com/ergoplatform/ergo/issues/1387 - fixes https://github.com/ergoplatform/ergo/issues/1023 --- .../scala/org/ergoplatform/nodeView/state/UtxoStateReader.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/scala/org/ergoplatform/nodeView/state/UtxoStateReader.scala b/src/main/scala/org/ergoplatform/nodeView/state/UtxoStateReader.scala index a2c10bd248..d891a6e30e 100644 --- a/src/main/scala/org/ergoplatform/nodeView/state/UtxoStateReader.scala +++ b/src/main/scala/org/ergoplatform/nodeView/state/UtxoStateReader.scala @@ -142,7 +142,7 @@ trait UtxoStateReader extends ErgoStateReader with UtxoSetSnapshotPersistence { * @param txs - transactions to generate proofs * @return proof for specified transactions and new state digest */ - def proofsForTransactions(txs: Seq[ErgoTransaction]): Try[(SerializedAdProof, ADDigest)] = persistentProver.synchronized { + def proofsForTransactions(txs: Seq[ErgoTransaction]): Try[(SerializedAdProof, ADDigest)] = synchronized { val rootHash = persistentProver.digest log.trace(s"Going to create proof for ${txs.length} transactions at root ${Algos.encode(rootHash)}") if (txs.isEmpty) { From 044cd32c6dd9c9e6fd18bca71f092ea2ee82dd21 Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Mon, 5 Feb 2024 18:34:20 +0300 Subject: [PATCH 19/23] maxHeadersInOneQuery eliminated --- src/main/scala/org/ergoplatform/http/api/BlocksApiRoute.scala | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/main/scala/org/ergoplatform/http/api/BlocksApiRoute.scala b/src/main/scala/org/ergoplatform/http/api/BlocksApiRoute.scala index 64a98e6044..41fc7d8ea9 100644 --- a/src/main/scala/org/ergoplatform/http/api/BlocksApiRoute.scala +++ b/src/main/scala/org/ergoplatform/http/api/BlocksApiRoute.scala @@ -44,8 +44,6 @@ case class BlocksApiRoute(viewHolderRef: ActorRef, readersHolder: ActorRef, ergo getModifierByIdR } - private val maxHeadersInOneQuery = ergoSettings.chainSettings.epochLength * 2 - private def getHistory: Future[ErgoHistoryReader] = (readersHolder ? GetDataFromHistory[ErgoHistoryReader](r => r)).mapTo[ErgoHistoryReader] @@ -105,7 +103,7 @@ case class BlocksApiRoute(viewHolderRef: ActorRef, readersHolder: ActorRef, ergo val headers = maxHeaderOpt .toIndexedSeq .flatMap { maxHeader => - history.headerChainBack(maxHeadersInOneQuery, maxHeader, _.height <= fromHeight + 1).headers + history.headerChainBack(MaxHeaders, maxHeader, _.height <= fromHeight + 1).headers } headers.toList.asJson } From 6d3da025f115dfa5e7208b0e087073491d5aebb3 Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Mon, 5 Feb 2024 19:46:24 +0300 Subject: [PATCH 20/23] TG and modules section added --- FAQ.md | 2 +- README.md | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/FAQ.md b/FAQ.md index a041b2a8dc..b8e6b45259 100644 --- a/FAQ.md +++ b/FAQ.md @@ -51,7 +51,7 @@ The details of the Ergo emission schedule and monetary supply can be found in th * Telegram: https://t.me/ergoplatform * Ecosystem: https://sigmaverse.io -* + * Github: https://github.com/ergoplatform/ergo * Documents: https://ergoplatform.org/en/documents/ diff --git a/README.md b/README.md index a296bdf8b2..70e9b957ff 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ Ergo utilizes three types of tests: 1) Unit and property tests: These can be run using the `sbt test` command. 2) Integration tests: These tests require Docker to be installed. Run them with the `sudo sbt it:test` command. -3) Bootstrapping tests: These tests are time-consuming as they verify that the node is syncing with the main network in various regimes. Docker is also required for these tests. Run them with the `sudo sbt it2:test` command. +3) Bootstrapping tests: These tests are time-consuming as they verify that the node is syncing with the main network in various modes. Docker is also required for these tests. Run them with the `sudo sbt it2:test` command. ## Setting up the Project in an IDE @@ -86,9 +86,13 @@ Ensure that the project can be built with sbt before opening it in an IDE. You m To open the project in IntelliJ IDEA, select File / Open and navigate to the project folder. This will initiate the Project Import Wizard, which uses the SBT configuration (build.sbt file) to generate the project configuration files for IDEA. You can view the project configuration in the `File / Project Structure...` dialog. If the import is successful, you should be able to compile the project in the IDE. +## Modules + +* ergo-core + ## Contributing to Ergo -Ergo is an open-source project and we welcome contributions from developers and testers! Join the discussion on [Ergo Discord](https://discord.gg/kj7s7nb) in the #development channel and check out our [Contributing documentation](https://docs.ergoplatform.com/contribute/). +Ergo is an open-source project and we welcome contributions from developers and testers! Join the discussion over [Ergo Discord](https://discord.gg/kj7s7nb) in #development channel, or Telegram: https://t.me/ErgoDevelopers. Please also check out our [Contributing documentation](https://docs.ergoplatform.com/contribute/). ## Frequently Asked Questions From 3c061a25a736392f5d039a4c004fa1778f153211 Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Mon, 5 Feb 2024 19:51:12 +0300 Subject: [PATCH 21/23] link to ergo-core readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 70e9b957ff..ef8a4095a9 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,7 @@ To open the project in IntelliJ IDEA, select File / Open and navigate to the pro ## Modules -* ergo-core +* [ergo-core](ergo-core/README.md) ## Contributing to Ergo From c2a9d16eab6e95809f45c3396cb2c1473fa73d24 Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Thu, 8 Feb 2024 23:41:16 +0300 Subject: [PATCH 22/23] modules section in readme --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ef8a4095a9..14f0ea32e6 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,11 @@ To open the project in IntelliJ IDEA, select File / Open and navigate to the pro ## Modules -* [ergo-core](ergo-core/README.md) +This repository has modular structure, so only parts which are needed for an application could be used: + +* [avldb](ergo-core/README.md) - implementation of authenticated AVL+ tree used in Ergo, with persistence +* [ergo-core](ergo-core/README.md) - functionality needed for an SPV client (P2P messages, block section stuctures, PoW, NiPoPoW) +* ergo-wallet - Java and Scala functionalities to sign and verify transactions ## Contributing to Ergo From 6d0bd4c3534a21712e9b90ed083de89e62f694fa Mon Sep 17 00:00:00 2001 From: Alexander Chepurnoy Date: Fri, 9 Feb 2024 21:22:39 +0300 Subject: [PATCH 23/23] avldb link fix --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 14f0ea32e6..53d2e5a64e 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,7 @@ To open the project in IntelliJ IDEA, select File / Open and navigate to the pro This repository has modular structure, so only parts which are needed for an application could be used: -* [avldb](ergo-core/README.md) - implementation of authenticated AVL+ tree used in Ergo, with persistence +* [avldb](avldb/README.md) - implementation of authenticated AVL+ tree used in Ergo, with persistence * [ergo-core](ergo-core/README.md) - functionality needed for an SPV client (P2P messages, block section stuctures, PoW, NiPoPoW) * ergo-wallet - Java and Scala functionalities to sign and verify transactions