Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert BigInt/Decimal to expected types for packed numbers #1391

Merged
merged 1 commit into from
Jan 10, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -772,7 +772,11 @@ trait ElementBase
// Validate that the number of bits does not exceed the maximum number of
// bits allowed for the type
if (
result.isDefined && repElement.isSimpleType && representation == Representation.Binary
result.isDefined &&
repElement.isSimpleType &&
representation == Representation.Binary &&
// Don't check bit length of packed binary numbers
(!optionBinaryNumberRep.isDefined || (binaryNumberRep == BinaryNumberRep.Binary))
) {
val nBits = result.get
primType match {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,8 @@
) extends PackedBinaryDecimalBaseParser(e, binaryDecimalVirtualPoint)
with HasKnownLengthInBits {

override def toBigInteger(num: Array[Byte]): JBigInteger = DecimalUtils.bcdToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.bcdToBigDecimal(num, scale)
override def toNumber(num: Array[Byte]): JBigDecimal =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change this to toNumber? The return type is not JNumber.

Is there scaladoc on the toNumber method? There needs to be to explain this naming convention. I don't really understand why this got renamed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The trait these classes mix in requires an implementation of def toNumber(num: Array[Byte]): A, where A <: Number. We could remove the A type parameter and just make it so it always returns a JNumber, but then implementations have to be a bit more careful to return the right thing.

As it is right now, the type parameter ensures the implementations of toNumber return the right kind of JNumber (some should always return a BigInteger, some should always return a BigDecimal). I'm not sure the type parameter is used anywhere else though, so maybe that's overkill and just adds unnecessary complication?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the type parameter is almost certainly the right thing, I just don't see a change that adds this toNumber super method declaration, yet it's not in the current Daffodil main branch, so when did this type parameter get added?

But what is the purpose of the toNumber method? I think it is "convert to a number type we can store in the Infoset object. This should be the most specific meaningful subtype of JNumber" that is used in the Infoset object representation. Or something like that.

Copy link
Contributor Author

@jadams-tresys jadams-tresys Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The top level declaration of toNumber is in PackedBinaryTraits.scala where it is simply declared as

    trait PackedBinaryConversion[A <: Number] {
        def toNumber(num: Array[Byte]): A
        ....

The actual implementations are in the classes that extend the PackedBinaryConversion trait, like BCDIntegerKnownLengthParser.

The function will only generate either BigInteger's or BigDecimal's depending on the calling classes type, so perhaps a better name for the function would be "toBigNumber". The BigNumber then gets converted to the appropriate PrimType by calling PrimNumberic.fromNumber(bigNumber). This call is made by the toPrimType function of the same trait.

DecimalUtils.bcdToBigDecimal(num, binaryDecimalVirtualPoint)

}

Expand All @@ -48,9 +47,8 @@
) extends PackedBinaryDecimalBaseParser(e, binaryDecimalVirtualPoint)
with HasRuntimeExplicitLength {

override def toBigInteger(num: Array[Byte]): JBigInteger = DecimalUtils.bcdToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.bcdToBigDecimal(num, scale)
override def toNumber(num: Array[Byte]): JBigDecimal =
DecimalUtils.bcdToBigDecimal(num, binaryDecimalVirtualPoint)

Check warning on line 51 in daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/BCDParsers.scala

View check run for this annotation

Codecov / codecov/patch

daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/BCDParsers.scala#L51

Added line #L51 was not covered by tests

}

Expand All @@ -64,9 +62,8 @@
) extends PackedBinaryDecimalBaseParser(e, binaryDecimalVirtualPoint)
with PrefixedLengthParserMixin {

override def toBigInteger(num: Array[Byte]): JBigInteger = DecimalUtils.bcdToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.bcdToBigDecimal(num, scale)
override def toNumber(num: Array[Byte]): JBigDecimal =
DecimalUtils.bcdToBigDecimal(num, binaryDecimalVirtualPoint)

override def childProcessors: Vector[Processor] = Vector(prefixedLengthParser)

Expand All @@ -83,19 +80,15 @@
) extends PackedBinaryIntegerBaseParser(e)
with HasRuntimeExplicitLength {

override def toBigInteger(num: Array[Byte]): JBigInteger = DecimalUtils.bcdToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.bcdToBigDecimal(num, scale)
override def toNumber(num: Array[Byte]): JBigInteger = DecimalUtils.bcdToBigInteger(num)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here the return type is different. toNumber returns a JBigInteger.

I'm surprised this even works. JBigInteger must be a sub-type of JBigDecimal.


}

class BCDIntegerKnownLengthParser(e: ElementRuntimeData, val lengthInBits: Int)
extends PackedBinaryIntegerBaseParser(e)
with HasKnownLengthInBits {

override def toBigInteger(num: Array[Byte]): JBigInteger = DecimalUtils.bcdToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.bcdToBigDecimal(num, scale)
override def toNumber(num: Array[Byte]): JBigInteger = DecimalUtils.bcdToBigInteger(num)

}

Expand All @@ -108,9 +101,7 @@
) extends PackedBinaryIntegerBaseParser(e)
with PrefixedLengthParserMixin {

override def toBigInteger(num: Array[Byte]): JBigInteger = DecimalUtils.bcdToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.bcdToBigDecimal(num, scale)
override def toNumber(num: Array[Byte]): JBigInteger = DecimalUtils.bcdToBigInteger(num)

override def childProcessors: Vector[Processor] = Vector(prefixedLengthParser)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,8 @@
) extends PackedBinaryDecimalBaseParser(e, binaryDecimalVirtualPoint)
with HasKnownLengthInBits {

override def toBigInteger(num: Array[Byte]): JBigInteger =
DecimalUtils.ibm4690ToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.ibm4690ToBigDecimal(num, scale)
override def toNumber(num: Array[Byte]): JBigDecimal =
DecimalUtils.ibm4690ToBigDecimal(num, binaryDecimalVirtualPoint)

}

Expand All @@ -49,10 +47,8 @@
) extends PackedBinaryDecimalBaseParser(e, binaryDecimalVirtualPoint)
with HasRuntimeExplicitLength {

override def toBigInteger(num: Array[Byte]): JBigInteger =
DecimalUtils.ibm4690ToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.ibm4690ToBigDecimal(num, scale)
override def toNumber(num: Array[Byte]): JBigDecimal =
DecimalUtils.ibm4690ToBigDecimal(num, binaryDecimalVirtualPoint)

Check warning on line 51 in daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/IBM4690PackedDecimalParsers.scala

View check run for this annotation

Codecov / codecov/patch

daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/IBM4690PackedDecimalParsers.scala#L51

Added line #L51 was not covered by tests

}

Expand All @@ -66,10 +62,8 @@
) extends PackedBinaryDecimalBaseParser(e, binaryDecimalVirtualPoint)
with PrefixedLengthParserMixin {

override def toBigInteger(num: Array[Byte]): JBigInteger =
DecimalUtils.ibm4690ToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.ibm4690ToBigDecimal(num, scale)
override def toNumber(num: Array[Byte]): JBigDecimal =
DecimalUtils.ibm4690ToBigDecimal(num, binaryDecimalVirtualPoint)

override def childProcessors: Vector[Processor] = Vector(prefixedLengthParser)

Expand All @@ -86,10 +80,8 @@
) extends PackedBinaryIntegerBaseParser(e)
with HasRuntimeExplicitLength {

override def toBigInteger(num: Array[Byte]): JBigInteger =
override def toNumber(num: Array[Byte]): JBigInteger =
DecimalUtils.ibm4690ToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.ibm4690ToBigDecimal(num, scale)

}

Expand All @@ -99,10 +91,8 @@
) extends PackedBinaryIntegerBaseParser(e)
with HasKnownLengthInBits {

override def toBigInteger(num: Array[Byte]): JBigInteger =
override def toNumber(num: Array[Byte]): JBigInteger =
DecimalUtils.ibm4690ToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.ibm4690ToBigDecimal(num, scale)

}

Expand All @@ -115,10 +105,8 @@
) extends PackedBinaryIntegerBaseParser(e)
with PrefixedLengthParserMixin {

override def toBigInteger(num: Array[Byte]): JBigInteger =
override def toNumber(num: Array[Byte]): JBigInteger =
DecimalUtils.ibm4690ToBigInteger(num)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.ibm4690ToBigDecimal(num, scale)

override def childProcessors: Vector[Processor] = Vector(prefixedLengthParser)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@
import org.apache.daffodil.lib.exceptions.Assert
jadams-tresys marked this conversation as resolved.
Show resolved Hide resolved
import org.apache.daffodil.lib.util.Maybe
import org.apache.daffodil.lib.util.MaybeChar
import org.apache.daffodil.runtime1.dpath.NodeInfo
import org.apache.daffodil.runtime1.dpath.InvalidPrimitiveDataException
import org.apache.daffodil.runtime1.dpath.NodeInfo.PrimType
import org.apache.daffodil.runtime1.infoset.DataValue.DataValueNumber
import org.apache.daffodil.runtime1.processors.ElementRuntimeData
import org.apache.daffodil.runtime1.processors.FieldDFAParseEv
import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
Expand All @@ -36,9 +38,22 @@

import passera.unsigned.ULong

trait PackedBinaryConversion {
def toBigInteger(num: Array[Byte]): JBigInteger
def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal
trait PackedBinaryConversion[A <: Number] {

/**
* Converts the byte array to either a BigInteger or Big Decimal as defined by
* the implementing class
*/
def toNumber(num: Array[Byte]): A

jadams-tresys marked this conversation as resolved.
Show resolved Hide resolved
def toPrimType(context: ElementRuntimeData, num: Array[Byte]): DataValueNumber = {
context.optPrimType.get match {
case pn: PrimType.PrimNumeric => pn.fromNumber(toNumber(num))
// Non-numeric types such as Time can still use these funcitons and
// expect BigIntegers as the output of the conversion
case _ => toNumber(num)
}
}
}

trait PackedBinaryLengthCheck {
Expand Down Expand Up @@ -74,7 +89,7 @@
override val context: ElementRuntimeData,
binaryDecimalVirtualPoint: Int
) extends PrimParser
with PackedBinaryConversion
with PackedBinaryConversion[JBigDecimal]
with PackedBinaryLengthCheck {
override lazy val runtimeDependencies = Vector()

Expand All @@ -95,28 +110,23 @@
}

try {
val bigDec = toBigDecimal(dis.getByteArray(nBits, start), binaryDecimalVirtualPoint)
start.simpleElement.overwriteDataValue(bigDec)
val dec = toPrimType(context, dis.getByteArray(nBits, start))
start.simpleElement.setDataValue(dec)
} catch {
case n: NumberFormatException => PE(start, "Error in packed data: \n%s", n.getMessage())
jadams-tresys marked this conversation as resolved.
Show resolved Hide resolved
case i: InvalidPrimitiveDataException =>
PE(start, "Error in packed data: \n%s", i.getMessage())

Check warning on line 118 in daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/PackedBinaryTraits.scala

View check run for this annotation

Codecov / codecov/patch

daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/PackedBinaryTraits.scala#L118

Added line #L118 was not covered by tests
}
}
}

abstract class PackedBinaryIntegerBaseParser(
override val context: ElementRuntimeData
) extends PrimParser
with PackedBinaryConversion
with PackedBinaryConversion[JBigInteger]
with PackedBinaryLengthCheck {
override lazy val runtimeDependencies = Vector()

val signed = {
context.optPrimType.get match {
case n: NodeInfo.PrimType.PrimNumeric => n.isSigned
// context.optPrimType can be of type date/time via ConvertZonedCombinator
case _ => false
}
}
protected def getBitLength(s: ParseOrUnparseState): Int

def parse(start: PState): Unit = {
Expand All @@ -134,13 +144,12 @@
}

try {
val int = toBigInteger(dis.getByteArray(nBits, start))
if (!signed && (int.signum != 1))
jadams-tresys marked this conversation as resolved.
Show resolved Hide resolved
PE(start, "Expected unsigned data but parsed a negative number")
else
start.simpleElement.overwriteDataValue(int)
val int = toPrimType(context, dis.getByteArray(nBits, start))
start.simpleElement.setDataValue(int)
} catch {
case n: NumberFormatException => PE(start, "Error in packed data: \n%s", n.getMessage())
case i: InvalidPrimitiveDataException =>
PE(start, "Error in packed data: \n%s", i.getMessage())
}
}
}
Expand All @@ -158,7 +167,7 @@
fieldDFAEv,
isDelimRequired
)
with PackedBinaryConversion {
with PackedBinaryConversion[JBigInteger] {

override def processResult(parseResult: Maybe[dfa.ParseResult], state: PState): Unit = {
Assert.invariant(
Expand All @@ -177,11 +186,13 @@
return
} else {
try {
val num = toBigInteger(fieldBytes)
val num = toPrimType(context, fieldBytes)
state.simpleElement.setDataValue(num)
} catch {
case n: NumberFormatException =>
PE(state, "Error in packed data: \n%s", n.getMessage())
case i: InvalidPrimitiveDataException =>
PE(state, "Error in packed data: \n%s", i.getMessage())

Check warning on line 195 in daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/PackedBinaryTraits.scala

View check run for this annotation

Codecov / codecov/patch

daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/PackedBinaryTraits.scala#L195

Added line #L195 was not covered by tests
}

if (result.matchedDelimiterValue.isDefined) state.saveDelimitedParseResult(parseResult)
Expand All @@ -205,7 +216,7 @@
fieldDFAEv,
isDelimRequired
)
with PackedBinaryConversion {
with PackedBinaryConversion[JBigDecimal] {

/**
* We are treating packed binary formats as just a string in iso-8859-1 encoding.
Expand Down Expand Up @@ -236,11 +247,13 @@
return
} else {
try {
val num = toBigDecimal(fieldBytes, binaryDecimalVirtualPoint)
val num = toPrimType(e, fieldBytes)
state.simpleElement.setDataValue(num)
} catch {
case n: NumberFormatException =>
PE(state, "Error in packed data: \n%s", n.getMessage())
case i: InvalidPrimitiveDataException =>
PE(state, "Error in packed data: \n%s", i.getMessage())

Check warning on line 256 in daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/PackedBinaryTraits.scala

View check run for this annotation

Codecov / codecov/patch

daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/PackedBinaryTraits.scala#L256

Added line #L256 was not covered by tests
}

if (result.matchedDelimiterValue.isDefined) state.saveDelimitedParseResult(parseResult)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,8 @@
) extends PackedBinaryDecimalBaseParser(e, binaryDecimalVirtualPoint)
with HasKnownLengthInBits {

override def toBigInteger(num: Array[Byte]): JBigInteger =
DecimalUtils.packedToBigInteger(num, packedSignCodes)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.packedToBigDecimal(num, scale, packedSignCodes)
override def toNumber(num: Array[Byte]): JBigDecimal =
DecimalUtils.packedToBigDecimal(num, binaryDecimalVirtualPoint, packedSignCodes)

}

Expand All @@ -51,11 +49,8 @@
) extends PackedBinaryDecimalBaseParser(e, binaryDecimalVirtualPoint)
with HasRuntimeExplicitLength {

override def toBigInteger(num: Array[Byte]): JBigInteger =
DecimalUtils.packedToBigInteger(num, packedSignCodes)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.packedToBigDecimal(num, scale, packedSignCodes)

override def toNumber(num: Array[Byte]): JBigDecimal =
DecimalUtils.packedToBigDecimal(num, binaryDecimalVirtualPoint, packedSignCodes)

Check warning on line 53 in daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/PackedDecimalParsers.scala

View check run for this annotation

Codecov / codecov/patch

daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/PackedDecimalParsers.scala#L53

Added line #L53 was not covered by tests
}

class PackedDecimalPrefixedLengthParser(
Expand All @@ -69,10 +64,8 @@
) extends PackedBinaryDecimalBaseParser(e, binaryDecimalVirtualPoint)
with PrefixedLengthParserMixin {

override def toBigInteger(num: Array[Byte]): JBigInteger =
DecimalUtils.packedToBigInteger(num, packedSignCodes)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.packedToBigDecimal(num, scale, packedSignCodes)
override def toNumber(num: Array[Byte]): JBigDecimal =
DecimalUtils.packedToBigDecimal(num, binaryDecimalVirtualPoint, packedSignCodes)

override def childProcessors: Vector[Processor] = Vector(prefixedLengthParser)

Expand All @@ -90,10 +83,8 @@
) extends PackedBinaryIntegerBaseParser(e)
with HasRuntimeExplicitLength {

override def toBigInteger(num: Array[Byte]): JBigInteger =
override def toNumber(num: Array[Byte]): JBigInteger =
DecimalUtils.packedToBigInteger(num, packedSignCodes)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.packedToBigDecimal(num, scale, packedSignCodes)

}

Expand All @@ -104,10 +95,8 @@
) extends PackedBinaryIntegerBaseParser(e)
with HasKnownLengthInBits {

override def toBigInteger(num: Array[Byte]): JBigInteger =
override def toNumber(num: Array[Byte]): JBigInteger =
DecimalUtils.packedToBigInteger(num, packedSignCodes)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.packedToBigDecimal(num, scale, packedSignCodes)

}

Expand All @@ -121,10 +110,8 @@
) extends PackedBinaryIntegerBaseParser(e)
with PrefixedLengthParserMixin {

override def toBigInteger(num: Array[Byte]): JBigInteger =
override def toNumber(num: Array[Byte]): JBigInteger =
DecimalUtils.packedToBigInteger(num, packedSignCodes)
override def toBigDecimal(num: Array[Byte], scale: Int): JBigDecimal =
DecimalUtils.packedToBigDecimal(num, scale, packedSignCodes)

override def childProcessors: Vector[Processor] = Vector(prefixedLengthParser)

Expand Down
Loading