CBOR serialization incorrect behaviour

[FerrumBrain]

0. Setup

We created the following class hierarchy for testing serialization library:

Value (org.plan.research)
    ArrayValue (org.plan.research)
    BooleanArrayValue (org.plan.research)
    BooleanValue (org.plan.research)
    ByteArrayValue (org.plan.research)
    ByteValue (org.plan.research)
    CharArrayValue (org.plan.research)
    CharValue (org.plan.research)
    CompositeNullableValue (org.plan.research)
    DefaultValueAlways (org.plan.research)
    DefaultValueNever (org.plan.research)
    DoubleArrayValue (org.plan.research)
    DoubleValue (org.plan.research)
    EnumValue (org.plan.research)
    FloatArrayValue (org.plan.research)
    FloatValue (org.plan.research)
    IntArrayValue (org.plan.research)
    IntValue (org.plan.research)
    ListValue (org.plan.research)
    LongArrayValue (org.plan.research)
    LongValue (org.plan.research)
    NullValue (org.plan.research)
    ObjectValue (org.plan.research)
    ShortArrayValue (org.plan.research)
    ShortValue (org.plan.research)
    StringValue (org.plan.research)

Value hierarchy tries to use most of the available serialization API and test it on all main data types available on Kotlin/JVM.
The exact implementation details are not important in most cases. We will highlight interesting implementation details whenever necessary.

CBOR bugs are mainly due to unhandled internal exceptions that should not be displayed to the user in the raw format.

1. Unhandled IllegalStateException

Byte 126 is interpreted as "read a string of 30 characters"

@Test
fun `unhandled illegal state exception`() {
    val byteArray = byteArrayOf(126)
    val serializer = Cbor.Default
    // Fails with "java.lang.IllegalStateException: Unexpected EOF, available 0 bytes, requested: 30"
    assertThrows<SerializationException> {
        serializer.decodeFromByteArray<String>(byteArray)
    }
}

2. Unhandled NegativeArraySizeException

Byte 40 is interpreted as an instruction to read -9 bytes.

@Test
fun `unhandled negative array size exception`() {
    val byteArray = byteArrayOf(127, 40)
    val serializer = Cbor.Default
    // Fails with "java.lang.NegativeArraySizeException: -9"
    assertThrows<SerializationException> {
        serializer.decodeFromByteArray<String>(byteArray)
    }
}

3. Unhandled StackOverflowError

Root of issue:

• CborParser class does not check for the end of the buffer
• ByteArrayInput returns -1 on read if it has reached the end of the buffer
• CborParser::readBytes interprets this -1 value as "read an indefinite number of bytes" and calls CborParser::readIndefiniteLengthBytes ; CborParser::readIndefiniteLengthBytes , meanwhile, calls CborParser::readBytes recursively

@Test
fun `unhandled stack overflow error`() {
    val byteArray = byteArrayOf(127, 0, 0)
    val serializer = Cbor.Default
    // Goes to infinite recursion:
    //   at kotlinx.serialization.cbor.internal.CborParser.readBytes(Decoder.kt:247)
    //   at kotlinx.serialization.cbor.internal.CborParser.readIndefiniteLengthBytes(Decoder.kt:514)
    assertThrows<SerializationException> {
        serializer.decodeFromByteArray<String>(byteArray)
    }
}

4. Unhandled ArrayIndexOutOfBounds

Option ignoreUnknownKeys=true tells the parser to skip unknown elements.
Byte 122 at position 67 is interpreted as the start of the element and encodes its length of -272646673.
In an attempt to skip this element, the parser moved to -272646673 bytes "ahead" in ByteArrayInput and sets the
current position to -272646606.

If ignoreUnknownKeys=false, this will fail with
"kotlinx.serialization.cbor.internal.CborDecodingException: CborLabel unknown: 31 for obj(status: kotlin.String, value: kotlin.collections.LinkedHashMap)"

@Test
fun `unhandled array index oob exception`() {
    val byteArray = byteArrayOf(
        -103, 7, 127, 127, -61, 111, 98, 106, 0, 0, -1, -66, -1, -9, -29, 47, 38, 38, 38, 38, 1, 38, 38, 38,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 38, 38, 38, 38, 38, 38, 111, 98, 106, -17, -65, -67, -17, -65, -67,
        -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, 122, -17, -65, -67, -17, -65, -67, -17,
        -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67,
        -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, 38, 38, 38, 38, 38,
        38, 38, 126, 126, 126, 38, 35, -128, -128, -128, -128, -128, -128, -128, -128, -128, 126, 126, 126,
        126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126,
        126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126,
        126, -67, -17, -65, -67, -17, 126, 126, 126, 126, 5, 0, 126, 126, 126, 126, 126, 126, 126, 126, 126,
        126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, -1, -1, -1, -1, -1, -1, -1,
        -1, 126, 126
    )
    val serializer = Cbor {
        ignoreUnknownKeys = true
    }
    // Fails with "java.lang.ArrayIndexOutOfBoundsException: Index -272646606 out of bounds for length 216"
    assertThrows<SerializationException> {
        serializer.decodeFromByteArray<Value>(byteArray)
    }
}

Bugs are found by fuzzing team @ PLAN Lab

Environment

• Kotlin version: 2.0.20
• Library version: 1.7.3
• Kotlin platforms: JVM
• Gradle version: 8.8