JsonWriterException: illegal char sequence of surrogate pair

[2072]

Hello,

Just stumble on this error (JsonWriterException: illegal char sequence of surrogate pair) while marshalling a large data structure containing the original file names of a real filesystem (with probably a bad file name somewhere it seems).

I've tried with withEscapeUnicode true and false but the result is the same. What can I do ?

Here is the back trace:

com.github.plokhotnyuk.jsoniter_scala.core.JsonWriter.encodeError(JsonWriter.scala:230),
com.github.plokhotnyuk.jsoniter_scala.core.JsonWriter.illegalSurrogateError(JsonWriter.scala:982),
com.github.plokhotnyuk.jsoniter_scala.core.JsonWriter.writeEncodedString(JsonWriter.scala:901),
com.github.plokhotnyuk.jsoniter_scala.core.JsonWriter.writeEscapedOrEncodedString(JsonWriter.scala:877),
com.github.plokhotnyuk.jsoniter_scala.core.JsonWriter.writeString(JsonWriter.scala:865),
com.github.plokhotnyuk.jsoniter_scala.core.JsonWriter.writeVal(JsonWriter.scala:258),

Jdk: openjdk 17.0.8 2023-07-18 LT
Scala: 2.13.11
Jsoniter: 2.21.3

Thanks

[plokhotnyuk]

Hi John,

While ABNF of JSON specification does allow strings for keys and values to contain bit sequences that cannot encode Unicode characters, this library does not allow that by design.

It could happen when strings for filenames are truncated at the middle of surrogate pair or just generated by random sequence of UTF-16 symbols.

The best solution would be renaming of affected files or fixing the software that produce them with such names.

Using the following custom codec you can print affected names for analysis:

implicit val codecThatPrintInvalidStrings: JsonValueCodec[String] = new JsonValueCodec[String] {
  override def decodeValue(in: JsonReader, default: String): String = in.readString(default)

  override def encodeValue(x: String, out: JsonWriter): _root_.scala.Unit =
    try {
      out.writeVal(x)
    } catch {
      case ex: JsonWriterException =>
        println("Has illegal surrogate pair: " + x)
        throw ex
    }

  override def nullValue: String = null
}

As a workaround, until the source problem is not fixed, you can serialize invalid surrogate pairs using a custom codec that store string as a raw value (see, out.writeRawVal), but in that case you should encode UTF-8 characters and escape \r, \t, \n, and \f values by your code (as an example using other JSON serializer).