[Kryo5][scala] Case class serialization issue: TestVarArgs(vargs: String*)

[roczei]

Describe the bug

Kryo5 cannot serialize the following scala case class which has one variable length argument:

case class TestVarArgs(vargs: String*)

It works with Kryo4 (4.0.2). Seems like this is a regression issue in Kryo5.

Error message:

[warn] either append it to `Global / excludeLintKeys` or call .withRank(KeyRanks.Invisible) on the key
[info] running com.roczei.testing.Main 
[error] com.esotericsoftware.kryo.kryo5.KryoException: java.lang.ClassCastException: class [Ljava.lang.String; cannot be cast to class java.lang.String ([Ljava.lang.String; and java.lang.String are in module java.base of loader 'bootstrap'
[error] Serialization trace:
[error] array (scala.collection.mutable.WrappedArray$ofRef)
[error] vargs (com.roczei.testing.TestVarArgs)
[error] 	at com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:101)
[error] 	at com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108)
[error] 	at com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642)
[error] 	at com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:70)
[error] 	at com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108)
[error] 	at com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:627)
[error] 	at com.roczei.testing.Main$.main(Main.scala:21)
[error] 	at com.roczei.testing.Main.main(Main.scala)
[error] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[error] Caused by: java.lang.ClassCastException: class [Ljava.lang.String; cannot be cast to class java.lang.String ([Ljava.lang.String; and java.lang.String are in module java.base of loader 'bootstrap')
[error] 	at com.esotericsoftware.kryo.kryo5.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:164)
[error] 	at com.esotericsoftware.kryo.kryo5.Kryo.writeObjectOrNull(Kryo.java:692)
[error] 	at com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:79)
[error] 	at com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108)
[error] 	at com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642)
[error] 	at com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:70)
[error] 	at com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108)
[error] 	at com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:627)
[error] 	at com.roczei.testing.Main$.main(Main.scala:21)
[error] 	at com.roczei.testing.Main.main(Main.scala)
[error] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[error] stack trace is suppressed; run last Compile / run for the full output
[error] (Compile / run) com.esotericsoftware.kryo.kryo5.KryoException: java.lang.ClassCastException: class [Ljava.lang.String; cannot be cast to class java.lang.String ([Ljava.lang.String; and java.lang.String are in module java.base of loader 'bootstrap')
[error] Serialization trace:
[error] array (scala.collection.mutable.WrappedArray$ofRef)
[error] vargs (com.roczei.testing.TestVarArgs)
[error] Total time: 1 s, completed 18 Apr 2023, 19:23:51

Other test results:

https://github.com/roczei/kryo_repro/blob/main/repro-logs-kryo5-bad.txt

To Reproduce

Example scala code:

import com.esotericsoftware.kryo.kryo5.Kryo
import com.esotericsoftware.kryo.kryo5.io.Input
import com.esotericsoftware.kryo.kryo5.io.Output
import com.esotericsoftware.kryo.kryo5.objenesis.strategy.StdInstantiatorStrategy

import java.io.{FileInputStream, FileOutputStream}

case class TestVarArgs(vargs: String*)

object Main {
  def main(args: Array[String]): Unit = {
    val kryo = new Kryo()
    kryo.setInstantiatorStrategy(new StdInstantiatorStrategy())
    kryo.register(classOf[TestVarArgs])
    kryo.register(Class.forName("scala.collection.immutable.ArraySeq$ofRef"))
    kryo.register(Class.forName("[Ljava.lang.String;"))
    val t = TestVarArgs("hey", "you", "guys")
    val output = new Output(new FileOutputStream("file.bin"))
    kryo.writeObject(output, t)
    output.close()
    val input = new Input(new FileInputStream("file.bin"))
    val i = kryo.readObject(input, classOf[TestVarArgs])
    input.close()
    assert(i.equals(t))
  }
}

Bad results (Kryo5):

git clone [email protected]:roczei/kryo_repro.git

cd scala_project_kryo5-2.13.8
sbt run

or

cd scala_project_kryo5-2.12.7
sbt run

or

cd scala_project_kryo5-2.11.12
sbt run

Console outputs: https://github.com/roczei/kryo_repro/blob/main/repro-logs-kryo5-bad.txt

Good result (Kryo4)

cd scala_project_kryo4-2.13.8
sbt run

Console output:

https://github.com/roczei/kryo_repro/blob/main/kryo4-good-logs.txt

OS: macosx but I think this is not OS related
JDK: Azul Systems, Inc. Java 11.0.16 / Azul Systems, Inc. Java 1.8.0_345 (probably it can be reproduced with other jave versions as well)
Scala versions: 2.11.12, 2.12.7, 2.13.8
Kryo Version: all Kryo5 versions

Additional context

I would like to finish this pull request what nicknezis has started (update chill to the newer Kryo 5): twitter/chill#514 and currently this is blocking me to finish this task (latest pull request): roczei/chill#1. Similar issue was mentioned in this kryo thread as well:

https://groups.google.com/g/kryo-users/c/6iBVAKG5p4g

[theigl]

This might have the same root cause as #940. I have a tentative fix for this, but I'm not 100% sure that it is fully backwards compatible.

Could you check if disabling generics optimization (kryo.setOptimizedGenerics(false)) works in your case as well?

[roczei]

Thanks @theigl! I have just tested your suggetion and it works:

roczei@roczei-MBP16 ~/github/kryo_repro/scala_project_kryo5-2.13.8 (main) $ git diff src/main/scala/Main.scala
diff --git a/scala_project_kryo5-2.13.8/src/main/scala/Main.scala b/scala_project_kryo5-2.13.8/src/main/scala/Main.scala
index 554c8a1..d4d3c2f 100644
--- a/scala_project_kryo5-2.13.8/src/main/scala/Main.scala
+++ b/scala_project_kryo5-2.13.8/src/main/scala/Main.scala
@@ -13,6 +13,7 @@ object Main {
   def main(args: Array[String]): Unit = {
     val kryo = new Kryo()
     kryo.setInstantiatorStrategy(new StdInstantiatorStrategy())
+    kryo.setOptimizedGenerics(false)
     kryo.register(classOf[TestVarArgs])
     kryo.register(Class.forName("scala.collection.immutable.ArraySeq$ofRef"))
     kryo.register(Class.forName("[Ljava.lang.String;"))
roczei@roczei-MBP16 ~/github/kryo_repro/scala_project_kryo5-2.13.8 (main) $ sbt run
[info] welcome to sbt 1.8.2 (Azul Systems, Inc. Java 1.8.0_345)
[info] loading global plugins from /Users/roczei/.sbt/1.0/plugins
[info] loading settings for project scala_project_kryo5-2-13-8-build from plugins.sbt ...
[info] loading project definition from /Users/roczei/github/kryo_repro/scala_project_kryo5-2.13.8/project
[info] loading settings for project root from build.sbt ...
[info] set current project to scala_project (in build file:/Users/roczei/github/kryo_repro/scala_project_kryo5-2.13.8/)
[warn] there's a key that's not used by any other settings/tasks:
[warn]  
[warn] * root / idePackagePrefix
[warn]   +- /Users/roczei/github/kryo_repro/scala_project_kryo5-2.13.8/build.sbt:8
[warn]  
[warn] note: a setting might still be used by a command; to exclude a key from this `lintUnused` check
[warn] either append it to `Global / excludeLintKeys` or call .withRank(KeyRanks.Invisible) on the key
[info] running com.roczei.testing.Main 
[success] Total time: 1 s, completed Apr 18, 2023 9:29:07 PM
roczei@roczei-MBP16 ~/github/kryo_repro/scala_project_kryo5-2.13.8 (main) $ cat file.bin 

he?yo?guy?roczei@roczei-MBP16 ~/github/kryo_repro/scala_project_kryo5-2.13.8 (main) $ 

[theigl]

OK great. I will revisit #940 in the coming days, but as I said, I'm not sure I can come up with a fix that is guaranteed not to break some other edge cases.

I would suggest you simply keep generics optimization disabled in your PR. I'm planning to disable it (or tone in down significantly) in Kryo 6. It is the source of most of the bugs that have been reported against Kryo 5 and the potential benefits are limited.

[roczei]

Ok, then I will disable it in my PR. Thanks!

[theigl]

I just pushed a fix for #940. Could you please test against the latest snapshot if this fixes your issue as well?

[roczei]

I just pushed a fix for #940. Could you please test against the latest snapshot if this fixes your issue as well?

I have just tested it and the latest 5.4.1-SNAPSHOT fixes my issue as well.

[theigl]

Fantastic thanks! I will release Kryo 5.5.0 later this week.