Renaissance 0.16.0

This release brings improved benchmark validation, numerous platform and library updates, and improved compatibility with recent Java versions. Even though the source code of most benchmarks remains unchanged (apart from fixes and changes due to introducing validation in several benchmarks), the actual code executed at runtime may be affected by the dependency updates and exhibit changes in performance.

Workload validation

Validation of workload results ensures that changes in the underlying libraries or the JVM do not break the workloads. This release adds or improves validation in 7 workloads, bringing Renaissance closer to having fully validated workloads (there are 4 workloads left). In most cases, adding validation does not impact the reported benchmark performance, because it is performed outside the timed phase. However, in some cases, adding validation required changing the timed part of the benchmark, which is the case of the future-genetic and gauss-mix benchmarks (see the next section for a summary of changes with performance impact).

• Add validation to fj-kmeans benchmark (#431)
• Add validation to future-genetic benchmark (#432)
• Add validation to als benchmark (#435)
• Improve validation in log-regression benchmark (#436)
• Add validation to movie-lens benchmark (#441)
• Add validation to philosophers benchmark (#446)
• Improve validation in gauss-mix validation (#448)

Workload changes (with impact on performance)

In some benchmarks, adding validation required changes to the benchmark code that impact performance. In the future-genetic benchmark, obtaining stable results for validation required associating random number generators with work items, not worker threads. This change eliminated work thread contention on a shared Random instance, leading to higher performance. In the gauss-mix benchmark, adding a more in-depth validation required training multiple models with reduced size and dimensionality of the input data.

In other cases, the benchmark code was modified to get a more reasonable behavior. In the philosophers benchmark, the rendering of system state snapshots (into string) was moved out of the transaction to avoid blocking the philosopher threads. In addition, the number of state snapshots was fixed to prevent the benchmark from performing variable amount of work. The movie-lens benchmark now loads input data directly from resource files, avoding unnecessary IO, but also uses Spark-provided methods to load CSV files to properly handle quoted values. This results in different partitioning defaults in subsequent computations, impacting performance.

• Avoid contention on shared Random in the future-genetic benchmark (#432)
• Train multiple models in the gauss-mix benchmark (#441)
• Reduce contention and make the amount of work in the philosophers benchmark constant (#447)
• Use Spark-provided methods to load CSV files and avoid extra IO in movie-lens (#455)

Externally visible changes

The chromosome_count parameter name in the future-genetic benchmark has been renamed to population_size to better match what it really represents in the benchmark.

• Refactor parameter and variable names for clarity in the future-genetic benchmark (#463)

Java Compatibility

To improve compatibility with Java 23, the hadoop-client-api library used by the Apache Spark framework has been patched to avoid using the deprecated security manager API which prevented running on Java 23 without using the -Djava.security.manager=allow. The database libraries used in the db-shootout benchmark were updated to recent versions and, together with a fix to Chronicle initialization, the benchmark is now compatible with current Java releases.

• Patch Hadoop Client API jar to avoid deprecated Java Security API on Java 23 (#453)
• Update benchmark dependencies (#456)
• Set 'java.class.path' temporarily during Chronicle intialization (#459)

Dependency updates

The Scala 2.12 and 2.13 versions used by various benchmarks were updated to versions 2.12.20 and 2.13.15, respectively. The philosophers and scala-stm-bench7 benchmarks are now using Scala 3.3.4, together with the other benchmarks brought to Scala 3 in previous releases. This leaves only the reactors benchmark in the Scala 2.12 camp.

The following primary frameworks/libraries used by various benchmarks we updated:

• apache-spark benchmarks now use Spark version 3.5.3
• twitter-finagle benchmarks now use Finagle version 24.2.0
• neo4j-analytics now uses Neo4J version 5.25.1
• future-genetic now uses Jenetics version 6.3.0
• rx now uses RXJava3 version 3.1.9
• scala-dotty now uses Scala 3 compiler version 3.3.4
• scala-stm benchmarks now use Scala STM version 0.11.1
• db-shootout benchmark now uses H2 MVStore version 2.3.232, MapDB
  version 3.1.0, and Chronicle Map version 3.26ea4.

There are also numerous updates to the secondary/transitive dependencies such as Arrow, ASM, Caffeine, Guava, Jackson, Jersey, JNA, Netty, Parquet, and many others. See the relevant pull requests (especially #456) for details.

• Switch to external scala-stm and update to version 0.11.1 (#444)
• Migrate scala-stm benchmarks from Scala 2.12 to Scala 3 (#445)
• Update benchmark dependencies (#456)

Build system and CI changes

• Update SBT to version 1.9.9 and drop SBT binaries (#450)
• Speed up CI by building Renaissance once and running it on multiple JDKs (#454)

Thanks

This release includes contributions from @lovisek, @BohdanQQ, @guilhas07, and @reneleonhardt in addition to @vhotspur and @lbulej. Many thanks!