About code changes that change original implementation

[xezon]

Hello.

This wiki page lists one deviation to original implementation:
https://github.com/TheAssemblyArmada/Thyme/wiki/Changes-To-Original

This makes me thinking it would be very beneficial to have project defines to guard certain code changes:
1. Changes to original implementation, such as bug fixes, improvements, new features
2. Changes to original implementation that will mismatch game states in multiplayer or block progress in singleplayer

Guarding such changes will
a) act as clean documentation as to which changes deviate from the original and which do not
b) allow programmer to compile out custom changes and get back to clean original state easily
c) allow publisher to pick appropriate changes for its Mod requirements

There are many bugs in the game that can be fixed under category 1. Therefore it likely is possible to publish a Thyme executable that is fully compatible with original game, allowing all users to play together. Such compatibility should ideally not be lost: CRC check can likely be faked, however it should only be faked if no category 2 changes are compiled in.

Maybe you already have that covered. In that case this post is void :)

[duncanspumpkin]

I feel that this would increase time to develop and slow down progress. It's always super annoying having multiple compile time options and it would be avoided as much as possible. We aren't going for a binary identical build and there is no reason to go for one.

[xezon]

Hi Duncan.

I think such compatibility is very important and essential. The neglectance of such, for the reasons you mentioned above, is not sound decision making. It's common practice to have CMake options, or in code project defines, that can compile in or out certain features. Cutting compatibility with retail game is a huge decision, which essentially would render Thyme useless for the majority of multiplayer matches played, because Thyme users would not be compatible with Retail users.

If this was already discussed in the past and a decision was made to not support compatibility with original game content and multiplayer, then ok we don't need to talk about it again. But if this is one of your objectives and you feel strongly about supporting full retail game compatibility, then certainly I am happy to be involved and invest the time necessary when available to make it compatible, which would already solve the time issue you mentioned. As for having many annoying compile options, I don't think this is problem at all.

If this discussion is still on the table, I think it would make sense to first talk about potential challenges given the current state of Thyme.

I suspect Game State Machine (or whatever it is named) will be biggest factor here. Any mismatches in game state will inevitably cause a mismatch to Retail game, so there can't be any deviations. Many parameters are build via INI files, but still I suspect game code carries a fundamental responsibility. On top of that there is the Networking part, which likely also plays a role as it needs to communicate all player game state changes reliably.

[OmniBlade]

While I don't have any real philosophical objection to having a kind of "bug fixes" ifdef that can be switched on or off, that alone isn't going to be sufficient to get multiplayer compatibility with the original game binary given the use of floating point for a lot of game state. CRC checks will fail on slight deviations in the order and precision of maths operations so we'd have to compile for the FPU rather than SSE2 as we currently do and be fastidious in checking the generated ASM. It would also make it even more difficult to work towards cross platform multiplayer to be tied to matching the floating point behaviour of the old msvc compiler. Its not even clear if the newer versions of the windows CRT behave the same regarding floating point functions like sin, cos, tan and such so even that might be a blocker.

I'm more of the opinion that the multiplayer community can adopt Thyme or not in the same way that they have adopted Gentool, but that its going to have to be an all or nothing where either everyone has to use it in a game or no one has to. If plausible solutions to keeping floating point maths generating the same binary results with the original game and between platforms can be devised I'd be willing to reconsider this position though.

[xezon]

Oh that is a really excellent point. Yes all math operations and math constants would have to match certainly. Ok so if I understand correctly there is a worry of support retail compatibility at the expense for other platform compatibility. For Game Mods like Shockwave, Contra, retail compatibility is certainly no concern. They can have their own binary for a new version release. For retail player retail compatibily will be more important than platform compatibility however. Because retail player wants to play matches against other players, and when there is no big player base for Thyme matches, then adoption can never happen.

As for GenTool, keep in mind it is fully compatible in Multiplayer with players who did not install GenTool. It only changes user facing functionality at runtime and therefore will not break any game states. All the bugs GenTool fixes and features it adds do not cause mismatches. I would estimate most issues this game has can be fixed without mismatching clients.

As for FPU, I am not so familiar with floating point difference. Maybe compiler options can be adjusted in a way that they use compatible operation?

For clang for example there is this:
https://clang.llvm.org/docs/CrossCompilation.html

-mcpu=<cpu-name>, like x86-64, swift, cortex-a15
-mfpu=<fpu-name>, like SSE3, NEON, controlling the FP unit available
-mfloat-abi=<fabi>, like soft, hard, controlling which registers to use for floating-point

We could write test program, run some common math operations, get their results from a compiled unit of Visual Studio 6.0 and compare runs of modern compilers against that. If there are mismatches, then we can see from there how to go on making it compatible.

What compilers and platforms does Thyme intent to be compatible with?

[OmniBlade]

Clang, GCC and MSVC I think are the primary compilers we care about.

FPU matching is difficult, very difficult, because different platforms supply different versions of trancedental functions like sin and cos even if you have the fpu precision matching (which it doesn't by default, GCC sets it to extended 80bit, while msvc uses 64bit). GCC warns against setting the precision to anything else because the maths functions assume the precision and it will affect the output. You also have fast math optimisation that was enabled for the original build that optimises division to 1 / value multiplication which again affects the exact result the computation generates.

[xezon]

As for math functions, it sounds like we would need to figure out which math library they used for this SAGE version and add it to Thyme, right? We would then forward all math function calls critical to Game State related functionality to functions provided by this original library. For starters we could assume it is the standard library for MSVC 6.0, but how do we test it and verify correctness?

[OmniBlade]

The library is msvcrt, wine just reimplements a number of functions with calls to the underlying OS math library. How stable are wine games when played against a native windows player?

[xezon]

I have no expertise about Wine so can't comment on this.

[OmniBlade]

My intention was to just include a math library statically linked into the engine to provide more consistent behaviour but since the msvc CRT isn't open source we couldn't use that one. If wine still plays nice then maybe it won't matter, I guess it depends on how often the calculations stray towards edge cases where different math libraries produce different results within the game state.

[xezon]

I think it would already be a first good step to get Windows binaries into a mode where it is compatible with Retail. If we are certain msvcrt (which one?) provides correct math implementations, then we can create program to generate a good number of values with common math operations and functions, write these out, and then build a test case to compare these benchmark results to output values of a different platform/compiler, essentially verifying compatibility of floating point results.

[xezon]

Compatibility to original game.dat may no longer be really necessary, other than for testing. The current plan is to push for a new ecosystem of Community Content and Patches, so retail compatibility would be off the table regardless of Thyme.