Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Further reverse engineering #5

Open
4 of 6 tasks
WizzardMaker opened this issue May 8, 2019 · 33 comments
Open
4 of 6 tasks

Further reverse engineering #5

WizzardMaker opened this issue May 8, 2019 · 33 comments
Labels
discussion Discussions about a topic

Comments

@WizzardMaker
Copy link
Member

WizzardMaker commented May 8, 2019

Figure out how

  • *.pol
  • *.dat
  • *.mcf
  • *.csv
  • *.raw
  • *.smk

are structured

@WizzardMaker
Copy link
Member Author

There is already C++ (Or rather old C) code that can decode the *.csv files. They are encoded with an xtRLE method. http://karlchenofhell.org/kram/xtRLE_dec.c is working 100%
Also here https://www.gog.com/forum/airline_tycoon_deluxe/editing_game_data_csv_files_to_customizes_cities_routes_etc

We have to add a smk decoder as well, for all the character animations in the rooms, a C# version is already in the open source StarCraft remake (https://github.com/kangaroo/scsharp/tree/master/SCSharp/SCSharp.Mpq.Smk), we should ask them if we can use and modify their code, would save us some work.

I don't know what the *.dat files contain, but the misc folder has *.res files which should contain all the dialog if my memory serves me right.

the *.mcf files contain the font data.

@WizzardMaker
Copy link
Member Author

The *.dat files hold the airport gfx position data.

I managed to extract the information of those files. I created a script which reads the file and then creates a scene from it.

The file has the following structure:
Header:
4 bytes (int) - Entries in file
4 bytes - ? often null

For each entry:
4 bytes (long) - file id (from brick.csv)
4 bytes (int) - x position
4 bytes (int) - y position
1 byte - other info, maybe room number
1 byte - null

There are still some artifacts, like misplaced tiles or multiple entries on the same position and its still missing a method to load animated entries.

Here is a little preview of the assembled airport from those files:

Imgur

@Serphentas
Copy link
Collaborator

Amazing work ! The preview looks great, despite the few details here and there. I think that's a good basis to build upon.

I'll try to write a CSV handler/editor so that we can customize routes in-game.

Serphentas added a commit that referenced this issue May 26, 2019

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
@Serphentas
Copy link
Collaborator

Serphentas commented May 26, 2019

Found info on xtRLE (but it's in German): http://www.karlchenofhell.org/kram.php?part=rhodan#xtRLE

@AWKIF2005
Copy link

AWKIF2005 commented May 26, 2019

I stumbled upon this project while trying desperately to find a way to decrypt the .csv files in ATD (specifically routen, planetyp, and city). First of all, I am beyond excited about the work you guys are doing to open ATD and bring it into the future. This is a classic game! I am not a coder or programmer, so most of what I see here goes completely over my head and I doubt I can be much help, but I will be following this project closely! Now to see if I can figure out how to use this script to open up and edit the .csv files...

Keep up the good work!

@Serphentas
Copy link
Collaborator

Thanks for the comment ! Much appreciated.

As for the script, it can only be used on relation.csv because other ".csv" files are either already CSV or something else (not xtRLE).

@WizzardMaker
Copy link
Member Author

As for the script, it can only be used on relation.csv because other ".csv" files are either already CSV or something else (not xtRLE).

That's not 100% true. I successfully decoded all csv files with the aforementioned c file, doesn't mean your files are the same as mine, but they should be decryptable

I am porting the c file to C# atm. That should show us whether your files can be decoded or not.

xtRLE is also used in the Dialog files, but the current implementation of the script has some problems with the file, but that should be fixable.

@Serphentas
Copy link
Collaborator

That's what I get for en/data/*.csv:

brick.csv:    ISO-8859 text, with CRLF line terminators
builds.csv:   ASCII text, with CRLF line terminators
city.csv:     ISO-8859 text, with CRLF line terminators
clan.csv:     ISO-8859 text, with very long lines, with CRLF line terminators
names.csv:    ISO-8859 text, with CRLF line terminators
piloten.csv:  Non-ISO extended-ASCII text, with CRLF, NEL line terminators
planetyp.csv: ISO-8859 text, with very long lines, with CRLF line terminators
pnames.csv:   ISO-8859 text, with CRLF line terminators
relation.csv: data
routen.csv:   ASCII text, with CR line terminators
stdpaper.csv: ISO-8859 text, with CRLF line terminators

Do you have a similar layout ?

@WizzardMaker
Copy link
Member Author

Well, seems like our versions differ a bit. My relation.csv and builds.csv are not encrypted and all other .csv files are encrypted with xtRLE

I always just look at the HEX content of the files to see whether they are encrypted or not

@AWKIF2005
Copy link

AWKIF2005 commented May 27, 2019

Perhaps the .csv files are in different states of coding depending on different versions or releases of the game. My Strategy First release of ATEvolution had the routen.csv file in regular text, so I was able to use Excel to easily mod the in-game route structure in ATE to my tastes, within some limits before the the game would start crashing on start up. I tried using the regular text routen.csv from ATE, manually added the new routes from ATDeluxe, following the same format, and then put that new routen.csv in the ATDeluxe data folder. It didn't work, the game crashes. I tried multiple variations of city spelling, etc, but no joy. Hence my desire to decode the actual routen.csv from ATD :) My version of ATDeluxe is a download from GOG games.

Interestingly, the builds.csv file is normal text in both versions, allowing easy modding of the "custom airplane" parts. I use this to create custom planes that mimic the specs of real aircraft that are not included in the original game (A330, A380, etc)

Hopefully part of the open ATD project will be to allow these types of mods more easily. Since I am a coding neanderthal, I would be happy to beta play the game for you guys, when it's ready! :)

@AWKIF2005
Copy link

WizzardMaker, sounds like you and I have the same version of the game as far as which .csv's are encrypted. Serphentas, your ATDeluxe routen.csv is already in normal text? Interesting!

@WizzardMaker
Copy link
Member Author

Serphentas is using the Linux version (as far as I can see it) which was ported over, so they probably changed some parts of how they handle the files.

@AWKIF2005
Copy link

That probably explains it then.

@WizzardMaker
Copy link
Member Author

Aaaand now we should be able to decrypt all the xtRLE files.

It seems like the c source from http://karlchenofhell.org/kram/xtRLE_dec.c had some problems with bigger files. I tested the following files with my C# decoder and all got decrypted successfully:

  • misc/ein_ger.res
  • misc/dlg_ger.res
  • misc/std_ger.res
  • data/brick.csv
  • data/city.csv
  • data/clan.csv
  • data/Names.csv
  • data/piloten.csv
  • data/planetyp.csv
  • data/pnames.CSV
  • data/routen.CSV
  • data/stdpapter.csv

I need to clean my code, but it should be committed in a couple of hours

@Serphentas
Copy link
Collaborator

Indeed, I run Linux and use the Linux version from Steam. Anyhow, just by looking at the first bytes of each file we can determine whether or not it is xtRLE-encoded and proceed with proper decoding accordingly.

@WizzardMaker this methodology could be implemented directly in ATDGameLoader.cs at import time, what do you think ?

@WizzardMaker
Copy link
Member Author

@WizzardMaker this methodology could be implemented directly in ATDGameLoader.cs at import time, what do you think ?

Question is, do we need to pack these informations in our game package? There isn't really a need for us to have all of those files in the godot editor

We would need to create a system to handle most of those files correctly first I think.

The brick.csv file is only used by the airport loader at the moment, that doesn't need to be imported into godot.

The clan.csv file handles the IDs and animations of the people, that is also something that should be only loaded once like for the airport loadout, so that we can create the animations.

The other files are kind of cosmetic, like the names, planes or the routes. These are special, as they are the main target of modders. We should maybe include these in godot as special resources, so that modders can modify them.

@AWKIF2005
Copy link

@WizzardMaker The other files are kind of cosmetic, like the names, planes or the routes. These are special, as they are the main target of modders. We should maybe include these in godot as special resources, so that modders can modify them.

I like this idea. Plus it would be interesting to see how (or if) these modded files play with the open version you guys are working on.

@Serphentas
Copy link
Collaborator

In this case, I think it would be wise to create and maintain a Markdown file (instead of a Wiki page, just to keep everything within Git) where we detail how every file type is structured and what purpose it serves. This will not only serve as a reference for game data design but also help modders in their task.

We could already use information found by @WizzardMaker regarding graphics files.

@Serphentas
Copy link
Collaborator

Also, do we want our own format (or a standard format) that we will use later on for any game file or do we want to import game data as is ? In the latter case, modding will imply conversion to "classic" format and then reimporting to the game.

@WizzardMaker
Copy link
Member Author

We should use our own format for all files we want to keep in the game.
We can make our lives easier by using the Godot Resource class. With that we can save and load it like any other resource in godot. We enable in editor editing and we create an easy way to mod our files without studying the original game file structure

@AWKIF2005
Copy link

Okay, I admit defeat on this. I pride myself on being able to "figure it out" when it comes to computer stuff, but after many hours of trial and error and dozens of Google searches over the last few days I cannot figure out how to get the xtRLE code to decrypt the .csv files. Coding and programming is just not something I have needed to use up until now. If only I had gone to school to be a programmer instead of an airline pilot ;)

I know my problem is probably a little beyond the scope of the OpenATDeluxe project, but any hints or pointers would be much appreciated, from one Airline Tycoon fan to another. I am running Windows 10 with Visual Studio 2019 installed. I can't figure out how to point the script to the .csv files or vise versa. I tried compiling the c# script using the cmd prompt (csc), but it throws an "error CS0246: The type or namespace name 'file' could not be found (are you missing a using directive or an assembly reference?)" for each of the three instances of "(file fileIn)". The first time I tried compiling it there was also an error about the using godot reference, so I took that line out since I am trying to do this just in Windows.

Is there a way to define the path to the .csv file in the c# code, allowing it to find the file and run the decryption? Is compiling the .cs file to an .exe even necessary? Or am I thinking about this all wrong?

Thanks for any tips you guys might be able to provide to this noob over here!

@WizzardMaker
Copy link
Member Author

You can't just run the code in visual studio, as we use functions from godot like the file handler

That's why your compile fails. You would need to either use godot or rewrite the code to use the .Net version of File, but that would need a bit more work

If you are currently only interested in the decrypted csv files of AT Deluxe, then write me an email and I will send them to you

@Serphentas
Copy link
Collaborator

As @WizzardMaker said, you need Godot unless you convert the script to native .NET. Otherwise, you can use this file (needs compilation) or simply this Python3 script.

@AWKIF2005
Copy link

@WizzardMaker emailing the decrypted .csv's to me would be incredible. Send em to shiz222@hotmail.com.

That makes sense. I didn't catch that the c# code was specific to godot. I did mess around with the project in godot a bit, but I couldn't get it pointed to at.exe properly, so I went back to trying the code in visual studio and cmd. I also tried the python script early on, but I don't think I got python set up properly on my 10 year old laptop (the only computer I currently have with me) :)

Folks like you guys make the internet awesome. Thank you. I will be following the project...

@Serphentas
Copy link
Collaborator

Thanks for the support, much appreciated. We're still far away from a playable state, realistically looking at the end of 2019, but we'll get there eventually :)

@WizzardMaker WizzardMaker added the discussion Discussions about a topic label Jun 1, 2019
@SyrusDrake
Copy link

Hey, sorry for kinda hijacking this thread.
I tried using the Python script kindly provided by you guys to decrypt the csv files on Linux and I either get the Error message

ValueError: Infile is not encoded with xtRLE

when trying it without specifying an output file or

ValueError: invalid literal for int() with base 10: ''

when specifying an output file.

Am I missing something?

@WizzardMaker
Copy link
Member Author

WizzardMaker commented Jun 30, 2019

OK, seems like I need to make the latest decryptor standalone.

Syrus, the python script was a direct port of a C script someone made for another game. We since have a much improved C# version which handles edge cases a lot better. I will push the standalone decryptor this/next week. You would need mono though to execute the program, as it's programmed in C#.

@SyrusDrake
Copy link

Cool, not sure if I'm smart enough to handle that but I'll give it a try as soon as it's out, thanks a lot ^^'

@WizzardMaker
Copy link
Member Author

So, for anyone finding this, I can't post the decrypted CSV files on GitHub (for now, this can be discussed when implementing the routes etc.), but maybe you will find answers in the GoG forum for the game. cough

@WizzardMaker
Copy link
Member Author

´*.raw´ files are the audio files of the game.
They are standard 44100 KHz wav files.
Example code:

audioFile = new AudioStreamSample()
byte[] data = System.IO.File.ReadAllBytes(PathToSound);
audioFile.SetData(data);
audioFile.MixRate = 44100;

A special sound class will be added to the repo soon.

@WizzardMaker
Copy link
Member Author

Smacker files can now be read!

Thanks to a modified scsharp code. I will push my additions later, after polish.

Question, should we save these animations as files, so that we can drag them in, or should we use scripts and load them at runtime?

image

@Serphentas
Copy link
Collaborator

´*.raw´ files are the audio files of the game.
They are standard 44100 KHz wav files.

Have you been able to play them on VLC or any other media software ? Even on hex editor, it looks far from a WAV (at least with my files).

@WizzardMaker
Copy link
Member Author

Playback right now is working inside my forked smacker branch, which will soon be pushed to the main branch. To play those files outside of the game use Audacity and import the files as raw/unformatted audio files with 8Bit PCM and 44100Hz when using the /video/ files. The /SOUND/ files use 16Bit PCM and 22050Hz

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion Discussions about a topic
Projects
None yet
Development

No branches or pull requests

4 participants