Sim Structure
| The Fundamentals → Sim Structure |
|---|
Written by Connor Jakubik
The Space Teams Platform source code is a large, complex project, but the portions of the software that the user is meant to interact with are built to follow intuitive design patterns. Space Teams uses an architecture somewhat similar to the Entity-Component-System (ECS) pattern. Space Teams Entities are unique objects representing something that exists in the simulation. Space Teams Parameters are similar to ECS components.
From: https://redis.io/nosql/key-value-databases/
Parameters hold all the state of the simulation, in key-value ParamMap data structures. Parameters can hold values of all variable types, including arrays, nested parameter maps, references to Entities, and more. Each Entity has parameters that describe its properties such as mass, location, and graphics representation. Space Teams programs constantly read and affect these parameters to compute and show the simulation.
Space Teams Systems are pieces of code that hold all of the behavior of the simulation. Systems are applied to Entities, and during simulation execution they affect the parameters of the entities to evolve the simulation over time. Systems typically have zero or very few dependencies on other Systems, meaning they can be specialized to a certain environmental effect (e.g. gravity), physical process (e.g. heat transfer), analysis computation (e.g. logging telemetry), or autonomous logic (e.g. path-planning) which can be applied on any Entity.
The Space Teams Platform is structured as a distributed system of programs, where the simulation data is replicated on all programs, and modifications to simulation state are transmitted to other programs to be eventually consistent. The Compute_Server is where all C++-defined simulation behaviors (C++ Systems) are executed. SpaceCRAFT, the Unreal Engine client program, is where any 3D graphics and sim interaction happens, including flat-screen and VR. Additional programs are started for things such as Python Systems. In all programs in the Platform, a copy of all of the Entities / Parameters exists. The general assumption for programs is that they can treat their copy of the simulation Entities and Parameters as the up-to-date current state of the simulation, as the change-replication happens in the background.
Any number of programs can join a simulation, which is agnostic to their properties besides being "another program". This means all Space Teams simulations are multiuser simulations, with zero modifications required compared to single-user, because one Unreal client and a Compute_Server is basically the same as an arbitrary number of Unreal clients being connected.
Space Teams Pro provides web services to authenticate users, manage Orgs, and provide easy methods for hosting multiuser simulations. Org access controls permissions for uploaded files, multiuser session joining, and other online details.
The user can create and edit simulations, upload and configure sim assets such as 3D models, and host/browse/join multiuser sim sessions all from the SpaceCRAFT unreal client program. The simulation editing tools are similar to those used to create games, but they are specialized for spaceflight simulation needs such as orbit adjustment, snapping objects to a planet's surface at a latitude/longitude, setting up reference frames, navigating to objects extremely far from one another, etc. The user can work with local files as well as using Space Teams' git tools for collaborative work.
To write simulation behavior in C++, a user duplicates our template System subclass through a tool to establish all the boilerplate code, then writes code inside the load(), init(), update() and other System member functions that will be called by the Compute_Server in a thread specific to that System during the simulation. The build system for Systems is CMake, and we compile to the C++20 standard. The user is free to link external libraries that are already present in Space Teams or that they include inside their self-contained packages.
For writing simulation behavior in Python instead of C++, Space Teams includes a Python API library. This library encapsulates most of the functionality of the Compute_Server. The Python scripts that use it are 100% "normal" Python, as opposed to "embedded" python scripting where you can't install pip packages. We include the installer for a particular version of Python alongside our application because we can't easily build Python API versions for more than that one version. Our install/setup scripts create a python Virtual Environment in your AppData/Local/SimDynamX/STPro/Versions/.../py folder, which is set up with our baseline set of python packages like numpy. Python System script processes are spun off from the Compute_Server process using .\sc_python.bat (for Windows) with special configuration options in order to connect to the running sim. The user can install more pip packages through .\sc_python_pip.bat install ____.
The Space Teams Platform has special requirements for timekeeping. During a multiuser simulation, it must be possible to:
- Change timescale to any value including zero and negative numbers.
- Do a time-jump to any other time point.
- Be able to represent time for any point in the relevant future and past.
- At least 1,000 years to cover past and future highly eccentric solar system object orbits.
- Provide threadsafe async access to read the time and write timescale and time-jump clock modifications.
- Synchronize clock modifications across a network to for a multiuser sim.
To fulfill these requirements, we implemented an eventual-consistency system which accumulates clock modifications over real-world "wall" time from all clients in the network, then recomputes the sim time from those modifications every time the clock is read. This means, with the assumption that all clients have a synchronized real-world "wall" time, they will have a synchronized sim-time (after clock modification messages have been received through the network).
Space Teams differs from other simulation platforms in that it has no "main loop", "tick numbers", or "synchronization points". All programs and threads within those programs are meant to operate with a completely independent timing scheme. To interact with the running sim, any thread establishes a "now simtime" by reading the sim clock, which is then used as the timestamp on any state changes that happen on that thread. For reading time-sensitive state such as location and orientation of an Entity, the thread uses time-based extrapolation to compensate for the difference between the now simtime and the timestamp of the existing state values. This timestamping and extrapolation approach results in time-corrected and smooth state values being available anywhere in the simulation platform, despite underlying network and compute time instability.
This approach maximizes the benefits of a distributed-systems architecture by allowing programs and threads to independently contribute to the simulation, and never wait on other programs. This also represents reality well by closely matching a real-world complex scenario with independent physical processes, device microcontrollers, and control loops. Want to run a System in a realtime sim at 100,000 Hz? It's possible in Space Teams.
The price of this parallelism is it requires threadsafe and asynchronously-replicated data structures to accomplish, which are much harder to develop than basic object-oriented data structures. Space Teams has a very well-matured solution to this in the GenParamMap data structure and related backend netcode.
There are nondeterministic results in the typical async simulation, where Systems are relying on the thread scheduler and sleep-times to trigger their updates. The results from sim to sim can differ within reasonable error or in some cases corrupted by a lapse in true update frequency due to either high sim timescale or high/irregular computation time per update cycle. To address this, Space Teams has a deterministic mode, which uses the same exact simulation code but replaces the system-clock timing with a deterministic scheduled-ahead sequence of updates. This then runs the simulation as fast as possible on a single thread, jumping directly from one system-update to the next, with the assumed-time jumping accordingly. This does not yet have a mechanism to work with sim behavior written in Python Systems as those are external processes not operating in lock-step with the Compute_Server, but read-only Python Systems will function as expected.
SimGlobals is a class with a set of functions that access simulation-wide state. SimGlobals are accessible from anywhere in the simulation. The SimEntity is a singleton Entity in SimGlobals, which has parameters like any other entity in the simulation. SimEntity does not allow Systems to be applied to it, as all Systems already can do SimGlobals::GetSimEntity(). SimGlobals also provides functions for accessing the SimClock, Adding/Deleting Entities, and more.
Main wiki page: Making Sim Configs
To make a simulation in Space Teams, one makes a Simulation Configuration (sim config) file. This can be done manually by duplicating a file, through SpaceCRAFT with the Create New Sim menu, or by saving the state from a running simulation. The Sim Config holds declarations of all the Entities that exist when the simulation starts, their initial conditions, System Instances that will be run, and tags for applying System Instances to certain Entities. On sim loading, programs in the Space Teams Platform independently parse the sim config to initialize their local copy of the simulation state.
On the Compute_Server or any Python System, console output as well as logging to a file are all handled by our custom-made Atomic Logger feature. This is a threadsafe logging framework that has configurable console and file outputs, does colored text output with ANSI escape sequences, and allows many separately-configured Logger objects to write to overlapping sets of output files / pipes with proper mutual-exclusion locking.
When an unhandled exception or other internal error occurs in either a C++ System or a Python System, unless a particular option is changed, the exception is caught and printed by the Fatal log function.
In the SpaceCRAFT Unreal Engine client, fatal errors are typically handled by popping up a modal window with the error message and asking if the user wants to quit or return to the menu. SpaceCRAFT doesn't have logging enabled in the User Release at this time (v0.25.0); that only happens in development builds.