Project overview
- Brief
- Approach
- Technical development
- Revisions, Streams and Deltas
- Network and data flow
- Network configurations and data flow scenarios
- Rest API
- Requirements gathering
The aim of AECDeltas is to define a workflow for the AEC industry that streamlines data exchange, so that anyone, from consultants to manufacturers, can access the same data from a single source.
This is accomplished through the definition of a novel transfer protocol, together with an open-source micro-services online framework. These enable the industry to exchange individual object-level changes across various applications, regardless of data format.
In summary, the AECDeltas:
- Defines a new common interchange schema that is open and shared across various systems;
- Specifies a new technology so that different architecture, engineering and construction applications can communicate easily over the Internet;
- Validates the proposed specification by creating a working reference implementation that connects open-source Speckle Works and 3DRepo with proprietary systems.
The approach of AECDeltas was defined by considering what is the "classical approach" for data sharing in the AEC industry.
The most common solution to date is to share entire models, which mean large data payload need to be sent. This makes data exchange cumbersome in most scenarios.
Additionally, the tracking of changes is usually absent. This is because many copies of the large models are usually stored for tracking the history of the work; the comparison between those models is usually written manually in spreadsheets or not tracked at all.
AECDeltas, is hinged on the concept of the Diff and of the exchange of Deltas.
A Diff is the result of the comparison between two models, that contains all the changes between them.
The Diff can be completed with authorship and security information to form a Delta, which can be then transferred among people to minimise the payload and ensure the tracking of changes.
This also enables more possibilities in collaboration. We shift the paradigm of collaboration based on models, or files, to a collaboration based on exchange of objects (the individual components of the models).
The underling basis for the Delta is the Diffing, which is a term borrowed from computer science.
In simple terms, the Diff is the result of comparing (Diffing) two Revisions of a certain project Model.
The diff is simply a collection of the objects that are different between the two revisions, categorised per their type: new, old or modified.
In more detail, if we consider the Modified objects, the Diff can also store the properties that have been modified across the two revisions. This is a more "detailed" version of the diff:
We distinguish the Diff depending on whether it contains or not the modified properties, in terms of level of detail. This helps in clarifying implementation, as the Property-Level diffing is more complex and not covered by AECDeltas.
Collection-level diffing (Level 1 diff).
Input being the two collections and the result is a list of changed objects. This is implemented as part of the AEC Deltas micro-services layer and is integral to our definition;
Property-level diffing (Level 2 diff).
Diffing of individual line-by-line properties only happens at the application layer and is left open to implementation as required by the receiving application. This, therefore, is not an enforced part of the AEC Deltas protocol, but can be delivered on top.
The delta container is AECDeltas' unit of exchange. Producing the Delta is based on the Diff, but the Delta is subject to its own defining principles:
A single object is the smallest unit of change.
Even if the smallest of properties on an object has been changed, the whole object needs to be flagged accordingly. For instance, a single vertex in a mesh change would require the whole mesh object to be updated.
All objects are immutable.
Any change on an object means the object has been modified, and thus a new object with a new ID/hash needs to be created to replace it and advance the revision further.
Each object has a unique and a shared ID.
Each object requires two types of ID: a unique ID (UID) that uniquely identifies it at a certain point in time. This changes every time the object changes. a persistent ID (PID) which does not change across various revisions of the same object over time. This allows to identify modified objects over time. It is generated once per object creation.
Not all objects have to have a visual representation.
There are objects that simply have no visual representation (such as a renderable mesh) attached. E.g. a steel beam can be represented by its load capacity and endpoints in order to perform structural analysis calculations even though it would not be renderable directly as is.
The Full Delta container specification can be found here.
The Delta contains 2 main blocks:
- the essential parts of the Diff between two revisions. This is to be only the minimal amount of data taken from a full Diff, needed to update a model stored elsewhere. This means: the entire “modified objects” are exchanged; we leave out “UnchangedObjects”; we do not send the entire Deleted objects but just their Id, as it suffices to perform deletion on the receiving end.
- Authorship and addressing data. All the data required to ensure a secure transmission of the information.
The following image is the json representation of the Delta.
For the Delta, see https://github.com/aecdeltas/aec-deltas-spec/wiki/Delta-Container-Specification.
Now that we introduced the main concepts, we can better define the terms we've used so far and better understand the relationship between them.
Revision – a given state in time of a collection of objects, which will be stored (recorded) in a Stream.
Stream – a series of Revisions, expressing a record of the states of a collection of objects over time.
It is also a means of sharing or categorising parts of a project that will undergo change.
Delta – the change between two given Revisions, authored, secured and stamped with DateTime. It can be used to update a Stream.
A full table of definitions, including comparisons with other terms common in the industry, can be found here: https://github.com/aecdeltas/aec-deltas-spec/wiki/Definitions.
Our definitions allow us to define an AECDeltas network, and how the data should flow within it.
The following image describes an example of a network of users within the AECDeltas project.
Alice, Bernard, Calvin and Dina are all users of the network. Alice stores her models into a Stream S0, from which Bernard and Calvin can access her data. Further, Bernard decides to store a specific comparison between his and Alice's data into another Stream S1. S1 can be separately accessed by Calvin and Dina.
How does the data actually flow between users, and through the stream?
Supposing that the Stream S0 is initially empty, Alice will send her whole model in the form of a new Revision. From there, Users Bernard and Calvin can ask for the Delta from S0. From receiving users, the Delta is always the unit of exchange; similarly, if Bernard wants to send an update to S0, he will share a Delta.
An additional advantage of this is that users retain hosting and control access to the streams. Effectively, Stream S0 and S1 might be hosted in a same server inside one organisation, or they might be stored on individual servers, depending on the desired set-up.
The AECDeltas specification considers all the possible network configurations and scenarios that may be decided by users.
The network configurations considered, illustrated in the following image, are:
single user/single server; single user/multiple server; multiple servers/multiple users.
The data flow scenarios are:
1. Push of an entirely new model/project.
A new model is to be stored in the server for the very first time.
2. Server is behind.
A clients wants to update the Server revision, sending the delta.
**3. Conflict:
-
Conflict: a client is behind.**
Client is on an out of date revision. They send an out-of-date delta.
- Conflict: two clients are targeting the same revision.
Two clients update the same revision simultaneously which leads to one of them being out of sync.
- Conflict: Client is pushing a revision already existing on the server.
This results in a null delta.
The flow-sequence diagrams formalise the specification with respect to:
- each possible network configuration
- each possible data flow scenario.
The sequence diagrams can be complex, as they need to consider multiple factors both on front-end and back-end.
Together with the REST API, they allow for seamless integration with other AECDeltas-compliant servers/services.
The full diagrams can be found here. An example of a sequence diagram is pictured in the image below:
The REST API defines the required end-points for an AECDeltas-compliant server.
The endpoints cover the variety of actions as per the network configurations described above; for this reason, they have been categorised as in the following image.
Each endpoint defines the HTTP Request, the required parameters and all possible Responses.
The Full REST API specification can be found here: https://github.com/aecdeltas/aec-deltas-spec/wiki/REST-API.
In order to demonstrate the usefulness of the AECDeltas protocol, a detailed study of the most common design and manufacturing activities has been conducted.
The selected activities have been transformed into an extensive list of user requirements.
These capture all of the relevant system actors and user stories in a standard software engineering format of “As a , I want a so that a ”.
In order to better describe the requirements, 10 distinctive user stories have been captured and illustrated through diagrams.
An example of a User Story and related diagram is shown in the following image.
In addition to the User requirements, the Consortium gathered formal systems requirements.
These have been split into 3 categories and subdivided by individual partner companies:
- Functional requirements (e.g. the ability to identify users by their cryptographic signature)
- Non-functional requirements (e.g. the ability to process large federated projects within 30 minutes)
- Domain-specific requirements (e.g. the need to support AEC-specific authoring tool)