Architecture Design Proposal

[pedro-psb]

Introduction

This modular design proposal goal is to assure flexibility and consistency in the development process, from the first codebase iteration to future improvements, optimizations and bugfixes.

By clearly separating modules responsibilities and defining communication policies (layered architecture), the project may enjoy the following advantages:

• parallel work: developers can work on different parts with clearer task scope and without much risk of merge conflicts.
• on-boarding: new contributors can understand and contribute faster and safer to the project
• caching: clear definition of API makes module-level caching easier
• side-effect safeguard: bug fixes, feature or optimizations should only affect a predictable portion of the codebase.

The main disadvantage is that changes that cannot be accommodated in this framework will be hard to implement. Nevertheless, as the project has received a lot of feedback and concrete usage in the last years, most core behaviour until now should be properly covered.

In this document, I'll present a mix of implementation and abstract structural ideas.

Architecture and data flow

General Structure

A Layer Architecture main goal is to establish communication protocols to make modules as independent as possible. Here are some definitions and motivations about the proposed structure.

Shell and Controllers

In Dynaconf 3, the Dynaconf class contains methods that are aimed at public use and internal use, which makes it harder to maintain and make deprecating more sensible (as internal methods are exposed, we can't change them freely). As a consequence, the user experience of auto-completion was not very good, either, as many internal methods were exposed.

In this context, the Shell layer is a thin structure that should contain only public resources and should trust the layer below to provide better internal calls. In this way, both developer and user experience will be clearer and easier to improve and optimize.

For example, a public method setting.set("foo", "bar") may only call a corresponding lib_core.set(...), which will handle all the necessary logic to perform the set operation in a safe environment/namespace. Any changes in set implementation won't have to worry about deprecation, renaming internal methods and variables, etc, as long as public set continues to have the same behaviour.

Also, although not represented in the diagram, the core_lib and core_cli may share some common low-level API and differentiate themselves only in the "upper side", that is, in the interaction with lib or cli shells.

Controllers and Services

Dynaconf 3 internal data flow was fairly complex. By trying to systematize how it worked, I've come up with an abstraction of services and controllers, where controllers arrange a set of services to fulfill the user's request. This separation goal is to make it clear what action are being taken upon the settings, which was a bit hard to track on the old codebase.

The service layer needs to provide only two things in this arrangement: storage and data processing. The controllers get some data, and may load more data, evaluate, merge and possibly save it back to storage. Then it converts it back to the final-user. Also, this provides some caching points, which may be implemented in later optimization steps.

Here is a sample data-flow:

Setting Internal Representation

To allow for a clear communication between Services and Controllers, it is important to have a common data-exchange format. Although using a regular dict may save costs compared to converting from a custom data-structure, it lacks some flexibility that would help operations in general, such as evaluating and merging. It's hard to conjecture about what is best, but I think it worth trying it out and doing some benchmark.

With that said, a Tree seemed appropriate for representing such a flexible setting data-structure. The main advantage how easy it is to traverse it in various ways and that nodes (or elements within nodes) can hold settings-metadata, which may be useful for Action processing.

For example, the evaluate() can pre-evaluate simple values, such as a raw_value with casting token, by saving them to real_value. At the same time, it can gather variable dependencies, sort them correctly and run the evaluation again just against those nodes (when convenient).

Action overview

The Action service was designed to accommodate most current features of Dynaconf3 under a common interface. Namely: multi-layer config (load and merge operations), lazy evaluation (evaluation operations) and validation (validation operations). This is where most complexity will live, therefore, it should be good to give it an extra attention.

The common denominator among those is that all of them are operations over a set of settings. But as each one of them have specific needs, they should have specific extra-parameters. One advantage of such an abstraction is to provide a clear way of understanding the role of them in the process and enforce the use of a common communication pattern.

Another important aspect to consider in each module implementation is the configurable nature of Dynaconf behaviour. Different internal config lead to specific merge, load or validation behaviour, and this mapping between configuration and functionality should be made explicit as possible. To achieve this, it would be beneficial to have scoped internal configs in a per-module basis. If this scoped-config is made available for the final user too, it would be easier for them to to find specific ways to configure specific parts of the program.

Considerations

This is an overview of a possible strategy for starting a new implementation of Dynaconf. I've started with the idea (and implementation experimentation) of the internal config as a Tree and, after some iterations, I thought of some possible ways things could go together. A lot of important details are left off, and I'll try to list some of them here:

• Immutability vs mutability in actions: If SettingTree objects are mutable, when an action (e.g., evaluate) is manipulating it, it will be changed in the Storage too, which is where changes should be persisted across runtime. This can be good for performance, because it saves the need for making a copy of the whole structure. On the other hand, it makes it harder to debug and breaks a little of the system predictability. For a start, we can try keeping the mutability, being aware that this might happen. Also, my initial proposal is a SettingTree which contains Setting object, and mutability can operate differently in those entities.
• Mapping configuration and functionality: although the idea of scoped configuration should be helpful, this is in a very conceptual state yet. My first thoughts about it is that configurations should be a special kind of SettingTree that is stored in Storage. At the same time, it would be nice to have it like an object with autocompletion when implementing each module. This is an important part of the system and should be treated carefully.

Conclusion

To summarize, the main differences compared to Dynaconf v3 are:

• Layer architecture: strict separation of modules into conceptual layers, which restrict with which parts of the system they can communicate.
• Creation of Shell Layer: providing better usability for final users and more flexibility for developing, testing and maintaining.
• Replace Box with SettingTree a Tree should bring more flexibility for the manipulation and storage. We can implement a lightweight solution for the object-dot-notation feature.
• Actions Layer formalization enforce how an action should process data internally by a common function signature.
• Storage Layer enforce a common way of accessing background data, that is, settings that are not the current active set of settings/environment.

As stated across the document, there are some uncertainty regardless performance (like the use of a Tree) and some not well-defined structures (like how internal configurations will work). Nevertheless, this should provide a good starting point for discussion.