Move Ansible hosts storage to file system #213
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, ansible-cmdb is storing parsed Ansible hosts and variables in dictionary variable in memory. While this works well for tens and hundreds of hosts, it does not scale for larger inventories with thousands hosts or more. The script would run for over 1 hour and finally crash with memory exception for me. Generating cmdb inventory for 10.000+ hosts is basically impossible.
This change moves hosts management to a separate AnsibleHosts class where it implements Python dictionary so that the data is saved to a temporary directory on disk rather than your RAM thus enabling linear processing of hosts where the free space on disk is your only limit.
This only changes the back-end of the hosts dictionary so that it implements functionality of the regular Python dictionary providing standard methods, such as:
This way it does not affect other parts of code which will utilize the dictionary as usual. Unit tests attached to verify the required functionality as well.