Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move Ansible hosts storage to file system #213

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

unitmatrix
Copy link

Currently, ansible-cmdb is storing parsed Ansible hosts and variables in dictionary variable in memory. While this works well for tens and hundreds of hosts, it does not scale for larger inventories with thousands hosts or more. The script would run for over 1 hour and finally crash with memory exception for me. Generating cmdb inventory for 10.000+ hosts is basically impossible.

This change moves hosts management to a separate AnsibleHosts class where it implements Python dictionary so that the data is saved to a temporary directory on disk rather than your RAM thus enabling linear processing of hosts where the free space on disk is your only limit.

This only changes the back-end of the hosts dictionary so that it implements functionality of the regular Python dictionary providing standard methods, such as:

hosts = AnsibleHosts() # create new object
hosts[key] = value     # set new value
item = hosts.get(key)  # get value by key
len = len(hosts)       # get number of items
items = hosts.items()  # iterate-able items
key in hosts           # check if key exists
etc.

This way it does not affect other parts of code which will utilize the dictionary as usual. Unit tests attached to verify the required functionality as well.

Aleksandr Cupacenko added 2 commits March 19, 2020 10:21
…es (10K+) file linear system based processing of parsed hosts data to reduce memory usage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant