Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactoring interface common routines #1079

Draft
wants to merge 1 commit into
base: devel
Choose a base branch
from
Draft

Refactoring interface common routines #1079

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
223 changes: 75 additions & 148 deletions ansible_runner/__main__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -381,109 +381,6 @@ def error(self, message):

super(AnsibleRunnerArgumentParser, self).error(message)


@contextmanager
def role_manager(vargs):
if vargs.get('role'):
role = {'name': vargs.get('role')}
if vargs.get('role_vars'):
role_vars = {}
for item in vargs['role_vars'].split():
key, value = item.split('=')
try:
role_vars[key] = ast.literal_eval(value)
except Exception:
role_vars[key] = value
role['vars'] = role_vars

kwargs = Bunch(**vargs)
kwargs.update(private_data_dir=vargs.get('private_data_dir'),
json_mode=vargs.get('json'),
ignore_logging=False,
project_dir=vargs.get('project_dir'),
rotate_artifacts=vargs.get('rotate_artifacts'))

if vargs.get('artifact_dir'):
kwargs.artifact_dir = vargs.get('artifact_dir')

if vargs.get('project_dir'):
project_path = kwargs.project_dir = vargs.get('project_dir')
else:
project_path = os.path.join(vargs.get('private_data_dir'), 'project')

project_exists = os.path.exists(project_path)

env_path = os.path.join(vargs.get('private_data_dir'), 'env')
env_exists = os.path.exists(env_path)

envvars_path = os.path.join(vargs.get('private_data_dir'), 'env/envvars')
envvars_exists = os.path.exists(envvars_path)

if vargs.get('cmdline'):
kwargs.cmdline = vargs.get('cmdline')

playbook = None
tmpvars = None

play = [{'hosts': vargs.get('hosts') if vargs.get('hosts') is not None else "all",
'gather_facts': not vargs.get('role_skip_facts'),
'roles': [role]}]

filename = str(uuid4().hex)

playbook = dump_artifact(json.dumps(play), project_path, filename)
kwargs.playbook = playbook
output.debug('using playbook file %s' % playbook)

if vargs.get('inventory'):
inventory_file = os.path.join(vargs.get('private_data_dir'), 'inventory', vargs.get('inventory'))
if not os.path.exists(inventory_file):
raise AnsibleRunnerException('location specified by --inventory does not exist')
kwargs.inventory = inventory_file
output.debug('using inventory file %s' % inventory_file)

roles_path = vargs.get('roles_path') or os.path.join(vargs.get('private_data_dir'), 'roles')
roles_path = os.path.abspath(roles_path)
output.debug('setting ANSIBLE_ROLES_PATH to %s' % roles_path)

envvars = {}
if envvars_exists:
with open(envvars_path, 'rb') as f:
tmpvars = f.read()
new_envvars = safe_load(tmpvars)
if new_envvars:
envvars = new_envvars

envvars['ANSIBLE_ROLES_PATH'] = roles_path
kwargs.envvars = envvars
else:
kwargs = vargs

yield kwargs

if vargs.get('role'):
if not project_exists and os.path.exists(project_path):
logger.debug('removing dynamically generated project folder')
shutil.rmtree(project_path)
elif playbook and os.path.isfile(playbook):
logger.debug('removing dynamically generated playbook')
os.remove(playbook)

# if a previous envvars existed in the private_data_dir,
# restore the original file contents
if tmpvars:
with open(envvars_path, 'wb') as f:
f.write(tmpvars)
elif not envvars_exists and os.path.exists(envvars_path):
logger.debug('removing dynamically generated envvars folder')
os.remove(envvars_path)

# since ansible-runner created the env folder, remove it
if not env_exists and os.path.exists(env_path):
logger.debug('removing dynamically generated env folder')
shutil.rmtree(env_path)


def print_common_usage():
print(textwrap.dedent("""
These are common Ansible Runner commands:
Expand Down Expand Up @@ -824,52 +721,82 @@ def main(sys_args=None):
if vargs.get('command') in ('transmit', 'worker', 'process'):
streamer = vargs.get('command')

# TODO: Remove / Refactor unused role context manager
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like it was removed?

with context:
with role_manager(vargs) as vargs:
run_options = dict(private_data_dir=vargs.get('private_data_dir'),
ident=vargs.get('ident'),
binary=vargs.get('binary'),
playbook=vargs.get('playbook'),
module=vargs.get('module'),
module_args=vargs.get('module_args'),
host_pattern=vargs.get('hosts'),
verbosity=vargs.get('v'),
quiet=vargs.get('quiet'),
rotate_artifacts=vargs.get('rotate_artifacts'),
ignore_logging=False,
json_mode=vargs.get('json'),
omit_event_data=vargs.get('omit_event_data'),
only_failed_event_data=vargs.get('only_failed_event_data'),
inventory=vargs.get('inventory'),
forks=vargs.get('forks'),
project_dir=vargs.get('project_dir'),
artifact_dir=vargs.get('artifact_dir'),
roles_path=[vargs.get('roles_path')] if vargs.get('roles_path') else None,
process_isolation=vargs.get('process_isolation'),
process_isolation_executable=vargs.get('process_isolation_executable'),
process_isolation_path=vargs.get('process_isolation_path'),
process_isolation_hide_paths=vargs.get('process_isolation_hide_paths'),
process_isolation_show_paths=vargs.get('process_isolation_show_paths'),
process_isolation_ro_paths=vargs.get('process_isolation_ro_paths'),
container_image=vargs.get('container_image'),
container_volume_mounts=vargs.get('container_volume_mounts'),
container_options=vargs.get('container_options'),
directory_isolation_base_path=vargs.get('directory_isolation_base_path'),
cmdline=vargs.get('cmdline'),
limit=vargs.get('limit'),
streamer=streamer,
suppress_env_files=vargs.get("suppress_env_files"),
)
try:
res = run(**run_options)
except Exception:
exc = traceback.format_exc()
if stderr_path:
open(stderr_path, 'w+').write(exc)
else:
sys.stderr.write(exc)
return 1
return(res.rc)
run_options = dict(private_data_dir=vargs.get('private_data_dir'),
ident=vargs.get('ident'),
binary=vargs.get('binary'),
playbook=vargs.get('playbook'),
module=vargs.get('module'),
module_args=vargs.get('module_args'),
host_pattern=vargs.get('hosts'),
verbosity=vargs.get('v'),
quiet=vargs.get('quiet'),
rotate_artifacts=vargs.get('rotate_artifacts'),
ignore_logging=False,
json_mode=vargs.get('json'),
omit_event_data=vargs.get('omit_event_data'),
only_failed_event_data=vargs.get('only_failed_event_data'),
inventory=vargs.get('inventory'),
forks=vargs.get('forks'),
project_dir=vargs.get('project_dir'),
artifact_dir=vargs.get('artifact_dir'),
role=vargs.get('role'),
roles_path=[vargs.get('roles_path')] if vargs.get('roles_path') else None,
process_isolation=vargs.get('process_isolation'),
process_isolation_executable=vargs.get('process_isolation_executable'),
process_isolation_path=vargs.get('process_isolation_path'),
process_isolation_hide_paths=vargs.get('process_isolation_hide_paths'),
process_isolation_show_paths=vargs.get('process_isolation_show_paths'),
process_isolation_ro_paths=vargs.get('process_isolation_ro_paths'),
container_image=vargs.get('container_image'),
container_volume_mounts=vargs.get('container_volume_mounts'),
container_options=vargs.get('container_options'),
directory_isolation_base_path=vargs.get('directory_isolation_base_path'),
cmdline=vargs.get('cmdline'),
limit=vargs.get('limit'),
streamer=streamer,
suppress_env_files=vargs.get("suppress_env_files"),
)
try:
output.display(run_options)
res = run(**run_options)
rc_actual = res.rc
except Exception:
exc = traceback.format_exc()
if stderr_path:
open(stderr_path, 'w+').write(exc)
else:
sys.stderr.write(exc)
rc_actual = 1

if vargs.get('project_dir'):
project_path = vargs.get('project_dir')
else:
project_path = os.path.join(vargs.get('private_data_dir'), 'project')

project_exists = os.path.exists(project_path)

env_path = os.path.join(vargs.get('private_data_dir'), 'env')
env_exists = os.path.exists(env_path)

envvars_path = os.path.join(vargs.get('private_data_dir'), 'env/envvars')
envvars_exists = os.path.exists(envvars_path)

if vargs.get('role'):
if not project_exists and os.path.exists(project_path):
logger.debug('removing dynamically generated project folder')
shutil.rmtree(project_path)
elif not envvars_exists and os.path.exists(envvars_path):
logger.debug('removing dynamically generated envvars folder')
os.remove(envvars_path)

# since ansible-runner created the env folder, remove it
if not env_exists and os.path.exists(env_path):
logger.debug('removing dynamically generated env folder')
shutil.rmtree(env_path)

return(rc_actual)

try:
with open(pidfile, 'r') as f:
Expand Down
2 changes: 1 addition & 1 deletion ansible_runner/config/runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ def prepare_env(self):
else:
self.cwd = self.project_dir

if 'fact_cache' in self.settings:
if self.settings.get('fact_cache'):
if 'fact_cache_type' in self.settings:
if self.settings['fact_cache_type'] == 'jsonfile':
self.fact_cache = os.path.join(self.artifact_dir, self.settings['fact_cache'])
Expand Down
4 changes: 2 additions & 2 deletions ansible_runner/interface.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,10 @@ def init_runner(**kwargs):
'''
# If running via the transmit-worker-process method, we must only extract things as read-only
# inside of one of these commands. That could be either transmit or worker.
if kwargs.get('streamer') not in ('worker', 'process'):
if kwargs.get('process_isolation') or kwargs.get('streamer') not in ('worker', 'process'):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines are what I was really looking for in here. Trying to use direct role execution from the CLI using an execution environment didn't work because of the path assignment behavior.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoa, wait. Just on the surface of it, it seems like making this change would break the transmit phase of streaming runner.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... or I misread the logic.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reconsidering, though, I'm not sure how it helps to do the dump_artifacts writing to disk in the worker and processor cases when we have process isolation? The worker should already have everything by unzipping everything that came over the wire from the transmit phase, and I don't think that the processor phase needs any of it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In practice, artifacts shouldn't be getting dumped because of suppress_env_files

dump_artifacts(kwargs)

if kwargs.get('streamer'):
if kwargs.get('streamer') or kwargs.get('process_isolation'):
# undo any full paths that were dumped by dump_artifacts above in the streamer case
private_data_dir = kwargs['private_data_dir']
project_dir = os.path.join(private_data_dir, 'project')
Expand Down