-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Local check runner #51
Merged
Merged
Changes from all commits
Commits
Show all changes
95 commits
Select commit
Hold shift + click to select a range
25cf982
Changes for local-check-runner utility.
dmichaels-harvard 89e6973
typo
dmichaels-harvard 61c6934
Merge branch 'master' into local-check-runner
dmichaels-harvard ed6e363
Adding support for local-check-runner utility.
dmichaels-harvard 0ff3ceb
Adding support for local-check-runner utility.
dmichaels-harvard 57720f7
Adding support for local-check-runner utility.
dmichaels-harvard a56d731
Adding support for local-check-runner utility.
dmichaels-harvard 3c5659c
Adding support for local-check-runner utility.
dmichaels-harvard 38f91d4
Adding support for local-check-runner utility.
dmichaels-harvard 485bdf7
Adding support for local-check-runner utility.
dmichaels-harvard 0fbbc50
Adding support for local-check-runner utility.
dmichaels-harvard be29cd1
Adding support for local-check-runner utility.
dmichaels-harvard dc89492
Adding support for local-check-runner utility.
dmichaels-harvard 040a4a7
Adding support for local-check-runner utility.
dmichaels-harvard 3c329ba
Adding support for local-check-runner utility.
dmichaels-harvard 8f321cf
Adding support for local-check-runner utility.
dmichaels-harvard 6479479
Adding support for local-check-runner utility.
dmichaels-harvard 86e9a65
Adding support for local-check-execution utility.
dmichaels-harvard 93c8890
Adding support for local-check-execution utility.
dmichaels-harvard 4d65058
Fix to test_fs_connection
dmichaels-harvard 2d6fa13
comments
dmichaels-harvard 86cddd0
Version update
dmichaels-harvard 2c7242f
Minor update to identity.py to respect REDIS_HOST_LOCAL.
dmichaels-harvard 8aecbd3
Allow override of Auth0 client/secret via env var for local dev.
dmichaels-harvard 47d3c44
Misc updates related to foursight-smaht.
dmichaels-harvard d5279bb
Misc updates related to foursight-smaht.
dmichaels-harvard a2d71d1
Minor test fix.
dmichaels-harvard 70e8f55
SMaHT related UI updates.
dmichaels-harvard 427863e
SMaHT related UI updates.
dmichaels-harvard 425f170
Minor UI fix for Foufront
dmichaels-harvard 1a3495f
Minor UI fix for Foufront
dmichaels-harvard e7f7e59
Minor SMaHT related updates.
dmichaels-harvard a28f07c
Minor SMaHT related updates.
dmichaels-harvard b8f0a46
Minor SMaHT related updates.
dmichaels-harvard 6d8d509
Minor SMaHT related updates.
dmichaels-harvard d035521
Fixup users
dmichaels-harvard 34a55e5
Fixup users
dmichaels-harvard 49e9123
Fixup users
dmichaels-harvard 2774547
Fixup users
dmichaels-harvard b72e045
Fixup users
dmichaels-harvard fce6e65
Fixup users
dmichaels-harvard b04015b
Fixup users
dmichaels-harvard d3bd79a
Fixup users
dmichaels-harvard bfb91db
Fixup users
dmichaels-harvard 85006df
Fixup users
dmichaels-harvard 8b6e76d
Fixup users
dmichaels-harvard 2df1f54
Fixup users
dmichaels-harvard 3624c96
Fixup users
dmichaels-harvard 1fca246
Fixup users
dmichaels-harvard 5231161
Fixup users
dmichaels-harvard 5bac39a
Fixup users
dmichaels-harvard 0872b7e
Fixup users
dmichaels-harvard e30f9e7
Fixup users
dmichaels-harvard 68d4e0d
Fixup users
dmichaels-harvard 42c7e18
Fixup users
dmichaels-harvard e7afba2
Minor updates to accounts component.
dmichaels-harvard e698a8e
Minor fix in accounts component.
dmichaels-harvard 1c44926
Minor fix in accounts component.
dmichaels-harvard 741367e
Minor fix in accounts component.
dmichaels-harvard 237fc04
Minor fix in accounts component.
dmichaels-harvard 521f8d1
Minor fix in accounts component.
dmichaels-harvard 5db0045
Minor fix in accounts component.
dmichaels-harvard 6f528d8
Minor fix in accounts component.
dmichaels-harvard bcaa9f0
Minor fix in accounts component.
dmichaels-harvard 55fe023
Minor fix in accounts component.
dmichaels-harvard 8e5471f
Minor fix in accounts component.
dmichaels-harvard ae1ef02
Minor fix in accounts component.
dmichaels-harvard a617f79
Minor fix in accounts component.
dmichaels-harvard 698dd1f
Minor fix in accounts component.
dmichaels-harvard fa558e1
Minor fix in accounts component.
dmichaels-harvard 2d5316d
Minor UI update.
dmichaels-harvard 2dcf80f
Support for ingestion sumbission endpoints.
dmichaels-harvard 8b0ec14
Support for ingestion sumbission endpoints.
dmichaels-harvard 68d4718
Support for ingestion sumbission endpoints.
dmichaels-harvard efac358
Support for ingestion sumbission endpoints.
dmichaels-harvard bd11700
Support for ingestion sumbission endpoints.
dmichaels-harvard 6727d36
Support for ingestion sumbission endpoints.
dmichaels-harvard 8960b9d
Support for ingestion sumbission endpoints.
dmichaels-harvard aa5622e
Support for ingestion sumbission endpoints.
dmichaels-harvard 8a367c0
Support for ingestion sumbission endpoints.
dmichaels-harvard c24eccb
Support for ingestion sumbission endpoints.
dmichaels-harvard 69d91e1
Support for ingestion sumbission endpoints.
dmichaels-harvard 1154ef4
Support for ingestion sumbission endpoints.
dmichaels-harvard ac11e6f
Support for ingestion sumbission endpoints.
dmichaels-harvard 2622ac9
Support for ingestion sumbission endpoints.
dmichaels-harvard 446e636
Support for ingestion sumbission endpoints.
dmichaels-harvard 841b863
versions
dmichaels-harvard 75c16ca
versions
dmichaels-harvard 217d2ce
versions
dmichaels-harvard a6c7774
versions
dmichaels-harvard 7fbaed7
Minor updates to ingestion submissions page.
dmichaels-harvard e999725
Minor updates to ingestion submissions page.
dmichaels-harvard 0f59871
Minor updates to ingestion submissions page.
dmichaels-harvard dc0c2f5
Minor updates to ingestion submissions page.
dmichaels-harvard 33fd703
Minor updates to ingestion submissions page.
dmichaels-harvard File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,6 @@ | ||
from typing import Optional | ||
from chalice import Chalice | ||
app = Chalice(app_name='foursight-core') | ||
app.request_args = lambda: app.current_request.to_dict().get("query_params", {}) | ||
app.request_arg = lambda name, default = None: app.current_request.to_dict().get("query_params", {}).get(name, default) | ||
app.request = lambda: app.current_request |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,70 @@ | ||
# TODO: Move to dcicutils. | ||
from collections import namedtuple | ||
from contextlib import contextmanager | ||
import io | ||
import sys | ||
from typing import Optional | ||
|
||
_real_stdout = sys.stdout | ||
_real_stderr = sys.stderr | ||
|
||
@contextmanager | ||
def captured_output(capture: bool = True): | ||
""" | ||
Context manager to capture any/all output to stdout or stderr, and not actually output it to stdout | ||
or stderr. Yields and object with a get_captured_output() method to get the output captured thus far, | ||
and another uncaptured_print() method to actually print the given output to stdout, even though output | ||
to stdout is being captured. Can be useful, for example, in creating command-line scripts which invoke | ||
code which outputs a lot of info, warning, error, etc to stdout or stderr, and we want to suprress that | ||
output; but with the yielded uncaptured_print() method output specific to the script can actually be | ||
output (to stdout); and/or can also optionally output any/all captured output, e.g. for debugging or | ||
troubleshooting purposes. Disable this capture, without having to restructure your code WRT the usage | ||
of the with-clause with this context manager, pass False as an argument to this context manager. | ||
""" | ||
|
||
original_stdout = _real_stdout | ||
original_stderr = _real_stderr | ||
captured_output = io.StringIO() | ||
|
||
def set_original_output() -> None: | ||
sys.stdout = original_stdout | ||
sys.stderr = original_stderr | ||
|
||
def set_captured_output() -> None: | ||
if capture: | ||
sys.stdout = captured_output | ||
sys.stderr = captured_output | ||
|
||
def uncaptured_print(*args, **kwargs) -> None: | ||
set_original_output() | ||
print(*args, **kwargs) | ||
set_captured_output() | ||
|
||
def uncaptured_input(message: str) -> str: | ||
set_original_output() | ||
value = input(message) | ||
set_captured_output() | ||
return value | ||
|
||
def get_captured_output() -> Optional[str]: | ||
return captured_output.getvalue() if capture else None | ||
|
||
try: | ||
set_captured_output() | ||
Result = namedtuple("Result", ["get_captured_output", "uncaptured_print", "uncaptured_input"]) | ||
yield Result(get_captured_output, uncaptured_print, uncaptured_input) | ||
finally: | ||
set_original_output() | ||
|
||
|
||
@contextmanager | ||
def uncaptured_output(): | ||
original_stdout = sys.stdout | ||
original_stderr = sys.stderr | ||
sys.stdout = _real_stdout | ||
sys.stderr = _real_stderr | ||
try: | ||
yield | ||
finally: | ||
sys.stdout = original_stdout | ||
sys.stderr = original_stderr |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
import os | ||
import importlib | ||
from collections import namedtuple | ||
import copy | ||
import importlib | ||
import json | ||
import logging | ||
import os | ||
from typing import Callable, Optional | ||
from dcicutils.env_base import EnvBase | ||
from dcicutils.env_utils import infer_foursight_from_env | ||
from dcicutils.misc_utils import json_leaf_subst | ||
|
@@ -39,7 +41,7 @@ def __init__(self, foursight_prefix, check_package_name='foursight_core', check_ | |
# which calls back to the locate_check_setup_file function AppUtilsCore here in foursight-core). | ||
if not os.path.exists(check_setup_file): | ||
raise BadCheckSetup(f"Did not locate the specified check setup file: {check_setup_file}") | ||
self.CHECK_SETUP_FILE = check_setup_file # for display/troubleshooting | ||
self.CHECK_SETUP_FILE = check_setup_file # for display/troubleshooting | ||
with open(check_setup_file, 'r') as jfile: | ||
self.CHECK_SETUP = json.load(jfile) | ||
logger.debug(f"foursight_core/CheckHandler: Loaded check_setup.json file: {check_setup_file} ...") | ||
|
@@ -374,31 +376,138 @@ def run_check_or_action(self, connection, check_str, check_kwargs): | |
Fetches the check function and runs it (returning whatever it returns) | ||
Return a string for failed results, CheckResult/ActionResult object otherwise. | ||
""" | ||
# make sure parameters are good | ||
error_str = ' '.join(['Info: CHECK:', str(check_str), 'KWARGS:', str(check_kwargs)]) | ||
if len(check_str.strip().split('/')) != 2: | ||
return ' '.join(['ERROR. Check string must be of form module/check_name.', error_str]) | ||
mod_name = check_str.strip().split('/')[0] | ||
check_name = check_str.strip().split('/')[1] | ||
check_method = None | ||
try: | ||
check_method = self._get_check_or_action_function(check_str) | ||
except Exception as e: | ||
return f"ERROR: {str(e)}" | ||
if not isinstance(check_kwargs, dict): | ||
return ' '.join(['ERROR. Check kwargs must be a dict.', error_str]) | ||
check_mod = None | ||
return "ERROR: Check kwargs must be a dictionary: {check_str}" | ||
return check_method(connection, **check_kwargs) | ||
|
||
def _get_check_or_action_function(self, check_or_action_string: str, check_or_action: str = "check") -> Callable: | ||
if len(check_or_action_string.strip().split('/')) != 2: | ||
raise Exception(f"{check_or_action.title()} string must be of form" | ||
"module_name/{check_or_action}_function_name: {check_or_action_string}") | ||
module_name = check_or_action_string.strip().split('/')[0] | ||
function_name = check_or_action_string.strip().split('/')[1] | ||
module = None | ||
for package_name in [self.check_package_name, 'foursight_core']: | ||
try: | ||
check_mod = self.import_check_module(package_name, mod_name) | ||
module = self.import_check_module(package_name, module_name) | ||
except ModuleNotFoundError: | ||
continue | ||
except Exception as e: | ||
raise e | ||
if not check_mod: | ||
return ' '.join(['ERROR. Check module is not valid.', error_str]) | ||
check_method = check_mod.__dict__.get(check_name) | ||
if not check_method: | ||
return ' '.join(['ERROR. Check name is not valid.', error_str]) | ||
if not self.check_method_deco(check_method, self.CHECK_DECO) and \ | ||
not self.check_method_deco(check_method, self.ACTION_DECO): | ||
return ' '.join(['ERROR. Check or action must use a decorator.', error_str]) | ||
return check_method(connection, **check_kwargs) | ||
if not module: | ||
raise Exception(f"Cannot find check module: {module_name}") | ||
function = module.__dict__.get(function_name) | ||
if not function: | ||
raise Exception(f"Cannot find check function: {module_name}/{function_name}") | ||
if not self.check_method_deco(function, self.CHECK_DECO) and \ | ||
not self.check_method_deco(function, self.ACTION_DECO): | ||
raise Exception(f"{check_or_action.title()} function must use" | ||
"@{check_or_action}_function decorator: {module_name}/{function_name}") | ||
return function | ||
|
||
@staticmethod | ||
def get_checks_info(search: str = None) -> list: | ||
checks = [] | ||
registry = Decorators.get_registry() | ||
for item in registry: | ||
info = CheckHandler._create_check_or_action_info(registry[item]) | ||
if search and search not in info.qualified_name.lower(): | ||
continue | ||
if info.is_check: | ||
checks.append(info) | ||
return sorted(checks, key=lambda item: item.qualified_name) | ||
|
||
@staticmethod | ||
def get_actions_info(search: str = None) -> list: | ||
actions = [] | ||
registry = Decorators.get_registry() | ||
for item in registry: | ||
info = CheckHandler._create_check_or_action_info(registry[item]) | ||
if search and search not in info.qualified_name.lower(): | ||
continue | ||
if info.is_action: | ||
actions.append(info) | ||
return sorted(actions, key=lambda item: item.qualified_name) | ||
|
||
@staticmethod | ||
def get_check_info(check_function_name: str, check_module_name: str = None) -> Optional[namedtuple]: | ||
return CheckHandler._get_check_or_action_info(check_function_name, check_module_name, "check") | ||
|
||
@staticmethod | ||
def get_action_info(action_function_name: str, action_module_name: str = None) -> Optional[namedtuple]: | ||
return CheckHandler._get_check_or_action_info(action_function_name, action_module_name, "action") | ||
|
||
@staticmethod | ||
def _get_check_or_action_info(function_name: str, | ||
module_name: str = None, kind: str = None) -> Optional[namedtuple]: | ||
|
||
function_name = function_name.strip(); | ||
if module_name: | ||
module_name = module_name.strip(); | ||
if not module_name: | ||
if len(function_name.split("/")) == 2: | ||
module_name = function_name.split("/")[0].strip() | ||
function_name = function_name.split("/")[1].strip() | ||
elif len(function_name.split(".")) == 2: | ||
module_name = function_name.split(".")[0].strip() | ||
function_name = function_name.split(".")[1].strip() | ||
registry = Decorators.get_registry() | ||
for name in registry: | ||
if not kind or registry[name]["kind"] == kind: | ||
item = registry[name] | ||
if item["name"] == function_name: | ||
if not module_name: | ||
return CheckHandler._create_check_or_action_info(item) | ||
if item["module"].endswith("." + module_name): | ||
return CheckHandler._create_check_or_action_info(item) | ||
|
||
@staticmethod | ||
def _create_check_or_action_info(info: dict) -> Optional[namedtuple]: | ||
|
||
def unqualified_module_name(module_name: str) -> str: | ||
return module_name.rsplit(".", 1)[-1] if "." in module_name else module_name | ||
|
||
def qualified_check_or_action_name(check_or_action_name: str, module_name: str) -> str: | ||
unqualified_module = unqualified_module_name(module_name) | ||
return f"{unqualified_module}/{check_or_action_name}" if unqualified_module else check_or_action_name | ||
|
||
Info = namedtuple("CheckInfo", ["kind", | ||
"is_check", | ||
"is_action", | ||
"name", | ||
"qualified_name", | ||
"file", | ||
"line", | ||
"module", | ||
"unqualified_module", | ||
"package", | ||
"github_url", | ||
"args", | ||
"kwargs", | ||
"function", | ||
"associated_action", | ||
"associated_check"]) | ||
return Info(info["kind"], | ||
info["kind"] == "check", | ||
info["kind"] == "action", | ||
info["name"], | ||
qualified_check_or_action_name(info["name"], info["module"]), | ||
info["file"], | ||
info["line"], | ||
info["module"], | ||
unqualified_module_name(info["module"]), | ||
info["package"], | ||
info["github_url"], | ||
info["args"], | ||
info["kwargs"], | ||
info["function"], | ||
info.get("action"), | ||
info.get("check")) | ||
|
||
Comment on lines
+479
to
+510
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does this really serve any purpose if you're not going to use outside of this function? You might consider moving this somewhere it can be used more widely ie: helper function that builds. |
||
def init_check_or_action_res(self, connection, check): | ||
""" | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like a good candidate for utils