[Issue #1578] Setup adding task information to all task logs

[chouinar]

Summary

Fixes #1578

Time to review: 10 mins

Changes proposed

Add a decorator method you can add to an ECS task entry function that automatically logs basic info about the environment, app, and AWS setup

Context for reviewers

This makes it so you can setup the app (running as an API or standalone script for an ECS task) to automatically include some general log info. In this case, the app name, environment, and the task name. Once we're running more consistently on ECS, it should also include some AWS info, although that needs to be tested still.

When we get proper log ingestion into New Relic, this will make searching for things much easier (eg. filter to all logs from a specific job type, or specific run) which can greatly aid in debugging.

Additional information

For example, using the export-opportunity-data task as an example, the logs change from:

➜  api git:(main) poetry run flask task export-opportunity-data
15:38:00.537  src.logging.config                   log_program_info             INFO     start src: CPython 3.12.0 Darwin, hostname Michaels-MacBook-Pro-2.local, pid 46294, user 503(michaelchouinard) hostname=Michaels-MacBook-Pro-2.local cpu_count=8 cpu_usable=unknown
15:38:00.537  src.logging.config                   log_program_info             INFO     invoked as: /Users/michaelchouinard/workspace/grants-equity/api/.venv/bin/flask task export-opportunity-data
15:38:00.537  root                                 init_app                     INFO     initialized flask logger
15:38:00.537  src.adapters.db.clients.postgres_config get_db_config                INFO     Constructed database configuration                 host=localhost dbname=app username=app password=*** port=5432 hide_sql_parameter_logs=True
15:38:00.560  src.adapters.db.clients.postgres_client check_db_connection          INFO     connected to postgres db                           dbname=app user=app host=localhost port=5432 options= dsn_parameters=user=app connect_timeout=10 dbname=app host=localhost hostaddr=::1 sslmode=allow sslcertmode=allow protocol_version=3 server_version=150007
15:38:00.560  src.adapters.db.clients.postgres_client verify_ssl                   WARNING  database connection is not using SSL
15:38:00.560  src.api.feature_flags.feature_flag_config initialize                   INFO     Constructed feature flag configuration             enable_opportunity_log_msg=False
15:38:00.562  src.adapters.search.opensearch_config get_opensearch_config        INFO     Constructed opensearch configuration               host=localhost port=9200 use_ssl=False verify_certs=False connection_pool_size=10
15:38:00.585  src.task.task                        run                          INFO     Starting ExportOpportunityDataTask                 app.name=src.app
15:38:00.657  src.task.opportunities.export_opportunity_data_task export_data_to_json          INFO     Creating Opportunity JSON extract                  json_extract_path=/tmp/opportunity_data-2024-09-23_15-38-00.json app.name=src.app
15:38:00.658  src.task.opportunities.export_opportunity_data_task export_opportunities_to_csv  INFO     Creating Opportunity CSV extract                   csv_extract_path=/tmp/opportunity_data-2024-09-23_15-38-00.csv app.name=src.app
15:38:00.660  src.task.task                        run                          INFO     Completed ExportOpportunityDataTask in 0.075 seconds csv_file=/tmp/opportunity_data-2024-09-23_15-38-00.csv json_file=/tmp/opportunity_data-2024-09-23_15-38-00.json records_exported=30 task_duration_sec=0.075 app.name=src.app

to

➜  api git:(chouinar/setup-ecs-bg-task) poetry run flask task export-opportunity-data
15:37:48.106  src.logging.config                   log_program_info             INFO     start src: CPython 3.12.0 Darwin, hostname Michaels-MacBook-Pro-2.local, pid 45829, user 503(michaelchouinard) hostname=Michaels-MacBook-Pro-2.local cpu_count=8 cpu_usable=unknown
15:37:48.107  src.logging.config                   log_program_info             INFO     invoked as: /Users/michaelchouinard/workspace/grants-equity/api/.venv/bin/flask task export-opportunity-data
15:37:48.107  root                                 init_app                     INFO     initialized flask logger                           app.name=src.app environment=local
15:37:48.107  src.adapters.db.clients.postgres_config get_db_config                INFO     Constructed database configuration                 host=localhost dbname=app username=app password=*** port=5432 hide_sql_parameter_logs=True app.name=src.app environment=local
15:37:48.147  src.adapters.db.clients.postgres_client check_db_connection          INFO     connected to postgres db                           dbname=app user=app host=localhost port=5432 options= dsn_parameters=user=app connect_timeout=10 dbname=app host=localhost hostaddr=::1 sslmode=allow sslcertmode=allow protocol_version=3 server_version=150007 app.name=src.app environment=local
15:37:48.147  src.adapters.db.clients.postgres_client verify_ssl                   WARNING  database connection is not using SSL               app.name=src.app environment=local
15:37:48.147  src.api.feature_flags.feature_flag_config initialize                   INFO     Constructed feature flag configuration             enable_opportunity_log_msg=False app.name=src.app environment=local
15:37:48.150  src.adapters.search.opensearch_config get_opensearch_config        INFO     Constructed opensearch configuration               host=localhost port=9200 use_ssl=False verify_certs=False connection_pool_size=10 app.name=src.app environment=local
15:37:48.150  src.task.ecs_background_task         _get_ecs_metadata            INFO     ECS metadata not available for local environments. Run this task on ECS to see metadata. app.name=src.app environment=local task_name=export-opportunity-data task_uuid=15517044-e28d-4e01-98b4-2fc90ffa813a
15:37:48.150  src.task.ecs_background_task         _ecs_background_task_impl    INFO     Starting ECS task export-opportunity-data          app.name=src.app environment=local task_name=export-opportunity-data task_uuid=15517044-e28d-4e01-98b4-2fc90ffa813a
15:37:48.176  src.task.task                        run                          INFO     Starting ExportOpportunityDataTask                 app.name=src.app environment=local task_name=export-opportunity-data task_uuid=15517044-e28d-4e01-98b4-2fc90ffa813a
15:37:48.241  src.task.opportunities.export_opportunity_data_task export_data_to_json          INFO     Creating Opportunity JSON extract                  json_extract_path=/tmp/opportunity_data-2024-09-23_15-37-48.json app.name=src.app environment=local task_name=export-opportunity-data task_uuid=15517044-e28d-4e01-98b4-2fc90ffa813a
15:37:48.243  src.task.opportunities.export_opportunity_data_task export_opportunities_to_csv  INFO     Creating Opportunity CSV extract                   csv_extract_path=/tmp/opportunity_data-2024-09-23_15-37-48.csv app.name=src.app environment=local task_name=export-opportunity-data task_uuid=15517044-e28d-4e01-98b4-2fc90ffa813a
15:37:48.244  src.task.task                        run                          INFO     Completed ExportOpportunityDataTask in 0.069 seconds csv_file=/tmp/opportunity_data-2024-09-23_15-37-48.csv json_file=/tmp/opportunity_data-2024-09-23_15-37-48.json records_exported=30 task_duration_sec=0.069 app.name=src.app environment=local task_name=export-opportunity-data task_uuid=15517044-e28d-4e01-98b4-2fc90ffa813a
15:37:48.244  src.task.ecs_background_task         _ecs_background_task_impl    INFO     Completed ECS task export-opportunity-data         ecs_task_duration_sec=0.094 status=success app.name=src.app environment=local task_name=export-opportunity-data task_uuid=15517044-e28d-4e01-98b4-2fc90ffa813a