Merge pull request #63 from cityofaustin/ch-update-logging

Update issues to Socrata logging

README.md

@@ -3,20 +3,26 @@ A bot that creates github issues from our Knack-based intake form.
 
 ## Get it going
 
-1. Configure these environmental variables, which you can grab from Airflow:
+1. Configure these environmental variables, which you can grab from 1Password:
 - KNACK_DTS_PORTAL_SERVICE_BOT_USERNAME
 - KNACK_DTS_PORTAL_SERVICE_BOT_PASSWORD
 - KNACK_API_KEY
 - KNACK_APP_ID
 - GITHUB_ACCESS_TOKEN
+- ZENHUB_ACCESS_TOKEN
+- SOCRATA_ENDPOINT
+- SOCRATA_API_KEY_ID
+- SOCRATA_API_KEY_SECRET
+- SOCRATA_APP_TOKEN
+- SOCRATA_RESOURCE_ID (of the Socrata dataset for issues)
 
 2. Pull the docker image (`atddocker/atd-service-bot`) or install the package dependencies: `pip install -r requirements.txt`
 
 3. Run `python intake.py`
 
 ## How it works
 
-The bot runs on Airflow and fetches new service requests from our Knack app. It geneterates a github issue and applies labels and assignees based on the definitions in `config/config.py`. With the github issue successfully created, the bot submits an "edit record" form in Knack, which sets the record's `github_transmission_status` to `sent`. The form submit also triggers email notifications to the requester and to our staff.
+The bot runs on Airflow and fetches new service requests from our Knack app. It generates a github issue and applies labels and assignees based on the definitions in `config/config.py`. With the github issue successfully created, the bot submits an "edit record" form in Knack, which sets the record's `github_transmission_status` to `sent`. The form submit also triggers email notifications to the requester and to our staff.
 
 ## How not to break the bot
 

issues_to_socrata.py

@@ -124,7 +124,7 @@ def main():
     logging.info("Converting timestamps...")
     convert_timestamps(issues)
 
-    logging.info("Fetching zenhub data...")
+    logging.info("Fetching Zenhub data...")
     zenhub_metadata = get_zenhub_metadata(WORKSPACE_ID, ZENHUB_ACCESS_TOKEN, REPO["id"])
     zenhub_metadata_index = create_zenhub_metadata_index(zenhub_metadata)
 
@@ -149,13 +149,16 @@ def main():
 
     logging.info(f"Uploading to Socrata...")
     first_chunk = True
+    count_processed = 0
     for chunk in chunks(issues, 1000):
         if first_chunk:
             # completely replace dataset to ensure deleted issues are flushed
             client.replace(SOCRATA_RESOURCE_ID, issues)
             first_chunk = False
         client.upsert(SOCRATA_RESOURCE_ID, issues)
-        logging.info(f"{len(chunk)} processed")
+        count_processed += len(chunk)
+        logging.info(f"{count_processed} processed of {len(issues)}")
+    logging.info(f"Done uploading issues to Socrata")
 
 
 if __name__ == "__main__":