Tickets/dm 46151 #20
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,271 @@ | ||
| { | ||
|
There was a problem hiding this comment. Line #1. # Paste contents of source_adapters.py in new cell below this one. Are you planning on having a different PR to implement that so it works in Times Square, or should we do this now? Reply via ReviewNB There was a problem hiding this comment. Argg, it should be done now! Thanks!! There was a problem hiding this comment. Line #5. recs = service_adapter.get_messages(limit=limit, I think we should change both instances of Reply via ReviewNB There was a problem hiding this comment. Agreed. Its a even worse than the obvious because exposure has that we might want to show for both "messages" and "exposures". There was a problem hiding this comment. Line #55. self.ignore_fields = list() I was curious about whether or not to use https://stackoverflow.com/questions/5790860/whats-the-difference-between-and-vs-list-and-dict Reply via ReviewNB There was a problem hiding this comment. I find it hard to read (see) the [] method. I think its the list() method makes it more obvious that my intent is to create a list that is always empty at that point. I think its more readable. I don't think performance of creating an empty list matters (if its even true). Your link talks more about time of dict(). There was a problem hiding this comment. Line #64. to = (timeout or self.timeout) I suggest Reply via ReviewNB There was a problem hiding this comment. Line #67. url_http_status_code = dict() see comment about Reply via ReviewNB There was a problem hiding this comment. Line #71. r = requests.get(url, timeout=(timeout or self.timeout)) use what is set in #64 Reply via ReviewNB |
||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "0", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Night Log" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "1", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Parameters. Set defaults here.\n", | ||
| "# Times Square replaces this cell with the user's parameters.\n", | ||
| "record_limit = '999'\n", | ||
| "day_obs = 'TODAY' # TODAY, YESTERDAY, YYYY-MM-DD\n", | ||
| "number_of_days = '1' # Total number of days of data to display (ending on day_obs)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "2", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Only use packages available in the Rubin Science Platform\n", | ||
| "import requests\n", | ||
| "from collections import defaultdict\n", | ||
| "import pandas as pd\n", | ||
| "from pprint import pp\n", | ||
| "from urllib.parse import urlencode\n", | ||
| "from IPython.display import display, Markdown, display_markdown\n", | ||
| "from matplotlib import pyplot as plt\n", | ||
| "import os\n", | ||
| "from datetime import datetime, date, timedelta" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "3", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "limit = int(record_limit)\n", | ||
| "\n", | ||
| "match day_obs.lower():\n", | ||
| " case 'today':\n", | ||
| " date = datetime.now().date()\n", | ||
| " case 'yesterday':\n", | ||
| " date = datetime.now().date()-timedelta(days=1)\n", | ||
| " case _:\n", | ||
| " date = datetime.strptime(dd, '%Y-%m-%d').date()\n", | ||
| "\n", | ||
| "days = int(number_of_days)\n", | ||
| "\n", | ||
| "# Thus: [min_day_obs,max_day_obs)\n", | ||
| "min_day_obs = (date - timedelta(days=days-1)).strftime('%Y%m%d') # Inclusive\n", | ||
| "max_day_obs = (date + timedelta(days=1)).strftime('%Y%m%d') # prep for Exclusive\n", | ||
| "\n", | ||
| "response_timeout = 3.05 # seconds, how long to wait for connection\n", | ||
| "read_timeout = 20 # seconds\n", | ||
| "timeout = (float(response_timeout), float(read_timeout))\n", | ||
| "\n", | ||
| "summit = 'https://summit-lsp.lsst.codes'\n", | ||
| "usdf = 'https://usdf-rsp-dev.slac.stanford.edu'\n", | ||
| "tucson = 'https://tucson-teststand.lsst.codes'\n", | ||
| "\n", | ||
| "# Use server=tucson for dev testing\n", | ||
| "server = os.environ.get('EXTERNAL_INSTANCE_URL', summit)\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "4", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "print(f'Report from {server} over {number_of_days} nights'\n", | ||
| " f' from {min_day_obs} to {date}. ')" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "5", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# For Times Square, comment out next line and past next cell with contents of local python file.\n", | ||
| "from lsst.ts.logging_and_reporting.source_adapters import ExposurelogAdapter, NarrativelogAdapter\n", | ||
| "# Once our logrep package has been installed in RSP, we can use the simpler \"import\"" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "6", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Paste contents of source_adapters.py in new cell below this one." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "7", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Exposure Log" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "8", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "service_adapter = ExposurelogAdapter(server_url=server)\n", | ||
| "source = f'{server}'\n", | ||
| "try:\n", | ||
| " recs = service_adapter.get_messages(limit=limit,\n", | ||
| " min_day_obs=min_day_obs,\n", | ||
| " max_day_obs=max_day_obs,\n", | ||
| " )\n", | ||
| "except Exception as err:\n", | ||
| " recs = []\n", | ||
| " msg = f'ERROR getting records from {server=}: {err=}'\n", | ||
| " raise Exception(msg)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "9", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "print(f'Retrieved {len(recs)} records')" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "10", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "if recs:\n", | ||
| " new_column_names = dict(message_text='message',\n", | ||
| " date_added='date'\n", | ||
| " )\n", | ||
| " df = pd.DataFrame(recs).rename(columns=new_column_names)\n", | ||
| " user_df = df[['date','message']]\n", | ||
| " \n", | ||
| " display_markdown(f'### Exposure log for {number_of_days} days {min_day_obs} to {max_day_obs}', raw=True)\n", | ||
| " display(user_df)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "11", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "gaps = service_adapter.get_observation_gaps()\n", | ||
| "for instrument, day_gaps in gaps.items():\n", | ||
| " display_markdown(f'### Date vs Observation Gap (minutes) for {instrument=!s}', raw=True)\n", | ||
| " x,y = zip(*day_gaps.items())\n", | ||
| " df = pd.DataFrame(dict(day=x,minutes=y))\n", | ||
| " df.plot.bar(x='day', y='minutes', title=f'{instrument=!s}')" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "12", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Narrative Log\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "13", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "service_adapter = NarrativelogAdapter(server_url=server)\n", | ||
| "try:\n", | ||
| " # date like '2000-01-02 12:00:00'\n", | ||
| " # str(datetime(2000, 1, 2, 12, 0, 0))\n", | ||
| " recs = service_adapter.get_messages(limit=limit,\n", | ||
| " min_date_end=str(datetime.strptime(min_day_obs,'%Y%m%d')),\n", | ||
| " max_date_end=str(datetime.strptime(max_day_obs,'%Y%m%d')),\n", | ||
| " )\n", | ||
| "except Exception as err:\n", | ||
| " recs = []\n", | ||
| " msg = f'ERROR getting records from {server=}: {err=}'\n", | ||
| " raise Exception(msg)\n", | ||
| "\n", | ||
| "print(f'Retrieved {len(recs)} records.')" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "14", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "ignore_fields = ['id']\n", | ||
| "new_column_names = dict(message_text='message',\n", | ||
| " date_added='date'\n", | ||
| " )\n", | ||
| "if recs:\n", | ||
| " df = pd.DataFrame(recs).rename(columns=new_column_names)\n", | ||
| " user_df = df[['date','message']]\n", | ||
| "\n", | ||
| " display(Markdown(f'## Narrative log (Style A) for {number_of_days} days {min_day_obs} to {max_day_obs}'))\n", | ||
| " display(Markdown(\"### <font color='red'>Choose display Style (or offer other suggestion)</font>\"))\n", | ||
| " with pd.option_context('display.max_rows', None, 'display.max_columns', None):\n", | ||
| " display(user_df)\n", | ||
| " \n", | ||
| " display(Markdown(f'## Narrative log (Style B)'))\n", | ||
| " for index,row in user_df.iterrows():\n", | ||
| " print(f\"{datetime.fromisoformat(user_df.iloc[0]['date']).date()}: \"\n", | ||
| " f\"{row['message']}\"\n", | ||
| " )" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "15", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.10.12" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # For use with a Times Square notebook | ||
| title: Night Log | ||
| description: Combined report from Summit logs | ||
| authors: | ||
| - name: Steve Pothier | ||
| slack: Steve Pothier | ||
| - name: Valerie Becker | ||
| slack: valerie becker | ||
| tags: | ||
| - reporting | ||
| - prototype | ||
| - exposure | ||
| parameters: | ||
| record_limit: | ||
| type: integer | ||
| description: Max number of records to output | ||
| default: 99 | ||
| minimum: 1 | ||
| maximum: 9999 | ||
| day_obs: | ||
| type: string | ||
| description: The night to report on. (YYYY-MM-DD, TODAY, YESTERDAY) | ||
| default: "TODAY" | ||
| number_of_days: | ||
| type: integer | ||
| description: Number of days to show (ending in day_obs) | ||
| default: 1 | ||
| minimum: 1 | ||
| maximum: 9 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Line #2. f' from {min_day_obs} to {date}. ')I think for our initial investigating we were allowing many days and different days, etc, but I think when possible/relevant we should shift the focus more towards 'what happened last night' and allow that to be the default assumption of what users would be interested in. yagni
This comment applies to record_limit, number_of_days, and this message
Reply via ReviewNB
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Without number_of_days greater than one, users may find it difficult to see if results look good. If they happen to have a day with no results (happens on usdf-dev frequently) the only remedy would be to try another day and see if that works. Repeat a few times and its frustrating. In this case to pick one day, we have to have a min and max day so we already have to have a number of days concept that finds its way into code. After everyone agrees on the format of displays, we can remove the number_of_days parameter from sidecar yaml file so user will always get the default of 1.