[WIP] Port R docs vignettes to epidatpy

[rzats]

Original vignettes can be found here: https://github.com/cmu-delphi/epidatr/tree/dev/vignettes

[rzats]

What other visualization would we like?

[dshemetov]

This looks pretty good, but let's change this to a plot of multiple lines, one for each of pa, ca, and fl. Hopefully the scale is comparable (otherwise we might need to choose a normalized signal).

[dshemetov]

Also in the versioned_data.rst, it would be nice to have a plot that demonstrates data revisions where a time series as_of an older versioned is plotted against a more up to date snapshot. There are some plots that demonstrate this in epipredict vignettes (though they do it in the context of forecasting). We don't need to do anything quite so fancy, you just might be able to use the signals used there to generate a simpler plot. Just one signal but at three different as_of_snapshots (maybe a week apart?) and a choice of date and location so that the revision is visible would be good enough.

[rzats]

Added both of these :)

[rzats]

Without this change the second query of docs_smoke_test fails, because in the dataset there's a sample size of O (the letter O, not zero).

[dshemetov]

😱

Could you isolate that row, get a reproducible API request, and make an issue about it in delphi-epidata? We definitely need to fix it in the database (and where-ever in acquisition that that O got introduced).

[rzats]

Ah, turns out I was dealing with a slightly misleading error message :) This column is supposed to have a float type, as documented here - that was the actual issue.

[dshemetov]

Thanks Rostyslav, this looks great. I made a few requests to fix a few examples, but overall this is excellent. We might just need to let sample_size be type text for now (wow). A better getting_started plot would be good (request in a comment) and a simple plot for revisions would be good (request in a comment). Everything else is looking great though.

A few more things:

• please remove the old unfinished docs: signals_covid.rst and covidcast_examples.rst
• move the epidatpy reference section below the intro guides in the left bar
• the signal-discovery page in epidatr has a table of available endpoints that produces a table like what is now in getting_started.rst... I would like to combine those; could you write a simple available_endpoints() function that pretty prints a table similar to what is there in R and use that function to produce a table in signal_discovery.rst?
• let's move the information about epiweeks and dates from getting_started.rst to the bottom of getting_started_with_epidatpy.rst
• after that we can remove getting_started.rst

[dshemetov]

This looks pretty good, but let's change this to a plot of multiple lines, one for each of pa, ca, and fl. Hopefully the scale is comparable (otherwise we might need to choose a normalized signal).

[dshemetov]

fix: could you update the columns listed? They don't match the dataframe above.

[dshemetov]

Suggested change

	- ``name`` - Name of signal.
	- ``name`` - Human-readable signal name.

[dshemetov]

Also in the versioned_data.rst, it would be nice to have a plot that demonstrates data revisions where a time series as_of an older versioned is plotted against a more up to date snapshot. There are some plots that demonstrate this in epipredict vignettes (though they do it in the context of forecasting). We don't need to do anything quite so fancy, you just might be able to use the signals used there to generate a simpler plot. Just one signal but at three different as_of_snapshots (maybe a week apart?) and a choice of date and location so that the revision is visible would be good enough.

[dshemetov]

😱

Could you isolate that row, get a reproducible API request, and make an issue about it in delphi-epidata? We definitely need to fix it in the database (and where-ever in acquisition that that O got introduced).

[rzats]

@dshemetov finished updating this, please take a look again!

One thing that's still a WIP:

the signal-discovery page in epidatr has a table of available endpoints that produces a table like what is now in getting_started.rst... I would like to combine those; could you write a simple available_endpoints() function that pretty prints a table similar to what is there in R and use that function to produce a table in signal_discovery.rst?

Here I used a somewhat similar approach to R, but a bit more complex as Python doesn't have a nice native "table" format.

My solution instead is to:

• grab all methods that start with pub_ or pvt_ from AEpiDataEndpoints;
• get their docstrings;
• combine the methods and the docstrings into a CSV file. I updated make doc so this file is regenerated before the docs;
• display this CSV file as a pretty-printed table in the signal discovery doc.

Let me know if this approach is OK, and I can clean up the docstrings themselves. Currently they have a couple of missing values and inconsistent formatting, and don't specify links to the original endpoints either.

[dshemetov]

From a comment above: maybe a lot of the work here can be simplified by using Jupyter notebooks directly using https://github.com/spatialaudio/nbsphinx/. Pandas offers quite a lot of configuration options for how to print its tables, see here, maybe you could use that and available_endpoints()?

[dshemetov]

This looks great, thanks for setting this all up Rostyslav! I took another pass through all the documentation, made sure all the language flows, and that we we covered all the major features (like caching and signal discovery).