Simple Metrics

[kamicut]

We would like to create a simple metrics page:

• Aggregation of SQL statements
• Write out to CSV or JSON
• cronnable / uploadable to s3 keyed by date
• capture the state on s3 with a "latest file" URL
• Example metrics: number of users, number of each entity (nodes, ways, relations)
• Docker file so that script can run as a command within osm-seed
• Nice to have: connect a jupyter style notebook to the CSV

If we want to graph things "over time", we will need to aggregate the CSV "snapshots" of the SQL across each metric across a time period: we'll need some kind of materialized view into those CSVs so that it's efficient (instead of downloading all the CSVs).

Question / nice to have: how do we backfill the CSV snapshots?

cc @geohacker @batpad

[danrademacher]

Is this underway, @kamicut ?

[kamicut]

This work is currently under way @danrademacher. I'd like to share a couple of the simple metrics that I've experimented with by querying the database dump:

Nodes/Ways/Relations over time (cumulative)

Nodes over time

NB: I've removed the entity counts from the initial import so that the charts show a sense of progression otherwise the charts become unreadable. This is because the node size of the initial import is really big compared to subsequent changes.

Other metrics / next steps

This type of chart can be done similarly for users over time.

Other metrics that I'd like to explore:

1. Top 10 active users
2. Changeset comments / word frequency
3. Most active areas using changeset boundaries (over time)

My immediate next steps are to create a script that will generate the data for these charts that we could potentially schedule.