Datasette 0.18: units

This release introduces support for units, contributed by Russ Garrett (#203). You can now optionally specify the units for specific columns using metadata.json. Once specified, units will be displayed in the HTML view of your table. They also become available for use in filters - if a column is configured with a unit of distance, you can request all rows where that column is less than 50 meters or more than 20 feet for example.

• Link foreign keys which don't have labels. [Russ Garrett]

  This renders unlabeled FKs as simple links.

  Also includes bonus fixes for two minor issues:

  • In foreign key link hrefs the primary key was escaped using HTML escaping rather than URL escaping. This broke some non-integer PKs.
  • Print tracebacks to console when handling 500 errors.

• Fix SQLite error when loading rows with no incoming FKs. [Russ Garrett]

  This fixes ERROR: conn=<sqlite3.Connection object at 0x10bbb9f10>, sql = 'select ', params = {'id': '1'} caused by an invalid query when loading incoming FKs.

  The error was ignored due to async but it still got printed to the console.

• Allow custom units to be registered with Pint. [Russ Garrett]

• Support units in filters. [Russ Garrett]

• Tidy up units support. [Russ Garrett]

  • Add units to exported JSON
  • Units key in metadata skeleton
  • Docs

• Initial units support. [Russ Garrett]

  Add support for specifying units for a column in metadata.json and rendering them on display using pint

Datasette 0.16: sort on mobile, better error handling

• Better mechanism for handling errors; 404s for missing table/database

  New error mechanism closes #193

  404s for missing tables/databases closes #184

• long_description in markdown for the new PyPI

• Hide Spatialite system tables. [Russ Garrett]

• Allow explain select / explain query plan select #201

• Datasette inspect now finds primary_keys #195

• Ability to sort using form fields (for mobile portrait mode) #199

  We now display sort options as a select box plus a descending checkbox, which means you can apply sort orders even in portrait mode on a mobile phone where the column headers are hidden.

Datasette 0.15: sort by column

The biggest new feature in this release is the ability to sort by column. On the table page the column headers can now be clicked to apply sort (or descending sort), or you can specify ?_sort=column or ?_sort_desc=column directly in the URL.

You can try this feature out on this fivethirtyeight data about the ages of different US politicians.

• table_rows => table_rows_count, filtered_table_rows => filtered_table_rows_count

  Renamed properties. Closes #194

• New sortable_columns option in metadata.json to control sort options.

  You can now explicitly set which columns in a table can be used for sorting using the _sort and _sort_desc arguments using metadata.json:

  {
      "databases": {
          "database1": {
              "tables": {
                  "example_table": {
                      "sortable_columns": [
                          "height",
                          "weight"
                      ]
                  }
              }
          }
      }
  }
  

  Refs #189

• Column headers now link to sort/desc sort - refs #189

• _sort and _sort_desc parameters for table views

  Allows for paginated sorted results based on a specified column.

  Refs #189

• Total row count now correct even if _next applied

• Use .custom_sql() for _group_count implementation (refs #150)

• Make HTML title more readable in query template (#180) [Ryan Pitts]

• New ?_shape=objects/object/lists param for JSON API (#192)

  New _shape= parameter replacing old .jsono extension

  Now instead of this:

  /database/table.jsono
  

  We use the _shape parameter like this:

  /database/table.json?_shape=objects
  

  Also introduced a new _shape called object which looks like this:

  /database/table.json?_shape=object
  

  Returning an object for the rows key:

  ...
  "rows": {
      "pk1": {
          ...
      },
      "pk2": {
          ...
      }
  }
  

  Refs #122

• Utility for writing test database fixtures to a .db file

  python tests/fixtures.py /tmp/hello.db

  This is useful for making a SQLite database of the test fixtures for interactive exploration.

• Compound primary key _next= now plays well with extra filters

  Closes #190

• Fixed bug with keyset pagination over compound primary keys

  Refs #190

• Database/Table views inherit source/license/source_url/license_url metadata

  If you set the source_url/license_url/source/license fields in your root metadata those values will now be inherited all the way down to the database and table templates.

  The title/description are NOT inherited.

  Also added unit tests for the HTML generated by the metadata.

  Refs #185

• Add metadata, if it exists, to heroku temp dir (#178) [Tony Hirst]

• Initial documentation for pagination

• Broke up test_app into test_api and test_html

• Fixed bug with .json path regular expression

  I had a table called geojson and it caused an exception because the regex was matching .json and not \.json

• Deploy to Heroku with Python 3.6.3

Datasette 0.14: customization edition

The theme of this release is customization: Datasette now allows every aspect of its presentation to be customized either using additional CSS or by providing entirely new templates.

Datasette's metadata.json format has also been expanded, to allow per-database and per-table metadata. A new datasette skeleton command can be used to generate a skeleton JSON file ready to be filled in with per-database and per-table details.

The metadata.json file can also be used to define canned queries, as a more powerful alternative to SQL views.

• extra_css_urls/extra_js_urls in metadata

  A mechanism in the metadata.json format for adding custom CSS and JS urls.

  Create a metadata.json file that looks like this:

  {
      "extra_css_urls": [
          "https://simonwillison.net/static/css/all.bf8cd891642c.css"
      ],
      "extra_js_urls": [
          "https://code.jquery.com/jquery-3.2.1.slim.min.js"
      ]
  }
  

  Then start datasette like this:

  datasette mydb.db --metadata=metadata.json
  

  The CSS and JavaScript files will be linked in the <head> of every page.

  You can also specify a SRI (subresource integrity hash) for these assets:

  {
      "extra_css_urls": [
          {
              "url": "https://simonwillison.net/static/css/all.bf8cd891642c.css",
              "sri": "sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI"
          }
      ],
      "extra_js_urls": [
          {
              "url": "https://code.jquery.com/jquery-3.2.1.slim.min.js",
              "sri": "sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g="
          }
      ]
  }
  

  Modern browsers will only execute the stylesheet or JavaScript if the SRI hash matches the content served. You can generate hashes using https://www.srihash.org/

• Auto-link column values that look like URLs (#153)

• CSS styling hooks as classes on the body (#153)

  Every template now gets CSS classes in the body designed to support custom styling.

  The index template (the top level page at /) gets this:

  <body class="index">
  

  The database template (/dbname/) gets this:

  <body class="db db-dbname">
  

  The table template (/dbname/tablename) gets:

  <body class="table db-dbname table-tablename">
  

  The row template (/dbname/tablename/rowid) gets:

  <body class="row db-dbname table-tablename">
  

  The db-x and table-x classes use the database or table names themselves IF they are valid CSS identifiers. If they aren't, we strip any invalid characters out and append a 6 character md5 digest of the original name, in order to ensure that multiple tables which resolve to the same stripped character version still have different CSS classes.

  Some examples (extracted from the unit tests):

  "simple" => "simple"
  "MixedCase" => "MixedCase"
  "-no-leading-hyphens" => "no-leading-hyphens-65bea6"
  "_no-leading-underscores" => "no-leading-underscores-b921bc"
  "no spaces" => "no-spaces-7088d7"
  "-" => "336d5e"
  "no $ characters" => "no--characters-59e024"
  

• datasette --template-dir=mytemplates/ argument

  You can now pass an additional argument specifying a directory to look for custom templates in.

  Datasette will fall back on the default templates if a template is not found in that directory.

• Ability to over-ride templates for individual tables/databases.

  It is now possible to over-ride templates on a per-database / per-row or per-table basis.

  When you access e.g. /mydatabase/mytable Datasette will look for the following:

  - table-mydatabase-mytable.html
  - table.html
  

  If you provided a --template-dir argument to datasette serve it will look in that directory first.

  The lookup rules are as follows:

  Index page (/):
      index.html
  
  Database page (/mydatabase):
      database-mydatabase.html
      database.html
  
  Table page (/mydatabase/mytable):
      table-mydatabase-mytable.html
      table.html
  
  Row page (/mydatabase/mytable/id):
      row-mydatabase-mytable.html
      row.html
  

  If a table name has spaces or other unexpected characters in it, the template filename will follow the same rules as our custom <body> CSS classes - for example, a table called "Food Trucks" will attempt to load the following templates:

  table-mydatabase-Food-Trucks-399138.html
  table.html
  

  It is possible to extend the default templates using Jinja template inheritance. If you want to customize EVERY row template with some additional content you can do so by creating a row.html template like this:

  {% extends "default:row.html" %}
  
  {% block content %}
  <h1>EXTRA HTML AT THE TOP OF THE CONTENT BLOCK</h1>
  <p>This line renders the original block:</p>
  {{ super() }}
  {% endblock %}
  

• --static option for datasette serve (#160)

  You can now tell Datasette to serve static files from a specific location at a specific mountpoint.

  For example:

  datasette serve mydb.db --static extra-css:/tmp/static/css
  

  Now if you visit this URL:

  http://localhost:8001/extra-css/blah.css
  

  The following file will be served:

  /tmp/static/css/blah.css
  

• Canned query support.

  Named canned queries can now be defined in metadata.json like this:

  {
      "databases": {
          "timezones": {
              "queries": {
                  "timezone_for_point": "select tzid from timezones ..."
              }
          }
      }
  }
  

  These will be shown in a new "Queries" section beneath "Views" on the database page.

• New datasette skeleton command for generating metadata.json (#164)

• metadata.json support for per-table/per-database metadata (#165)

  Also added support for descriptions and HTML descriptions.

  Here's an example metadata.json file illustrating custom per-database and per-table metadata:

  {
      "title": "Overall datasette title",
      "description_html": "This is a <em>description with HTML</em>.",
      "databases": {
          "db1": {
              "title": "First database",
              "description": "This is a string description & has no HTML",
              "license_url": "http://example.com/",
          "license": "The example license",
              "queries": {
                "canned_query": "select * from table1 limit 3;"
              },
              "tables": {
                  "table1": {
                      "title": "Custom title for table1",
                      "description": "Tables can have descriptions too",
                      "source": "This has a custom source",
                      "source_url": "http://example.com/"
                  }
              }
          }
      }
  }
  

• Renamed datasette build command to datasette inspect (#130)

• Upgrade to Sanic 0.7.0 (#168)

  https://github.com/channelcat/sanic/releases/tag/0.7.0

• Package and publish commands now accept --static and --template-dir

  Example usage:

  datasette package --static css:extra-css/ --static js:extra-js/ \
    sf-trees.db --template-dir templates/ --tag sf-trees --branch master
  

  This creates a local Docker image that includes copies of the templates/, extra-css/ and extra-js/ directories. You can then run it like this:

  docker run -p 8001:8001 sf-trees
  

  For publishing to Zeit now:

  datasette publish now --static css:extra-css/ --static js:extra-js/ \
    sf-trees.db --template-dir templates/ --name sf-trees --branch master
  

• HTML comment showing which templates were considered for a page (#171)

Datasette 0.13: foreign key, search and filters

0.13 (2017-11-24)

• Search now applies to current filters.

  Combined search into the same form as filters.

  Closes #133

• Much tidier design for table view header.

  Closes #147

• Added ?column__not=blah filter.

  Closes #148

• Row page now resolves foreign keys.

  Closes #132

• Further tweaks to select/input filter styling.

  Refs #86 - thanks for the help, @natbat!

• Show linked foreign key in table cells.

• Added UI for editing table filters.

  Refs #86

• Hide FTS-created tables on index pages.

  Closes #129

• Add publish to heroku support [Jacob Kaplan-Moss]

  datasette publish heroku mydb.db

  Pull request #104

• Initial implementation of ?_group_count=column.

  URL shortcut for counting rows grouped by one or more columns.

  ?_group_count=column1&_group_count=column2 works as well.

  SQL generated looks like this:

  select "qSpecies", count(*) as "count"
  from Street_Tree_List
  group by "qSpecies"
  order by "count" desc limit 100
  

  Or for two columns like this:

  select "qSpecies", "qSiteInfo", count(*) as "count"
  from Street_Tree_List
  group by "qSpecies", "qSiteInfo"
  order by "count" desc limit 100
  

  Refs #44

• Added --build=master option to datasette publish and package.

  The datasette publish and datasette package commands both now
  accept an optional --build argument. If provided, this can be used
  to specify a branch published to GitHub that should be built into
  the container.

  This makes it easier to test code that has not yet been officially
  released to PyPI, e.g.:

  datasette publish now mydb.db --branch=master
  

• Implemented ?_search=XXX + UI if a FTS table is detected.

  Closes #131

• Added datasette --version support.

• Table views now show expanded foreign key references, if possible.

  If a table has foreign key columns, and those foreign key tables
  have label_columns, the TableView will now query those other
  tables for the corresponding values and display those values as
  links in the corresponding table cells.

  label_columns are currently detected by the inspect() function,
  which looks for any table that has just two columns - an ID column
  and one other - and sets the label_column to be that second non-ID
  column.

• Don't prevent tabbing to "Run SQL" button (#117) [Robert
  Gieseke]

  See comment in #115

• Add keyboard shortcut to execute SQL query (#115) [Robert
  Gieseke]

• Allow --load-extension to be set via environment variable.

• Add support for ?field__isnull=1 (#107) [Ray N]

• Add spatialite, switch to debian and local build (#114) [Ariel
  Núñez]

• Added --load-extension argument to datasette serve.

  Allows loading of SQLite extensions. Refs #110.

Datasette 0.12

• Added __version__, now displayed as tooltip in page footer
  (#108).

• Added initial docs, including a changelog (#99).

• Turned on auto-escaping in Jinja.

• Added a UI for editing named parameters (#96).

  You can now construct a custom SQL statement using SQLite named parameters (e.g. :name) and datasette will display form fields for editing those parameters. Here's an example which lets you see the most popular names for dogs of different species registered through various dog registration schemes in Australia.

• Pin to specific Jinja version. (#100).

• Default to 127.0.0.1 not 0.0.0.0. (#98).

• Added extra metadata options to publish and package commands.
  (#92).

  You can now run these commands like so:

  datasette now publish mydb.db \
      --title="My Title" \
      --source="Source" \
      --source_url="http://www.example.com/" \
      --license="CC0" \
      --license_url="https://creativecommons.org/publicdomain/zero/1.0/"
  

  This will write those values into the metadata.json that is packaged
  with the app. If you also pass --metadata=metadata.json that file
  will be updated with the extra values before being written into the
  Docker image.

• Added simple production-ready Dockerfile (#94) [Andrew Cutler]

• New ?_sql_time_limit_ms=10 argument to database and table page
  (#95)

• SQL syntax highlighting with Codemirror (#89) [Tom Dyson]