Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Javascript Plugin API (Custom panels, column menu items with JS actions) #2052

Conversation

hydrosquall
Copy link
Contributor

@hydrosquall hydrosquall commented Apr 2, 2023

Motivation

  • Allow plugins that add data visualizations datasette-vega, datasette-leaflet, and datasette-nteract-data-explorer to co-exist safely
  • Standardize APIs / hooks to ease development for new JS plugin developers (better compat with datasette-lite) through standardized DOM selectors, methods for extending the existing Table UI. This has come up as a feature request several times (see research notes for examples)
  • Discussion w/ @simonw about a general-purpose Datasette JS API

Changes

Summary: Provide 2 new surface areas for Datasette JS plugin developers. See alpha documentation

  1. Custom column header items: https://a.cl.ly/Kou97wJr
  2. Basic "panels" controlled by buttons: https://a.cl.ly/rRugWobd

User Facing Changes

  • Allow creating menu items under table header that triggers JS (instead of opening hrefs per the existing menu_link hook). Items can respond to any column metadata provided by the column header (e.g. label). The proof of concept plugins log data to the console, or copy the column name to clipboard.
  • Allow plugins to register UI elements in a panel controller. The parent component handles switching the visibility of active plugins.
    • Because native button elements are used, the panel is keyboard-accessible - use tab / shift-tab to cycle through tab options, and enter to select.
    • There's room to improve the styling, but the focus of this PR is on the API rather than the UX.

(plugin) Developer Facing Changes

  • Dispatch a datasette_init CustomEvent when the datasetteManager is finished loading.
  • Provide manager.registerPlugin API for adding new functionality that coordinates with Datasette lifecycle events.
  • Provide a manager.selectors map of DOM elements that users may want to hook into.
    • Updated table.js to use refer to these to motivating keeping things in sync
  • Allow plugins to register themselves with 2 hooks:
    • makeColumnActions: Add items to menu in table column headers. Users can provide a label, and either href or onClick with full access to the metadata for the clicked column (name, type, misc. booleans)
    • makeAboveTablePanelConfigs: Add items to the panel. Each panel has a unique ID (namespaced within that plugin), a render function, and a string label.

See this file for example plugin usage.

Core Developer Facing Changes

  • Modified table.js to make use of the datasetteManager API.
  • Added example plugins to the demos/plugins folder, and stored the test js in the statics/ folder

Testing

For Datasette plugin developers, please see the alpha-level documentation .

To run the examples:

datasette serve fixtures.db  --plugins-dir=demos/plugins/

Open local server: http://127.0.0.1:8001/fixtures/facetable

Open to all feedback on this PR, from API design to variable naming, to what additional hooks might be useful for the future.

My focus was more on the general shape of the API for developers, rather than on the UX of the test plugins.

Design notes

  • The manager tab panel could be a separate plugin if the implementation is too custom.
  • The makeColumnHeaderItems benefits from hooking into the logic of table.js
  • I wanted to offer this to the Datasette core, since the datasette-manager would be more powerful if it were connected to lifecycle and JS events that are part of the existing table.js.
  • Non-goals:
    • Dependency management (for now) - there's no "build" step, we don't know when new plugins will be added. While there are some valid use cases (for example, allow multiple plugins to wait for a global leaflet object to be loaded), I don't see enough use-cases to justify doing this yet.
    • Enabling single-page-app features - for now, most datasette actions lead to a new page being loaded. SPA development offers some benefits (no page jumping after clicking on a link), but also complexity that doesn't need to be in the core Datasette project.

Research Notes

  • Relocated to a comment, as this isn't required to review when evaluating the plugin. Including it just for those who are curious.

@hydrosquall hydrosquall changed the title feat: Datasette JS Plugin API (Panel Manager + Column menu items with custom actions feat: Javascript Plugin API (Custom panels, column menu items with JS actions) Apr 2, 2023

manager.registerPlugin("column-name-plugin", {
version: 0.1,
getColumnHeaderItems: (columnMeta) => {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd call this getColumnActions() to match with the names of other server-side action plugins: https://docs.datasette.io/en/stable/plugin_hooks.html#table-actions-datasette-actor-database-table-request

@hydrosquall
Copy link
Contributor Author

Notes from 1:1 - it is possible to pass in URL params into a ObservableHQ notebook: https://observablehq.com/@bherbertlc/pass-values-as-url-parameters

@hydrosquall
Copy link
Contributor Author

hydrosquall commented Apr 16, 2023

Javascript Plugin Docs (alpha)

Motivation

The Datasette JS Plugin API allows developers to add interactive features to the UI, without having to modify the Python source code.

Setup

No external/NPM dependencies are needed.

Plugin behavior is coordinated by the Datasette manager. Every page has 1 manager.

There are 2 ways to add your plugin to the manager.

  1. Read window.__DATASETTE__ if the manager was already loaded.
const manager = window.__DATASETTE__;
  1. Wait for the datasette_init event to fire if your code was loaded before the manager is ready.
document.addEventListener("datasette_init", function (evt) {
  const { detail: manager } = evt;
  
  // register plugin here
});
  1. Add plugin to the manager by calling manager.registerPlugin in a JS file. Each plugin will supply 1 or more hooks with
  • unique name (YOUR_PLUGIN_NAME)
  • a numeric version (starting at 0.1),
  • configuration value, the details vary by hook. (In this example, getColumnActions takes a function)
manager.registerPlugin("YOUR_PLUGIN_NAME", {
    version: 0.1,
    makeColumnActions: (columnMeta) => {
      return [
        {
          label: "Copy name to clipboard",
          // evt = native click event
          onClick: (evt) => copyToClipboard(columnMeta.column),
        }
      ];
    },
  });

There are 2 plugin hooks available to manager.registerPlugin:

  • makeColumnActions - Add items to the cog menu for headers on datasette table pages
  • makeAboveTablePanelConfigs - Add items to "tabbed" panel above the <table/> on pages that use the Datasette table template.

While there are additional properties on the manager, but it's not advised to depend on them directly as the shape is subject to change.

  1. To make your JS file available as a Datasette plugin from the Python side, you can add a python file resembling this to your plugins directory. Note that you could host your JS file anywhere, it doesn't have to be served from the Datasette statics folder.

I welcome ideas for more hooks, or feedback on the current design!

Examples

See the example plugins file for additional examples.

Hooks API Guide

makeAboveTablePanelConfigs

Provide a function with a list of panel objects. Each panel object should contain

  1. A unique string id
  2. A string label for the tab
  3. A render function. The first argument is reference to an HTML Element.

Example:

 manager.registerPlugin("panel-plugin-graphs", {
    version: 0.1,
    makeAboveTablePanelConfigs: () => {
      return [
        {
          id: 'first-panel',
          label: "My new panel",
          render: node => {
            const description = document.createElement('p');
            description.innerText = 'Hello world';
            node.appendChild(description);
          }
        }
      ];
    },
  });

makeColumnActions

Provide a function that returns a list of action objects. Each action object has

  1. A string label for the menu dropdown label
  2. An onClick render function.

Example:

  manager.registerPlugin("column-name-plugin", {
    version: 0.1,
    getColumnActions: (columnMeta) => {
      
      // Info about selected column. 
      const { columnName, columnNotNull, columnType, isPk } = columnMeta;

      return [
        {
          label: "Copy name to clipboard",
          onClick: (evt) => copyToClipboard(column),
        }
      ];
    },
  });

The getColumnActions callback has access to an object with metadata about the clicked column. These fields include:

  • columnName: string (name of the column)
  • columnNotNull: boolean
  • columnType: sqlite datatype enum (text, number, etc)
  • isPk: Whether this is the primary key: boolean

You can use this column metadata to customize the action config objects (for example, handling different summaries for text vs number columns).

@hydrosquall
Copy link
Contributor Author

Research notes

Copy link
Collaborator

@asg017 asg017 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey! This is awesome: I really dig the pure vanilla JS plugin system (including the "InitDatasette" custom event + window.__DATASETTE__).

  • Left some very nitty comments, not too important
  • For the plugin methods getColumnActions and getAboveTablePanelConfigs: Maybe they should have more "action-y" names like addColumnActions and addAboveTablePanelConfigs? It seems like the get* prefix is there because they're lifecycle/hook names, but it seems odd to be to have a method called get*() that returns new items
  • I know TypeScript can be a little controversial, but for APIs like this (well defined callbacks and return objects) I find types to be extremely helpful. I think we can distribute an optional flat single-file types.d.ts file like so:
interface ColumnMeta {
  column: string;
  columnNotNull: boolean;
  columnType: string;
  isPk: boolean;
}

interface ColumnAction {
  label: string;
  onClick: (event: MouseEvent) => void;
}

interface AboveTablePanelConfig {
  id: string;
  label: string;
  render: (node: HTMLElement) => void;
}

export interface Plugin {
  version: number;
  getColumnActions(columnMeta: ColumnMeta): ColumnAction[];
  getAboveTablePanelConfigs(): AboveTablePanelConfig[];
}
export interface Manager {
  plugins: Map<String, Plugin>;
  registerPlugin(name: string, plugin: Plugin): void;
}

And then, one could use these types with JSDoc in plain JS files like so:

/**
 * @typedef {import('./types').Manager} Manager
 * @typedef {import('./types').Plugin} Plugin
 */

document.addEventListener("InitDatasette", function (evt) {
  const { detail: manager } = evt;
  addPlugins(manager);
});

/**
 * @param {Manager} manager
 */
function addPlugins(manager) {
  manager.registerPlugin("column-name-plugin", {
    version: 0.1,
    getColumnActions: (columnMeta) => { ... }
  });
}

Then your editor will have type suggestions for manager, columnMeta, etc.

Probably not necessary for this PR, but happy to contribute a types.d.ts when this is finalized!

  • Maybe we can make it easier to do pure-js datasette plugins? For example the example_js_manager_plugins.py file is fairly small that just uses extra_js_urls. Maybe there can be CLI options like:
datasette fixtures.db --plugin-js path/to/table-example-plugins.js`

And then do the PERMITTED_VIEWS filtering in JS rather than Python. Or even autodetect JS files in --plugins-dir/ and server those automatically (though I doubt that's backwards compatible)

@@ -0,0 +1,202 @@
// Custom events for use with the native CustomEvent API
const DATASETTE_EVENTS = {
INIT: "InitDatasette", // returns datasette manager instance in evt.detail
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Maybe "DatasetteInit" instead? Or "datasette_init"? Not sure if there's a common convention for custom event names...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm flexible, I saw a bunch of opinions about this here:

https://stackoverflow.com/questions/19071579/javascript-dom-event-name-convention

It looks like it might be safe to err on the side of all-lowercase. datasette_init or datasette.init both have a reasonable look to them.

plugins: new Map(),

registerPlugin: (name, pluginMetadata) => {
if (datasetteManager.plugins.get(name)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: .has() over .get()

Comment on lines 62 to 65
* - column: string
* - columnNotNull: 0 or 1
* - columnType: sqlite datatype enum (text, number, etc)
* - isPk: 0 or 1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could columnNotNull and isPk be booleans instead? Also, they appear to be strings right now ("0" and "1" rather than 0 and 1, at least according to the "Log column metadata to console" example

Also, maybe column -> columnName? or just name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that makes sense, right now I'm passing through the data unparsed directly from the DOMStringMap.

Probably more ergonomic to use these precise datatypes and more opinionated field names, great suggestions - I'll apply these soon.

*/
renderAboveTablePanel: () => {

const aboveTablePanel = document.querySelector(DOM_SELECTORS.aboveTablePanel);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This returns null on tables that have no rows: ex in fixtures on http://localhost:8001/fixtures/123_starts_with_digits , I get:

caught TypeError: Cannot read properties of null (reading 'querySelector')
    at Object.renderAboveTablePanel (datasette-manager.js:95:50)
    at Object.registerPlugin (datasette-manager.js:51:24)
    at addPlugins (table-example-plugins.js:34:11)
    at HTMLDocument.<anonymous> (table-example-plugins.js:8:3)
    at initializeDatasette (datasette-manager.js:193:12)
    at HTMLDocument.<anonymous> (datasette-manager.js:201:3)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting- in those cases, do you think it still makes sense for the plugin to attempt to display (in which case there should be a fallback mount point, or the mount point should always show up regardless of whether there is data), or should a panel like this only show up for views that have some rows?

I think it's the first case, but wanted to check what you think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decided to always provide a hook so this won't be an issue

06b4829

but also added a bit of JS safety in case the template structure changes in the future.

cf504fe

@hydrosquall
Copy link
Contributor Author

Thanks for the thoughtful review and generous examples @asg017 ! I'll make the changes you suggested soon. Bonus thoughts inlined below.

comments

These were very much appreciated, it's important to a plugin system that details like this feel right! I'll address them in batch later in the week.

I know TypeScript can be a little controversial

FWIW I am in favor of doing Typescript - I just wanted to keep the initial set of files in this PR as simple as possible to review. Really appreciate you scaffolding this initial set of types + I think it would be a welcome addition to maintain a set of types.d.ts files.

I'm entertaining the idea of writing the actual source code in Typescript as long as the compiled output is readable b/c it can be tricky to keep the types and plain JS files in sync. Curious if you have encountered projects that are good at preventing drift.

Maybe they should have more "action-y" names

This is a great observation. I'm inclined towards something like make* or build* since to me add* make me think the thing the method is attached to is being mutated, but I agree that any of these may be clearer than the current get* setup. I'll go through and update these.

Maybe we can make it easier to do pure-js datasette plugins?

I really like this idea! It'll be easier to get contributors if they don't have to touch the python side at all.

And then do the PERMITTED_VIEWS filtering in JS rather than Python.

One cost of doing this is that pages that won't use the JS would still have to load the unused code (given that I'm not sending up anything complex like lazy loading). But hopefully the manager core size is close to negligible, and it won't be a big deal.

Copy link
Collaborator

@asg017 asg017 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! Very eager to get this in, I want to add a hook for adding new features to the SQL editor with custom codemirror components

@cldellow
Copy link
Contributor

cldellow commented May 2, 2023

Thanks for putting this together! I've been slammed with work/personal stuff so haven't been able to actually prototype anything with this. :(

tl;dr: I think this would be useful immediately as is. It might also be nice if the plugins could return Promises.

The long version: I read the design notes and example plugin. I think I'd be able to use this in datasette-ui-extras for my lazy-facets feature.

The lazy-facets feature tries to provide a snappier user experience. It does this by altering how suggested facets work.

First, at page render time:
(A) it lies to Datasette and claims that no columns support facets, this avoids the lengthy delays/timeouts that can happen if the dataset is large.
(B) there's a python plugin that implements the extra_body_script hook, to write out the list of column names for future use by JavaScript

Second, at page load time: there is some JavaScript that:
(C) makes AJAX requests to suggest facets for each column - it makes 1 request per column, using the data from (B)
(D) wires up the column menus to add Facet-by-this options for each facet

With the currently proposed plugin scheme, I think (D) could be moved into the plugin. I'd do the ajax requests, then register the plugin.

If the plugin scheme also supported promises, I think (B) and (C) could also be moved into the plugin.

Does that make sense? Sorry for the wall of text!

@simonw
Copy link
Owner

simonw commented Jul 10, 2023

I tried running this locally just now. I made one edit:

diff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py
index 7bdb9f3f..f9dfa8e6 100644
--- a/demos/plugins/example_js_manager_plugins.py
+++ b/demos/plugins/example_js_manager_plugins.py
@@ -15,6 +15,6 @@ def extra_js_urls(view_name):
     if view_name in PERMITTED_VIEWS:
         return [
             {
-                "url": f"/-/demos/plugins/static/table-example-plugins.js",
+                "url": f"/static/table-example-plugins.js",
             }
         ]

And then started it running like this:

wget https://datasette.io/content.db
datasette content.db \
  --plugins-dir demos/plugins \
  --static static:datasette/demos/plugins/static

It didn't quite work for me - I got this error on a table page:

image

And this error on a query page:

image

from datasette import hookimpl

# Test command:
# datasette fixtures.db --plugins-dir=demos/plugins/
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be:

datasette fixtures.db \
  --plugins-dir=demos/plugins/ \
  --static static:demos/plugins/static

Then move datasette/demos/plugins/static/table-example-plugins.js to demos/plugins/static/table-example-plugins.js to get that to work.

// DATASETTE_EVENTS.INIT event to avoid the habit of reading from the window.

window.__DATASETTE__ = datasetteManager;
console.debug("Datasette Manager Created!");
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was worried that there might be browsers in which this would cause an error (because console.debug might not be defined), but as far as I can tell this has been supported in every modern browser for years at this point: https://console.spec.whatwg.org/ and https://developer.mozilla.org/en-US/docs/Web/API/console#browser_compatibility - so this is fine.

Comment on lines 34 to 35
const datasetteManager = {
VERSION: '1.0a2',
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should make this available in the base.html template, something like this:

<script>
datasetteVersion = '{{ datasette_version }}';
</script>

Then the datasette-manager.js script, which is loaded later, can pick it up from there.

The {{ datasette_version }} variable is already set and is used by the footer template here:

Powered by <a href="https://datasette.io/" title="Datasette v{{ datasette_version }}">Datasette</a>

Copy link
Contributor Author

@hydrosquall hydrosquall Jul 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brilliant! Confirmed that this works:

https://a.cl.ly/Z4uGrYg8

@hydrosquall hydrosquall force-pushed the cameron.yick/experimental-datasette-javascript-api branch from 4b6fa80 to eb1f408 Compare July 11, 2023 12:49
@hydrosquall
Copy link
Contributor Author

Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the table and database views.

@hydrosquall hydrosquall requested a review from simonw July 11, 2023 12:54
@codecov
Copy link

codecov bot commented Jul 12, 2023

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (3feed1f) 92.46% compared to head (8ae479c) 92.69%.
Report is 112 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2052      +/-   ##
==========================================
+ Coverage   92.46%   92.69%   +0.22%     
==========================================
  Files          38       40       +2     
  Lines        5750     6047     +297     
==========================================
+ Hits         5317     5605     +288     
- Misses        433      442       +9     

see 19 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@20after4
Copy link

This is such a well thought out contribution. I don't think I've seen such a thoroughly considered PR on any project in recent memory.

@simonw
Copy link
Owner

simonw commented Oct 12, 2023

Weird, the cog check is failing in CI.

Run cog --check docs/*.rst
  cog --check docs/*.rst
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib
Check failed
Checking docs/authentication.rst
Checking docs/binary_data.rst
Checking docs/changelog.rst
Checking docs/cli-reference.rst
Checking docs/configuration.rst  (changed)

@simonw
Copy link
Owner

simonw commented Oct 12, 2023

Oh! I think I broke Cog on main and these tests are running against this branch rebased against main.

simonw added a commit that referenced this pull request Oct 12, 2023
@simonw
Copy link
Owner

simonw commented Oct 12, 2023

I'm landing this despite the cog failures. I'll fix them on main if I have to.

@simonw simonw merged commit 452a587 into simonw:main Oct 13, 2023
6 of 11 checks passed
@hydrosquall hydrosquall deleted the cameron.yick/experimental-datasette-javascript-api branch October 14, 2023 17:49
simonw added a commit that referenced this pull request Feb 6, 2024
simonw added a commit that referenced this pull request Feb 7, 2024
Closes #2243

* Changelog for jinja2_environment_from_request and plugin_hook_slots
* track_event() in changelog
* Remove Using YAML for metadata section - no longer necessary now we show YAML and JSON examples everywhere.
* Configuration via the command-line section - #2252
* JavaScript plugins in release notes, refs #2052
* /-/config in changelog, refs #2254

Refs #2052, #2156, #2243, #2247, #2249, #2252, #2254
simonw added a commit that referenced this pull request Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants