Skylighting Syntax Files

At the time of this writing, Matterhorn uses the Skylighting package at version 0.6. That version of Skylighting is licensed under the GPL, which causes Matterhorn to be in violation of the GPL because we have licensed Matterhorn under the BSD license. Skylighting 0.6 is licensed under the GPL because its binary form bundles syntax specifications for the languages it supports, and those specifications are GPL-licensed. Version 0.7 of Skylighting remedied this problem by splitting the package up into a core package devoid of such bundled specifications, and a wrapper package that provides the bundled specifications. The latter package is compatible with 0.6 but the core package requires that the specifications be loaded manually from XML files on disk.

This page is about capturing some details about how we want to deal with migrating to the new Skylighting core package that became available with version 0.7. Moving to Skylighting 0.7 entails bundling its XML syntax specifications with Matterhorn, which brings us out of violation with the GPL since those files will not be part of the binary and will be included with their own licensing terms.

High-level concerns:

• We need to ensure that the user experience of upgrading Matterhorn after this has been addressed is smooth.

• We need to ensure that syntaxes supported by Matterhorn after addressing this issue match the set of supported syntaxes of older versions of Matterhorn.

Concrete issues and proposal:

• We need to capture and distribute Skylighting XML syntax specifications with Matterhorn. For this, I propose that we:

  • Write a script to fetch the XML files associated with a specific version of Skylighting-core.
  • Run the script to fetch the XML files.
  • Check those files into Matterhorn's git repository.
  • Include those files in Matterhorn releases.

  This approach makes it possible to ensure that we're getting versions of the XML files that will work with the version of Skylighting we use (in case those files' formats change). It also makes it easy to try out new files in development without commiting to using them, and it makes it possible to ship XML files without being dependent on access to Skylighting, such as if it goes away or is somehow inaccessible.

• We need to ensure that Matterhorn knows where to find these XML files at runtime without onerous or privileged installation requirements for the user. For this, I propose:

  • Matterhorn loads syntax files at startup. It does this by loading all known syntax files that it can locate in a sequence of directories.

  • The sequence of directories indicates precedence, with directories earlier in the sequence taking precedence over later entries, on a per-syntax basis. So, for example, if a syntax for Haskell is loaded from an earlier directory, syntax files for Haskell in later directories are ignored.

  • Syntax file loading is expensive, based on my experience working with Skylighting, so these files should be loaded asynchronously at startup, but they shouldn't be loaded on demand (e.g. upon getting a post using a particular syntax to format code) due to the time required to parse the XML files.

  • The list of paths should be configurable via a setting in the confuration file but should default to: ~/.config/matterhorn/syntax, MH/syntax/, where MH refers to the directory containing the matterhorn binary. This is because if we distribute the syntax files with Matterhorn, this behavior makes it possible to load syntax files from the distribution without installation. It does assume that users just run the binary out of the distribution without moving it, though, which brings me to the last point:

  • Matterhorn needs to provide an installation procedure, which is something we haven't done before this point. This would allow the user to install Matterhorn globally along with syntax files. Without an installer, a moved Matterhorn binary in a fresh installation has no way to know where to find syntax files. With an installer, we can assume they live somewhere like /usr/local/share/matterhorn/syntax.