Add webfont and script to subset a font. #35

Szunti · 2018-09-22T18:03:19Z

How about this for #34?

A python script (python 3.6+, but easy to backport) using the fontTools package to subset multiple fontfiles until all the codepoints can be displayed. Configured by a json file.
Also @font-face rule and HanaMinA subsetted webfont.

Hanazono font was chosen because for Han Nom:

couldn't find a license
I don't know if it matters but it's a Vietnamese font, as far as I know Hanazono is Japanese

Do you want anything different?

make_woff.py --help

usage: ./make_woff.py conffile

conffile is in JSON format. Example with explanatory comments (don't include
these comments in the JSON file):
{
  // List of files or globs relative to the config file's dir.
  "dataFiles": [
    "../data/pages/en-GB/**/*.txt"
  ],
  // List of individual code points and ranges (list with start an end).
  "excludeCodepoints": [
    [0, 127]
  ],
  // The order of fonts are important, fonts will be checked for every code
  // point until one has a glyph. First element is the input, second is the
  // output.
  "fonts": [
    ["HanaMinA.ttf", "../HanaMinA.woff"],
    ["HanaMinB.ttf", "../HanaMinB.woff"]
  ],
  // Missing characters will be written to this file in utf-8. relative to
  // config file's location
  "missingOutput": "missing.txt",
  // if true, only print which fonts are used and missing characters
  "justCheck": false
}

Subsetted HanaMinA also included.

Pomax · 2018-09-23T17:54:41Z

style.css

 .nrGrammardata {
  background: white;
+  font-family: serif, HanaMinA;


This will always pick serif because it will always work as per the CSS spec, so you'll have to swap these around. The generic family class keyword always has to come last.

The idea was if the serif font has CJK glyphs than the reader probably likes the default font and don't really need the HanaMinA substitute. But if the default serif doesn't have CJK glyphs, which is I guess common for learners, they have their nice readable latin font and only CJK glyphs fall back to HanaMinA.

Alternative is to have a good looking latin font before HanaMinA and serif at the end.

But I have no intention of honoring the reader's locally installed fonts: the reason we're subsetting is to ensure that everyone sees the same thing, with a font that covers exactly what is necessary =)

HanaMinA's latin is really ugly here. Added Times first.

Pomax · 2018-09-23T17:55:11Z

style.css

@@ -52,7 +58,7 @@
  height: 500px;
  text-align: right;
  font-size: 13px;
-  font-family: Arial;
+  font-family: Arial, HanaMinA;


Probably good to add a , sans-serif since we're touching this rule anyway.

Pomax · 2018-09-23T17:55:31Z

webfont/make_woff.py

+
+
+if sys.version_info[0:2] < (3, 6):
+    error("This script requires Python version 3.6")


3.6 or higher?

Pomax · 2018-09-23T17:58:23Z

webfont/make_woff.py

+    import fontTools.merge
+    import fontTools.misc.loggingTools
+except ImportError:
+    error("Please install the fontTools python module.")


it feels like we should be able to invoke pip here.

accepted = ask("FontTools dependency is missing, would you like to install them now? [Y/n]"); if accepted is True: runPipInstalls() else: error("Dependencies not installed, exiting...") exit(1)

There are just so many possibilities. Do you want it to be installed with the distribution's package manager or pip? Install globally (which also needs root permission probably) or only for the current user or maybe in a virtual environment? But I can make something if it's really necessary.

Offering the choice, only makes it more usable. We can add in the option to use pip (which is the defacto manager these days) and if someone says "no" then they get exactly as much functionality as if we just immediately exit. So worst case: things stay the same. Best case: the script takes care of it and just continues.

Let's add in a pip install (and if it complains about permissions, we can make it say "could not install dependencies, please install the following packages manually:" and then a list of the dependencies necessary)

Pomax · 2018-09-23T17:59:25Z

webfont/make_woff.py

+
+class BaseFilter():
+    def feed(self, codepoint):
+        """Let the filter handle the codepoint.


Let a subclassing filter handle the codepoint because without overriding this function literally breaks python.

Pomax · 2018-09-23T18:04:19Z

webfont/make_woff.py

+    uniq_filt = UniqueFilter()
+    pipeline.add_filter(uniq_filt)
+
+    excl_filt = ExcludeFilter()


The ASCII range takes up near-as-makes-no-difference nothing compared to the rest of the data that gets loaded, and having the Hanazono-specific latin characters in the font means that stretches that end up being marked as "use Japanese typesetting, even for the latin bits" will look better later.

Let's take out the exclusion option.

Pomax · 2018-09-23T18:04:29Z

webfont/make_woff.py

+            return False
+
+
+class ExcludeFilter(BaseFilter):


see https://github.com/Pomax/nrGrammar/pull/35/files#r219706575

Pomax · 2018-09-23T18:04:48Z

webfont/woff.conf

+  "dataFiles": [
+    "../data/pages/en-GB/**/*.txt"
+  ],
+  "excludeCodepoints": [


https://github.com/Pomax/nrGrammar/pull/35/files#r219706575

Pomax

left some review comments as normal comments.

Pomax · 2018-09-23T18:06:16Z

Having both PHP and Python as build tools might cause some problems when it comes to adding CI to this, but that can be a later concern. I've left a few comments but overall this looks fine to me. Thank you for helping get this implemented.

Useful to ignore missing TAB and LFD, those don't have glyphs in font files.

Pomax · 2018-09-24T16:14:00Z

style.css

@@ -1,11 +1,12 @@
 @font-face {
  font-family: HanaMinA;
-  src: url(HanaMinA.woff);
+  src: local("HanaMinA")


The idea is still to offer exactly the font necessary so that everyone sees exactly the same, so let's remove local(), because there's no guarantee their local font is the same font version as the one used to build the woff subset font.

If the initial .woff is too big up front, let's use woff2 instead and slice it so that it loads incrementally based on what's necessary per chapter.

323K as woff, 241K as woff2

brotli showing off its power. Very nice!

woff2 is used if the output filename ends in '.woff2' else woff used.

Szunti · 2018-09-25T03:09:13Z

Unless slicing for sections are needed, I think I'm done.

Add webfont and script to subset a font.

c0ecd24

Subsetted HanaMinA also included.

Szunti force-pushed the webfont branch from 53fbb01 to c0ecd24 Compare September 22, 2018 18:06

Pomax reviewed Sep 23, 2018

View reviewed changes

webfont/make_woff.py Outdated

if sys.version_info[0:2] < (3, 6):

error("This script requires Python version 3.6")

Copy link

Owner

Pomax Sep 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3.6 or higher?

Pomax reviewed Sep 23, 2018

View reviewed changes

webfont/make_woff.py

return False

class ExcludeFilter(BaseFilter):

Copy link

Owner

Pomax Sep 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see https://github.com/Pomax/nrGrammar/pull/35/files#r219706575

Pomax reviewed Sep 23, 2018

View reviewed changes

Pomax requested changes Sep 23, 2018

View reviewed changes

Szunti added 3 commits September 23, 2018 21:16

Doc fixes.

7b1b7ca

Move exclusion of codepoints after the subsets.

a005efe

Useful to ignore missing TAB and LFD, those don't have glyphs in font files.

Move justCheck option to command line

6d4bacb

Szunti force-pushed the webfont branch from 0b9b842 to 6d4bacb Compare September 24, 2018 02:35

Fixing CSS font order

2fece03

Pomax requested changes Sep 24, 2018

View reviewed changes

Szunti added 3 commits September 24, 2018 19:18

Add ability for multiple outfiles and woff2 format

9bf14f8

woff2 is used if the output filename ends in '.woff2' else woff used.

Remove local() from @font-face and add woff hint

ee5370a

Use pip install --user for missing packages

294f9dc

Add woff2 font, keep woff as fallback.

3bb73dc

Szunti force-pushed the webfont branch from 2728e59 to 3bb73dc Compare September 25, 2018 12:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add webfont and script to subset a font. #35

Add webfont and script to subset a font. #35

Szunti commented Sep 22, 2018

Pomax Sep 23, 2018

Szunti Sep 23, 2018

Pomax Sep 24, 2018

Szunti Sep 24, 2018

Pomax Sep 23, 2018

Pomax Sep 23, 2018

Pomax Sep 23, 2018

Szunti Sep 23, 2018

Pomax Sep 24, 2018

Pomax Sep 23, 2018 •

edited

Loading

Pomax Sep 23, 2018

Pomax Sep 23, 2018

Pomax Sep 23, 2018

Pomax left a comment

Pomax commented Sep 23, 2018 •

edited

Loading

Pomax Sep 24, 2018 •

edited

Loading

Szunti Sep 24, 2018

Pomax Sep 25, 2018

Szunti commented Sep 25, 2018



		if sys.version_info[0:2] < (3, 6):
		error("This script requires Python version 3.6")

Add webfont and script to subset a font. #35

Are you sure you want to change the base?

Add webfont and script to subset a font. #35

Conversation

Szunti commented Sep 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Pomax Sep 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Pomax left a comment

Choose a reason for hiding this comment

Pomax commented Sep 23, 2018 • edited Loading

Pomax Sep 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Szunti commented Sep 25, 2018

Pomax Sep 23, 2018 •

edited

Loading

Pomax commented Sep 23, 2018 •

edited

Loading

Pomax Sep 24, 2018 •

edited

Loading