Add equation support

[kurabirko]

This PR adds support for equation input in FormattedText structs. Equation syntax is similar to Latex, i.e. \( ... \) for inline equations, \[ ... \] for display mode equations. Since dollar sign is commonly used ,I decided against Tex syntax. This can be changed of course. This PR itself does not do much by itself; it just wraps equations in a span tag. Still, this can come in handy when using a third party framework like KaTeX to render the equation, possibly as a Publish plugin. Finally, I wrote a few tests but I am not sure if they cover every edge case. It would be nice to have more tests.

[russellgordon]

I would love to see this pull request approved. For the moment I have worked around the lack of equation support by including images in my posts, but that's not ideal long-term.

[russellgordon]

@kurabirko I've been playing with this a little bit from your fork. I'm probably missing something, but when I include this in the markdown file for an article:

\[
y = a\left(x-h\right)^2+k
\]

The resulting HTML is as follows:

<p>[ y = aleft(x-hright)^2+k ]</p>

As you can see, the backslashes are stripped out. I don't see how this would get rendered to the desired output:

... by KaTeX? I can see that the backslashes should be being retained. What am I doing wrong?

Just want to try this out, and if I can, help with perhaps writing some tests, documentation, or code if necessary.

[russellgordon]

Tentatively, I think some changes are needed.

First, a question. What is the best way for me to amend the code in this pull request? Forgive me for having to ask, as I'm new to making contributions in an open source project like this. I know how to make a fork from the main repository, make a branch, make changes, push, and issue a pull request. What I'm wondering: is there a way for me to build upon the @kurabirko pull request? Or should I be creating a new pull request with my own proposed changes?

In any case, I think there are two issues at play:

1. The delimiters for a LaTeX equation aren't making it through to the HTML. Aren't those necessary for KaTeX to find and render the equation?

So for example, this is a test that I think is correct:

    func testDisplayStandardFormQuadraticSingleLine() {
        let html = MarkdownParser().html(from: #"\[y = a\left(x-h\right)^2+k\]"#)
        XCTAssertEqual(html, #"<p><span class="math display">\[y = a\left(x-h\right)^2+k\]</span></p>"#)
    }

Note the inclusion of the opening \[ and closing \] in the assertion above. When I manually place that HTML in a document that has the following in the header:

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.css" integrity="sha384-AfEj0r4/OFrOo5t7NnNe46zW/tFgW6x/bCJG8FqQCEo3+Aro6EYUG4+cU+KJWu/X" crossorigin="anonymous">

    <!-- The loading of KaTeX is deferred to speed up page rendering -->
    <script defer src="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.js" integrity="sha384-g7c+Jr9ZivxKLnZTDUhnkOnsh30B4H0rpLUpJ4jAIKs4fnJI+sEnkvrMWph2EDg4" crossorigin="anonymous"></script>

    <!-- To automatically render math in text elements, include the auto-render extension: -->
    <script defer src="https://cdn.jsdelivr.net/npm/[email protected]/dist/contrib/auto-render.min.js" integrity="sha384-mll67QQFJfxn0IYznZYonOWZ644AWYC+Pt2cHqMaRhXVrursRwvLnLaebdGIlYNa" crossorigin="anonymous"
        onload="renderMathInElement(document.body);"></script>

... the math renders correctly. Without the opening \[ and closing \] the math does not render.

2. Next, equations spanning multiple lines aren't handled well. Consider these tests:

    func testDisplayStandardFormQuadraticMultiLine() {
        let html = MarkdownParser().html(from: #"""
                                                \[
                                                    y = a\left(x-h\right)^2+k
                                                \]
                                                """#)
        XCTAssertEqual(html, #"""
                                <p><span class="math display">
                                \[
                                y = a\left(x-h\right)^2+k
                                \]
                                </span></p>
                             """#)
    }

    func testDisplayMultiLineProgression() {
        let html = MarkdownParser().html(from: #"""
                                                \[
                                                \begin{aligned}
                                                y&=\left(x-r\right)\left(x-s\right)\\
                                                y&=\left(x-\left(-7\right)\right)\left(x-\left(-2\right)\right)\\
                                                y&=\left(x+7\right)\left(x+2\right)\\
                                                y&=x^2+9x+14\\
                                                y&=x^2+bx+c\\
                                                \end{aligned}
                                                \]
                                                """#)
        XCTAssertEqual(html, #"""
                                <p><span class="math display">
                                \[
                                \begin{aligned}
                                y&=\left(x-r\right)\left(x-s\right)\\
                                y&=\left(x-\left(-7\right)\right)\left(x-\left(-2\right)\right)\\
                                y&=\left(x+7\right)\left(x+2\right)\\
                                y&=x^2+9x+14\\
                                y&=x^2+bx+c\\
                                \end{aligned}
                                \]
                                </span></p>
                                """#)
    }

Again, both of those equations render when I place that HTML manually in a page, and have the KaTeX includes in the header as noted above. Here is an example of the KaTeX website rendering both:

Respectfully I think our goal for Ink supporting LaTeX equations is that anything KaTeX can render, we should pass through from Markdown to HTML untouched.

Now, thinking out loud... could this be implemented in the same way that Ink handles Swift code blocks? Ink just ignores everything between matching \[ and \] in Markdown? Doesn't strip out the backslashes, et cetera?

I'd love to contribute by writing some tests at a minimum.

I will continue digging through the code and seeing if I can figure out what to do. Sorry that I'm such a newb. 😅 I hope my comment here makes sense. I welcome feedback on my suggestions. Thanks, all!

[russellgordon]

@kurabirko I've made a pull request on your fork that implements the changes I mentioned in my most recent comment. Please let me know what you think.

[russellgordon]

Now just a question – I've verified that the command line version of Ink, with the changes I've made a pull request for, correctly parses the LaTeX in my Markdown file.

See the input Markdown file and the output HTML.

What I can't figure out is how to get my local copy of Publish to use this revised code (until the pull requests for equation support are, hopefully, merged at some point in the future).

For example, I've merged the changes from my math branch into the master branch of the fork I've made of Ink. I then set a new release version, of 0.5.1.

I then changed Package.swift in my copy of Publish:

// swift-tools-version:5.2

/**
*  Publish
*  Copyright (c) John Sundell 2019
*  MIT license, see LICENSE file for details
*/

import PackageDescription

let package = Package(
    name: "Publish",
    products: [
        .library(name: "Publish", targets: ["Publish"]),
        .executable(name: "publish-cli", targets: ["PublishCLI"])
    ],
    dependencies: [
//        .package(name: "Ink", url: "https://github.com/johnsundell/ink.git", from: "0.2.0"),
        .package(name: "Ink", url: "https://github.com/russellgordon/Ink.git", from: "0.5.1"),
        .package(name: "Plot", url: "https://github.com/johnsundell/plot.git", from: "0.4.0"),
        .package(name: "Files", url: "https://github.com/johnsundell/files.git", from: "4.0.0"),
        .package(name: "Codextended", url: "https://github.com/johnsundell/codextended.git", from: "0.1.0"),
        .package(name: "ShellOut", url: "https://github.com/johnsundell/shellout.git", from: "2.3.0"),
        .package(name: "Sweep", url: "https://github.com/johnsundell/sweep.git", from: "0.4.0")
    ],
    targets: [
        .target(
            name: "Publish",
            dependencies: [
                "Ink", "Plot", "Files", "Codextended", "ShellOut", "Sweep"
            ]
        ),
        .target(
            name: "PublishCLI",
            dependencies: ["PublishCLICore"]
        ),
        .target(
            name: "PublishCLICore",
            dependencies: ["Publish"]
        ),
        .testTarget(
            name: "PublishTests",
            dependencies: ["Publish", "PublishCLICore"]
        )
    ]
)

I then deleted the .build folder in the Publish directory, ran make, saw that the code from my repository was pulled and compiled, but when I invoke publish run I'm still seeing the old behaviour where equations were not parsed correctly.

Does anyone have a suggestion on what to do? @JohnSundell ? Thanks in advance. Sorry to post so many times today on this pull request. I'm building a website that I'll be using, potentially, to post lessons to my students remotely as a teacher this year. So I'm hoping to get equation support working before September. 😅

[russellgordon]

🤦🏻‍♂️

This comment thread is going to be a great example of vulnerability and being OK with learning in public when I share it with my senior students later this year! 😅

I forked Publish as well, and added a couple of tests to see if Markdown with equations would be handled correctly, using the changes I've proposed.

My theory was that something in Publish might be causing backslash characters to be removed prior to Markdown processing. I figured could explain the results I'm seeing – tests in Ink show that equations are being parsed accurately – but when generating a site using Publish, equations are not recognized.

So here's a screen capture of the tests I wrote in my fork of Publish:

... as you can see, they pass. But, when I build the site with Publish, the very same Markdown does not parse correctly.

I've learned a bit about how to use Swift Package manager – I'm now pointing right at the math branch in my fork of Ink:

// swift-tools-version:5.2

/**
*  Publish
*  Copyright (c) John Sundell 2019
*  MIT license, see LICENSE file for details
*/

import PackageDescription

let package = Package(
    name: "Publish",
    products: [
        .library(name: "Publish", targets: ["Publish"]),
        .executable(name: "publish-cli", targets: ["PublishCLI"])
    ],
    dependencies: [
//        .package(name: "Ink", url: "https://github.com/johnsundell/ink.git", from: "0.2.0"),
        .package(url: "https://github.com/russellgordon/Ink.git", .branch("math")),
        .package(name: "Plot", url: "https://github.com/johnsundell/plot.git", from: "0.4.0"),
        .package(name: "Files", url: "https://github.com/johnsundell/files.git", from: "4.0.0"),
        .package(name: "Codextended", url: "https://github.com/johnsundell/codextended.git", from: "0.1.0"),
        .package(name: "ShellOut", url: "https://github.com/johnsundell/shellout.git", from: "2.3.0"),
        .package(name: "Sweep", url: "https://github.com/johnsundell/sweep.git", from: "0.4.0")
    ],
    targets: [
        .target(
            name: "Publish",
            dependencies: [
                "Ink", "Plot", "Files", "Codextended", "ShellOut", "Sweep"
            ]
        ),
        .target(
            name: "PublishCLI",
            dependencies: ["PublishCLICore"]
        ),
        .target(
            name: "PublishCLICore",
            dependencies: ["Publish"]
        ),
        .testTarget(
            name: "PublishTests",
            dependencies: ["Publish", "PublishCLICore"]
        )
    ]
)

... and I've verified that the code changes I've made in Ink are being used to build Publish:

... and yet, Markdown files with embedded equations aren't being handled correctly:

I'm going to keep digging around in Ink, but I'm really stumped here. 🧐

[russellgordon]

For the time being, I'm just going to insert equations in my documents like this:

<p><span class="math display">\[y = a\left(x-h\right)^2+k\]</span></p>

The backslashes are left alone, as you'd expect, since that is raw HTML.

I have not been able to figure out where, but I'm certain that somewhere in the Publish pipeline, \ characters are being stripped out of Markdown files, either before they are sent to Ink, or in some part of Ink that I'm not aware of.

[russellgordon]

Final comment – for anyone reading through this thread, and wanting to get KaTeX rendered equations working on their Publish sites while this pull request is considered – here is a diff showing how I set this up on my site (my website repository is private so that's a link to a self-contained HTML file).

Note: The implementation is super ugly.

1. It requires raw HTML around the LaTeX in your post's Markdown file (otherwise \ characters get removed from the LaTeX, see my comments above).
2. I've completely duplicated the .head method (see issue 56 on the Publish project) so that I could get the Katex javascript included only on the pages that need it, and made the use of my alternate <head> dependent on whether the custom metadata tag math: true is present in a page's Markdown.

Down the road, my hope is that:

• we find a way to work out this pull request, so there's no need for raw HTML to wrap an equation
• something happens with Publish so that custom additions to the <head> are easier
• perhaps look at whether equations could be rendered server-side with KaTeX – I'd rather like to avoid Javascript on my site if possible, but as a teacher of mathematics with a need to get my site running for possible remote instruction in September, I have to be pragmatic for now

Whew... thanks all. Sorry for flooding this PR with my comments.

[kurabirko]

Hi @russellgordon and thanks for the detailed comments. I did not have much time so am just replying. Sorry for this. About the two point you made,

1. The delimiters for a LaTeX equation aren't making it through to the HTML. Aren't those necessary for KaTeX to find and render the equation?

I think these are necessary only for the auto rendering extension to parse the input and find equations. Katex functions render and renderToString do need (and possibly fail with) the delimiters. You can test this with node.

 echo '\sqrt{2}' |  npx katex

The code above gives you the following output

<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msqrt><mn>2</mn></msqrt></mrow><annotation encoding="application/x-tex">\sqrt{2}
</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.04em;vertical-align:-0.13278em;"></span><span class="mord sqrt"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.90722em;"><span class="svg-align" style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="mord" style="padding-left:0.833em;"><span class="mord">2</span></span></span><span style="top:-2.86722em;"><span class="pstrut" style="height:3em;"></span><span class="hide-tail" style="min-width:0.853em;height:1.08em;"><svg width='400em' height='1.08em' viewBox='0 0 400000 1080' preserveAspectRatio='xMinYMin slice'><path d='M95,702
c-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14
c0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54
c44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10
s173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429
c69,-144,104.5,-217.7,106.5,-221
l0 -0
c5.3,-9.3,12,-14,20,-14
H400000v40H845.2724
s-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7
c-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z
M834 80h400000v40h-400000z'/></svg></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.13278em;"><span></span></span></span></span></span></span></span></span>

I plugged this into an HTML file along with the KaTeX css. It works fine as you can see. I think it is best to leave delimiter choice to the Ink modifier that actually render the equation.

Next, equations spanning multiple lines aren't handled well.

Yes there's something wonky going on. And not only on multi line equations. I will look into your pr about this. However, it also contains the addition of delimiters so I think it would be better if you make them into two separate PRs. I will comment this on my repo as well as a reminder. We can continue discussing this there not to clutter this PR any further.

perhaps look at whether equations could be rendered server-side with KaTeX – I'd rather like to avoid Javascript on my site if possible, but as a teacher of mathematics with a need to get my site running for possible remote instruction in September, I have to be pragmatic for now

Well actually I was working on something like this. I am using a WKWebView (or possibly a JSContext from JavaScriptCore. But this performs poorly on ios afaik.) to render strings using javascript. I don't have much time these days though, so development is reeally slow.

[kurabirko]

Oh and I recently came across the following delimiters

• ${ and }$ for inline equations
• $${ and }$$ for display equations

This is quite nice because it is plain TeX compatible. These tokens also do not end up together often so I don't think there will be cases where dollars signs are mistaken for math tokens. What do you think about this? It seems better than abusing escape syntax.

[kurabirko]

Hi again, I created a simple plugin to render equations using KaTeX. It's available in kurabirko/KaTeXPublishPlugin. I will create a release tag once this PR is merged.