Markdown support

Dockerfile

@@ -14,10 +14,12 @@ RUN apt-get update && \
             libgmp10 \
             libgmpxx4ldbl \
             openjdk-8-jdk \
+            pandoc \
             python3-minimal \
             python3-pip \
             python3-plastex \
             python3-yaml \
+            rsvg-convert \
             sudo \
             texlive-fonts-recommended \
             texlive-lang-cyrillic \

README.md

@@ -207,13 +207,13 @@ The dependencies needed to *build/install* problemtools can be installed with:
 
 And the dependencies needed to *run* problemtools can be installed with:
 
-    sudo apt install ghostscript libgmpxx4ldbl python3-minimal python-pkg-resources python3-plastex python3-yaml texlive-fonts-recommended texlive-lang-cyrillic texlive-latex-extra texlive-plain-generic tidy
+    sudo apt install ghostscript libgmpxx4ldbl pandoc python3-minimal python-pkg-resources python3-plastex python3-yaml rsvg-convert texlive-fonts-recommended texlive-lang-cyrillic texlive-latex-extra texlive-plain-generic tidy
 
 ### Fedora
 
 On Fedora, these dependencies can be installed with:
 
-    sudo dnf install boost-regex gcc gmp-devel gmp-c++ python3 python3-pyyaml texlive-latex texlive-collection-fontsrecommended texlive-fancyhdr texlive-subfigure texlive-wrapfig texlive-import texlive-ulem texlive-xifthen texlive-overpic texlive-pbox tidy ghostscript
+    sudo dnf install boost-regex gcc gmp-devel gmp-c++ pandoc python3 python3-pyyaml rsvg-convert texlive-latex texlive-collection-fontsrecommended texlive-fancyhdr texlive-subfigure texlive-wrapfig texlive-import texlive-ulem texlive-xifthen texlive-overpic texlive-pbox tidy ghostscript
 
 Followed by:
 

admin/docker/Dockerfile.minimal

@@ -20,10 +20,12 @@ RUN apt update && \
     apt install -y \
         ghostscript \
         libgmpxx4ldbl \
+        pandoc \
         python-pkg-resources \
         python3-minimal \
         python3-yaml \
         python3-plastex \
+        rsvg-convert \
         texlive-fonts-recommended \
         texlive-lang-cyrillic \
         texlive-latex-extra \

debian/control

@@ -8,7 +8,7 @@ Homepage: https://github.com/Kattis/problemtools
 
 Package: kattis-problemtools
 Architecture: any
-Depends: ${shlibs:Depends}, ${python3:Depends}, ${misc:Depends}, python3-plastex, python3-pkg-resources, texlive-plain-generic, texlive-fonts-recommended, texlive-latex-extra, texlive-lang-cyrillic, tidy, ghostscript, dvisvgm
+Depends: ${shlibs:Depends}, ${python3:Depends}, ${misc:Depends}, pandoc, python3-plastex, python3-pkg-resources, rsvg-convert, texlive-plain-generic, texlive-fonts-recommended, texlive-latex-extra, texlive-lang-cyrillic, tidy, ghostscript, dvisvgm
 Recommends: gcc, g++
 Description: Kattis Problem Tools
  These are tools to manage and verify problem packages in the

examples/README.md

@@ -24,4 +24,5 @@ more than one language.
 ## oddecho
 
 This is an example of a *scoring* problem where submissions can get
-different scores depending on which test groups they solve. It also demonstrates how an input validator might check different constraints for different test groups.
+different scores depending on which test groups they solve. It also demonstrates how an input validator might check different constraints for different test groups. The swedish statement showcases how to use images, footnotes
+and tables in Markdown.

examples/different/problem.yaml

@@ -5,6 +5,11 @@
 ## Author of the problem (default: null)
 # author:
 
+# The problem name
+# En may be omitted, as there is only one language
+name:
+    en: A Different Problem
+
 ## Where the problem was first used (default: null)
 source: Kattis
 # source_url:

examples/guess/problem.yaml

@@ -2,6 +2,9 @@ source: Kattis
 license: cc by-sa
 
 validation: custom interactive
+name:
+  sv: Gissa talet
+  en: Guess the Number
 
 # Override standard limits: say that the TLE solutions provided should
 # be at least 4 times above the time limit in order for us to be

examples/guess/problem_statement/problem.sv.md

@@ -0,0 +1,20 @@
+Jag tänker på ett hemligt tal mellan $1$ and $100$, kan du gissa vilket?
+Givet en gissning kommer jag att berätta om din gissning 
+var för stor, för liten eller rätt. Du får bara $10$ gissningar, använd 
+dem klokt!
+
+
+## Interaktion
+Ditt program ska skriva ut gissningar om talet.
+En gissning är en rad som enbart innehåller ett heltal mellan $1$ och $1000$.
+Efter varje gissning måste du flusha standard out.
+
+Efter varje gissning kan du läs svaret på standard in.
+Detta svar är ett av tre ord:
+
+- `lower` om talet jag tänker på är lägre än din gissning,
+- `higher` om talet jag tänker på är högre än din gissning, eller
+- `correct` om din gissning är korrekt.
+
+Efter att ha gissat rätt ska du avsluta ditt program.
+Om du gissar fel $10$ gånger får du inga fler chanser och ditt program kommer avbrytas.

examples/hello/problem.yaml

@@ -1,5 +1,8 @@
 source: Kattis
 license: public domain
+name:
+    sv: Hej Världen!
+    en: Hello World!
 
 # Fix memory limit at 512 MB.  (Note that for most problems, this
 # should not be done.  It is only done in this case because we include

examples/oddecho/problem.yaml

@@ -2,6 +2,8 @@ license: cc by-sa
 author: Johan Sannemo
 source: Principles of Algorithmic Problem Solving
 type: scoring
-name: Echo
+name:
+    en: Odd Echo
+    sv: Udda Eko
 grading:
     show_test_data_groups: true

examples/oddecho/problem_statement/problem.sv.md

@@ -0,0 +1,33 @@
+**EKO! Eko! Ek...**
+
+![](echo_cave.jpg)
+
+Du älskar att skrika i grottor för att höra dina ord ekade tillbaka till dig. Tyvärr, som en hårt arbetande mjukvaruingenjör, har du
+inte tid för att komma ut och skrika i grottor så ofta. Istället skulle du vilja implementera ett program som fungerar som en ersättning för en grotta.
+
+Ibland vill du mata in några ord i programmet och få dem ekade tillbaka till dig. Men, som det är välkänt, om du skriker för snabbt i en grotta kan ekot störa de nya ord du säger. [^1] Mer specifikt, vartannat ord du säger kommer att störa ekot av ditt tidigare ord. Därför kommer endast det första, tredje, femte och så vidare ordet faktiskt att producera ett eko.
+
+Din uppgift är att skriva ett program som simulerar detta beteende.
+
+## Indata
+
+Den första raden av indata innehåller ett heltal $N$ ($1 \le N \le 10$).
+
+De följande $N$ raderna innehåller vardera ett ord. Varje ord är högst $100$ bokstäver långt och innehåller endast bokstäverna `a-z`.
+
+## Utdata
+
+Skriv ut de ord som har udda index (dvs. första, tredje, femte och så vidare) i inmatningen.
+
+
+## Poängsättning
+
+Din lösning kommer att testas på en mängd testfallsgrupper.
+För att få poäng för en grupp så måste du klara alla testfall i gruppen.
+
+| Grupp | Poäng | Begränsningar            |
+|-------|-------|--------------------------|
+| 1     | 1     | $N$ är alltid $5$        |
+| 2     | 1     | Inga ytterligare begränsningar |
+
+[^1]: [https://sv.wikipedia.org/wiki/Interferens](https://sv.wikipedia.org/wiki/Interferens)

problemtools/md2html.py

@@ -0,0 +1,140 @@
+#! /usr/bin/env python3
+# -*- coding: utf-8 -*-
+import os.path
+import string
+import argparse
+import json
+import subprocess
+
+from . import statement_common
+
+
+FOOTNOTES_STRING = '<section class="footnotes" role="doc-endnotes">'
+
+def convert(problem: str, options: argparse.Namespace) -> bool:
+    """Convert a Markdown statement to HTML
+
+    Args:
+        problem: path to problem directory
+        options: command-line arguments. See problem2html.py
+    """
+    problembase = os.path.splitext(os.path.basename(problem))[0]
+    destfile = string.Template(options.destfile).safe_substitute(problem=problembase)
+
+    statement_path = statement_common.find_statement(problem, extension="md", language=options.language)
+
+    if statement_path is None:
+        raise Exception('No markdown statement found')
+
+    if not os.path.isfile(statement_path):
+        raise Exception(f"Error! {statement_path} is not a file")
+
+
+    _copy_images(statement_path,
+                 lambda img_name: handle_image(os.path.join(problem, "problem_statement", img_name)))
+    command = ["pandoc", statement_path, "-t" , "html"]
+    statement_html = subprocess.run(command, capture_output=True, text=True,
+                                    shell=False, check=True).stdout
+
+    templatepaths = [os.path.join(os.path.dirname(__file__), 'templates/markdown_html'),
+                     os.path.join(os.path.dirname(__file__), '../templates/markdown_html'),
+                     '/usr/lib/problemtools/templates/markdown_html']
+    templatepath = next((p for p in templatepaths
+                              if os.path.isdir(p) and os.path.isfile(os.path.join(p, "default-layout.html"))),
+                             None)
+
+    if templatepath is None:
+        raise Exception('Could not find directory with markdown templates')
+
+    problem_name = statement_common.get_problem_name(problem, options.language)
+
+    html_template = _substitute_template(templatepath, "default-layout.html",
+           statement_html=statement_html,
+           language=options.language,
+           title=problem_name or "Missing problem name",
+           problemid=problembase)
+
+    samples = "".join(statement_common.format_samples(problem, to_pdf=False))
+
+    html_template = inject_samples(html_template, samples)
+    html_template = replace_hr_in_footnotes(html_template)
+
+    with open(destfile, "w", encoding="utf-8", errors="xmlcharrefreplace") as output_file:
+        output_file.write(html_template)
+
+    if options.css:
+        with open("problem.css", "w") as output_file:
+            with open(os.path.join(templatepath, "problem.css"), "r") as input_file:
+                output_file.write(input_file.read())
+
+    return True
+
+
+def handle_image(src: str) -> None:
+    """This is called for every image in the statement
+    Copies the image from the statement to the output directory
+
+    Args:
+        src: full file path to the image
+    """
+    file_name = os.path.basename(src)
+
+    if not os.path.isfile(src):
+        raise Exception(f"File {file_name} not found in problem_statement")
+    if os.path.isfile(file_name):
+        return
+    with open(src, "rb") as img:
+        with open(file_name, "wb") as out:
+            out.write(img.read())
+
+
+def json_dfs(data, callback) -> None:
+    """Traverse all items in a JSON tree, find all images, and call callback for each one"""
+    if isinstance(data, dict):
+        for key, value in data.items():
+            # Markdown-style images
+            if key == 't' and value == 'Image':
+                callback(data['c'][2][0])
+            else:
+                json_dfs(value, callback)
+
+    elif isinstance(data, list):
+        for item in data:
+            json_dfs(item, callback)
+
+
+def _copy_images(statement_path, callback):
+    command = ["pandoc", statement_path, "-t" , "json"]
+    statement_json = subprocess.run(command, capture_output=True,
+                                    text=True, shell=False, check=True).stdout
+    json_dfs(json.loads(statement_json), callback)
+
+
+def inject_samples(html, samples):
+    if FOOTNOTES_STRING in html:
+        pos = html.find(FOOTNOTES_STRING)
+    else:
+        pos = html.find("</body>")
+    html = html[:pos] + samples + html[pos:]
+    return html
+
+
+def replace_hr_in_footnotes(html_content):
+    if not FOOTNOTES_STRING in html_content:
+        return html_content
+    footnotes = html_content.find(FOOTNOTES_STRING)
+    hr_pos = html_content.find("<hr />", footnotes)
+    return html_content[:hr_pos] + """
+<p>
+    <b>Footnotes</b>
+</p>
+""" + html_content[6 + hr_pos:]
+
+
+def _substitute_template(templatepath: str, templatefile: str, **params) -> str:
+    """Read the markdown template and substitute in things such as problem name,
+    statement etc using python's format syntax.
+    """
+    with open(os.path.join(templatepath, templatefile), "r", encoding="utf-8") as template_file:
+        html_template = template_file.read() % params
+    return html_template

problemtools/problem2html.py

@@ -4,67 +4,21 @@
 import os.path
 import string
 import argparse
-import logging
 import subprocess
 
-from . import template
+from . import tex2html
+from . import md2html
+from . import statement_common
 
 def convert(options: argparse.Namespace) -> None:
-    # PlasTeX.Logging statically overwrites logging and formatting, so delay loading
-    import plasTeX.TeX
-    import plasTeX.Logging
-    from .ProblemPlasTeX import ProblemRenderer
-    from .ProblemPlasTeX import ProblemsetMacros
-
     problem = os.path.realpath(options.problem)
 
+    if not os.path.isdir(problem):
+        raise Exception(f"Problem does not exist: {problem}")
+
     problembase = os.path.splitext(os.path.basename(problem))[0]
     destdir = string.Template(options.destdir).safe_substitute(problem=problembase)
     destfile = string.Template(options.destfile).safe_substitute(problem=problembase)
-    imgbasedir = string.Template(options.imgbasedir).safe_substitute(problem=problembase)
-
-    if options.quiet:
-        plasTeX.Logging.disableLogging()
-    else:
-        plasTeX.Logging.getLogger().setLevel(getattr(logging, options.loglevel.upper()))
-        plasTeX.Logging.getLogger('status').setLevel(getattr(logging, options.loglevel.upper()))
-
-    texfile = problem
-    # Set up template if necessary
-    with template.Template(problem, language=options.language) as templ:
-        texfile = open(templ.get_file_name(), 'r')
-
-        origcwd = os.getcwd()
-
-        # Setup parser and renderer etc
-
-        # plasTeX version 3 changed the name of this argument (and guarding against this
-        # by checking plasTeX.__version__ fails on plastex v3.0 which failed to update
-        # __version__)
-        try:
-            tex = plasTeX.TeX.TeX(myfile=texfile)
-        except Exception:
-            tex = plasTeX.TeX.TeX(file=texfile)
-
-        ProblemsetMacros.init(tex)
-
-        tex.ownerDocument.config['general']['copy-theme-extras'] = options.css
-        if not options.headers:
-            tex.ownerDocument.userdata['noheaders'] = True
-        tex.ownerDocument.config['files']['filename'] = destfile
-        tex.ownerDocument.config['images']['filenames'] = 'img-$num(4)'
-        tex.ownerDocument.config['images']['enabled'] = False
-        tex.ownerDocument.config['images']['imager'] = 'none'
-        tex.ownerDocument.config['images']['base-url'] = imgbasedir
-        # tell plasTeX where to search for problemtools' built-in packages
-        tex.ownerDocument.config['general']['packages-dirs'] = [os.path.join(os.path.dirname(__file__), 'ProblemPlasTeX')]
-
-        renderer = ProblemRenderer()
-
-        if not options.quiet:
-            print('Parsing TeX source...')
-        doc = tex.parse()
-        texfile.close()
 
     # Go to destdir
     if destdir:
@@ -75,12 +29,13 @@ def convert(options: argparse.Namespace) -> None:
     try:
         if not options.quiet:
             print('Rendering!')
-        renderer.render(doc)
 
-        # Annoying: I have not figured out any way of stopping the plasTeX
-        # renderer from generating a .paux file
-        if os.path.isfile('.paux'):
-            os.remove('.paux')
+        origcwd = os.getcwd()
+
+        if statement_common.find_statement_extension(problem, options.language) == "tex":
+            tex2html.convert(problem, options)
+        else:
+            md2html.convert(problem, options)
 
         if options.tidy:
             with open(os.devnull, 'w') as devnull: