Add support for markdown statements. #191

jsannemo · 2021-03-17T17:49:21Z

This PR adds rudimentary support for markdown statements, together with some extra markdown statements for example problems (and a scoring example problem).

Supported:

samples, including interactive
tables
inline math mode using 
problem2html

Planned to be added when I add problems that use the feature:

footnotes
images

Not supported:

problem2pdf
everything else

This is completely untested in combination with Kattis problem installation, since that's a pain to test locally for me.

RagnarGrootKoerkamp · 2021-03-17T23:35:05Z

Nice! We've been considering to add this to the spec as well.

I think it's important to spec this well. Some things that come to mind:

I'd prefer to also allow the usual $ delimiters.
We should spec whether the first line must be the problem title or not. Latex has \problemname{NAME} on the first line, so I think it makes sense to have # NAME as the first line of a markdown statement.
- Alternatively, we could render the title from the name in the problem.yaml automatically.
Similarly, statements should have \subsection{Input} or the equivalent \begin{Input} ... \end{Input} (which is nice because you can't accidentally mistype Input or put \section instead of \subsection). Some form of checking may be nice. (Or encourage using the latex commands directly, but it's ugly.)
I think Pandoc (which I've been trying in BAPCtools for md->tex->pdf rendering) would still support things like $\newcommand$ , which is nice to extract common bounds. Does this implementation also do this?
For images, one way would be to do it directly via the latex syntax: $\illustration{0.3}{image.jpg}{CC-BY}$ . (Ah, but you're probably doing the md->html rendering directly, so maybe it's better to somehow parse the standard markdown image inclusion separately for html/pdf rendering. Sounds non-trivial though.)

jsannemo · 2021-03-17T23:44:10Z

Nice! We've been considering to add this to the spec as well.

I think it's important to spec this well. Some things that come to mind:

I'd prefer to also allow the usual $ delimiters.

I always felt that syntax in MD to be slightly iffy. In LaTeX escaping $ is part of the syntax, in markdown it is not. You'd have to resort to heuristics to detect math instead, which feels much harder to spec properly.

We should spec whether the first line must be the problem title or not. Latex has \problemname{NAME} on the first line, so I think it makes sense to have # NAME as the first line of a markdown statement.

This PR was based on the spec, which suggests that the name entry in problem.yaml should be used instead (and I agree with the spec).

Alternatively, we could render the title from the name in the problem.yaml automatically.

That is what I assume the --title option to problem2html does, but I didn't actually check that verifyproblem passes it

Similarly, statements should have \subsection{Input} or the equivalent \begin{Input} ... \end{Input} (which is nice because you can't accidentally mistype Input or put \section instead of \subsection). Some form of checking may be nice.

Except for interactive problems, or just problems in general where communication is done in other ways (such as IOI-style with e.g. language specific wrappers). But either way, this feels like something tp be discussed in a new issue.

I think Pandoc (which I've been trying in BAPCtools for md->tex->pdf rendering) would still support things like $\newcommand$ , which is nice to extract common bounds. Does this implementation also do this?

No

For images, one way would be to do it directly via the latex syntax: $\illustration{0.3}{image.jpg}{CC-BY}$ . (Ah, but you're probably doing the md->html rendering directly, so maybe it's better to somehow parse the standard markdown image inclusion separately for html/pdf rendering. Sounds non-trivial though.)

The plan is to add a post-processor for img tags that fixes links to images pointing to an image file.

RagnarGrootKoerkamp · 2021-03-18T00:05:34Z

Nice! We've been considering to add this to the spec as well.
I think it's important to spec this well. Some things that come to mind:

I'd prefer to also allow the usual $ delimiters.

I always felt that syntax in MD to be slightly iffy. In LaTeX escaping $ is part of the syntax, in markdown it is not. You'd have to resort to heuristics to detect math instead, which feels much harder to spec properly.

Hmmm yeah - I'm a bit torn on this. $ is supported by pandoc which is the de facto standard for math in markdown, but probably doesn't have a nice clean spec. I think not allowing $ (and thus parsing $ as normal text) is going to be ugly as well.

Also came across this, which actually argues against \( because that's an escaped parenthesis in plain markdown.

On this topic, we also have to choose a base markdown spec - probably commonmark is the most universal one (as opposed to e.g. github flavored markdown).

We should spec whether the first line must be the problem title or not. Latex has \problemname{NAME} on the first line, so I think it makes sense to have # NAME as the first line of a markdown statement.

This PR was based on the spec, which suggests that the name entry in problem.yaml should be used instead (and I agree with the spec).

Hmm where exactly did you read this? This is all tricky since officially the name field in problem.yaml isn't part of the ICPC spec... Although it is what I'm doing for latex as well.
IMO this entire problem name part of the spec could use some cleaning up. Actually I'd prefer to only put the name in the problem.yaml and not again as the first line of the statement, as you currently propose, but the asymmetry with latex statements is annoying.

Alternatively, we could render the title from the name in the problem.yaml automatically.

That is what I assume the --title option to problem2html does, but I didn't actually check that verifyproblem passes it

Similarly, statements should have \subsection{Input} or the equivalent \begin{Input} ... \end{Input} (which is nice because you can't accidentally mistype Input or put \section instead of \subsection). Some form of checking may be nice.

Except for interactive problems, or just problems in general where communication is done in other ways (such as IOI-style with e.g. language specific wrappers). But either way, this feels like something tp be discussed in a new issue.

See #180 ;) But yeah, it's definitely out of scope to add commands/environments for all possible use cases. But in both BAPC and NWERC we had some potential bugs where people used the wrong type of section heading so I'd like to somehow guard against this.

I think Pandoc (which I've been trying in BAPCtools for md->tex->pdf rendering) would still support things like $\newcommand$ , which is nice to extract common bounds. Does this implementation also do this?

No

Ack. Actually I'd be ok with having slightly less latex capabilities in a markdown file - if needed you can always revert to the equivalent latex. This means we'd have to make sure that the layout for e.g. lists is exactly equivalent between the two.

For images, one way would be to do it directly via the latex syntax: $\illustration{0.3}{image.jpg}{CC-BY}$ . (Ah, but you're probably doing the md->html rendering directly, so maybe it's better to somehow parse the standard markdown image inclusion separately for html/pdf rendering. Sounds non-trivial though.)

The plan is to add a post-processor for img tags that fixes links to images pointing to an image file.

Makes sense.

CC @niemela, who has also been wanting/pushing this.

RagnarGrootKoerkamp · 2021-03-18T00:10:47Z

For reference, GitLab uses:

$`abc`$

for inline maths (which I don't like so much) and

```math
abc
```

for display math, which is quite nice actually, although I'm not sure whether pandoc does this.

Actually, if we're doing maths, we should definitely support both inline and display maths.

jsannemo · 2021-03-18T00:14:05Z

On Thu, 18 Mar 2021, 01:05 Ragnar Groot Koerkamp, ***@***.***> wrote: Nice! We've been considering to add this to the spec as well. I think it's important to spec this well. Some things that come to mind: - I'd prefer to also allow the usual $ delimiters. I always felt that syntax in MD to be slightly iffy. In LaTeX escaping $ is part of the syntax, in markdown it is not. You'd have to resort to heuristics to detect math instead, which feels much harder to spec properly. Hmmm yeah - I'm a bit torn on this. $ is supported by pandoc which is the de facto standard for math in markdown, but probably doesn't have a nice clean spec. I think not allowing $ (and thus parsing $ as normal text) is going to be ugly as well..Also came across this <https://talk.commonmark.org/t/ignore-latex-like-math-mode-or-parse-it/1926/3>, which actually argues against \( because that's an escaped parenthesis in plain markdown.

That is true (the Kattis implementation parses is configured to parse math and \( before escapes for this reason). I am not married to \( at all though, and am happy to use another sufficiently easy-to-distinguish and simple-to-type delimiter.

On this topic, we also have to choose a base markdown spec - probably commonmark is the most universal one (as opposed to e.g. github flavored markdown). - We should spec whether the first line must be the problem title or not. Latex has \problemname{NAME} on the first line, so I think it makes sense to have # NAME as the first line of a markdown statement. This PR was based on the spec, which suggests that the name entry in problem.yaml should be used instead (and I agree with the spec). Hmm where exactly did you read this? This is all tricky since officially the name field in problem.yaml isn't part of the ICPC spec... Although it is what I'm doing for latex as well. IMO this entire problem name part of the spec could use some cleaning up. Actually I'd prefer to only put the name in the problem.yaml and not again as the first line of the statement, as you currently propose, but the asymmetry with latex statements is annoying.

I thought LaTeX statements can use the name mechanism already today.

- Alternatively, we could render the title from the name in the problem.yaml automatically. That is what I assume the --title option to problem2html does, but I didn't actually check that verifyproblem passes it - Similarly, statements should have \subsection{Input} or the equivalent \begin{Input} ... \end{Input} (which is nice because you can't accidentally mistype Input or put \section instead of \subsection). Some form of checking may be nice. Except for interactive problems, or just problems in general where communication is done in other ways (such as IOI-style with e.g. language specific wrappers). But either way, this feels like something tp be discussed in a new issue. See #180 <#180> ;) But yeah, it's definitely out of scope to add commands/environments for all possible use cases. But in both BAPC and NWERC we had some potential bugs where people used the wrong type of section heading so I'd like to somehow guard against this. - I think Pandoc (which I've been trying in BAPCtools for md->tex->pdf rendering) would still support things like $\newcommand$, which is nice to extract common bounds. Does this implementation also do this? No Ack. Actually I'd be ok with having slightly less latex capabilities in a markdown file - if needed you can always revert to the equivalent latex. This means we'd have to make sure that the layout for e.g. lists is exactly equivalent between the two.

For me, markdown is convenient when I churn out problems or assist people who do not know LaTeX in preparing simple contests. That it lacks all the options of LaTeX is IMO fine.

…

- For images, one way would be to do it directly via the latex syntax: $\illustration{0.3}{image.jpg}{CC-BY}$. (Ah, but you're probably doing the md->html rendering directly, so maybe it's better to somehow parse the standard markdown image inclusion separately for html/pdf rendering. Sounds non-trivial though.) The plan is to add a post-processor for img tags that fixes links to images pointing to an image file. Makes sense. CC @niemela <https://github.com/niemela>, who has also been wanting/pushing this. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#191 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHIMXOAPDS5CSDA5UJLMK3TEE7V7ANCNFSM4ZLB3UGQ> .

jsannemo · 2021-03-18T00:16:57Z

On Thu, 18 Mar 2021, 01:11 Ragnar Groot Koerkamp, ***@***.***> wrote: For reference, GitLab uses: $`abc`$ for inline maths (which I don't like so much)

The advantage is that it is simpler to preprocess for extensionless markdown implementations since you can care less about avoiding math delimiters in `` blocks (something which \( also suffers from).

andd ```math abc for display math, which is quite nice actually, although I'm not sure whether pandoc does this. Actually, if we're doing maths, we should definitely support both inline and display maths. Definitely. I started with inline since it felt like a smaller syntax to

decide on. I like ```math too.

…

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#191 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHIMXOXDGTO7UJPW5ARCW3TEFAJRANCNFSM4ZLB3UGQ> .

pehrsoderman

First of all, this is a feature that I will love to get integrated with problemtools!

I gave this a readthrough and gave some general comments regarding style (problemtools isn't quite updated to the style we strive for in the rest of Kattis, but at least type hints and doc strings will make it a bit nicer).

I think it would be nice if @austrin could have a look at it. Also, I think getting the mathmode right is kind of important.

pehrsoderman · 2021-03-23T22:59:47Z

problemtools/templates/markdown/default-layout.html

+});
+</script>
+<script type="text/javascript"
+        src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_HTMLorMML">


Hard-coding links to a CDN is a bit of a maintenance nightmare. This needs to be rethought.

pehrsoderman · 2021-03-23T23:00:34Z

problemtools/tex2html.py

@@ -0,0 +1,65 @@
+#! /usr/bin/env python3


This file lacks documentation and type hints. It will save us headaches later if that is added.

pehrsoderman · 2021-03-23T23:01:47Z

problemtools/problem2pdf.py

@@ -8,13 +8,16 @@
 from . import template


-def convert(problem, options=None):
+def convert(problem, options=None, ignore_markdown=False):


Add documentation and type hints.

pehrsoderman · 2021-03-23T23:05:23Z

problemtools/problem2html.py

+            if not os.path.isdir(destdir):
+                os.makedirs(destdir)
+
+        if md2html.get_markdown_statement(problem, options.language):


A warning if both types are present would be nice. Hidden priorities like here are often an annoyance.

pehrsoderman · 2021-03-23T23:07:31Z

problemtools/md2html.py

@@ -0,0 +1,146 @@
+#! /usr/bin/env python3


This file lacks documentation and type hints.

pehrsoderman · 2021-03-23T23:11:40Z

examples/different/problem_statement/problem.sv.md

+Skriv ett program som beräknar differensen mellan icke-negativa heltal.
+
+## Indata
+Indatan består av flera testfall (minst \(1\) och som mest \(40\)), ett per rad.


I am not sure I am overly fond of the $ and $ syntax for math mode in markdown.

pehrsoderman · 2021-04-26T14:37:56Z

@jsannemo Any progress here?

pehrsoderman · 2023-05-01T13:01:10Z

@jsannemo Any progress? Also, see comments above...

Add support for markdown statements.

591f24e

jsannemo requested review from ghamerly and pehrsoderman March 17, 2021 17:49

Remove forgotten refactored thing

7c5bb53

jsannemo assigned ghamerly and pehrsoderman Mar 17, 2021

jsannemo added the enhancement label Mar 17, 2021

pehrsoderman requested changes Mar 23, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for markdown statements. #191

Add support for markdown statements. #191

jsannemo commented Mar 17, 2021 •

edited

Loading

RagnarGrootKoerkamp commented Mar 17, 2021 •

edited

Loading

jsannemo commented Mar 17, 2021 •

edited

Loading

RagnarGrootKoerkamp commented Mar 18, 2021 •

edited

Loading

RagnarGrootKoerkamp commented Mar 18, 2021 •

edited

Loading

jsannemo commented Mar 18, 2021 via email

jsannemo commented Mar 18, 2021 via email

pehrsoderman left a comment

pehrsoderman Mar 23, 2021

pehrsoderman Mar 23, 2021

pehrsoderman Mar 23, 2021

pehrsoderman Mar 23, 2021

pehrsoderman Mar 23, 2021

pehrsoderman Mar 23, 2021

pehrsoderman commented Apr 26, 2021

pehrsoderman commented May 1, 2023

Add support for markdown statements. #191

Are you sure you want to change the base?

Add support for markdown statements. #191

Conversation

jsannemo commented Mar 17, 2021 • edited Loading

RagnarGrootKoerkamp commented Mar 17, 2021 • edited Loading

jsannemo commented Mar 17, 2021 • edited Loading

RagnarGrootKoerkamp commented Mar 18, 2021 • edited Loading

RagnarGrootKoerkamp commented Mar 18, 2021 • edited Loading

jsannemo commented Mar 18, 2021 via email

jsannemo commented Mar 18, 2021 via email

pehrsoderman left a comment

Choose a reason for hiding this comment

pehrsoderman Mar 23, 2021

Choose a reason for hiding this comment

pehrsoderman Mar 23, 2021

Choose a reason for hiding this comment

pehrsoderman Mar 23, 2021

Choose a reason for hiding this comment

pehrsoderman Mar 23, 2021

Choose a reason for hiding this comment

pehrsoderman Mar 23, 2021

Choose a reason for hiding this comment

pehrsoderman Mar 23, 2021

Choose a reason for hiding this comment

pehrsoderman commented Apr 26, 2021

pehrsoderman commented May 1, 2023

jsannemo commented Mar 17, 2021 •

edited

Loading

RagnarGrootKoerkamp commented Mar 17, 2021 •

edited

Loading

jsannemo commented Mar 17, 2021 •

edited

Loading

RagnarGrootKoerkamp commented Mar 18, 2021 •

edited

Loading

RagnarGrootKoerkamp commented Mar 18, 2021 •

edited

Loading