How do you retain positioning information? #88

NullVoxPopuli · 2022-01-22T03:19:22Z

NullVoxPopuli
Jan 22, 2022

The act of parsing, and the stringifying html totally messes up the document (makes it impossible to use rehype for codemods)

For example, https://jsbin.com/selazix/edit?html,output

  let transformed = await unified()
    .use(rehypeParse)
    .use(rehypeStringify)
    .process(sourceFile);

  let result = transformed.value;

  console.log(result);

before:

const sourceFile = `<!DOCTYPE html>
<html>
<head>
  <title>Test</title>
  
  {{content-for "head"}}
  
  <link href="...">
  <link href="...">
  <link href="...">
</head>
<body>
  {{content-for "body"}}

</body>
</html>
`;

after:

<!doctype html><html><head>
  <title>Test</title>
  
  </head><body>{{content-for "head"}}
  
  <link href="...">
  <link href="...">
  <link href="...">


  {{content-for "body"}}



</body></html>

casing isn't even maintained

Answered by ChristianMurphy

Jan 23, 2022

Hey @NullVoxPopuli! Sorry you ran into a spot of trouble.
This sounds a lot like remarkjs/remark#922
Here you are using an HTML parser, to parse an HTML-like (but not HTML) templating language.

The handlebars like syntax ({{content-for "head"}}) is throwing off the interpretation.
pre-processing the text before handing off to rehype could work, or rehype could be customized to parse the handlebars-like-syntax.

View full answer

wooorm · 2022-01-22T10:39:47Z

wooorm
Jan 22, 2022
Maintainer

totally messes up the document (makes it impossible to use rehype for codemods)

Prettier and babel formats markdown, HTML, JavaScript, etc. unified does the same. This is intentional. It’s how ASTs work.

ESLint doesn’t change anything normally, because it doesn’t format things.
You can do the same as ESLint with unified if you want to.

3 replies

NullVoxPopuli Jan 22, 2022
Author

You can do the same as ESLint with unified if you want to.

How?

wooorm Jan 22, 2022
Maintainer

By taking the original document, combining it with positional info on nodes, and then slicing the document

NullVoxPopuli Jan 22, 2022
Author

right, I understand conceptually, it's the execution and integration with rehype that I'm struggling with.

in the mean time, I'm going to use posthtml, as it seems better suited for codemods on userland code

ChristianMurphy · 2022-01-23T18:36:56Z

ChristianMurphy
Jan 23, 2022
Maintainer

Hey @NullVoxPopuli! Sorry you ran into a spot of trouble.
This sounds a lot like remarkjs/remark#922
Here you are using an HTML parser, to parse an HTML-like (but not HTML) templating language.

The handlebars like syntax ({{content-for "head"}}) is throwing off the interpretation.
pre-processing the text before handing off to rehype could work, or rehype could be customized to parse the handlebars-like-syntax.

4 replies

NullVoxPopuli Jan 23, 2022
Author

This sounds a lot like remarkjs/remark#922

nay, it has nothing to do with that.
I'm building a generic preset / apply tool with some high-level apis:

  await html.insertText('app/index.html', {
    text: `<link integrity="" rel="stylesheet" href="{{rootURL}}assets/tailwind.css">\n`,
    beforeFirst: 'link',
  });

which, with posthtml, looks like this

/**
 * Transform HTML
 *
 * Uses [posthtml](https://posthtml.org/#/) for transformation.
 * See: [posthtml plugins](https://github.com/posthtml/posthtml-plugins/blob/master/src/pluginsNames.json) for examples.
 *
 * @example
 * the `before` option is to insert HTML before the first tag specified -- `link` in this case.
 *
 * ```js
 * import { html } from 'ember-apply';
 *
 * await html.transform(filePath, (tree) => {
 *   tree.match(({ tag: 'head' }, 'head') => {
 *     if (node.content) {
 *       node.content.unshift({ tag: 'noscript', content: 'Please enable javascript' });
 *     }
 *
 *     return node;
 *   });
 * });
 * ```
 *
 * Example borrowed from posthtml-noscript
 *
 * @param {string} filePath
 * @param {Plugin | Plugin[]} plugin
 */
export async function transform(filePath, plugin) {
  let code = (await fs.readFile(filePath)).toString();

  let plugins = Array.isArray(plugin) ? plugin : [plugin];
  let result = await posthtml(plugins).process(code);

  await fs.writeFile(filePath, result.html);
}

/**
 * Add a string of HTML to an HTML document.
 *
 * @example
 * the `before` option is to insert HTML before the first tag specified -- `link` in this case.
 * ```js
 * import { html } from 'ember-apply';
 *
 * await html.inserText('app/index.html', {
 *   text: `<link integrity="" rel="stylesheet" href="{{rootURL}}assets/tailwind.css">`,
 *   beforeFirst: 'link',
 * });
 * ```
 *
 * @typedef {object} InsertBeforeFirstOptions
 * @property {string} beforeFirst the HTML tag to insert `text` before
 * @property {string} text the HTML to inject before the first tag matching `tag`
 *
 * @param {string} filePath
 * @param {InsertBeforeFirstOptions} options
 */
export async function insertText(filePath, { beforeFirst, text }) {
  let code = (await fs.readFile(filePath)).toString();

  if (code.includes(text)) {
    return;
  }

  /**
   * @param {string} tag
   * @param {string} text
   */
  function insertBeforeFirst(tag, text) {
    /** @type {Plugin} */
    return (tree) => {
      let found = false;
      /** @type {Node} */
      let previousNode;

      // Types are wrong for Node. tag: false is how you say "array of any content"
      // @ts-expect-error
      tree.walk((node) => {
        if (!found && node.tag === tag) {
          found = true;

          if (typeof previousNode === 'string') {
            let parts = previousNode.split('\n');
            let indentation = parts[parts.length - 1];

            return { tag: false, content: [text, indentation, node], attrs: {} };
          }

          return { tag: false, content: [text, node], attrs: {} };
        }

        previousNode = node;

        return node;
      });

      return tree;
    };
  }

  return await transform(filePath, insertBeforeFirst(beforeFirst, text));
}

Here you are using an HTML parser, to parse an HTML-like (but not HTML) templating language.
The handlebars like syntax ({{content-for "head"}}) is throwing off the interpretation.

imo, it's irrelevant, and shouldn't matter because that should be interpreted as plain text, due to being in the body of the body tag.

ChristianMurphy Jan 23, 2022
Maintainer

imo, it's irrelevant, and shouldn't matter because that should be interpreted as plain text

It is relevant, parse5 and by extension rehype applies HTML semantics.
{{content-for "head"}} is interpreted as plain text, but it is plain text which appears where plain text cannot in a valid document (in the <head>).
Invalid documents will be handled the same as they would by a browser, for example the same document run through document.write(), will produce the same output rehype does. https://codesandbox.io/s/html-handling-j8sme

NullVoxPopuli Jan 23, 2022
Author

gotchya -- I didn't know that, thanks! Guess posthtml is very forgiving then
but also, that's really annoying. 🤔 lol

I have some in-progress rehype/remark work on a custom parser over here: NullVoxPopuli/glimdown#2
But, it's slow progress, because all the examples I'm looking are ridiculously split apart, so it's hard to casually trace all the packages (I'm also not doing "heads down" time to learn it all, cause it's just side-project territory atm)

ChristianMurphy Jan 23, 2022
Maintainer

but also, that's really annoying

It can be,
and HTML recovery modes can be confusing, I've gotten tripped up a few times as well (a recent example being inikulin/parse5#387)

Guess posthtml is very forgiving then

It handles error recovery different.
Even at the tree level https://astexplorer.net/#/gist/49d3abb211938d9f39cd1c881ff5a9d8/20ba9329373a08c0866d4fc50287e4a4d5e3d09b compared to https://astexplorer.net/#/gist/49d3abb211938d9f39cd1c881ff5a9d8/130beea91c7bbf001c038302a0e48a3a0dde3427
Another option, if you want to leverage the rehype ecosystem, but not have parse5 as the parser, would be to bridge posthtml into https://github.com/syntax-tree/hast
Similar to how https://github.com/syntax-tree/hast-util-from-parse5 bridges from parse5 (this is the default rehype parser) and https://github.com/syntax-tree/hast-util-from-dom does the same thing, but with the DOM APIs.

I have some in-progress rehype/remark work on a custom parser over here

Nice!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rehype

How do you retain positioning information? #88

{{title}}

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

rehype

How do you retain positioning information? #88

NullVoxPopuli Jan 22, 2022

Replies: 2 comments · 7 replies

wooorm Jan 22, 2022 Maintainer

NullVoxPopuli Jan 22, 2022 Author

wooorm Jan 22, 2022 Maintainer

NullVoxPopuli Jan 22, 2022 Author

ChristianMurphy Jan 23, 2022 Maintainer

NullVoxPopuli Jan 23, 2022 Author

ChristianMurphy Jan 23, 2022 Maintainer

NullVoxPopuli Jan 23, 2022 Author

ChristianMurphy Jan 23, 2022 Maintainer

NullVoxPopuli
Jan 22, 2022

Replies: 2 comments 7 replies

wooorm
Jan 22, 2022
Maintainer

NullVoxPopuli Jan 22, 2022
Author

wooorm Jan 22, 2022
Maintainer

NullVoxPopuli Jan 22, 2022
Author

ChristianMurphy
Jan 23, 2022
Maintainer

NullVoxPopuli Jan 23, 2022
Author

ChristianMurphy Jan 23, 2022
Maintainer

NullVoxPopuli Jan 23, 2022
Author

ChristianMurphy Jan 23, 2022
Maintainer