Quarto does not preserve the render order mentioned in _quarto.yaml

[AliSajid]

Bug description

Previously, under the 1.4 release, it was possible to control the order of execution of the qmd files by specifying them as an array in the metadata.

This is no longer possible with the versions going from 1.5 onwards. Controlling order of execution is important as evidenced by issues like #10463 , #10490, and this discussion.

Steps to reproduce

https://github.com/maxdrohde/quarto_render_order

Expected behavior

Specifying the order should render documents in order.

Actual behavior

The files are executed in arbitrary order.

Your environment

• IDE: Neovim
• • NVIM v0.10.1
  • Build type: Release
  • LuaJIT 2.1.1724512491
• OS: macos Sonoma 14.3.1

Quarto check output

Quarto 1.5.557
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.11: OK
      Dart Sass version 1.69.5: OK
      Deno version 1.37.2: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.5.57
      Path: /Applications/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: (external install)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: TinyTex
      Path: /Users/aimami/Library/TinyTeX/bin/universal-darwin
      Version: 2022

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.12.5
      Path: /Users/aimami/.local/share/mise/installs/python/3.12/bin/python3
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[✓] Checking R installation...........OK
      Version: 4.4.1
      Path: /opt/homebrew/Cellar/r/4.4.1/lib/R
      LibPaths:
        - /Users/aimami/Library/Caches/org.R-project.R/R/renv/library/drugfindR-edc455ab/macos/R-4.4/aarch64-apple-darwin23.4.0
        - /Users/aimami/Library/Caches/org.R-project.R/R/renv/sandbox/macos/R-4.4/aarch64-apple-darwin23.4.0/8b0c8122
      knitr: 1.47
      rmarkdown: 2.26

[✓] Checking Knitr engine render......OK

[cscheid]

Sorry, but we do need a full repro.

Change the order specified in #3 and run it again.

I don't understand what this pull request has to do with the report.

Can you please provide us with a git repository that we can clone from, where you see this behavior? Thank you.

[maxdrohde]

@cscheid I provided an example using my GitHub repository in my issue #10490. Thanks again for looking into this!

[cscheid]

Can you give us a small repo?

[maxdrohde]

I will try again. I tried making one before I submitted the issue with only a couple .qmd files but I couldn't get it to reproduce the ordering issue.

[cscheid]

Thanks!

[maxdrohde]

@cscheid I just figured it out, please see here: https://github.com/maxdrohde/quarto_render_order

When I run quarto render from the terminal, I see

❯ quarto render
[1/2] 2-file2.qmd
[2/2] 1-file1.qmd


processing file: 1-file1.qmd
1/3
2/3 [unnamed-chunk-1]
3/3
output file: 1-file1.knit.md


Output created: _output/1-file1.html

but the _quarto_yml is

project:
  title: "Ordering Test"
  output-dir: _output
  render:
    - 1-file1.qmd
    - 2-file2.qmd
author: "Maximilian Rohde"
date: today
format:
  html:
    toc: true

[cscheid]

Thank you, I can repro that!

[cscheid]

The issue is somehow related to engine determination. If you remove the code cell from 1-file1.qmd, the order changes.

[AliSajid]

Thank you @maxdrohde. @cscheid I meant to say repeat step 3, which github autolinked to Issue 3. Apologies.

[AliSajid]

Is there any update on this issue? This is a major blocker on a part of our pipeline and is holding up publication of multiple papers. We'd appreciate any work on this.

[cderv]

I believe the regression was introduced in

• ProjectContext - take responsibility over engine determination and markdown resolution #8787

specifically at commit d75246d

This change modified projectContext logic and introduced a Promise.all() which makes the resulting order different than input order. So order can change after engine detection. Which is what you observe @cscheid.

quarto-cli/src/project/project-context.ts

Lines 799 to 817 in 52d5a6e

	const renderFiles = metadata?.project[kProjectRender];
	if (renderFiles) {
	const exclude = projIgnoreGlobs.concat(outputDir ? [outputDir] : []);
	const resolved = resolvePathGlobs(dir, renderFiles, exclude, {
	mode: "auto",
	});
	await Promise.all(
	(ld.difference(resolved.include, resolved.exclude) as string[])
	.map((file) => {
	if (Deno.statSync(file).isDirectory) {
	return addDir(file);
	} else {
	return addFile(file);
	}
	}),
	);
	} else {
	await addDir(dir);
	}

I can confirm that Quarto 1.5.16 before PR is merged does not have the issue, and git bisect confirms it. This is the root cause for several issues

• Can't control the render order on GitHub Actions ubuntu-latest #10463
• render --to pdf and render --to docx has image path problems while render --to html works #11039
• New version of Quarto not respecting render order #10490

[cscheid]

This is a major blocker on a part of our pipeline and is holding up publication of multiple papers.

Hi. I just want to mildly suggest that you could engage in our discussions in a more productive way - specifically, the implication that our bugs are the cause of your work being hindered is really not helpful. For context, I was an academic researcher and professor for more than a decade (and have a number of papers to my name), and one lesson that was imparted in me (and I on my students) is that it pays to be conservative about your tooling. We take this seriously, too, and older releases of Quarto are always available to download and use, and you have a number of options in how to use them, including using GitHub actions that allow you to select any Quarto version.

In the worst case, for example, you could convert your manuscript to a single .qmd file and then ordering would not be an issue.

We hear you, and we will fix this (as cderv has pointed out here and has a PR open). But do try to be more considerate when engaging in discussions. Thank you.

[cscheid]

This is closed by #11062 and will be in 1.5 and 1.6 releases soon.

[AliSajid]

Thank you so much. We really appreciate all the work you do.