Epic: single source of truth for document/project information

[cscheid]

A number of bugs in Quarto happen because different parts of the code are responsible for reasoning about a .qmd document. We should have a single source of truth on this, and that should be what drives the output of, for example,quarto inspect.

This is related (but simpler) to our Execution Planner, and solving this will make it easier to do the planner well.

• Resource files are not watched in preview mode #3185
• engine: should be respected globally at the _quarto.yml level #3157
• Image resources not enumerated in inspect output #3053
• communicate format output to all engines  #2216
• using resources: field in _quarto.yml #3766
• shortcode meta author fails in any file other than index.qmd when rendering a book to html, works everywhere for PDF #8675
• params should work with metadata-files #3663
• quarto should report found extensions #3510
• quarto inspect needs to use the same sources of truth as quarto render #8563

Resolution of CLI parameters into metadata or pandoc args

• Wrong arg forwarding from preview to render #8591

format configuration

• Images not following global figure alignment unless has caption #4415

include shortcode resolution

• include shortcodes must be resolved earlier than they are #8779

See also:

• Preview - improve format detection #8702
• partitionedMarkdownForInput and projectIndex should be direct responsibilities of ProjectContext.

[jkrumbiegel]

I see there's been some work in this direction, with the addition of the ProjectContext in #8771. For our users, both not being able to specify engine: julia in the _quarto.yml and not being able to use .jl script files specifying engine: julia are still annoying roadblocks that I'd like to remove. @cscheid could you tell me if you think this can be meaningfully done by an outside contributor, or are the necessary changes too intertwined with your larger refactoring plans for the code base? If it's possible, it would be great to get a rough idea from you which parts are missing.

I was looking at the fileExecutionEngine function. In principle, the logic seems relatively straightforward. If an engine is specified in either frontmatter or project, this should take precedence over "file claiming". An engine specified in the frontmatter should take precedence over one specified in the project. With your new ProjectContext, the information needed is already there. However, markdownExecutionEngine currently expects file claiming to have run before it, and I'm not sure which way this conflict should be resolved.

[cderv]

not being able to specify engine: julia in the _quarto.yml

For reference, project wide engine setting is tracked at:

• engine: should be respected globally at the _quarto.yml level #3157

This is a limitation across engine currently I believe.

being able to use .jl script files specifying engine: julia

For script file, we started discussion at

• Script files not picked up by julia engine #10034

Is this the same topic ?

I am just cross-referencing to keep track of related topics so that we can all follow.

[jkrumbiegel]

Yes, thanks, should've just linked those again directly.

[cscheid]

@jkrumbiegel Yes, I definitely want to fix this. We're nearing a 1.6 release and won't have time to do this between now and then (we plan to release early November). But I would like to spend some time on this in 1.7. Could we revisit this then?

I hope to be able to split the work such that some of it can be tackled by the engine code directly (and so you should be able to implement it in the pure-Julia engine).

[jkrumbiegel]

Ok sure, if there's nothing I could do now, let's revisit it in November. But I could also get started now if I had a rough outline how to approach these two issues. Whatever works better for you :)

[cscheid]

For our users, both not being able to specify engine: julia in the _quarto.yml

I think this one will be harder for you to fix, because it involves the code where we bind engines/"targets" to documents. It is pretty messy and mostly outside of an engine's responsibilities.

and not being able to use .jl script files specifying engine: julia are still annoying roadblocks that I'd like to remove.

I think this could be easier (barring unexpected snags in our codebase). I think this one is entirely a matter of implementing claimsFile in the Julia engine. For contrast with the knitr and jupyter engines, see:

quarto-cli/src/execute/rmd.ts

Lines 68 to 71 in 8eac0b5

	claimsFile: (file: string, ext: string) => {
	return kRmdExtensions.includes(ext.toLowerCase()) ||
	isKnitrSpinScript(file);
	},

quarto-cli/src/execute/jupyter/jupyter.ts

Lines 148 to 151 in 0097478

	claimsFile: (file: string, ext: string) => {
	return kJupyterNotebookExtensions.includes(ext.toLowerCase()) ||
	isJupyterPercentScript(file);
	},

While the julia engine has:

quarto-cli/src/execute/julia.ts

Line 70 in 9f4e188

claimsFile: (file: string, ext: string) => false,

When an engine claims a file, then it's also responsible for providing the converted markdown for that file. See eg the jupyter engine code:

quarto-cli/src/execute/jupyter/jupyter.ts

Lines 162 to 165 in 0097478

	} else if (isJupyterPercentScript(file)) {
	return Promise.resolve(
	asMappedString(markdownFromJupyterPercentScript(file)),
	);

[jkrumbiegel]

Hm are you saying I should check within claimsFile whether the percent script has a engine: julia setting? Because otherwise I can't claim jl files without it being a breaking change. That's why I didn't do it at the time.

To me it seems like a logic bug that engines can claim file types which happens before an engine: whatever setting can even be considered. Shouldn't it be the other way around, check for engine settings otherwise fall back to file type association?

[cscheid]

Because otherwise I can't claim jl files without it being a breaking change. That's why I didn't do it at the time.

I only now see that jupyter.ts claims .jl percent scripts. That does complicate things quite a bit. I now think that this is also a November conversation. Should we put something on the calendar so that we can have a discussion about it?

[jkrumbiegel]

That would be nice. Should I send you an email?

[cscheid]

Yes, please: [email protected]