Choice lists are only included in the output if they are referenced in a select.

[lognaturel]

Software and hardware versions

pyxform v1.12.0

Problem description

Currently, pyxform only includes choice lists used in selects in its output.

Steps to reproduce the problem

Convert a form that has multiple choice lists but only one of them used in a select (example, remove the calculate).

Expected behavior

My ideal would be to include lists that are referenced in an instance expression (see example above) and to output a warning when a choice list is ignored. I do think it's reasonable to omit choice lists that really aren't used -- I've seen folks who have a standard choices sheet with many choices that they use. The problem now is that there's a way to use choice lists that's not detected by pyxform.

[lindsay-stevens]

Relevant forum discussion: workflow for randomisation requires hidden reference to choice list.

Per the design principles we have "user needs" in both cases, then:

1. Power and simplicity: remove extra step(s) to get indirectly unreferenced list data into the XForm to enable workflows, vs.
2. Prioritise performance: users should avoid adding too much extraneous XLSForm list data because it might slow down form load times.

On the second point, if there is test data on how big (in MB or choice items) you have to go before there's a noticeable impact on form performance, that could be helpful info for the docs, or it could be written into pyxform as a warning.

[tiritea]

FWIW, I dont think pyxform should (optimistically?) guess as to what the form designer does/not want in their form; rather, if it has been explicitly specified - in the XLSForm - then the default assumption should be that it appropriately appear in the ultimate generated XForm definition (unless explicitly specified otherwise). Notwithstanding appropriate warnings from pyxform on elements which have been identified as either redundant or could potentially significantly impact deployed performance; eg image size

[lognaturel]

remove extra step(s) to get indirectly unreferenced list data into the XForm to enable workflows

I think that's pretty compelling!

I dont think pyxform should (optimistically?) guess as to what the form designer does/not want in their form

In this case it could make the correct decision, it's just not fully implemented. But still, I agree it generally makes sense to treat the XLSForm as another view on the XForm.

because it might slow down form load times.

I think the bigger concern is form size and transfer time on poor connections. Form load time being high is a slight inconvenience but you have to get to 100s of thousands of rows before it really makes a difference on modern devices.

Shall we move forward? I don't think we need to specifically solicit further feedback on the forum unless one of you does. This is unlikely to affect very many people.

[lognaturel]

We've seen a couple of users run into this recently -- they wanted to use a list to look things up and the list wasn't included in the form definition.

[tiritea]

BTW, its not really a blocking issue because - as I do - you can just add a single select/multi question and make it permanently hidden with relevant=false. So in practice its more just an annoyance than anything, especially if you forget to do this and end up with errors in your resulting XForm.

[lognaturel]

Yes, absolutely! But for folks who don't know about this issue it's really hard to go from the error message they see to including a select. I also just saw a case where there were 7 lists to query and it felt very annoying to have to specify 7 non-relevant selects. I ended up recommending combining the lists into one with a column to separate out the 7 "sub-lists".

[tiritea]

Agree. It's really rather difficult to be 100% sure when something is/not 'used' in XForm, due to the eccentricities you can get away with with XPath [I'm more guilty than most... :-) ]. Hence my leaning to if its in the XLSForm then just blanket include it in the XForm, perhaps with a warning if we (pyxform) think it might not be used. That is, dont optimize the translation unless we are 110% sure it doesn't break something.