Iterator-based SeqObject

[bertptrs]

Since version 0.27 this crate provides a method to adapt an arbitrary object into a conceptual sequence, using the SeqObject trait which can then be used to iterate over. This can then be turned into a Value and used in the template.

One thing it doesn't allow you to do, is iterate over objects in a streaming fashion. The requirement of random access, baked into the trait, invalidates doing as much. You have to know ahead of time how many objects you have, and you have to have random access on the underlying object.

In Python on the other hand, it's possible to use generators to lazily load values and iterate over them as you go.

This mainly comes into play when writing multi-gigabyte files based on millions of iterable items.

Something like Value::from_iterator(val: impl<Iterator> + 'static) would solve this. Not to be confused with the current FromIterator impl, which materializes the list into a Vec. This is explicitly undesirable.

[rkrzr]

To add some context to what @bertptrs said:

jinja2 has a template.stream(...) method which can take an iterator of items and then return a TemplateStream. Example:

 generated_stream = jj2_env.from_string(some_template).stream(
      items=item_generator, **new_template_context
  )

This is very useful when working with a large number of items that need to be rendered, since they don't all need to be loaded into memory first. Is there a way to achieve this in minijinja as well?

We looked at SeqObject to replicate this, however it requires random access in its get_item method:

fn get_item(&self, idx: usize) -> Option<Value>;

[mitsuhiko]

This seems to conflate two very different problems. One is iterating over something that has no known length, the other is streaming a large template. The latter (eg: what stream does in Jinja2 on the template) you can do today just fine by using render_to_write.

Iterators in MiniJinja don't really exist, they are just vectors or sequences internally. Since there is no other loop equivalent it's quite likely it won't be possible to accomplish this without abusing recursion. I will look into what it takes to add iterator support but it's not something that is super trivial.

[mitsuhiko]

I looked into this a bit now. Today the iterator types you can get from Value are all ExactSizeIterator. That is also leverage by the iterator system for calculating the reverse indexes. Unfortunately also ValueKind is exhaustive which means that an Iterator type cannot really be added. This might require backwards incompatible changes.

Maybe this is something to look at together with #400

[mitsuhiko]

In order for this to be changed it most likely will either require some hacks or at least to do the changes here: #418

[mitsuhiko]

Turns out part of this is fixable even without a major. I hacked something up in #426. Would be curious to know if this works for you. You can basically now pass Value::from_iterator(0..10) or whatever you desire to the template.

[rkrzr]

Great, thanks for looking into this so quickly. We'll give this a try next week and will let you know if it works for us.

[rkrzr]

We have now tried this for one use case and I'm happy to report that it's working for us. Thanks for adding support for this!