Allow other lifetimes in tokens

[414owen]

At the moment, I think Logos forbids having more than one lifetime parameter on a token type, and assumes that the lifetime, if present, is that of the input.

This isn't a fair assumption. I actually have things I want to arena-allocate inside the payload of my tokens, so I need to be able to use the lifetime of the arena.

[jeertmans]

Hi @414owen, it's true that it is not a smart constraint.

A workaround might be to require the user to explicitly mention which lifetime refers to the source, if multiple lifetimes are passed.

What do you think?

E.g.,

// OK - defaults to 'source
#[derive(Logos)]
enum<'source> Token<'source> {
}

// OK - 'other lifetime has nothing to do with source
#[derive(Logos)]
#[logos(lifetime = None)]
enum<'other> Token {
}

// OK - defaults to 'source
#[derive(Logos)]
enum<'source> Token<'source> {
}

// OK
#[derive(Logos)]
#[logos(lifetime = 'source)]
enum<'source, 'other> Token {
}

// NOT OK - emits an error saying the source lifetime must be explicitly specified
#[derive(Logos)]
enum<'source, 'other> Token {
}

// NOT OK - emits an error saying the specified lifetime was not found
#[derive(Logos)]
#[logos(lifetime = 'source)]
enum<'other_source, 'other> Token {
}

// NOT OK - emits an error saying the specified lifetime was not found
#[derive(Logos)]
#[logos(lifetime = 'source)]
enum Token {
}

The second example is to provide backward compatibility with existing code.

[414owen]

@jeertmans That API looks really nice! I'd love to see this in logos.

[jeertmans]

I don't have time at the moment to work on this, so I'll label this as good first issue, because I think it should be relatively easy to implement.