Accessing decoder only model's hidden states with Executor API

[tkhanipov]

Hello,

We have a decoder only Llama-like model and we need to access its hidden states. Currently for this purpose we patch both the Python LLaMAModel and the C++ GptSession classes. In the former we do hidden_states.mark_output(...) to make these visible and in the latter we access them with outputBuffer.at(...) here.

We would like to timely switch to the new Executor API (with the current API announced to be deprecated) but cannot find the proper way to achieve what we want there.

What would be the easiest way to access the hidden states in the new API? Is it possible to do within something like executorExampleBasic?

The returnEncoderOutput setting looked promising but it does not seem to work for a decoder only model (as expected from the comment): Result::encoderOutput is empty.