Is the causal attention really works here?

[charlesxu90]

Found out that Efficient attention doesn't work on Causal attention scenario as mentioned here.
cmsflash/efficient-attention#4

So I doubt if the causal in this code really works?

[lucidrains]

@charlesxu90 yea it works

[charlesxu90]

@lucidrains Thanks for answering me. Really appreciated!

Causal self-attention requires a triangle-like attention mask to mask out future tokens. In this code, I did find the interface you left for input_mask.

However, I didn't find the spot you initiate the attention mask. That's something confused me.

Thanks.