Positional Encodings of Attention Mechanisms: Sinusoidal, Rotary Positional Encodings (Part 2)
Towards a principled approach to encode the ‘position’ of a sequence. Absolute and Relative.
Recent Posts
Positional Encodings of Attention Mechanisms: Weighted Sums and Attention(Part 1)
How do you design a model such that, despite the limited context of weighted sums, manages to capture the key properties of the given data?