tranformers notes

tranformers - attention is all you need

Example - German -> English Translation Model

Encoder
Decoder

Say we have a german sentence :

Wie geht es dir

English :

How are you

Because neural networks don't understand text, we need to convert it to numbers or tensors.

first we convert the sentence into smaller sequences -> tokens

Each token is converted into a word embedding that captures its meaning.

Positional encodings keeps track of the meaning of the encoding.