Mathematical Foundations of LLMs in the "Knowledge Cartography" Diagram

The mathematical equations in the diagram represent the foundational mechanics of Transformer-based Large Language Models (LLMs), focusing on attention mechanisms, vector embeddings, and layer transformations.

Scaled Dot-Product Attention

Near the bottom of the top circle, the formula is rendered artistically as $Q = \sigma\left(\frac{QK^T}{\sqrt{d_s}}\right) / \sqrt{d_k}$. This directly mimics the canonical Scaled Dot-Product Attention equation:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

• Query ($Q$), Key ($K$), and Value ($V$): Matrices created by multiplying input token representations by learned weight matrices. $Q$ represents the token seeking context, $K$ acts as the index to match against, and $V$ contains the contextual content.
• Dot Product ($QK^T$): Calculates pairwise similarity scores between every token in an input sequence to determine how much "attention" tokens pay to one another.
• Scaling ($\sqrt{d_k}$): Divides the dot product by the square root of the key dimension size ($d_k$). This prevents variance from blowing up in high dimensions, which would otherwise push the softmax activation ($\sigma$) into regions with vanishing gradients.

Embedding Vector Space ($x = \vec{v} - t$)

The lower circle (Embedding Vector Space) and the equation $x = (\vec{v} - t)$ represent vector geometry.

• LLM Connection: Before processing text, LLMs map discrete words into high-dimensional continuous vector spaces ($\mathbb{R}^d$). Semantic relationships are captured as spatial distances and directional vectors (e.g., conceptual vector arithmetic like $\vec{v}_{\text{king}} - \vec{v}_{\text{man}} + \vec{v}_{\text{woman}} \approx \vec{v}_{\text{queen}}$). The 3D grid illustrates how words like "Planet" and "Word" occupy distinct geometric coordinates.

Linear Transformations & Layer Normalization ($A = \frac{u}{b} + \frac{c}{c}$, $\bar{A} = zv + th$)

The upper equations display stylized linear combinations, scaling ratios, and variable offsets.

• LLM Connection: Modern LLMs consist of stacked hidden layers (as shown in the Layer Schematic on the left). Passing data through these layers requires continuous linear projections ($W \cdot x + b$), feed-forward transformations, and normalization layers (LayerNorm/RMSNorm) to stabilize signal flow across deep networks.