| 1 | <?xml version='1.0' encoding='UTF-8'?> |
| 2 | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 720" width="1280" height="720"> |
| 3 | <defs> |
| 4 | <filter id="figFrame" x="-5%" y="-5%" width="110%" height="110%"> |
| 5 | <feGaussianBlur in="SourceAlpha" stdDeviation="6" /> |
| 6 | <feOffset dx="0" dy="3" /> |
| 7 | <feFlood flood-color="#1A365D" flood-opacity="0.08" /> |
| 8 | <feComposite in2="SourceAlpha" operator="in" /> |
| 9 | <feMerge><feMergeNode /><feMergeNode in="SourceGraphic" /></feMerge> |
| 10 | </filter> |
| 11 | </defs> |
| 12 | |
| 13 | <g id="bg"> |
| 14 | <rect width="1280" height="720" fill="#FFFFFF" /> |
| 15 | </g> |
| 16 | |
| 17 | <g id="header"> |
| 18 | <rect x="60" y="56" width="6" height="40" fill="#3182CE" /> |
| 19 | <text x="84" y="88" font-family="Georgia, "Microsoft YaHei", serif" font-size="36" font-weight="700" fill="#1A365D">h parallel projections, then concatenate</text> |
| 20 | <text x="84" y="118" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#4A5568" letter-spacing="1.5">MULTI-HEAD ATTENTION · §3.2.2 · h = 8 · d_k = d_v = 64</text> |
| 21 | </g> |
| 22 | |
| 23 | <g id="formula-band"> |
| 24 | <rect x="60" y="160" width="1160" height="124" rx="12" fill="#F5F7FA" stroke="#E2E8F0" stroke-width="1.5" /> |
| 25 | <rect x="60" y="160" width="6" height="124" rx="3" fill="#3182CE" /> |
| 26 | <text x="100" y="186" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="12" font-weight="700" fill="#3182CE" letter-spacing="1.5">EQUATION</text> |
| 27 | <image href="../images/formula_002.png" x="200" y="192" width="780" height="92" preserveAspectRatio="xMidYMid meet" /> |
| 28 | </g> |
| 29 | |
| 30 | <g id="paper-figure"> |
| 31 | <rect x="80" y="316" width="380" height="350" rx="12" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1.5" filter="url(#figFrame)" /> |
| 32 | <image href="../images/attention_p4_1.png" x="92" y="328" width="356" height="326" preserveAspectRatio="xMidYMid meet" /> |
| 33 | <text x="270" y="662" text-anchor="middle" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" font-style="italic" fill="#4A5568">Figure 2 (right) · Multi-Head Attention</text> |
| 34 | </g> |
| 35 | |
| 36 | <g id="explanation"> |
| 37 | <text x="500" y="350" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1.5">WHY MULTI-HEAD INSTEAD OF SINGLE</text> |
| 38 | |
| 39 | <g id="point-1"> |
| 40 | <rect x="500" y="370" width="6" height="64" fill="#1A365D" /> |
| 41 | <text x="522" y="392" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Project Q, K, V into h subspaces</text> |
| 42 | <text x="522" y="416" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Independent learned linear projections W_iᴽ, W_iᴷ, W_iⱽ — one</text> |
| 43 | <text x="522" y="436" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">set per head; each head sees a different view of the inputs.</text> |
| 44 | </g> |
| 45 | |
| 46 | <g id="point-2"> |
| 47 | <rect x="500" y="450" width="6" height="64" fill="#1A365D" /> |
| 48 | <text x="522" y="472" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Run scaled DPA in each head in parallel</text> |
| 49 | <text x="522" y="496" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">d_k = d_v = d_model / h = 64. Total compute stays comparable</text> |
| 50 | <text x="522" y="516" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">to a single full-dimensional head.</text> |
| 51 | </g> |
| 52 | |
| 53 | <g id="point-3"> |
| 54 | <rect x="500" y="530" width="6" height="64" fill="#1A365D" /> |
| 55 | <text x="522" y="552" font-family="Georgia, "Microsoft YaHei", serif" font-size="18" font-weight="700" fill="#1A365D">Concat heads, project with Wᴼ</text> |
| 56 | <text x="522" y="576" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">Concatenated head outputs go through a final linear map back</text> |
| 57 | <text x="522" y="596" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="15" fill="#1A202C">to d_model.</text> |
| 58 | </g> |
| 59 | |
| 60 | <rect x="500" y="612" width="710" height="56" rx="8" fill="#EBF4FB" /> |
| 61 | <rect x="500" y="612" width="4" height="56" fill="#3182CE" /> |
| 62 | <text x="516" y="636" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="13" font-weight="700" fill="#3182CE" letter-spacing="1">THE PAYOFF</text> |
| 63 | <text x="516" y="656" font-family="Georgia, "Microsoft YaHei", serif" font-size="15" fill="#1A202C" font-style="italic">Each head can attend to different representation subspaces at different positions.</text> |
| 64 | </g> |
| 65 | |
| 66 | <g id="footer"> |
| 67 | <text x="60" y="694" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">Source: Vaswani et al. 2017, §3.2.2 Multi-Head Attention</text> |
| 68 | <text x="1220" y="694" text-anchor="end" font-family="Arial, "Microsoft YaHei", sans-serif" font-size="11" fill="#A0AEC0">08</text> |
| 69 | </g> |
| 70 | </svg> |
| 71 |