Loading files
This file is pinned to the selected release. Download the original to open it in your design tool.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large">
<title>Four-Layer Transformer Trained On A CPU | Brian Greenforest</title>
<meta name="author" content="Brian Greenforest">
<meta name="description" content="A complete generative Transformer run: 4 layers, 16 heads, 128-dimensional embeddings, 128-token context, 361-token vocabulary, and 834,644 parameters trained on a CPU.">
<link rel="canonical" href="https://greenforest.io/four-layer-tiny-transformer-training-run.html">
<meta property="og:title" content="Four-Layer Transformer Trained On A CPU">
<meta property="og:description" content="Follow an 834,644-parameter generative Transformer from its model shape and training command through 53,130 CPU iterations and recognizable story output.">
<meta property="og:type" content="article">
<meta property="og:url" content="https://greenforest.io/four-layer-tiny-transformer-training-run.html">
<meta property="article:published_time" content="2024-06-22">
<link rel="icon" href="/favicon.ico">
<!-- greenforest:share-metadata:start -->
<!-- greenforest:preview-source:none -->
<meta property="og:site_name" content="Greenforest I/O">
<meta property="og:image" content="https://greenforest.io/social-previews/four-layer-tiny-transformer-training-run.png">
<meta property="og:image:secure_url" content="https://greenforest.io/social-previews/four-layer-tiny-transformer-training-run.png">
<meta property="og:image:type" content="image/png">
<meta property="og:image:width" content="1200">
<meta property="og:image:height" content="630">
<meta property="og:image:alt" content="Four-Layer Transformer Trained On A CPU - Greenforest I/O share preview">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="https://greenforest.io/social-previews/four-layer-tiny-transformer-training-run.png">
<meta name="twitter:image:alt" content="Four-Layer Transformer Trained On A CPU - Greenforest I/O share preview">
<script src="/site-analytics.js" defer></script>
<!-- greenforest:share-metadata:end -->
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Four-Layer Transformer Trained On A CPU",
"author": {
"@type": "Person",
"name": "Brian Greenforest"
},
"datePublished": "2024-06-22",
"dateModified": "2026-07-28",
"description": "A complete generative Transformer run with 4 layers, 16 heads, 128-dimensional embeddings, a 128-token context, a 361-token vocabulary, and 834,644 parameters trained on a CPU.",
"mainEntityOfPage": "https://greenforest.io/four-layer-tiny-transformer-training-run.html",
"publisher": {
"@type": "Person",
"name": "Brian Greenforest"
}
}
</script>
<script src="/common-script.js" type="text/javascript"></script>
</head>
<body>
<header><h1 class="page-title">Four-Layer Transformer Trained On A CPU</h1></header>
<address>June 22, 2024</address>
<section class="abstract">
<p class="lead">An 834,644-parameter Transformer learned recognizable story structure on a CPU in 53,130 recorded iterations.</p>
<p>Four layers, 16 attention heads, 128-dimensional embeddings, a 128-token context, and a custom 361-token vocabulary keep the complete generative mechanism compact enough to inspect from initialization through output.</p>
<p>The model shape, training and inference commands, loss history, source change, and generated samples appear together, turning a machine-learning result into a reproducible engineering narrative.</p>
</section>
<main>
<section>
<h2>Open The Complete Run</h2>
<p>A GitHub pull request carries the training log and script in one file. It records the architecture, training command, inference command, loss progression, and generated output from the same run.</p>
<p><a href="https://github.com/omnipresentalgorithm/simpler-llama/pull/1" target="_blank" rel="noopener">GitHub PR: 0.8M parameters model training</a></p>
<p><a href="https://raw.githubusercontent.com/avesus/llama2.c/e7e79b283a344279d15f7761a525d5da222074e7/train-dali.sh" target="_blank" rel="noopener">Raw training script and log</a></p>
</section>
<section>
<h2>Model Shape</h2>
<div class="article-fact-table">
<div class="article-fact-row"><div class="article-fact-key">Layers</div><div>4</div></div>
<div class="article-fact-row"><div class="article-fact-key">Attention heads</div><div>16</div></div>
<div class="article-fact-row"><div class="article-fact-key">Key/value heads</div><div>16</div></div>
<div class="article-fact-row"><div class="article-fact-key">Embedding dimension</div><div>128</div></div>
<div class="article-fact-row"><div class="article-fact-key">Context length</div><div>128 tokens</div></div>
<div class="article-fact-row"><div class="article-fact-key">Vocabulary</div><div>361 tokens</div></div>
<div class="article-fact-row"><div class="article-fact-key">Dropout</div><div>0.15</div></div>
<div class="article-fact-row"><div class="article-fact-key">Logged parameters</div><div>834,644 total from 832,128 decayed and 2,516 non-decayed parameters</div></div>
<div class="article-fact-row"><div class="article-fact-key">Training device</div><div>CPU</div></div>
<div class="article-fact-row"><div class="article-fact-key">Tokens per iteration</div><div>8,192</div></div>
</div>
</section>
<section>
<h2>From Initialization To Generative Behavior</h2>
<p>The command initializes a new model from scratch with a custom 361-token vocabulary, batch size 64, 128-token sequence length, learning rate <code>3e-4</code>, weight decay <code>0.1</code>, a 2,500-iteration warmup, and a target maximum of 100,000 iterations.</p>
<p>Training starts at loss 13.0864 with validation loss 13.0707. By step 53,130, logged values occupy roughly the 0.75 to 0.86 range as the model begins producing recognizable story structure.</p>
<p>The inference command loads <code>./outminimagic23/ckpt.pt</code> and <code>./data/tok361.model</code>, then runs with <code>--max_new_tokens=10240</code>, <code>--top_k=200</code>, and <code>--temperature=0.8</code>.</p>
</section>
<section>
<h2>The Pseudo-Inverse Unembedding</h2>
<p>The implementation computes a pseudo-inverse of the embedding matrix, uses it for unembedding, and propagates gradients through that transformation. This change makes the output projection depend directly on the learned embedding geometry.</p>
</section>
<section>
<h2>Story Structure Emerges</h2>
<p>The samples form characters, actions, simple causal turns, and moral-like endings. Repetition and unstable references reveal the pressure of the compact vocabulary and model shape, while the recognizable narrative structure shows what 834,644 trained parameters can organize.</p>
</section>
<section>
<h2>Why The Run Matters</h2>
<p>The entire path—from tensor dimensions and tokenizer size through optimization settings, checkpoint selection, loss progression, and output—fits on one engineering surface. That density makes the run useful for teaching, adaptation, and direct experimentation with architecture changes.</p>
<p>Use the source to reproduce the run, change the vocabulary or model geometry, and watch each decision alter the learning curve and generated structure.</p>
</section>
</main>
<nav class="article-links">
<p>Next:</p>
<a href="/linkedin-archive/2025-08-19-cheap-pixelless-textures-with-2d-sdfs.html">Cheap Pixelless Textures With 2D SDFs</a>
<p>See another compact generative system turn a mathematical field into a visual result through executable source.</p>
<p>More:</p>
<a href="/shader-before-gpu-renderman-ai-hardware.html">A History of Programmable Shaders: RenderMan, GPUs, CUDA, and AI</a>
<p>Follow programmable parallel graphics from RenderMan shading through GPU convolution and dedicated matrix engines.</p>
<a href="/">Back to greenforest.io</a>
</nav>
</body>
</html>
44b2ffa79This file is pinned to the selected release. Download the original to open it in your design tool.