Train GPT-2-class language models from scratch in C/CUDA

nanoeuler

A GPT-2-class language model built entirely from scratch in C/CUDA - no PyTorch, no autograd, no ML libraries. The forward and backward passes are written and verified by hand, and the whole training pipeline lives in this repo: a hand-written byte-level BPE tokenizer, pretraining on a books + web corpus, and supervised fine-tuning into a chat model (RLHF/DPO planned). It runs on CPU (libm + OpenMP) for a small showcase model, and a full from-scratch CUDA engine - cuBLAS matmuls, a hand-written FlashAttention, validated against a CPU reference by a full-model gradient check - trains a ~116M-parameter model on a single RTX 4070.

Status & honesty. This is a research/educational artifact, built in public. At ~116M parameters trained on a single consumer GPU, it is a text generator in the spirit of GPT-2-small: fluent-ish English, no real world knowledge. It is not a capable assistant - the chat model demonstrates that the pretrain→SFT pipeline works end to end, it is not a useful chatbot. The point of the project is the from-scratch engineering and the complete, understandable training pipeline.

make check              # verify the backward pass (gradient check, double precision)
make                    # build the training binary
./nanoeuler train       # train the small showcase model (~0.76M params)
./nanoeuler train big   # train the larger model (~10M params; meant for a GPU)
./nanoeuler chat        # REPL: type a prompt, the model continues it

Why "Euler"?

A residual block computes

x = x + f(x)

Read it as a step of numerical integration. The forward-Euler method advances an ordinary differential equation dx/dt = f(x) by

x(t+Δt) = x(t) + Δt · f(x(t))

With step size Δt = 1 this is exactly the residual update. So a deep residual network is a discretized ODE: depth is integration time, and each layer integrates the hidden state forward by one Euler step. This is the view behind work like Neural ODEs (a ResNet is the Euler discretization of a continuous flow). The project is named after Leonhard Euler, who gave us that integration method.

Example output

A sample from the ~116M model after a partial pretraining run on the books + web corpus (prompt Alessandro eat a):

Alessandro eat a icing textile: the satisfied by the servants in order to keep your weight
[Using to a heated, collaborated young people that attend the metric process where the rank
is authorized and to contain the sedentary. Some state lawyers were able to insert ...

The content is not meaningful, but notice what it learned on its own: real grammar, long clauses, and an encyclopedic register picked up from the web data. This is the expected behaviour of a small model trained on a single GPU - fluent shape, shallow substance. More training and (far) more data improve fluency; world knowledge needs scale this project does not pretend to have.

Architecture

Decoder-only transformer with the building blocks common to current models:

• RMSNorm (pre-norm, no bias)
• Rotary position embeddings (RoPE) applied to queries and keys
• SwiGLU feed-forward: down(silu(gate(x)) * up(x))
• Grouped-query attention (GQA): query heads share a smaller set of key/value heads
• Multi-token prediction (MTP): K output heads predict the next K tokens; the auxiliary heads improve the learned representation and enable speculative decoding. Generation uses head 0.
• No biases anywhere.
• Byte-level BPE tokenizer, hand-written, with GPT-2-style pretokenization (a single leading space attaches to the following word, so spaces are not wasted as standalone tokens). Merges are learned on a sample of the corpus; the GPU model uses a 4096-token vocabulary (~3.4 bytes/token on English).

Each block is x = x + attn(rmsnorm(x)) followed by x = x + swiglu(rmsnorm(x)). A residual connection x = x + f(x) is one step of the forward-Euler method for the ODE dx/dt = f(x) - hence the name, and a nod to Leonhard Euler.

Configurations:

The CPU small model trains in a few hours on 12 cores and is a self-contained showcase. The ~116M GPU model is the real pipeline: it pretrains on the books + web mix and is then fine-tuned into a chat model (see below). The head size is 64 (768/12), which fits the FlashAttention kernel.

Verified backward pass

Hand-written back-propagation is easy to get subtly wrong, so every analytic gradient is compared against a central finite difference. The check runs in double precision so floating-point cancellation does not hide correct gradients:

$ make check
  tok      : max rel err 1.02e-04
  qkvw     : max rel err 7.20e-07
  gatew    : max rel err 6.86e-08
  ...
max relative error: 1.02e-04
>>> backward OK (error < 1e-2)

Every parameter tensor is checked, including the less obvious backward passes of RoPE, SwiGLU, GQA, and MTP.

Build and performance

make builds with -O3 -march=native -ffast-math -fopenmp. Matrix multiplies and attention are parallelized with OpenMP and vectorized; on a 12-core machine the training loop uses all cores. make check builds a separate double-precision binary used only for the gradient check.

No external dependencies. Tested with gcc 13 on Linux.

Scope

This is a from-scratch text generator and a complete, understandable training pipeline - not a product. A model of this size trained on one GPU produces fluent-looking English with little real knowledge; the fine-tuned chat model answers in assistant form but its content is shallow. A usable conversational model needs orders of magnitude more parameters, data and compute (a ~135M model only becomes a basic assistant after ~600B training tokens; this repo trains on a far smaller corpus on a single GPU). The goal is to own every piece - every parameter, every gradient, the tokenizer, the kernels, the pretraining and the fine-tuning.

GPU engine (CUDA)

cuda/nanoeuler_cuda.cu is a full from-scratch CUDA port - forward, backward, training and inference on the GPU. Every kernel is validated on the device against a CPU reference, and the whole model has a GPU gradient check (GPU grads vs CPU grads to ~1e-6).

Kernels: matmul (delegated to cuBLAS with TF32 tensor cores), RMSNorm, RoPE, grouped-query attention with a hand-written FlashAttention (tiled, online softmax, no T×T matrix in memory), SwiGLU, softmax/cross-entropy and AdamW. FlashAttention made the training step about 3× faster.

Build (RTX 40-series = Ada = sm_89; the host-compiler flag avoids a gcc ICE on the large file):

cd cuda
nvcc -O3 -arch=sm_89 -Xcompiler -fno-tree-reassoc,-fno-tree-copy-prop nanoeuler_cuda.cu -o nanoeuler_cuda -lcublas

Modes:

./nanoeuler_cuda              # run all kernel self-tests (GPU vs CPU)
./nanoeuler_cuda g            # full-model gradient check (GPU grads vs CPU)
./nanoeuler_cuda t            # pretrain from scratch, checkpoint to ../nanoeuler.bin every 5k steps
./nanoeuler_cuda tr           # resume pretraining from the latest ../nanoeuler.bin checkpoint
./nanoeuler_cuda i "It was"   # autoregressive generation on GPU
./nanoeuler_cuda s            # supervised fine-tune on Alpaca, save ../nanoeuler_chat.bin
./nanoeuler_cuda c            # interactive chat with the fine-tuned model

Training checkpoints every 5000 steps, so a long run can be stopped (Ctrl-C) and resumed with tr. A model trained on the GPU is saved in the CPU program's format, so ./nanoeuler chat can also load and run it.

Chatbot: pretrain then fine-tune (SFT)

The chat pipeline is two stages. First pretrain the ~116M base on the books + web mix (./nanoeuler_cuda t, resumable with tr). Then supervised fine-tuning turns it into an assistant: ./nanoeuler_cuda s loads the pretrained base, renders each Alpaca example with the standard instruction template, and trains with the loss masked to the response tokens only (prompt and padding positions get a target of -1, which the cross-entropy kernel turns into zero gradient). The result is saved to nanoeuler_chat.bin; ./nanoeuler_cuda c then wraps each line you type in the same template and samples a reply, stopping at the </s> end marker.

After fine-tuning the model answers in the right shape - it follows the instruction→response format, writes complete sentences and stops on its own. The content, though, is shallow and often wrong: this is a small model trained on a single GPU, so it has little world knowledge to express. SFT teaches the model how to respond, not what it knows - that comes from pretraining and scale. This is a faithful, fully-from-scratch demonstration that the pretrain→SFT pipeline works end to end, not a capable assistant.

Data

Pretraining uses a real books + web mix:

• Books - data/get_gutenberg.sh downloads ~95 public-domain Project Gutenberg classics (Austen, Dickens, Dostoevsky, Tolstoy, Melville, the complete Shakespeare, ...). Each book's Project Gutenberg license header/footer is stripped (only the text between the *** START ... *** / *** END ... *** markers is kept) so the model trains on prose.
• Web - data/get_web.sh pulls a slice of FineWeb-Edu (high-quality educational web text) straight from the Hugging Face parquet files using the DuckDB CLI (a single static binary - no Python, no libraries).

Then concatenate them into the pretraining corpus the trainer reads:

sh data/get_gutenberg.sh                       # books  -> data/gutenberg.txt
sh data/get_web.sh                             # web    -> data/web.txt (~1 GB by default)
cat data/gutenberg.txt data/web.txt > data/pretrain.txt
sh data/get_alpaca.sh                          # instruction data for SFT -> data/alpaca.json

Corpora and model checkpoints are git-ignored (regenerable).

Roadmap

• ✅ Hand-written byte-level BPE with GPT-2-style pretokenization.
• ✅ From-scratch CUDA engine (cuBLAS + FlashAttention), validated by a full-model gradient check.
• ✅ Pretraining on a books + web mix, with checkpoint/resume.
• ✅ Supervised fine-tuning (Alpaca) with response-masked loss → a chat model.
• ⏳ DPO (preference optimization) - the alignment stage, next to build.
• ⏳ Scale the model and data (toward ~270M) and publish a trained checkpoint people can try.

Files

nanoeuler.c             CPU model: forward, backward, training, sampling, chat REPL
cuda/nanoeuler_cuda.cu  GPU engine: BPE, kernels, FlashAttention, pretrain/SFT/infer/chat, gradient check
data/get_gutenberg.sh   downloads + cleans the Gutenberg books corpus
data/get_web.sh         downloads a FineWeb-Edu web slice via the DuckDB CLI (no Python)
data/get_alpaca.sh      downloads the Alpaca instruction data for fine-tuning
Makefile  LICENSE  shakespeare.txt  .gitignore

data/get_gutenberg.sh

#!/bin/sh
# Build a pretraining corpus from public-domain Project Gutenberg books.
# Produces data/gutenberg.txt. Re-runnable. No dependencies beyond wget.
# Unknown/removed IDs are skipped, so the list can be edited freely.
cd "$(dirname "$0")"
# A broad spread of public-domain classics in English. Multi-MB works
# (War and Peace, Les Miserables, the complete Shakespeare, Ulysses,
# Middlemarch, Don Quixote, the Karamazovs...) dominate the size.
IDS="
1342 84 1661 2701 11 98 1400 1232 76 345 2542 1080 5200 1260 768 2600 1184 174 2814 1952
158 161 105 141 121
74 86 3176
164 103 1268
35 36 5230
1023 766 580 730 963 786
244 2097 108 834 2852
2554 600 28054
1399
1257 1259
120 43
215 910
12
145 550
110
969
33 77
1727 6130
1497 1656
2680 1998 4363 205
1322
100 1513 1524 1533 1522
219
4300
844 902
514
289
113 17396
271
16
521 370
829
996
135
2413
26
131
1228
61
3300
23
408
203
73
209 2833
284
45
236 2226
55
41
2147
16328
21
2591
1597
"
: > gutenberg.txt
n=0
for id in $IDS; do
  printf 'fetching %s ... ' "$id"
  if wget -q --timeout=60 -O "book_$id.txt" "https://www.gutenberg.org/cache/epub/$id/pg$id.txt"; then
    # Strip the Project Gutenberg license header/footer: keep only the text
    # between the "*** START ... ***" and "*** END ... ***" markers. Books
    # without the markers are kept whole (guarded by grep) so none are lost.
    if grep -q 'START OF TH.* PROJECT GUTENBERG' "book_$id.txt"; then
      awk '/START OF TH.* PROJECT GUTENBERG/{p=1;next} /END OF TH.* PROJECT GUTENBERG/{p=0} p' "book_$id.txt" >> gutenberg.txt
    else
      cat "book_$id.txt" >> gutenberg.txt
    fi
    rm -f "book_$id.txt"; n=$((n+1)); echo "ok"
  else
    echo "skip"; rm -f "book_$id.txt"
  fi
done
printf 'fetched %s books | corpus size: %s bytes\n' "$n" "$(wc -c < gutenberg.txt)"

data/get_web.sh

#!/bin/sh
# Download a slice of FineWeb-Edu (high-quality educational web text) as plain text,
# using the DuckDB CLI (a single static binary -- no Python, no libraries) to read the
# Hugging Face parquet files directly via the hf:// scheme and export the text column.
#
#   sh data/get_web.sh [N_DOCS]
#
# N_DOCS defaults to 300000 (~1 GB, ~250M tokens). Pass a larger number to scale up;
# DuckDB reads parquet row-groups lazily, so a LIMIT only downloads what it needs.
cd "$(dirname "$0")"
DOCS="${1:-300000}"

# locate duckdb, or install the single-binary CLI (official installer, no Python)
DUCKDB="$(command -v duckdb 2>/dev/null)"
[ -z "$DUCKDB" ] && [ -x "$HOME/.duckdb/cli/latest/duckdb" ] && DUCKDB="$HOME/.duckdb/cli/latest/duckdb"
if [ -z "$DUCKDB" ]; then
  echo "installing the duckdb cli (single binary)..."
  curl -fsSL https://install.duckdb.org | sh
  DUCKDB="$HOME/.duckdb/cli/latest/duckdb"
fi

echo "exporting $DOCS FineWeb-Edu documents -> web.txt (this downloads a few GB) ..."
"$DUCKDB" -c "
  INSTALL httpfs; LOAD httpfs;
  COPY (
    SELECT text
    FROM 'hf://datasets/HuggingFaceFW/fineweb-edu/sample/10BT/*.parquet'
    LIMIT $DOCS
  ) TO 'web.txt' (HEADER false, QUOTE '', DELIMITER E'\n');
"
printf 'web corpus: %s bytes\n' "$(wc -c < web.txt 2>/dev/null || echo 0)"
echo "next: combine with the books -> cat gutenberg.txt web.txt > pretrain.txt"

data/get_alpaca.sh

#!/bin/sh
# Download the Alpaca instruction-tuning dataset (52k instruction/input/output triples,
# public domain CC0). Produces data/alpaca.json, consumed by `./nanoeuler_cuda s`.
cd "$(dirname "$0")"
URL="https://raw.githubusercontent.com/tatsu-lab/stanford_alpaca/main/alpaca_data.json"
printf 'fetching alpaca_data.json ... '
if wget -q --timeout=120 -O alpaca.json "$URL"; then
  printf 'ok | %s bytes\n' "$(wc -c < alpaca.json)"
else
  echo 'failed'; rm -f alpaca.json; exit 1
fi