Train and run byte-level transformer models on CUDA
gpt is a hackable CUDA/C transformer implementation that trains on raw bytes, with causal attention, RoPE, and AdamW.
Why it matters
Train custom autoregressive transformer models that predict sequences byte-by-byte, enabling you to generate text, model DNA/RNA sequences, or process any binary data stream using a hackable CUDA implementation.
Outcomes
What it gets done
Train GPT-style transformers on text or arbitrary byte streams using CUDA acceleration
Generate coherent text sequences with configurable temperature and context length
Process any binary data format including DNA, compressed files, images, or executables
Customize model architecture with configurable layers, dimensions, and attention mechanisms
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/markusheimerl-gpt | bash Overview
Tiny hackable CUDA language model
gpt is a hackable CUDA/C implementation of a byte-level generative pretrained transformer, with causal self-attention, rotary positional encoding, and AdamW training built from scratch. Use it to study or extend a from-scratch transformer implementation. Not a production LLM serving stack or a tool for using pretrained weights from elsewhere.
What it does
gpt is a minimal, hackable CUDA/C implementation of a generative pretrained transformer: an autoregressive model that processes raw byte sequences (a 256-value vocabulary, no tokenizer) and learns to predict the next byte from previous context. Because it operates on bytes rather than text tokens, the same architecture is content-agnostic - the README notes it can model DNA/RNA sequences, compressed data, images, audio, video, or executable binaries, not just text.
When to use - and when NOT to
Use it to study or hack on a from-scratch transformer implementation - every architectural piece (causal self-attention with rotary positional encoding, a swish-activated feed-forward block, residual connections, AdamW training with decoupled weight decay) is implemented directly rather than through a framework like PyTorch, and BLAS handles the matrix math for real training speed on a GPU. It is a small, byte-level reference implementation for understanding or extending the internals of a GPT-style model, not a production LLM serving stack or a tool for working with pretrained weights from elsewhere.
Inputs and outputs
Input: raw training data, any byte stream, plus a Makefile-driven build; the sample config uses d_model=512, hidden=1024, 16 layers, sequence length 1024. Output: a trained model that autoregressively generates further bytes from a prompt, sampled at a temperature (0.70 in the shipped example) until an end-of-text token or the token budget is reached.
sudo apt update
sudo apt install -y clang make time libopenblas-dev nvidia-cuda-toolkit git curl
git clone https://github.com/markusheimerl/gpt && cd gpt/
make data
make run -j 6
make infer
On a model trained on short children's-story-style text, prompting with "Once upon a time, there was a" produces coherent multi-paragraph short stories ending in an <|endoftext|> marker - a working demonstration of the full pretrain-then-generate loop rather than just the architecture in isolation.
Integrations
Built directly against CUDA and BLAS (via libopenblas-dev) with a plain Makefile build (make data, make run, make infer) and no external ML framework dependency - clang compiles the C/CUDA sources directly.
Who it's for
Developers who want to read, modify, or learn from a complete, dependency-light transformer implementation - attention, positional encoding, feed-forward, optimizer, and training loop all in hackable C/CUDA - rather than treating a transformer as a black box behind a high-level framework. The final transformer layer's output is projected to logits over all 256 possible byte values, turned into probabilities via softmax, and trained end to end with cross-entropy loss to maximize the probability of the correct next byte at every position.
Source README
gpt
A generative pretrained transformer implementation
This project implements an autoregressive sequence model using a transformer architecture as well as a pretraining pipeline. The model processes sequences of bytes (8-bit tokens), learning to predict the next byte given previous context. While this implementation trains on text data, the architecture is agnostic to the content. It can model any byte stream, including, but not limited to, DNA/RNA sequences, compressed data, images, audio, video, or executable binaries.
The architecture begins with a token embedding layer that converts each byte into a continuous vector representation.
The core of the model is a multi-layer transformer that processes the embedded sequences. Each transformer layer consists of two main components: a causal self-attention mechanism and a feed-forward network, both wrapped with residual connections. The causal attention ensures that predictions for each position can only depend on previous positions, which is essential for autoregressive generation. The attention mechanism computes query, key, and value projections, applies rotational positional encoding to the queries and keys to encode relative positions, computes scaled dot-product attention with a causal mask, and projects the result back. The feed-forward network applies two linear transformations with a swish activation, a smooth, non-monotonic function that multiplies its input by its sigmoid, in between.
After processing through all transformer layers, a linear projection maps the final hidden states to logits over the vocabulary (all 256 possible byte values). These logits are converted to probabilities using the softmax function, and the model is trained to maximize the probability of the correct next byte using cross-entropy loss.
The training process uses the AdamW optimizer, which enhances the standard Adam optimizer by decoupling weight decay from the gradient-based update. AdamW maintains exponential moving averages of both gradients and squared gradients, using these to adapt the learning rate for each parameter individually. The weight decay acts as L2 regularization, encouraging the model to use smaller weights and improving generalization.
The implementation uses BLAS (Basic Linear Algebra Subprograms) for efficient matrix operations, allowing the model to train effectively on modern hardware.
How to run
Ubuntu
sudo apt update
sudo apt install -y clang make time libopenblas-dev nvidia-cuda-toolkit git curl
git clone https://github.com/markusheimerl/gpt && cd gpt/
make data
make run -j 6
make infer
Sample outputs
Prompted with "Once upon a time, there was a":
markus@thinkpad:~/gpt$ make infer
Loaded: d_model=512 hidden=1024 layers=16 vocab=256 seq_len=1024
Generating 995 tokens (T=0.70, seed=1779612639)
Once upon a time, there was a compassionate little girl named Lily. She loved to play with her friends in the park. One day, she saw a small bird on a tree. The bird was sad because its wing was hurt.
Lily asked the bird, "Do you have any hurt wing?" The bird said, "Yes, I don't have any hurt wing." Lily wanted to help the bird, so she tried to make it feel better. But the bird was too big and her hurt wing still did not want to hurt Lily's wing.
Lily had an idea. She found a long stick and brought it to the bird. The bird said, "Thank you, Lily! You saved me!" The bird felt better and thanked Lily. They played together in the park all day. They were very happy and became best friends.
<|endoftext|>
Once upon a time, there was a little girl named Mia. Mia loved to study with her toys. She had a big box full of toys in her room. One day, Mia found a new toy. The toy was a small doll. The doll had a pretty dress and smiled a little.
Mia took the doll outside to play. She studied hard and felt the dress on her f
markus@thinkpad:~/gpt$ make infer
Loaded: d_model=512 hidden=1024 layers=16 vocab=256 seq_len=1024
Generating 995 tokens (T=0.70, seed=1779612665)
Once upon a time, there was a little boy named Tim. Tim had a big tree in his yard. He loved to run and play in the tree. One day, he saw a perfect bird in his yard. The bird was sad because it could not find its mom.
Tim wanted to help the bird. He kneeled down and looked all around. He saw a little girl named Sue. Sue was playing with a ball. Tim asked her, "How can I be like your bird?" Sue smiled and said, "You can be my friend."
Tim helped the bird get close to Sue. Sue was so happy and thanked Tim. They became good friends and played together in the tree. The bird sang a song and they all lived happily ever after.
<|endoftext|>
Once upon a time, there was a little boy. He was very careful as he walked around a park. One day, he saw an unusual thing called a rabbit. The rabbit hopped over to the thing and asked the other animals if they had seen it. The other animals thought it was a funny sight.
The rabbit and the other animals were very curious. They asked the other animal, "What do you think is a fun
markus@thinkpad:~/gpt$
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.