Latest posts
- Building intuition about LLM parameter countsJul 10, 2026
When I was building my GPT-2 implementation in JAX, I started with just token embeddings for the input, and a separate output head (as I was not using weight tying). It wasn't an LLM -- no Transformer blocks, no attention, no feed-forward networks. I was somewhat surprised when I noticed that even that stripped-down model had 77 million parameters with the "small" settings I was using to train --
- Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090Jul 24, 2026
I mentioned I'd got a new RTX 3090 on a group chat, and a friend said: I know this is not really your thing... but let me know how quickly it runs Qwen 3.6 35bn MoE. With only 24gb of VRAM you’ll need to use a 4-bit quantized version and you won’t get a massive context window. But it should still be pretty cool. He's right that it's not really been my thing -- I've been focusing on my own LL
- Why do OpenAI's GPT-2 weights beat mine?Jul 29, 2026
When I finished my project training an LLM from scratch, I was left with a minor mystery. Why were my models worse at instruction-following than the original OpenAI GPT-2 small weights? I had an evaluation that I was running, based on the instruction fine-tuning code in chapter 7 of "Build a Large Language Model (from Scratch)". The process was to train a model on samples from the Alpaca instruct
- Why do OpenAI's GPT-2 weights beat mine? Part two: the bugfixJul 30, 2026
I'm digging into why my GPT-2 style models score worse on an instruction-following eval than OpenAI's original weights; I gave the details in this post. While I was writing up the results of my first experiment into possible causes, I ran the post past ChatGPT -- I always use an "editorial board" of AIs to check my posts for flow, style, and any technical errors (though all writing is always mine)
- Why do OpenAI's GPT-2 weights beat mine? Part three: testing overtrainingJul 31, 2026
The GPT-2-style models that I've been training work really well, and I've even managed to train some that perform better than the original OpenAI small model in terms of cross entropy loss on a test set. But as I wrote previously, there's a mystery: why do they perform worse on my instruction fine-tuning evaluation? I had various theories about why that might be, and to me, the most plausible-see
- How I use AI on this blogJul 31, 2026
Inspired by this LessWrong post, I thought I'd write about how I use AI here. This is less in the interest of disclosure, more to provide a snapshot of what I'm doing right now so that I can revisit it in the future and see how it changes. And hey, maybe it'll be of interest to you, dear readers. If I were to summarise my working philosophy in fewer than ten words, it would be: AIs identify prob
- A quick(ish) Chinchilla checkAug 07, 2026
I recently overtrained a couple of GPT-2 style models, training them both on 40 tokens per parameter rather than the 20 per parameter that is generally regarded as "Chinchilla-optimal". The normal heuristic is that instead of doing that, you should scale up the number of tokens and the number of parameters equally -- so I would have been better off scaling up the model by 2 and the token count by
- Use the built-in GELU, don't roll your own!Aug 20, 2026
Unsurprisingly, PyTorch's own built-in GELU function is faster than the hand-rolled one I've been using to date. But I was surprised at how much faster using it made things when training my models. I discovered this accidentally just now while working on something unrelated, but am logging the details here for anyone else that might find it useful. The headline numbers: the same code, training t
- Adding diagrams to my static site generator with D2Aug 25, 2026
A lot of the time when I've been writing posts for this blog, I've felt that a diagram would really help. But they're a pain to produce well, and I think I underuse them as a result. I wanted to fix that, and wound up adding D2 support to my static site generator. I think it works pretty well! In the past, I've tried drawing my own diagrams in LibreOffice and exporting as SVG, but my complete l
- Why do OpenAI's GPT-2 weights beat mine? Part four: digging into dropoutAug 27, 2026
I'm still digging into a mystery about the models I've been training; although an increasing number of them beat the OpenAI GPT-2 small weights on the narrow technical measure of the loss they get on a test set, they're not as good at an instruction-fine-tuning test. While reading about MoE models, I came across this paragraph in the Switch Transformers paper: Our paper considers the common NLP ap