Writing about systems that work in the real world.
Technical articles on artificial intelligence, machine learning, and production systems.
2026The idea of recursive self-improvement (RSI) can be traced to I. J. Good (1965), who described an “ultraintelligent machine” as a system capable of exceeding human performance in all intellectual activities and of designing superior machines to improve itself. Yudkowsky (2008) later used the term “recursive self-improvement” to refer to a particular feedback loop: an AI applies its current intelligence to enhance the cognitive machinery that generates its intelligence. In contemporary AI, this feedback loop may mean a model directly rewriting its own weights; more broadly, it can involve improving the training pipeline and the deployment system, which then supports a stronger successor model with better performance across economically valuable tasks. The pace of AI research progress has been shown to be drastically accelerated in frontier labs (Anthropic; OpenAI).
2026Scaling laws represent one of the most important empirical results in deep learning. The core observation is straightforward: training loss $L$ tends to decline in a predictable way as we increase model size $N$, dataset size $D$, and compute $C$. Empirically, this relationship follows a power-law curve, which shows up as an approximately straight line on a log-log plot. More broadly, scaling laws provide a framework for characterizing how compute, loss, model size, and data relate to one another. In essence, they address how to allocate scarce compute optimally between $N$ and $D$.
2025We extend special thanks to John Schulman for extensive, highly valuable feedback and hands-on edits to this post. Test-time compute (Graves et al. 2016, Ling, et al. 2017, Cobbe et al. 2021) and chain-of-thought (CoT) (Wei et al. 2022, Nye et al. 2021) have produced substantial gains in model performance, while also surfacing many new research questions. This post reviews recent progress on using test-time compute (that is, “thinking time”) effectively, and explains why it can be beneficial.
2024Reward hacking arises when a reinforcement learning (RL) agent takes advantage of defects, loopholes, or ambiguities in the reward function to obtain high reward, without truly learning or accomplishing the intended objective. This behavior occurs because RL environments are frequently imperfect, and because specifying a reward function with complete fidelity is inherently difficult. As language models increasingly generalize across a wide range of tasks and RLHF becomes a de facto approach for alignment training, reward hacking during RL training of language models has become a significant practical challenge. Examples include models learning to alter unit tests in order to pass coding tasks, or producing responses that contain biases crafted to match a user’s preferences, both of which are concerning and likely represent major obstacles to real-world deployment of more autonomous AI model use cases.
2024In large language models, “hallucination” typically describes output that is unfaithful, fabricated, inconsistent, or nonsensical. In practice, the term has been used more broadly to cover many kinds of model errors. Here, I narrow the scope of “hallucination” to situations in which the model produces fabricated content that is not grounded in either the provided context or established world knowledge. There are two forms of hallucination: In-context hallucination: The model output should align with, and remain consistent with, the source material provided in context. Extrinsic hallucination: The model output should be grounded in the pre-training dataset. However, because the pre-training dataset is so large, retrieving it and detecting conflicts for each generation is prohibitively expensive. If we treat the pre-training corpus as a proxy for world knowledge, the practical goal becomes ensuring that the model’s output is factual and verifiable against external world knowledge. Equally important, when the model does not know a fact, it should explicitly say so. This post focuses on extrinsic hallucination. To avoid hallucination, LLMs must (1) be factual and (2) acknowledge when they do not know the answer, when applicable.
2024Over the past few years, diffusion models have achieved strong performance in image synthesis. The research community is now extending these methods to a more demanding setting: video generation. This problem strictly generalizes the image case because a single image can be viewed as a one-frame video, but it is substantially more challenging for several reasons. First, video generation introduces additional constraints on temporal consistency across frames, which in turn requires the model to encode richer world knowledge. Second, compared with text or images, acquiring large-scale collections of high-quality, high-dimensional video data is more difficult, even more so for paired text-video datasets. 🥑 Required Pre-read: Please ensure you have read the previous blog on “What are Diffusion Models,” for image generation before continuing here.
2024[Special thanks to Ian Kivlichan for many helpful pointers (e.g., the 100+ year old Nature paper “Vox populi”) and for thoughtful feedback. 🙏 ] High-quality data is the fuel that powers training for modern deep learning models. Most task-specific labeled datasets are produced through human annotation, whether for classification tasks or for RLHF labeling (which can be represented in a classification format) used to train LLM alignment. Although many ML techniques discussed in this post can improve data quality, human data collection ultimately depends on close attention to detail and careful execution. The community recognizes the importance of high-quality data, yet there remains a subtle perception that “Everyone wants to do the model work, not the data work” (Sambasivan et al. 2021).
2023The deployment of large language models in real-world settings accelerated dramatically following the release of ChatGPT. We (including my team at OpenAI, shoutout to them) have devoted substantial effort to building safe default behaviors into the model during the alignment process (for example, via RLHF). Even so, adversarial attacks, or jailbreak prompts, may still induce the model to produce undesired outputs. A significant portion of the foundational work on adversarial attacks focuses on images, where attacks operate in a continuous, high-dimensional space. In contrast, attacks on discrete data such as text have generally been viewed as considerably more difficult because they lack direct gradient signals. My earlier post on Controllable Text Generation is closely related to this subject, since attacking LLMs is, in effect, an attempt to control the model so that it generates a particular class of (unsafe) content.
2023Designing agents that use an LLM (large language model) as the central controller is a compelling idea. Several proof-of-concept demonstrations, including AutoGPT, GPT-Engineer, and BabyAGI, provide motivating examples. The potential of LLMs goes well beyond producing polished copy, stories, essays, and programs; they can also be positioned as powerful general-purpose problem solvers.
2023Prompt engineering, also called in-context prompting, encompasses techniques for communicating with an LLM in order to guide its behavior toward desired outcomes without modifying the model’s weights. It is largely an empirical discipline, and the impact of specific prompt-engineering approaches can differ substantially across models; as a result, it often demands extensive experimentation and practical heuristics. This post focuses exclusively on prompt engineering for autoregressive language models, and does not cover Cloze tests, image generation, or multimodal models. At a fundamental level, prompt engineering is concerned with alignment and model steerability. For additional background, see my previous post on controllable text generation.
2023Since my previous post on “The Transformer Family” (approximately three years ago), many improvements to Transformer architectures have been proposed. In this update, I performed a substantial refactor and expansion of that 2020 post: I reorganized the section hierarchy and strengthened many sections by incorporating more recent papers. Version 2.0 is a strict superset of the earlier version and is roughly twice as long. Notations Symbol Meaning $d$ The model size, hidden state dimension, positional encoding size. $h$ The number of heads in the multi-head attention layer. $L$ The segment length of the input sequence. $N$ The total number of attention layers in the model, excluding MoE. $\mathbf{X} \in \mathbb{R}^{L \times d}$ The input sequence, where each element has been mapped into an embedding vector of shape $d$, equal to the model size. $\mathbf{W}^k \in \mathbb{R}^{d \times d_k}$ The key weight matrix. $\mathbf{W}^q \in \mathbb{R}^{d \times d_k}$ The query weight matrix. $\mathbf{W}^v \in \mathbb{R}^{d \times d_v}$ The value weight matrix. Often we have $d_k = d_v = d$. $\mathbf{W}^k_i, \mathbf{W}^q_i \in \mathbb{R}^{d \times d_k/h}; \mathbf{W}^v_i \in \mathbb{R}^{d \times d_v/h}$ The per-head weight matrices. $\mathbf{W}^o \in \mathbb{R}^{d_v \times d}$ The output weight matrix. $\mathbf{Q} = \mathbf{X}\mathbf{W}^q \in \mathbb{R}^{L \times d_k}$ The query embedding inputs. $\mathbf{K} = \mathbf{X}\mathbf{W}^k \in \mathbb{R}^{L \times d_k}$ The key embedding inputs. $\mathbf{V} = \mathbf{X}\mathbf{W}^v \in \mathbb{R}^{L \times d_v}$ The value embedding inputs. $\mathbf{q}_i, \mathbf{k}_i \in \mathbb{R}^{d_k}, \mathbf{v}_i \in \mathbb{R}^{d_v}$ Row vectors in the query, key, and value matrices, $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$. $S_i$ A collection of key positions for the $i$-th query $\mathbf{q}_i$ to attend to. $\mathbf{A} \in \mathbb{R}^{L \times L}$ The self-attention matrix between an input sequence of length $L$ and itself. $\mathbf{A} = \text{softmax}(\mathbf{Q}\mathbf{K}^\top / \sqrt{d_k})$. $a_{ij} \in \mathbf{A}$ The scalar attention score between query $\mathbf{q}_i$ and key $\mathbf{k}_j$. $\mathbf{P} \in \mathbb{R}^{L \times d}$ Position encoding matrix, where the $i$-th row $\mathbf{p}_i$ is the positional encoding for input $\mathbf{x}_i$. Transformer Basics The Transformer (referred to here as the “vanilla Transformer” to distinguish it from enhanced variants; Vaswani, et al., 2017) uses an encoder-decoder architecture, as is common in many NMT models. Later work showed that simplified Transformer variants can also deliver strong performance on language modeling tasks, such as encoder-only BERT and decoder-only GPT.
2023[Updated on 2023-01-24: add a small section on Distillation.] Large transformer models are now widely used, delivering state-of-the-art (SoTA) results across a broad range of tasks. However, they are both costly to train and expensive to operate. The very high inference cost, in terms of both latency and memory footprint, remains a major bottleneck to deploying powerful transformers for real-world, large-scale workloads. Why is inference for large transformer models so difficult, In addition to the growing size of SoTA models, two primary factors contribute to the inference challenge (Pope et al. 2022):
2022Neural networks are widely recognized as over-parameterized and can often fit data easily, achieving near-zero training loss while still generalizing reasonably well to a test dataset. Despite the fact that these parameters are randomly initialized, the optimization process can reliably produce comparably strong results. This behavior also persists when the number of model parameters exceeds the number of training data points. The neural tangent kernel (NTK) (Jacot et al. 2018) is a kernel-based framework for describing how neural networks evolve during gradient-descent training. It provides key insight into why sufficiently wide neural networks can consistently converge to a global minimum when trained to minimize an empirical loss. In this post, we take a detailed look at the motivation and definition of the NTK, and we also cover the proof of deterministic convergence across different random initializations for neural networks in the infinite-width limit by characterizing the NTK in that regime.
2022For many years, researchers have explored methods for converting images into text, including image captioning and visual question answering. In traditional architectures, these systems typically use an object-detection network as the vision encoder to extract visual features, and then generate text using a text decoder. Given the breadth of prior work, this post focuses on a single approach to vision-language tasks: extending pre-trained generalized language models so they can accept and interpret visual signals.
2022This is Part 3 of the series on learning with insufficient data (see Part 1 and Part 2). In this installment, we consider two strategies for producing synthetic training data.
2022This is Part 2 of our discussion on how to proceed when you have only a limited amount of labeled data for supervised learning tasks. In this installment, we introduce a controlled amount of human labeling effort, constrained by a fixed budget. Accordingly, we must be deliberate and efficient in deciding which samples to send for labeling.
2021When supervised learning tasks must be addressed with only a small amount of labeled data, four commonly cited approaches are typically considered.
2021[Updated on 2022-03-13: added expert choice routing.] [Updated on 2022-06-10]: Greg and I wrote a shorter, improved version of this post, published on the OpenAI Blog: “Techniques for Training Large Neural Networks”
2021[Updated on 2021-09-19: I strongly recommend this blog post on score-based generative modeling by Yang Song (the author of several foundational papers cited in the references)]. [Updated on 2022-08-27: Added classifier-free guidance, GLIDE, unCLIP, and Imagen. [Updated on 2022-08-31: Added latent diffusion model. [Updated on 2024-04-13: Added progressive distillation, consistency models, and the Model Architecture section.
2021Contrastive representation learning aims to construct an embedding space where pairs of similar samples are mapped near one another, while dissimilar pairs are separated by large distances. This paradigm can be used in both supervised and unsupervised scenarios. In the unsupervised case, contrastive learning is regarded as one of the most effective methods in self-supervised learning.
2021Large pretrained language models are trained on extensive collections of online data. As a result, they inevitably absorb certain toxic behaviors and biases present on the Internet. These pretrained models are highly capable and have demonstrated strong performance across many NLP tasks. However, deploying them safely in practical, real-world applications requires robust safety controls over the model’s generation process.
2021[Updated on 2021-02-01: Updated to version 2.0 with several work added and many typos fixed.] [Updated on 2021-05-26: Add P-tuning and Prompt Tuning in the “prompt design” section.] [Updated on 2021-09-19: Add “unlikelihood training”.]
2020[Updated on 2020-11-12: Added an example of closed-book factual QA using the OpenAI API (beta). A model that can answer any question based on factual knowledge can enable many useful, practical applications, including serving as a chatbot or an AI assistant🤖. In this post, we review several common approaches to building an open-domain question answering system.
2020Even though many of today’s most widely used and successful model architectures are crafted by human experts, this does not imply that the full space of possible network architectures has been thoroughly explored or that we have already converged on the best choice. We are more likely to identify an optimal solution by using a systematic, automated approach to discover and learn high-performance model architectures.
2020[Updated on 2020-06-17: Add “exploration via disagreement” in the “Forward Dynamics” section. The exploration versus exploitation trade-off is a fundamental issue in Reinforcement Learning. Ideally, we want an RL agent to discover the best solution as quickly as possible. At the same time, locking into a particular strategy too early, without sufficient exploration, is risky, since it may trap the agent in local minima or cause complete failure. Many modern RL algorithms that directly optimize for high returns can exploit effectively and with notable efficiency, whereas exploration remains comparatively less settled and continues to be treated as an open area of study.]
2020[Updated on 2023-01-27: After nearly three years, I completed a major refactoring of this post to incorporate a number of new Transformer models released since 2020. The revised and expanded version of this post is available here: The Transformer Family Version 2.0. Please refer to that post for this topic.]
2020[Updated on 2020-02-03: Added a mention of PCG in the “Task-Specific Curriculum” section.] [Updated on 2020-02-04: Added a new “curriculum through distillation” section.]
2019[Updated on 2020-01-09: Added a new section on Contrastive Predictive Coding.] [Updated on 2020-04-13: Added a “Momentum Contrast” section covering MoCo, SimCLR, and CURL.] [Updated on 2020-07-08: Added a “Bisimulation” section covering DeepMDP and DBC.] [Updated on 2020-09-12: Added MoCo V2 and BYOL to the “Momentum Contrast” section.] [Updated on 2021-05-31: Removed the “Momentum Contrast” section and added a pointer to a full post on “Contrastive Representation Learning”.]
2019Stochastic gradient descent is a widely used default for optimizing deep learning models, but it is not the only viable approach. Black-box optimization algorithms allow you to evaluate a target function $f(x): \mathbb{R}^n \to \mathbb{R}$ even when you do not know the exact analytic form of $f(x)$ and therefore cannot compute gradients or the Hessian matrix. Representative black-box optimization methods include Simulated Annealing, Hill Climbing, and the Nelder-Mead method.
2019In my earlier post on meta-learning, I framed the problem primarily in the context of few-shot classification. Here, I would like to examine scenarios in which we aim to “meta-learn” Reinforcement Learning (RL) tasks by building an agent that can solve previously unseen tasks quickly and efficiently.
2019In robotics, one of the most challenging problems is enabling a model trained in simulation to transfer reliably to the real world. Because deep reinforcement learning (RL) algorithms are sample-inefficient and collecting data on real robots is expensive, models often must be trained in a simulator, which in principle can generate an unlimited amount of data. In practice, however, the reality gap between simulation and the physical world frequently causes failures during deployment on real hardware. This gap is driven by mismatches in physical parameters (for example, friction, kp, damping, mass, density) and, more critically, by inaccurate physical modeling (for example, collisions involving soft surfaces).
2019[Updated on 2019-05-27: add the section on Lottery Ticket Hypothesis.] If you entered the deep learning field with a background in traditional machine learning, you may have asked a recurring question: given that a typical deep neural network contains an enormous number of parameters, and that its training error can often be driven to perfection, should it not suffer from severe overfitting, If so, how can it generalize to out-of-sample data points,
2019[Updated on 2019-02-14: add ULMFiT and GPT-2.] [Updated on 2020-02-29: add ALBERT.] [Updated on 2020-10-25: add RoBERTa.] [Updated on 2020-12-13: add T5.] [Updated on 2020-12-30: add GPT-3.] [Updated on 2021-11-13: add XLNet, BART and ELECTRA; Also updated the Summary section.] Are they ELMo and BERT, (Image source: here) We saw remarkable progress in NLP in 2018. Large-scale, pre-trained language models such as OpenAI GPT and BERT delivered strong results across a broad range of language tasks while relying on generic model architectures. This concept parallels how ImageNet classification pre-training benefits many vision tasks (*). However, unlike vision classification pre-training, this straightforward and effective NLP approach does not require labeled data during pre-training, which lets us push training scale and explore larger experiments, up to our practical limits.
2018In Part 3, we examined models in the R-CNN family. These approaches are all region-based object detection algorithms. Although they can deliver high accuracy, they may be too slow for certain applications, such as autonomous driving. In Part 4, we focus exclusively on fast object detection models, including SSD, RetinaNet, and models in the YOLO family.
2018[Updated on 2019-10-01: with thanks to Tianhao, this post is now available in a Chinese translation!]
2018To date, I have discussed two categories of generative models, GANs and VAEs. Neither model class learns the real data probability density function explicitly, $p(\mathbf{x})$ (where $\mathbf{x} \in \mathcal{D}$), because doing so is extremely difficult. For example, in a generative model that uses latent variables, $p(\mathbf{x}) = \int p(\mathbf{x}\vert\mathbf{z})p(\mathbf{z})d\mathbf{z}$ is very difficult to compute, since it is intractable to enumerate all possible values of the latent code $\mathbf{z}$.
2018[Updated on 2019-07-18: add a section on VQ-VAE & VQ-VAE-2.] [Updated on 2019-07-26: add a section on TD-VAE.] The autoencoder was introduced as a neural network approach for reconstructing high-dimensional data by passing inputs through a narrow bottleneck layer positioned midway through the model (oops, this is probably not true for Variational Autoencoder, and we will investigate it in details in later sections). One useful side effect is dimensionality reduction: the bottleneck layer learns a compressed latent encoding. This low-dimensional representation can serve as an embedding vector in a range of applications (i.e. search), support data compression, or expose the underlying generative factors in the data.
2018[Updated on 2018-10-28: Added Pointer Network and a link to my Transformer implementation.] [Updated on 2018-11-06: Added a link to the Transformer model implementation.] [Updated on 2018-11-18: Added Neural Turing Machines.] [Updated on 2019-07-18: Corrected an error involving the use of the term “self-attention” when introducing the show-attention-tell paper, and moved that content to the Self-Attention section.] [Updated on 2020-04-07: A follow-up post on improved Transformer models is here.]
2018The complete implementation is available in lilianweng/deep-reinforcement-learning-gym. In the previous two posts, I introduced the algorithms behind a range of deep reinforcement learning models. Now it is time to get hands-on and apply those models in practice. This implementation is built using Tensorflow and the OpenAI Gym environment. The full version of the code used in this tutorial is available in [lilian/deep-reinforcement-learning-gym].
2018[Updated on 2018-06-30: Added two new policy gradient methods, SAC and D4PG.] [Updated on 2018-09-30: Added a new policy gradient method, TD3.] [Updated on 2019-02-09: Added SAC with automatically adjusted temperature.] [Updated on 2019-06-26: Thanks to Chanseok, this post is also available in Korean.] [Updated on 2019-09-12: Added a new policy gradient method, SVPG.] [Updated on 2019-12-22: Added a new policy gradient method, IMPALA.] [Updated on 2020-10-15: Added a new policy gradient method, PPG, along with additional discussion in PPO.] [Updated on 2021-09-19: Thanks to Wenhao & 爱吃猫的鱼, this post is also available in Chinese1 & Chinese2.]
2018[Updated on 2020-09-03: Revised the SARSA and Q-learning algorithms to make their differences more explicit. [Updated on 2021-09-19: Thanks to 爱吃猫的鱼, this post is now available in Chinese].
2018These algorithms are implemented for the Bernoulli bandit setting in lilianweng/multi-armed-bandit. Exploitation vs Exploration : The exploration versus exploitation dilemma appears across many areas of everyday life. For example, if your favorite restaurant is just around the corner and you go there every day, you can be confident about what you will get, but you may forgo opportunities to discover an even better alternative. Conversely, if you constantly try new places, you are very likely to end up eating unpleasant food from time to time. In the same way, online advisors aim to strike a balance between ads that are already known to be most attractive and new ads that could prove even more successful.
2017[Updated on 2018-12-20: Removed YOLO from this article. Part 4 will cover multiple fast object detection algorithms, including YOLO.] [Updated on 2018-12-27: Added sections on bbox regression and R-CNN tricks.] In the “Object Detection for Dummies” series, Part 1 covered foundational image-processing concepts, including gradient vectors and HOG. Part 2 then reviewed classic convolutional neural network architecture designs for classification, along with early object-recognition models such as Overfeat and DPM. In this third installment, we will examine a set of models in the R-CNN (“Region-based CNN”) family.
2017Part 1 of the “Object Detection for Dummies” series covered three foundations: (1) the concept of an image gradient vector, and how the HOG algorithm aggregates information from all gradient vectors within a single image; (2) the operation of image segmentation algorithms for identifying regions that may contain objects; and (3) how the Selective Search algorithm improves segmentation results to produce stronger region proposals.
2017I have never worked in computer vision, and I do not understand how the “magic” works when an autonomous car is configured to distinguish a stop sign from a pedestrian wearing a red hat. To motivate myself to study the mathematics behind object recognition and detection algorithms, I am writing a series of posts on this topic, “Object Detection for Dummies.” This first post begins with very basic concepts in image processing and introduces several methods for image segmentation. It does not cover deep neural networks. Deep learning models for object detection and recognition will be discussed in Part 2 and Part 3.
2017Human language is expressed as free text. To enable a machine learning model to interpret and process natural language, we must convert free-text words into numeric representations. One of the simplest approaches is one-hot encoding, where each unique word corresponds to a single dimension in the resulting vector, and a binary value indicates whether the word is present (1) or absent (0).
2017Professor Naftali Tishby passed away in 2021. I hope this post will introduce his compelling idea of the information bottleneck to a wider audience. Recently, I watched Prof. Naftali Tishby’s talk, “Information Theory in Deep Learning,” and found it highly engaging. In the talk, he explained how information theory can be used to examine the growth and transformation of deep neural networks during training. Using the Information Bottleneck (IB) method, he proposed a new learning bound for deep neural networks (DNN), since traditional learning theory breaks down in the presence of an exponentially large number of parameters. He also highlighted a sharp observation about DNN training: it proceeds in two distinct phases. First, the network is trained to fully represent the input data and to minimize the generalization error; next, it learns to discard irrelevant details by compressing the input representation.
2017[Updated on 2018-09-30: thanks to Yoonju, we have this post translated in Korean!] [Updated on 2019-04-18: this post is also available on arXiv.] Generative adversarial networks (GANs) have delivered strong performance across many generative tasks that aim to reproduce rich real-world content, including images, human language, and music. The approach is motivated by game theory: two models, a generator and a critic, compete against one another, and through this competition they can simultaneously improve. Even so, training GANs is often difficult in practice, and common problems include instability during training and failures to converge.
2017Machine learning models are increasingly being deployed in high-stakes domains such as health care, justice systems, and the financial industry. As a result, it is essential to understand how these models arrive at their decisions and to ensure that their decision-making processes comply with ethical requirements and legal regulations.
2017In Part 2 of this tutorial, I continue the discussion of stock price prediction by extending the recurrent neural network built in Part 1 so that it can handle multiple stocks. To help the model differentiate patterns across different price sequences, I include stock symbol embedding vectors as part of the input.
2017This tutorial walks through building a recurrent neural network in TensorFlow to predict stock market prices. The complete, runnable code is available at github.com/lilianweng/stock-rnn. If you are not yet familiar with recurrent neural networks or LSTM cells, you may want to review my previous post.
2017Many of you have likely watched, or at least heard about, the 2016 match series between AlphaGo and the professional Go player Lee Sedol. Lee holds the highest rank of nine dan and has won many world championships. There is no doubt that he is one of the best Go players in the world, yet he lost the series 1-4 against AlphaGo. Prior to this, Go was widely viewed as an intractable game for computers to master because its simple rules generate an exponential number of possible board-position variations, far more than in Chess. This event clearly underscored 2016 as a landmark year for AI. As a result of AlphaGo, substantial attention has been drawn to the progress of AI.
2026A practical guide to choosing between deterministic software, machine learning, model calls, RAG, workflows, and agents, with production trade-offs and evaluation criteria.