Prompt engineering, also called in-context prompting, encompasses techniques for communicating with an LLM in ways that steer its behavior toward desired outcomes without updating model weights. It is largely an empirical discipline: the impact of specific prompting methods can vary substantially across models, so extensive experimentation and practical heuristics are often necessary.

This post focuses exclusively on prompt engineering for autoregressive language models. It does not cover cloze tests, image generation, or multimodal models. Fundamentally, prompt engineering targets alignment and model steerability. See my previous post on controllable text generation.

[My personal spicy take] In my view, some prompt engineering papers are not worth eight pages, because many of these tricks can be conveyed in one or a few sentences, and much of the remaining content is benchmarking. A shared, easy-to-use benchmarking infrastructure would likely provide more value to the community. In contrast, iterative prompting or external tool use can be difficult to set up, and it is also non-trivial to coordinate the broader research community to adopt a shared infrastructure.

Basic Prompting

Zero-shot and few-shot learning are the two most fundamental prompting approaches. They were popularized by many LLM papers and are widely used to benchmark LLM performance.

Zero-Shot

Zero-shot learning means providing the task text to the model and directly requesting the result.

(All sentiment analysis examples are from SST-2.)

Text: i'll bet the video game is a lot more fun than the film.
Sentiment:

Few-shot

Few-shot learning supplies a set of high-quality demonstrations for the target task, where each demonstration includes both the input and the desired output. By seeing strong examples first, the model can better infer human intent and the criteria for acceptable answers. As a result, few-shot learning often outperforms zero-shot prompting. The tradeoff is increased token usage and the risk of hitting context-length limits when the input and output texts are long.

Text: (lawrence bounces) all over the stage, dancing, running, sweating, mopping his face and generally displaying the wacky talent that brought him fame in the first place.
Sentiment: positive

Text: despite all evidence to the contrary, this clunker has somehow managed to pose as an actual feature movie, the kind that charges full admission and gets hyped on tv and purports to amuse small children and ostensible adults.
Sentiment: negative

Text: for the first time in years, de niro digs deep emotionally, perhaps because he's been stirred by the powerful work of his co-stars.
Sentiment: positive

Text: i'll bet the video game is a lot more fun than the film.
Sentiment:

Many studies have examined how to construct in-context examples for best performance, and they have found that prompt format, the choice of training examples, and example ordering can yield dramatically different results, ranging from near-random guessing to near state of the art.

Zhao et al. (2021) studied few-shot classification and argued that several biases in LLMs (GPT-3 in their experiments) contribute to this high variance: (1) Majority label bias occurs when the label distribution in the demonstrations is imbalanced; (2) Recency bias refers to a tendency to repeat the most recent label; (3) Common token bias reflects that LLMs tend to generate frequent tokens more often than rare ones. To address these biases, they proposed calibrating the model’s label probabilities so that, when the input string is N/A, the output label probabilities become uniform.

Tips for Example Selection

• Select examples that are semantically similar to the test example using $k$-NN clustering in embedding space (Liu et al., 2021).

• To choose a diverse and representative set of examples, Su et al. (2022) proposed a graph-based method: (1) Construct a directed graph $G=(V, E)$ using cosine similarity between sample embeddings (for example, from SBERT or other embedding models), where each node points to its $k$ nearest neighbors; (2) Initialize a selected set $\mathcal{L}=\emptyset$ and a remaining set $\mathcal{U}$. Score each sample $u \in \mathcal{U}$ by $ \text{score}(u) = \sum_{v \in \{v \mid (u, v) \in E, v\in \mathcal{U}\}} s(v)\quad\text{where }s(v)=\rho^{- \vert \{\ell \in \mathcal{L} \vert (v, \ell)\in E \}\vert},\quad\rho > 1 $ such that $s(v)$ becomes small when many of $v$’s neighbors have already been selected, thereby encouraging diversity in the chosen samples.

• Rubin et al. (2022) proposed training embeddings, using contrastive learning, that are tailored to a single training dataset for selecting in-context learning samples. Given each training pair $(x, y)$, the quality of an example $e_i$ (a formatted input-output pair) can be evaluated by a conditional probability from the LM: $\text{score}(e_i) = P_\text{LM}(y \mid e_i, x)$. For each training pair, examples with top-$k$ and bottom-$k$ scores are treated as positive and negative candidate sets, respectively, and used for contrastive learning.

• Some researchers have explored using Q-Learning for sample selection (Zhang et al. 2022).

• Inspired by uncertainty-based active learning, Diao et al. (2023) proposed identifying examples that show high disagreement or high entropy across multiple sampling trials, and then annotating those examples for inclusion in few-shot prompts.

Tips for Example Ordering

• A common recommendation is to keep examples diverse and relevant to the test input, and to randomize their order to mitigate majority label bias and recency bias.
• Neither increasing model size nor adding more training examples reduces variance across different permutations of in-context examples. An ordering that works well for one model may perform poorly for another. With a limited validation set, consider selecting an order that avoids extremely unbalanced predictions or excessive overconfidence (Lu et al. 2022).

Instruction Prompting

Few-shot demonstrations communicate intent by showing the model what to do, effectively encoding task instructions through examples. However, few-shot prompting can be expensive in token usage and can further constrain input length due to limited context. This raises a natural question: why not provide the instruction directly?

Instructed LMs (for example, InstructGPT and natural instruction) fine-tune a pretrained model on high-quality (task instruction, input, ground truth output) tuples, improving the LM’s ability to infer user intent and follow instructions. RLHF (Reinforcement Learning from Human Feedback) is a common approach. Instruction-following fine-tuning improves alignment with human intent and can significantly reduce the communication burden.

When working with instruction-tuned models, describe requirements in detail. Aim to be specific and precise, and avoid phrasing requirements primarily as “do not do X”; instead, clearly state what the model should do.

Please label the sentiment towards the movie of the given movie review. The sentiment label should be "positive" or "negative". 
Text: i'll bet the video game is a lot more fun than the film. 
Sentiment:

Clarifying the intended audience is another effective way to provide instructions.

• For example, to produce educational materials for kids:

Describe what is quantum physics to a 6-year-old.

• And for safe content:

... in language that is safe for work.

In-context instruction learning (Ye et al. 2023) combines few-shot learning with instruction prompting. It places multiple demonstrations from different tasks into the prompt, with each demonstration containing an instruction, a task input, and an output. Note that their experiments were limited to classification tasks, and the instruction prompt includes all label options.

Definition: Determine the speaker of the dialogue, "agent" or "customer".
Input: I have successfully booked your tickets.
Ouput: agent

Definition: Determine which category the question asks for, "Quantity" or "Location".
Input: What's the oldest building in US?
Ouput: Location

Definition: Classify the sentiment of the given movie review, "positive" or "negative".
Input: i'll bet the video game is a lot more fun than the film.
Output:

Self-Consistency Sampling

Self-consistency sampling (Wang et al. 2022a) samples multiple outputs using temperature > 0 and then selects the best candidate from the resulting set. The selection criterion depends on the task. A general-purpose approach is majority voting. For tasks that are straightforward to validate, such as programming problems with unit tests, you can execute the candidate solutions and verify correctness via the tests.

Chain-of-Thought (CoT)

Chain-of-thought (CoT) prompting (Wei et al. 2022) encourages the model to produce a sequence of short sentences that lay out the reasoning step by step, often called reasoning chains or rationales, culminating in the final answer. CoT is most beneficial for complex reasoning tasks, particularly when using large models (for example, more than 50B parameters). For simpler tasks, CoT typically provides only modest gains.

Types of CoT prompts

There are two primary forms of CoT prompting:

• Few-shot CoT: Provide a small number of demonstrations, each including a manually written (or model-generated) high-quality reasoning chain.

(All math reasoning examples are from GSM8k.)

Question: Tom and Elizabeth have a competition to climb a hill. Elizabeth takes 30 minutes to climb the hill. Tom takes four times as long as Elizabeth does to climb the hill. How many hours does it take Tom to climb up the hill?
Answer: It takes Tom 30*4 = <<30*4=120>>120 minutes to climb the hill.
It takes Tom 120/60 = <<120/60=2>>2 hours to climb the hill.
So the answer is 2.
===
Question: Jack is a soccer player. He needs to buy two pairs of socks and a pair of soccer shoes. Each pair of socks cost $9.50, and the shoes cost $92. Jack has $40. How much more money does Jack need?
Answer: The total cost of two pairs of socks is $9.50 x 2 = 
lt;<9.5*2=19>>19.
The total cost of the socks and the shoes is $19 + $92 = 
lt;<19+92=111>>111.
Jack need $111 - $40 = 
lt;<111-40=71>>71 more.
So the answer is 71.
===
Question: Marty has 100 centimeters of ribbon that he must cut into 4 equal parts. Each of the cut parts must be divided into 5 equal parts. How long will each final cut be?
Answer:

• Zero-shot CoT: Use a natural-language cue such as Let's think step by step to explicitly prompt the model to generate reasoning chains first, and then follow with Therefore, the answer is to elicit the final answer (Kojima et al. 2022). A similar cue is Let's work this out it a step by step to be sure we have the right answer (Zhou et al. 2022).

Question: Marty has 100 centimeters of ribbon that he must cut into 4 equal parts. Each of the cut parts must be divided into 5 equal parts. How long will each final cut be?
Answer: Let's think step by step.

Tips and Extensions

• Self-consistency sampling can improve reasoning accuracy by generating a diverse set of answers and then selecting the majority vote (Wang et al. 2022a).

• Another ensemble strategy is to randomize the example order or to replace human-written rationales with model-generated ones, injecting randomness across multiple trials. Then aggregate outputs via majority vote to determine the final answer (Wang et al. 2022b).

• When training examples include only correct answers (easy to verify) but not rationales, you can apply STaR (Self-Taught Reasoner; Zelikman et al. 2022): (1) Ask the LLM to generate reasoning chains and retain only those that lead to correct answers; (2) Fine-tune the model on the retained rationales, and repeat until convergence. Note that higher temperature is more likely to produce incorrect rationales that nevertheless arrive at correct answers. If training examples lack ground-truth answers, consider using majority votes as the “correct” answers.

• Prompts that demonstrate higher reasoning complexity can yield better performance, where complexity is measured by the number of reasoning steps in the chain. When separating steps, newline \n performs better than step i, period ., or semicolon ; (Fu et al. 2023).

• Complexity-based consistency explicitly favors more complex chains by taking a majority vote among only the top $k$ most complex generations (Fu et al. 2023).

• Shum et al. (2023) later reported that, in their experiments on GSM8k, CoT prompts containing only complex examples improve accuracy on complex questions but degrade performance on simple questions.

• Replacing Q: with Question: was found to be beneficial (Fu et al. 2023).

• Ye & Durrett (2022) found that including explanations in prompts provides small-to-moderate gains for NLP tasks involving reasoning over text (for example, QA and NLI), with effects that vary by model. They observed that explanations are more likely to be nonfactual than inconsistent (that is, whether the explanation entails the prediction). Nonfactual explanations are most likely to produce incorrect predictions.

• Self-Ask (Press et al. 2022) repeatedly prompts the model to ask follow-up questions to iteratively construct a thought process. These follow-up questions can be answered using search engine results. Related approaches include IRCoT (Interleaving Retrieval CoT; Trivedi et al. 2022) and ReAct (Reason + Act; Yao et al. 2023), which combine iterative CoT prompting with calls to Wikipedia APIs to retrieve relevant entities and content, then insert that information back into the prompt context.

• Tree of Thoughts (Yao et al. 2023) generalizes CoT by exploring multiple reasoning alternatives at each step. It decomposes a problem into multiple thought steps and generates multiple thoughts per step, forming a tree. Search can proceed via BFS or DFS, and each state can be evaluated by a classifier (prompted) or by majority vote.

Automatic Prompt Design

A prompt can be viewed as a sequence of prefix tokens that increases the probability of producing the desired output given an input. Under this view, prompts can be treated as trainable parameters and optimized directly in embedding space using gradient descent. Examples include AutoPrompt (Shin et al., 2020), Prefix-Tuning (Li & Liang (2021)), P-tuning (Liu et al. 2021), and Prompt-Tuning (Lester et al. 2021). This section in my “Controllable Neural Text Generation” post covers these methods well. The overall progression from AutoPrompt to Prompt-Tuning is toward increasingly simplified setups.

APE (Automatic Prompt Engineer; Zhou et al. 2022) searches a pool of model-generated instruction candidates and filters them using a chosen scoring function, ultimately selecting the highest-scoring instruction.

1. Prompt the LLM to generate instruction candidates using a small set of demonstrations formatted as input-output pairs, for example: {{Given desired input-output pairs}}\n\nThe instruction is.

2. Given a dataset of $\mathcal{D}_\text{train} = \{(x, y)\}$, the goal is to find an instruction $\rho$ such that $\rho^* = \arg\max_\rho \mathbb{E}_{(x, y) \in \mathcal{D}_\text{train}} [f(\rho, x, y)]$, where $f(.)$ is a per-sample scoring function, such as execution accuracy $\mathbb{1}[\text{LM}(.\vert \rho, x)=y]$ or log probability: $p_\text{LM}(y \mid \rho, x)$.

3. Apply an iterative Monte Carlo search to refine top candidates by proposing semantically similar variants via prompts such as Generate a variation of the following instruction while keeping the semantic meaning.\n\nInput: ...\n\nOutput:...

For automatically constructing chain-of-thought prompts, Shum et al. (2023) proposed augment-prune-select, a three-step workflow:

1. Augment: Generate multiple pseudo chains of thought for each question using few-shot or zero-shot CoT prompts.
2. Prune: Remove pseudo chains whose generated answers do not match the ground truth.
3. Select: Use a variance-reduced policy gradient approach to learn a probability distribution over selected examples, treating the distribution over examples as a policy and validation accuracy as the reward.

Zhang et al. (2023) instead used clustering to sample questions and then generate chains. They noted that LLMs tend to make certain categories of mistakes, and that one class of errors can cluster in embedding space. By sampling only one or a few examples from frequent error clusters, the method can reduce the number of demonstrations reflecting a single recurring error type and produce a more diverse set of examples.

1. Question clustering: Embed questions and run $k$-means clustering.
2. Demonstration selection: Select representative questions from each cluster, that is, one demonstration per cluster. Within each cluster, samples are ranked by distance to the centroid, and those closest to the centroid are selected first.
3. Rationale generation: Use zero-shot CoT to generate reasoning chains for the selected questions, then construct a few-shot prompt and run inference.

Augmented Language Models

The survey on augmented language models by Mialon et al. (2023) provides an excellent overview of several categories of language models augmented with reasoning capabilities and the ability to use external tools. I recommend it.

Retrieval

Many tasks require knowledge that postdates the model’s pretraining cutoff or resides in an internal or private knowledge base. In such cases, the model will not have the needed context unless it is explicitly supplied in the prompt. Many Open Domain Question Answering methods therefore perform retrieval over a knowledge base and then incorporate the retrieved content into the prompt. End-to-end accuracy depends on the quality of both retrieval and generation.

Lazaridou et al. (2022) examined augmenting LLMs by retrieving documents via Google Search. Given a question $q$, they extract clean text from 20 URLs returned by Google, producing a document set. Because the documents are long, each is split into 6-sentence paragraphs, $\{p\}$. They rank paragraphs by TF-IDF cosine similarity between the query and evidence paragraphs. Only the top-ranked paragraph is included in the prompt to generate an answer $a$.

For closed-book QA, they format each demonstration as follows to build few-shot prompts. They found that swapping the question and the evidence (which increases the distance between questions and answers) consistently reduces performance across datasets.

Evidence: ...
Question: ...
Answer: ...

The answer probability is computed in three different ways:

1. RAG style, $p(a_i \mid q) = \sum_{i=1}^n p_\text{tf-idf} (p_i \mid q) \cdot p_\text{LM}(a_i \mid q, p_i)$, where $p_\text{tf-idf} (p_i \mid q)$ is the normalized cosine similarity between TF-IDF passage representations and question representations.
2. Noisy channel inference, $p(a_i\mid q) = \frac{p_\text{LM}(q \mid a_i, p_i) \cdot p_\text{LM}(a_i \mid p_i)}{p_\text{LM}(q \mid p_i)}$
3. Product-of-Experts (PoE), which combines all of the probabilities above, along with $p_\text{LM}(p_i \mid q)$.

Based on their experiments across generation and classification tasks, the three reranking scores perform in the order PoE > Noisy channel > RAG. Among the individual probabilities, $p_\text{LM}(a \mid q, p_i)$ and $p_\text{LM}(q \mid p_i, a)$ are the most informative. $p_\text{LM}(q \mid p_i, a)$ measures how well the LM can explain the question given the evidence paragraph and the answer, and it can reliably rerank answer candidates.

On the SituatedQA dataset, which includes questions grounded in different dates, they observed that even though the LM (with a pretraining cutoff in 2020) can access up-to-date information via Google Search, performance on post-2020 questions remains much worse than on pre-2020 questions. This suggests discrepancies or conflicts between contextual information and the model’s internal parametric knowledge.

It is also reported to be helpful to rely on “internal retrieval”, that is, to have the model generate knowledge about a topic before answering the question (Liu et al. 2022). First, extract knowledge using the following template:

Generate some knowledge about the input. Examples:

Input: What type of water formation is formed by clouds?
Knowledge: Clouds are made of water vapor.

Input: {question}
Knowledge:

Then, using the model-generated knowledge, prompt the LM further to obtain the answer.

Programming Language

Both PAL (Program-aided language models; Gao et al. 2022) and PoT (Program of Thoughts prompting; Chen et al. 2022) prompt an LLM to generate programming-language statements to solve natural-language reasoning problems, thereby offloading the solution step to a runtime such as a Python interpreter. This setup separates complex computation from reasoning, and it depends on an LM with sufficiently strong coding ability.

External APIs

TALM (Tool Augmented Language Models; Parisi et al. 2022) is a language model extended with text-to-text API calls. The LM is prompted to emit |tool-call and tool input text, conditioned on the task input, to assemble API requests. When |result appears, the designated tool API is invoked, and the returned output is appended to the token sequence. The model then produces the final answer after the |output token.

TALM uses a self-play strategy to iteratively bootstrap a dataset of tool-usage examples and fine-tune the LM on that data. In this setting, self-play means the model repeatedly interacts with a tool API and expands the dataset based on whether introducing a new tool API improves model outputs. Toolformer adopts the same general idea, described in more detail below. The overall pipeline loosely resembles a reinforcement learning process in which the LM serves as the policy network and is optimized with policy gradients using a binary reward signal.

Toolformer (Schick et al. 2023) is an LM that can invoke external tools through simple APIs. It is constructed in a self-supervised way and requires only a small number of demonstrations for each API. The Toolformer toolbox includes:

• Calculator, to compensate for the LM’s limited precision in mathematics;
• Q&A system, to mitigate unfaithful content and hallucinations;
• Search engine, to supply up-to-date information beyond the pretraining cutoff time;
• Translation system, to improve performance for low-resource languages;
• Calendar, to help the LM track the progression of time.

Toolformer training proceeds as follows:

1. Prompting to annotate candidate API calls. A pretrained LM is asked to annotate a dataset using few-shot learning with examples that demonstrate API usage. Formatting example:

   

- Each API call is represented as a tuple of (API name, corresponding input), $c=(a_c, i_c)$ and its corresponding result is denoted as $r$. The API call sequences with and without results are labeled as follows, respectively:

    <div>
    $
    \begin{aligned}
    e(c) &= \langle\texttt{API}\rangle a_c(i_c) \langle\texttt{/API}\rangle \\
    e(c, r) &= \langle\texttt{API}\rangle a_c(i_c) \to r \langle\texttt{/API}\rangle
    \end{aligned}
    $
    </div>

- Sample API calls based on the probabilities $p_\text{LM}(\langle\texttt{API}\rangle \mid \text{prompt}(\mathbf{x}), \mathbf{x}_{1:i})$ and select top $k$ candidate positions for doing API calls at position $i$ if the probability is larger than a threshold.

- Then we sample potential API calls from the LM given the sequence $[\text{prompt}(\mathbf{x}), x_1, \dots, x_{i-1}, \langle\texttt{API}\rangle]$ as prefix and $\langle\texttt{/API}\rangle$ as suffix.

2. Filtering annotations based on whether API calls help the model predict future tokens. A self-supervised loss is used to determine which API calls are genuinely beneficial.

   • Run each API call $c_i$ to obtain the corresponding result $r_i$.

   • Compute a weighted cross-entropy loss for the LM over tokens $x_i, \dots, x_n$ when the model is prefixed with the prompt. Two variants are evaluated: one includes the API result, and the other replaces it with the empty sequence $\varepsilon$.

     $ \begin{aligned} L^+_i &= L_i(e(c_i, r_i)) \\ L^-_i &= \min(L_i(\varepsilon), L_i(e(c_i, \varepsilon))) \\ \end{aligned} $

     Only API calls with $L^-_i - L^+_i$ exceeding a threshold are retained. This indicates that including the API call and its result helps the model better predict subsequent tokens.

3. Fine-tuning the LM on the annotated dataset. The updated training sequences are formed as $\mathbf{x}^* = x_{1:i-1}, e(c_i, r_i), x_{i:n}$ . Training uses a mixture of the original dataset (for example, a subset of CCNet, as described in the paper) and the augmented dataset.

At inference time, decoding proceeds until the model emits the “$\to$ " token, which signals that the next step expects a response from an API call.

At present, Toolformer does not support chained tool usage (that is, feeding the output of one tool into another) or interactive tool usage (that is, incorporating an API response after a human selection step). Both represent promising directions for extending the model.

Citation

Cited as:

Weng, Lilian. (Mar 2023). Prompt Engineering. Lil’Log. https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/.

Or

@article{weng2023prompt,
  title   = "Prompt Engineering",
  author  = "Weng, Lilian",
  journal = "lilianweng.github.io",
  year    = "2023",
  month   = "Mar",
  url     = "https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/"
}

Useful Resources

• OpenAI Cookbook provides many in-depth examples of how to use LLMs efficiently.
• LangChain, a library for combining language models with other components to build applications.
• Prompt Engineering Guide repo contains a fairly comprehensive collection of educational materials on prompt engineering.
• learnprompting.org
• PromptPerfect
• Semantic Kernel

References

[1] Zhao et al. “Calibrate Before Use: Improving Few-shot Performance of Language Models.” ICML 2021

[2] Liu et al. “What Makes Good In-Context Examples for GPT-3?” arXiv preprint arXiv:2101.06804 (2021).

[3] Lu et al. “Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity.” ACL 2022

[4] Ye et al. “In-Context Instruction Learning.” arXiv preprint arXiv:2302.14691 (2023).

[5] Su et al. “Selective annotation makes language models better few-shot learners.” arXiv preprint arXiv:2209.01975 (2022).

[6] Rubin et al. “Learning to retrieve prompts for in-context learning.” NAACL-HLT 2022

[7] Wei et al. “Chain of thought prompting elicits reasoning in large language models.” NeurIPS 2022

[8] Wang et al. “Self-Consistency Improves Chain of Thought Reasoning in Language Models.” ICLR 2023.

[9] Diao et al. “Active Prompting with Chain-of-Thought for Large Language Models.” arXiv preprint arXiv:2302.12246 (2023).

[10] Zelikman et al. “STaR: Bootstrapping Reasoning With Reasoning.” arXiv preprint arXiv:2203.14465 (2022).

[11] Ye & Durrett. “The unreliability of explanations in few-shot in-context learning.” arXiv preprint arXiv:2205.03401 (2022).

[12] Trivedi et al. “Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.” arXiv preprint arXiv:2212.10509 (2022).

[13] Press et al. “Measuring and narrowing the compositionality gap in language models.” arXiv preprint arXiv:2210.03350 (2022).

[14] Yao et al. “ReAct: Synergizing reasoning and acting in language models.” ICLR 2023.

[15] Fu et al. “Complexity-based prompting for multi-step reasoning.” arXiv preprint arXiv:2210.00720 (2022).

[16] Wang et al. “Rationale-augmented ensembles in language models.” arXiv preprint arXiv:2207.00747 (2022).

[17] Zhang et al. “Automatic chain of thought prompting in large language models.” arXiv preprint arXiv:2210.03493 (2022).

[18] Shum et al. “Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data.” arXiv preprint arXiv:2302.12822 (2023).

[19] Zhou et al. “Large Language Models Are Human-Level Prompt Engineers.” ICLR 2023.

[20] Lazaridou et al. “Internet augmented language models through few-shot prompting for open-domain question answering.” arXiv preprint arXiv:2203.05115 (2022).

[21] Chen et al. “Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks.” arXiv preprint arXiv:2211.12588 (2022).

[22] Gao et al. “PAL: Program-aided language models.” arXiv preprint arXiv:2211.10435 (2022).

[23] Parisi et al. “TALM: Tool Augmented Language Models” arXiv preprint arXiv:2205.12255 (2022).

[24] Schick et al. “Toolformer: Language Models Can Teach Themselves to Use Tools.” arXiv preprint arXiv:2302.04761 (2023).

[25] Mialon et al. “Augmented Language Models: a Survey” arXiv preprint arXiv:2302.07842 (2023).

[26] Yao et al. “Tree of Thoughts: Deliberate Problem Solving with Large Language Models.” arXiv preprint arXiv:2305.10601 (2023).