Building agents that use an LLM (large language model) as the primary controller is an appealing idea. Several proof-of-concept demos, including AutoGPT, GPT-Engineer, and BabyAGI, provide motivating examples. The potential of LLMs goes far beyond producing well-written copy, stories, essays, and programs; they can also be positioned as powerful general-purpose problem solvers.

Agent System Overview

In an LLM-powered autonomous agent system, the LLM serves as the agent’s brain and is supported by several core components:

• Planning
  • Subgoals and decomposition: The agent breaks a large task into smaller, tractable subgoals, which makes complex tasks easier to handle efficiently.
  • Reflection and refinement: The agent can critique and reflect on prior actions, learn from mistakes, and adjust future steps accordingly, improving final output quality.
• Memory
  • Short-term memory: In-context learning (see Prompt Engineering) can be viewed as using the model’s short-term memory during learning.
  • Long-term memory: This enables the agent to retain and recall (infinite) information over long periods, commonly by using an external vector store with fast retrieval.
• Tool use
  • The agent learns to call external APIs to obtain information not contained in the model weights (which are often difficult to modify after pre-training), including up-to-date information, code execution capabilities, access to proprietary sources, and more.

Component One: Planning

Complex tasks typically require many steps. To execute them successfully, an agent must identify those steps and plan ahead.

Task Decomposition

Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for improving model performance on challenging tasks. The model is instructed to “think step by step,” which encourages additional test-time computation to decompose difficult problems into smaller, simpler steps. CoT turns a large task into multiple manageable tasks and offers insight into how the model is reasoning.

Tree of Thoughts (Yao et al. 2023) generalizes CoT by exploring multiple reasoning alternatives at each step. It begins by decomposing a problem into multiple thought steps and generating multiple candidate thoughts per step, yielding a tree structure. The search procedure may use BFS (breadth-first search) or DFS (depth-first search), and each state can be evaluated either by a classifier (prompt-based) or by majority vote.

Task decomposition can be performed (1) by the LLM using simple prompts such as "Steps for XYZ.\n1." or "What are the subgoals for achieving XYZ?", (2) via task-specific instructions (for example, "Write a story outline." when writing a novel), or (3) with human input.

A separate approach, LLM+P (Liu et al. 2023), relies on an external classical planner for long-horizon planning. It uses the Planning Domain Definition Language (PDDL) as an intermediate interface for representing the planning problem. In this workflow, the LLM (1) translates the task into “Problem PDDL,” then (2) asks a classical planner to produce a PDDL plan using an existing “Domain PDDL,” and finally (3) translates the resulting PDDL plan back into natural language. In effect, the planning step is delegated to an external tool, assuming domain-specific PDDL and an appropriate planner are available, which is common in certain robotics settings but less common in many other domains.

Self-Reflection

Self-reflection is an essential capability that enables autonomous agents to improve iteratively by revisiting prior action choices and correcting earlier errors. This is especially important in real-world settings where trial and error is unavoidable.

ReAct (Yao et al. 2023) combines reasoning and acting within an LLM by expanding the action space into a mix of task-specific discrete actions and language. Discrete actions allow the LLM to interact with an environment (for example, by calling a Wikipedia search API), while language prompts the LLM to generate reasoning traces in natural language.

The ReAct prompt template includes explicit steps that encourage the LLM to think, with an approximate format like the following:

Thought: ...
Action: ...
Observation: ...
... (Repeated many times)

Across experiments on both knowledge-intensive tasks and decision-making tasks, ReAct outperforms the Act-only baseline, in which the Thought: … step is removed.

Reflexion (Shinn & Labash 2023) is a framework that equips agents with dynamic memory and self-reflection mechanisms to enhance reasoning. Reflexion uses a standard RL setup: the reward model provides a simple binary reward, and the action space follows the ReAct formulation, where the task-specific action space is augmented with language to support more sophisticated reasoning steps. After each action $a_t$, the agent computes a heuristic $h_t$ and may optionally decide to reset the environment to begin a new trial, depending on the outcome of self-reflection.

The heuristic determines when a trajectory is inefficient or includes hallucination and therefore should be terminated. Inefficient planning refers to trajectories that run too long without success. Hallucination is defined as a sequence of repeated identical actions that produce the same observation from the environment.

Self-reflection is produced by showing the LLM two-shot examples, where each example consists of a pair (failed trajectory, ideal reflection that guides future changes to the plan). These reflections are then stored in the agent’s working memory, up to three at a time, and used as context when querying the LLM.

Chain of Hindsight (CoH; Liu et al. 2023) pushes a model to improve its own outputs by explicitly providing a sequence of prior outputs, each paired with feedback annotations. Human feedback data consists of $D_h = \{(x, y_i , r_i , z_i)\}_{i=1}^n$, where $x$ is the prompt, each $y_i$ is a model completion, $r_i$ is the human rating of $y_i$, and $z_i$ is the corresponding hindsight feedback written by a human. Assume these feedback tuples are ranked by reward, $r_n \geq r_{n-1} \geq \dots \geq r_1$ The training procedure is supervised fine-tuning, where each data example is a sequence of the form $\tau_h = (x, z_i, y_i, z_j, y_j, \dots, z_n, y_n)$, where $\leq i \leq j \leq n$. The model is fine-tuned to predict only $y_n$ conditioned on the sequence prefix, enabling it to self-reflect and generate better outputs from the feedback sequence. At test time, the model can optionally receive multiple rounds of instructions from human annotators.

To reduce overfitting, CoH includes a regularization term that maximizes the log-likelihood on the pre-training dataset. To mitigate shortcutting and copying (since feedback sequences contain many repeated words), the method randomly masks 0% - 5% of past tokens during training.

In their experiments, the training data combines WebGPT comparisons, summarization from human feedback, and human preference dataset.

The core intuition behind CoH is to provide, in context, a history of sequentially improved outputs and train the model to continue that trend and generate better results. Algorithm Distillation (AD; Laskin et al. 2023) applies a similar idea to cross-episode trajectories in reinforcement learning, where an algorithm is represented as a long history-conditioned policy. Because an agent interacts with an environment repeatedly and tends to improve slightly across episodes, AD concatenates the learning history and feeds it to the model. The next predicted action is therefore expected to perform better than actions taken in earlier trials. The objective is to learn the RL process itself, rather than training a task-specific policy.

The paper hypothesizes that any algorithm capable of generating learning histories can be distilled into a neural network by applying behavioral cloning over actions. The history is produced by a set of source policies, each trained on a specific task. During training, for each RL run, a random task is sampled and a subsequence of multi-episode history is used for learning, so that the distilled policy becomes task-agnostic.

In practice, the model’s context window is limited, so episodes must be short enough to assemble multi-episode histories. Multi-episodic contexts of 2-4 episodes are required to learn a near-optimal in-context RL algorithm. The emergence of in-context RL depends on having sufficiently long context.

Compared with three baselines, ED (expert distillation, behavioral cloning from expert trajectories rather than from learning history), source policy (used to generate trajectories for distillation via UCB), and RL^2 (Duan et al. 2017; treated as an upper bound because it requires online RL), AD exhibits in-context RL with performance approaching RL^2 despite using only offline RL, and it learns substantially faster than the other baselines. When conditioned on partial training histories from the source policy, AD also improves more rapidly than the ED baseline.

Component Two: Memory

(A big thank you to ChatGPT for helping me draft this section. Through my conversations with ChatGPT, I learned a great deal about the human brain and data structures for fast MIPS.)

Types of Memory

Memory can be described as the processes used to acquire, store, retain, and later retrieve information. Human cognition includes multiple types of memory.

1. Sensory Memory: The earliest stage of memory, which preserves impressions of sensory input (visual, auditory, etc.) after the original stimulus has ended. Sensory memory typically lasts only a few seconds. Subcategories include iconic memory (visual), echoic memory (auditory), and haptic memory (touch).

2. Short-Term Memory (STM) or Working Memory: Stores information that we are currently aware of and that is needed to perform complex cognitive activities such as learning and reasoning. Short-term memory is commonly estimated to hold about 7 items (Miller 1956) and to persist for 20-30 seconds.

3. Long-Term Memory (LTM): Can store information for extremely long durations, from days to decades, with essentially unlimited capacity. LTM includes two subtypes:

   • Explicit / declarative memory: Memory for facts and events that can be consciously recalled, including episodic memory (events and experiences) and semantic memory (facts and concepts).
   • Implicit / procedural memory: Unconscious memory involving skills and routines performed automatically, such as riding a bike or typing.

We can loosely map these concepts as follows:

• Sensory memory corresponds to learning embedding representations for raw inputs, including text, images, or other modalities.
• Short-term memory corresponds to in-context learning. It is brief and finite because it is limited by the Transformer’s finite context window length.
• Long-term memory corresponds to an external vector store that the agent can attend to at query time through fast retrieval.

Maximum Inner Product Search (MIPS)

External memory can reduce the impact of a finite attention span. A common approach is to store embedding representations in a vector database that supports fast maximum inner-product search (MIPS). To accelerate retrieval, the usual choice is an approximate nearest neighbors (ANN)​ method that returns approximately the top k nearest neighbors, trading a small amount of accuracy for substantial speed.

Common ANN options for fast MIPS include:

• LSH (Locality-Sensitive Hashing): Introduces a hashing function so that similar inputs are mapped to the same bucket with high probability, while the number of buckets is much smaller than the number of inputs.
• ANNOY (Approximate Nearest Neighbors Oh Yeah): Uses random projection trees, a collection of binary trees where each internal node is a hyperplane that splits the space in half and each leaf stores a data point. Trees are built independently and randomly, which partially resembles hashing. During search, ANNOY queries all trees, repeatedly follows the half closest to the query, and aggregates results. The idea is related to a KD tree but is significantly more scalable.
• HNSW (Hierarchical Navigable Small World): Inspired by small world networks, where most nodes are reachable from any other within a small number of steps (for example, the “six degrees of separation” property in social networks). HNSW constructs hierarchical layers of small-world graphs, with the bottom layer containing the actual data points. Intermediate layers act as shortcuts that speed up search. During retrieval, HNSW starts from a random node in the top layer and moves toward the target; when it can no longer get closer, it drops to the next layer, continuing until it reaches the bottom. Moves in upper layers can traverse large distances in the space, while moves in lower layers improve search precision.
• FAISS (Facebook AI Similarity Search): Assumes that in high-dimensional space, distances between points follow a Gaussian distribution, implying clustering among data points. FAISS performs vector quantization by partitioning the space into clusters and then refining quantization within clusters. Search first identifies candidate clusters with coarse quantization, then searches within each candidate cluster using finer quantization.
• ScaNN (Scalable Nearest Neighbors): The key innovation is anisotropic vector quantization. It quantizes a data point $x_i$ to $\tilde{x}_i$ so that the inner product $\langle q, x_i \rangle$ matches the original distance $\angle q, \tilde{x}_i$ as closely as possible, rather than selecting the closest quantization centroid.

For additional MIPS algorithms and performance comparisons, see ann-benchmarks.com.

Component Three: Tool Use

Tool use is a notable and differentiating human characteristic. Humans create, modify, and apply external objects to accomplish goals beyond our physical and cognitive constraints. Providing LLMs with external tools can substantially expand what they can do.

MRKL (Karpas et al. 2022), short for “Modular Reasoning, Knowledge and Language,” is a neuro-symbolic architecture for autonomous agents. A MRKL system is designed to include a set of “expert” modules, while a general-purpose LLM acts as a router that directs queries to the most appropriate expert module. These modules may be neural (for example, deep learning models) or symbolic (for example, a math calculator, a currency converter, or a weather API).

They also ran an experiment fine-tuning an LLM to call a calculator, using arithmetic as the test case. The study found that verbal math problems were more difficult than explicitly stated math expressions because LLMs (the 7B Jurassic1-large model) could not reliably extract the correct arguments for basic arithmetic. The results emphasize that even when external symbolic tools are reliable, knowing when to use tools and how to use them is crucial, and that this capability depends on the LLM.

Both TALM (Tool Augmented Language Models; Parisi et al. 2022) and Toolformer (Schick et al. 2023) fine-tune a LM so that it learns to use external tool APIs. The dataset is expanded based on whether adding a new API call annotation improves the quality of the model output. For additional details, see the “External APIs” section of Prompt Engineering.

ChatGPT Plugins and OpenAI API function calling are practical examples of LLMs enhanced with tool-use capabilities. Tool APIs can be provided by third-party developers (as with Plugins) or defined by the developer (as with function calls).

HuggingGPT (Shen et al. 2023) is a framework that uses ChatGPT as a task planner, selects models hosted on the HuggingFace platform based on model descriptions, and summarizes the final response using the execution results.

The system consists of four stages:

(1) Task planning: The LLM serves as the brain and parses the user request into multiple tasks. Each task includes four attributes: task type, ID, dependencies, and arguments. Few-shot examples are used to guide the LLM’s task parsing and planning.

Instruction:

(2) Model selection: The LLM assigns tasks to expert models, framing the choice as a multiple-choice question. The LLM is given a list of candidate models to choose from. Because context length is limited, filtering by task type is required.

Instruction:

(3) Task execution: Expert models run the assigned tasks and record the results.

Instruction:

(4) Response generation: The LLM consumes the execution results and returns a summarized response to the user.

To deploy HuggingGPT in real-world settings, several challenges must be addressed: (1) Efficiency improvements are needed, since both multiple LLM inference rounds and interactions with other models slow the end-to-end workflow; (2) The system depends on a long context window to communicate complex task content; (3) Greater stability is needed in both LLM outputs and external model services.

API-Bank (Li et al. 2023) is a benchmark designed to evaluate tool-augmented LLMs. It includes 53 commonly used API tools, a complete tool-augmented LLM workflow, and 264 annotated dialogues containing 568 API calls. The APIs span a wide range of functions, including search engines, calculators, calendar queries, smart-home control, schedule management, health-data management, account authentication workflows, and more. Because there are many APIs, the LLM first uses an API search engine to locate an appropriate API, then consults the corresponding documentation to execute the call.

Within the API-Bank workflow, LLMs must make several decisions, and the accuracy of each decision can be evaluated step by step. These decisions include:

1. Determine whether an API call is necessary.
2. Select the appropriate API to call. If an initial attempt is insufficient, LLMs may need to iteratively revise the API inputs (for example, by choosing better search keywords for a Search Engine API).
3. Generate a response based on the API results. If the results are unsatisfactory, the model can decide to refine its approach and call the API again.

This benchmark evaluates an agent’s tool-use capabilities across three levels:

• Level-1 evaluates the ability to call the API. Given an API description, the model must decide whether to call the API, invoke it correctly, and respond appropriately to the returned output.
• Level-2 examines the ability to retrieve the API. The model must search for candidate APIs that could satisfy the user’s requirement and learn how to use them by reading documentation.
• Level-3 assesses the ability to plan API usage beyond retrieval and calling. Given ambiguous user requests (for example, scheduling group meetings or booking flights, hotels, or restaurants for a trip), the model may need to perform multiple API calls to complete the task.

Case Studies

Scientific Discovery Agent

ChemCrow (Bran et al. 2023) is a domain-specific example in which an LLM is augmented with 13 expert-designed tools to complete tasks spanning organic synthesis, drug discovery, and materials design. The workflow, implemented in LangChain, aligns with what was described earlier in ReAct and MRKLs, combining CoT reasoning with tools tailored to the target tasks:

• The LLM receives a list of tool names, descriptions of their intended use, and details about the required input and expected output.
• The model is then instructed to answer a user prompt, using the provided tools when needed. The instruction recommends following the ReAct format: Thought, Action, Action Input, Observation.

A notable finding is that, although an LLM-based evaluation concluded that GPT-4 and ChemCrow perform almost equivalently, expert human evaluations focused on solution completeness and chemical correctness showed ChemCrow outperforming GPT-4 by a large margin. This suggests a potential weakness in using an LLM to evaluate its own performance in domains that require deep expertise. When the model lacks domain knowledge, it may not recognize its own errors and therefore may not reliably judge the correctness of its outputs.

Boiko et al. (2023) similarly investigated LLM-empowered agents for scientific discovery, focusing on autonomous design, planning, and execution of complex scientific experiments. This agent can use tools to browse the Internet, read documentation, run code, call robotics experimentation APIs, and leverage other LLMs.

For instance, when asked to "develop a novel anticancer drug", the model produced the following reasoning steps:

1. inquired about current trends in anticancer drug discovery;
2. selected a target;
3. requested a scaffold targeting these compounds;
4. Once the compound was identified, the model attempted its synthesis.

The authors also discussed risks, particularly involving illicit drugs and bioweapons. They created a test set consisting of known chemical weapon agents and asked the agent to synthesize them. Of 11 requests, 4 (36%) were accepted, resulting in a synthesis solution, and the agent attempted to consult documentation to carry out the procedure. The remaining 7 out of 11 were rejected. Among these 7 rejections, 5 occurred after a web search, while 2 were rejected based solely on the prompt.

Generative Agents Simulation

Generative Agents (Park, et al. 2023) is a highly engaging experiment in which 25 virtual characters, each controlled by an LLM-powered agent, live and interact in a sandbox environment inspired by The Sims. Generative agents aim to create believable simulacra of human behavior for interactive applications.

The design combines an LLM with memory, planning, and reflection mechanisms, enabling agents to act in ways conditioned on past experience and to interact with other agents.

• Memory stream: a long-term memory module (external database) that stores a comprehensive list of an agent’s experiences in natural language.
  • Each element is an observation, an event directly provided by the agent. - Inter-agent communication can trigger new natural language statements.
• Retrieval model: brings relevant context forward to inform the agent’s behavior, based on relevance, recency, and importance.
  • Recency: more recent events receive higher scores.
  • Importance: separates mundane memories from core memories, by asking the LM directly.
  • Relevance: measures how related an item is to the current situation or query.
• Reflection mechanism: aggregates memories into higher-level inferences over time and steers the agent’s future behavior. These are higher-level summaries of past events (<- note that this differs slightly from self-reflection above).
  • Prompt the LM with the 100 most recent observations and ask it to generate the 3 most salient high-level questions given a set of observations/statements, then ask the LM to answer those questions.
• Planning & Reacting: converts reflections and environmental information into actions.
  • Planning is primarily intended to optimize believability both in the moment and over time.
  • Prompt template: {Intro of an agent X}. Here is X's plan today in broad strokes: 1)
  • Planning and reacting take into account relationships between agents, as well as observations made by one agent about another.
  • Environment information is represented in a tree structure.

This simulation produces emergent social behaviors, including information diffusion, relationship memory (for example, two agents continuing an earlier conversation topic), and coordination of social events (for example, hosting a party and inviting many others).

Proof-of-Concept Examples

AutoGPT has attracted substantial attention as a demonstration of how autonomous agents can be built with an LLM serving as the primary controller. It has significant reliability issues due to its natural-language interface, but it remains an interesting proof-of-concept. A large portion of the AutoGPT codebase is devoted to format parsing.

Below is the system message used by AutoGPT, where {{...}} denotes user-provided inputs:

You are {{ai-name}}, {{user-provided AI bot description}}.
Your decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.

GOALS:

1. {{user-provided goal 1}}
2. {{user-provided goal 2}}
3. ...
4. ...
5. ...

Constraints:
1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.
2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.
3. No user assistance
4. Exclusively use the commands listed in double quotes e.g. "command name"
5. Use subprocesses for commands that will not terminate within a few minutes

Commands:
1. Google Search: "google", args: "input": "<search>"
2. Browse Website: "browse_website", args: "url": "<url>", "question": "<what_you_want_to_find_on_website>"
3. Start GPT Agent: "start_agent", args: "name": "<name>", "task": "<short_task_desc>", "prompt": "<prompt>"
4. Message GPT Agent: "message_agent", args: "key": "<key>", "message": "<message>"
5. List GPT Agents: "list_agents", args:
6. Delete GPT Agent: "delete_agent", args: "key": "<key>"
7. Clone Repository: "clone_repository", args: "repository_url": "<url>", "clone_path": "<directory>"
8. Write to file: "write_to_file", args: "file": "<file>", "text": "<text>"
9. Read file: "read_file", args: "file": "<file>"
10. Append to file: "append_to_file", args: "file": "<file>", "text": "<text>"
11. Delete file: "delete_file", args: "file": "<file>"
12. Search Files: "search_files", args: "directory": "<directory>"
13. Analyze Code: "analyze_code", args: "code": "<full_code_string>"
14. Get Improved Code: "improve_code", args: "suggestions": "<list_of_suggestions>", "code": "<full_code_string>"
15. Write Tests: "write_tests", args: "code": "<full_code_string>", "focus": "<list_of_focus_areas>"
16. Execute Python File: "execute_python_file", args: "file": "<file>"
17. Generate Image: "generate_image", args: "prompt": "<prompt>"
18. Send Tweet: "send_tweet", args: "text": "<text>"
19. Do Nothing: "do_nothing", args:
20. Task Complete (Shutdown): "task_complete", args: "reason": "<reason>"

Resources:
1. Internet access for searches and information gathering.
2. Long Term memory management.
3. GPT-3.5 powered Agents for delegation of simple tasks.
4. File output.

Performance Evaluation:
1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.
2. Constructively self-criticize your big-picture behavior constantly.
3. Reflect on past decisions and strategies to refine your approach.
4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.

You should only respond in JSON format as described below
Response Format:
{
    "thoughts": {
        "text": "thought",
        "reasoning": "reasoning",
        "plan": "- short bulleted\n- list that conveys\n- long-term plan",
        "criticism": "constructive self-criticism",
        "speak": "thoughts summary to say to user"
    },
    "command": {
        "name": "command name",
        "args": {
            "arg name": "value"
        }
    }
}
Ensure the response can be parsed by Python json.loads

GPT-Engineer is another project that generates an entire code repository from a task described in natural language. GPT-Engineer is instructed to think through a set of smaller components to build and to request user input to clarify questions when needed.

The following is a sample task-clarification conversation sent to the OpenAI ChatCompletion endpoint by GPT-Engineer. User inputs are wrapped in {{user input text}}.

[
  {
    "role": "system",
    "content": "You will read instructions and not carry them out, only seek to clarify them.\nSpecifically you will first summarise a list of super short bullets of areas that need clarification.\nThen you will pick one clarifying question, and wait for an answer from the user.\n"
  },
  {
    "role": "user",
    "content": "We are writing {{a Super Mario game in python. MVC components split in separate files. Keyboard control.}}\n"
  },
  {
    "role": "assistant",
    "content": "Summary of areas that need clarification:\n1. Specifics of the Super Mario game (e.g. level design, characters, gameplay mechanics)\n2. Details about the MVC components (e.g. which components are in each file)\n3. Keyboard control implementation (e.g. which keys to use, how to handle input)\n\nClarifying question:\nCan you provide more details about the Super Mario game, such as level design, characters, and gameplay mechanics?"
  },
  {
    "role": "user",
    "content": "{{There are 10 levels in total. The main character is a plumber named Mario, who can walk and jump. It is a classical platform game just like Super Mario. The main character moves from left to right, trying to get to the destination, where there are many obstacles and attacks from enemies in the process.}}\n\nIs anything else unclear? If yes, only answer in the form:\n{remaining unclear areas} remaining questions.\n{Next question}\nIf everything is sufficiently clear, only answer \"Nothing more to clarify.\"."
  },
  {
    "role": "assistant",
    "content": "Remaining unclear areas: 2 remaining questions.\nCan you provide more information about how the MVC components are split into separate files?"
  },
  {
    "role": "user",
    "content": "{{Make your own assumptions and state them explicitly before starting}}"
  }
]

After these clarifications, the agent switched into a code-writing mode using a different system message.

System message:

CODE

Conversation samples:

[
  {
    "role": "system",
    "content": "You will get instructions for code to write.\nYou will write a very long answer. Make sure that every detail of the architecture is, in the end, implemented as code.\nMake sure that every detail of the architecture is, in the end, implemented as code.\n\nThink step by step and reason yourself to the right decisions to make sure we get it right.\nYou will first lay out the names of the core classes, functions, methods that will be necessary, as well as a quick comment on their purpose.\n\nThen you will output the content of each file including ALL code.\nEach file must strictly follow a markdown code block format, where the following tokens must be replaced such that\nFILENAME is the lowercase file name including the file extension,\nLANG is the markup code block language for the code's language, and CODE is the code:\n\nFILENAME\n```LANG\nCODE\n```\n\nYou will start with the \"entrypoint\" file, then go to the ones that are imported by that file, and so on.\nPlease note that the code should be fully functional. No placeholders.\n\nFollow a language and framework appropriate best practice file naming convention.\nMake sure that files contain all imports, types etc. Make sure that code in different files are compatible with each other.\nEnsure to implement all code, if you are unsure, write a plausible implementation.\nInclude module dependency or package manager dependency definition file.\nBefore you finish, double check that all parts of the architecture is present in the files.\n\nUseful to know:\nYou almost always put different classes in different files.\nFor Python, you always create an appropriate requirements.txt file.\nFor NodeJS, you always create an appropriate package.json file.\nYou always add a comment briefly describing the purpose of the function definition.\nYou try to add comments explaining very complex bits of logic.\nYou always follow the best practices for the requested languages in terms of describing the code written as a defined\npackage/project.\n\n\nPython toolbelt preferences:\n- pytest\n- dataclasses\n"
  },
 #  … same conversation as earlier, ended with "Make your own assumptions and state them explicitly before starting".
  {
    "role": "assistant",
    "content": "Assumptions:\n1. Model: The model will contain the game's data, such as level information, character states, and enemy positions.\n2. View: The view will handle the game's visuals, including rendering the game objects, backgrounds, and updating the display.\n3. Controller: The controller will manage user input, such as keyboard controls, and update the model accordingly.\n\nWith these assumptions in mind, is there any specific aspect of the keyboard control implementation you would like me to clarify?"
  },
  {
    "role": "user",
    "content": "Please now remember the steps:\n\nThink step by step and reason yourself to the right decisions to make sure we get it right.\nFirst lay out the names of the core classes, functions, methods that will be necessary, As well as a quick comment on their purpose.\n\nThen you will output the content of each file including ALL code.\nEach file must strictly follow a markdown code block format, where the following tokens must be replaced such that\nFILENAME is the lowercase file name including the file extension,\nLANG is the markup code block language for the code's language, and CODE is the code:\n\nFILENAME\n```LANG\nCODE\n```\n\nPlease note that the code should be fully functional. No placeholders.\n\nYou will start with the \"entrypoint\" file, then go to the ones that are imported by that file, and so on.\nFollow a language and framework appropriate best practice file naming convention.\nMake sure that files contain all imports, types etc. The code should be fully functional. Make sure that code in different files are compatible with each other.\nBefore you finish, double check that all parts of the architecture is present in the files.\n"
  }
]

Challenges

After reviewing key ideas and demonstrations of LLM-centered agents, several shared limitations become apparent:

• Finite context length: Limited context capacity constrains how much historical information, detailed instruction, API call context, and returned output can be included. System designs must operate within this restricted communication bandwidth. At the same time, mechanisms such as self-reflection, which aim to learn from prior mistakes, would benefit substantially from long or even unbounded context windows. Although vector stores and retrieval can provide access to a broader knowledge pool, their representational power is weaker than full attention.

• Challenges in long-term planning and task decomposition: Planning across long histories and effectively exploring the solution space remain difficult. LLMs often struggle to revise plans in response to unexpected errors, reducing robustness compared to humans, who typically improve through trial and error.

• Reliability of the natural-language interface: Current agent systems use natural language as the interface between LLMs and external components such as memory and tools. However, model outputs are not consistently reliable, since LLMs may introduce formatting errors and can occasionally display rebellious behavior (for example, refusing to follow an instruction). As a result, much of the agent demonstration code emphasizes parsing and validating model output.

Citation

Cited as:

Weng, Lilian. (Jun 2023). “LLM-powered Autonomous Agents”. Lil’Log. https://lilianweng.github.io/posts/2023-06-23-agent/.

Or

@article{weng2023agent,
  title   = "LLM-powered Autonomous Agents",
  author  = "Weng, Lilian",
  journal = "lilianweng.github.io",
  year    = "2023",
  month   = "Jun",
  url     = "https://lilianweng.github.io/posts/2023-06-23-agent/"
}

References

[1] Wei et al. “Chain of thought prompting elicits reasoning in large language models.” NeurIPS 2022

[2] Yao et al. “Tree of Thoughts: Dliberate Problem Solving with Large Language Models.” arXiv preprint arXiv:2305.10601 (2023).

[3] Liu et al. “Chain of Hindsight Aligns Language Models with Feedback “ arXiv preprint arXiv:2302.02676 (2023).

[4] Liu et al. “LLM+P: Empowering Large Language Models with Optimal Planning Proficiency” arXiv preprint arXiv:2304.11477 (2023).

[5] Yao et al. “ReAct: Synergizing reasoning and acting in language models.” ICLR 2023.

[6] Google Blog. “Announcing ScaNN: Efficient Vector Similarity Search” July 28, 2020.

[7] https://chat.openai.com/share/46ff149e-a4c7-4dd7-a800-fc4a642ea389

[8] Shinn & Labash. “Reflexion: an autonomous agent with dynamic memory and self-reflection” arXiv preprint arXiv:2303.11366 (2023).

[9] Laskin et al. “In-context Reinforcement Learning with Algorithm Distillation” ICLR 2023.

[10] Karpas et al. “MRKL Systems A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning.” arXiv preprint arXiv:2205.00445 (2022).

[11] Nakano et al. “Webgpt: Browser-assisted question-answering with human feedback.” arXiv preprint arXiv:2112.09332 (2021).

[12] Parisi et al. “TALM: Tool Augmented Language Models”

[13] Schick et al. “Toolformer: Language Models Can Teach Themselves to Use Tools.” arXiv preprint arXiv:2302.04761 (2023).

[14] Weaviate Blog. Why is Vector Search so fast? Sep 13, 2022.

[15] Li et al. “API-Bank: A Benchmark for Tool-Augmented LLMs” arXiv preprint arXiv:2304.08244 (2023).

[16] Shen et al. “HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace” arXiv preprint arXiv:2303.17580 (2023).

[17] Bran et al. “ChemCrow: Augmenting large-language models with chemistry tools.” arXiv preprint arXiv:2304.05376 (2023).

[18] Boiko et al. “Emergent autonomous scientific research capabilities of large language models.” arXiv preprint arXiv:2304.05332 (2023).

[19] Joon Sung Park, et al. “Generative Agents: Interactive Simulacra of Human Behavior.” arXiv preprint arXiv:2304.03442 (2023).

[20] AutoGPT. https://github.com/Significant-Gravitas/Auto-GPT

[21] GPT-Engineer. https://github.com/AntonOsika/gpt-engineer