agent.md

Agent

Agentic Design Frameworks

• Agentic Design Frameworks focus on managing autonomous or semi-autonomous AI agents for complex tasks.
• e.g., Autogen, and crewAI

Agent Design Patterns

• The Rise and Potential of Large Language Model Based Agents: A Survey: The papers list for LLM-based agents [cnt] / git [14 Sep 2023]
• AgentBench Evaluating LLMs as Agents: Assess LLM-as Agent’s reasoning and decision-making abilities. [7 Aug 2023]
• Agentic Design Patterns ref [Mar 2024]
  • Reflection: LLM self-evaluates to improve.
    • Self-Refine [30 Mar 2023]
    • Reflexion [20 Mar 2023 ]
    • CRITIC [19 May 2023]
  • Tool use: LLM uses tools for information gathering, action, or data processing.
    • Gorilla [24 May 2023]
    • MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action [20 Mar 2023]
    • Efficient Tool Use with Chain-of-Abstraction Reasoning [30 Jan 2024]
  • Planning: LLM devises and executes multistep plans to reach goals.
    • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models [28 Jan 2022]
    • HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face [30 Mar 2023]
    • Understanding the planning of LLM agents: A survey [5 Feb 2024]
  • Multi-agent collaboration: Multiple AI agents collaborate for better solutions.
    • Communicative Agents for Software Development [16 Jul 2023]
    • AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation [16 Aug 2023]
    • MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework [1 Aug 2023]
    • Framework: Autogen / LangGraph / crewAI
• Generate the code ref [Jun 2024]
  • AgentCoder: Multiagent-Code Generation with Iterative Testing and Optimisation [20 Dec 2023]
  • LDB: A Large Language Model Debugger via Verifying Runtime Execution Step by Step [25 Feb 2024]
  • SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering [6 May 2024]

---

• Generative AI Design Patterns for Agentic AI Systems: Design Patterns for Agentic solutions in Azure [May 2023]
• Exploring Generative AI (martinfowler.com): Memos on how LLMs are being used to enhance software delivery practices, including Toochain, Test-Driven Development (TDD) with GitHub Copilot, pair programming, and multi-file editing. [26 Jul 2023 ~ ]
• Cognitive Architectures for Language Agents: Cognitive Architectures for Language Agents (CoALA). Procedural (how to perform tasks), Semantic (long-term store of knowledge), Episodic Memory (recall specific past events) ref [5 Sep 2023]
• Language Agent Tree Search Method (LATS): LATS leverages an external environment and an MCTS (Monte Carlo Tree Search)-based search [6 Oct 2023]
• AI Agents That Matter: AI agent evaluations for optimizing both accuracy and cost. Focusing solely on accuracy can lead to overfitting and high costs. retry, warming, escalation [1 Jul 2024]
• Automated Design of Agentic Systems: Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them. [15 Aug 2024]
• The Different Ochestration Frameworks:💡Orchestration frameworks for LLM applications: Micro-orchestration / Macro-orchestration / Agentic Design Frameworks / Optimizer frameworks [11 Oct 2024]
• Agent-as-a-Judge: Evaluate Agents with Agents. DevAI, a new benchmark of 55 realistic automated AI development tasks. Agent-as-a-Judge > LLM-as-a-Judge > Human-as-a-Judge [14 Oct 2024]
• Generative Agent Simulations of 1,000 People: a generative agent architecture that simulates more than 1,000 real individuals using two-hour qualitative interviews. 85% accuracy in General Social Survey. [15 Nov 2024]

Tool use: LLM to Master APIs

• Berkeley Function-Calling Leaderboard V2 [Aug 2024]

• Gorilla: An API store for LLMs: [cnt]: Gorilla: Large Language Model Connected with Massive APIs git [24 May 2023]

  1. Used GPT-4 to generate a dataset of instruction-api pairs for fine-tuning Gorilla.
  2. Used the abstract syntax tree (AST) of the generated code to match with APIs in the database and test set for evaluation purposes.

  Another user asked how Gorilla compared to LangChain; Patil replied: LangChain is a terrific project that tries to teach agents how to use tools using prompting. Our take on this is that prompting is not scalable if you want to pick between 1000s of APIs. So Gorilla is a LLM that can pick and write the semantically and syntactically correct API for you to call! A drop in replacement into LangChain! cite [04 Jul 2023]

• Meta: Toolformer: [cnt]: Language Models That Can Use Tools, by MetaAI git [9 Feb 2023]

• ToolLLM: [cnt]: : Facilitating Large Language Models to Master 16000+ Real-world APIs git [31 Jul 2023]

• APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets [26 Jun 2024]

• ToolShed: Toolshed Knowledge Bases & Advanced RAG-Tool Fusion, optimized for storing and retrieving tools in a vector database for large-scale agents. To address the limitations of primary methods, two approaches are: 1. tuning-based tool calling via LLM fine-tuning, and 2. retriever-based tool selection and planning. [18 Oct 2024]

Computer use

• Anthropic Claude's computer use:
  1. OpenInterpreter starts to support Computer Use API
  2. Agent.exe: Electron app to use computer use APIs. [Oct 2024]
• OSS
  1. x-ref > UFO: Windows Control
  2. Self-Operating Computer Framework: A framework to enable multimodal models to operate a computer. [Nov 2023]
  3. Open-Interface: LLM backend (GPT-4V, etc), supporting Linux, Mac, Windows. [Jan 2024]
  4. Computer Use OOTB: Out-of-the-box (OOTB) GUI Agent for Windows and macOS. [Oct 2024]

Agent Framework

1. Huginn: A hackable version of IFTTT or Zapier on your own server for building agents that perform automated tasks. [Mar 2013]
2. Botpress Cloud: The open-source hub to build & deploy GPT/LLM Agents. [Nov 2016]
3. n8n: A workflow automation tool for integrating various tools. [Jan 2019]
4. phidata: Build AI Assistants with memory, knowledge, and tools [May 2022]
5. Cheshire-Cat (Stregatto): Framework to build custom AIs with memory and plugins [Feb 2023]
6. MetaGPT: Multi-Agent Framework. Assign different roles to GPTs to form a collaborative entity for complex tasks. e.g., Data Interpreter [Jun 2023]
7. SuperAGI: Autonomous AI Agents framework [May 2023]
8. AgentVerse: Primarily providing: task-solving and simulation. [May 2023]
9. OpenAgents: Three distinct agents: Data Agent for data analysis, Plugins Agent for plugin integration, and Web Agent for autonomous web browsing. [Aug 2023]
10. AgentOps:Python SDK for AI agent monitoring, LLM cost tracking, benchmarking. [Aug 2023]
11. Lagent: Inspired by the design philosophy of PyTorch. A lightweight framework for building LLM-based agents. [Aug 2023]
12. Autogen:💡Customizable and conversable agents framework [Aug 2023]
13. LangGraph: Built on top of LangChain [Aug 2023]
14. crewAI: Framework for orchestrating role-playing, autonomous AI agents. [Oct 2023]
15. XAgent: Autonomous LLM Agent for complex task solving like data analysis, recommendation, and model training [Oct 2023]
16. Qwen-Agent: Agent framework built upon Qwen1.5, featuring Function Calling, Code Interpreter, RAG, and Chrome extension. [Sep 2023]
17. Open AI Assistant API [6 Nov 2023]
18. Burr: Create an application as a state machine (graph/flowchart) for managing state, decisions, human feedback, and workflows. [Jan 2024]
19. TaskingAI: A BaaS (Backend as a Service) platform for LLM-based Agent Development and Deployment. [Jan 2024]
20. AgentScope: To build LLM-empowered multi-agent applications. [Jan 2024]
21. composio: Integration of Agents with 100+ Tools [Feb 2024]
22. maestro: A Framework for Claude Opus, GPT, and local LLMs to Orchestrate Subagents [Mar 2024]
23. GPT Computer Assistant(GCA): an AI agent framework designed to make computer use. [May 2024]
24. AWS: Multi-Agent Orchestrator: a framework for managing multiple AI agents and handling complex conversations. [Jul 2024]
25. SwarmZero: SwarmZero's SDK for building AI agents, swarms of agents. [Aug 2024]
26. Bee Agent Framework: IBM. The TypeScript framework for building scalable agentic applications. [Oct 2024]
27. Agent-S: To build intelligent GUI agents that autonomously learn and perform complex tasks on your computer. [Oct 2024]
28. OpenAI Swarm: An experimental and educational framework for lightweight multi-agent orchestration. [11 Oct 2024]
29. Dynamiq: An orchestration framework for RAG, agentic AI, and LLM applications [Sep 2024]
30. PydanticAI: Agent Framework / shim to use Pydantic with LLMs. Model-agnostic. Type-safe. [29 Oct 2024]
31. Microsoft Agent Frameworks x-ref
32. Agent Framework used in MLE-bench: GPT-4o (AIDE) earned the highest score x-ref
    • AIDE: The state-of-the-art machine learning engineer agent [Apr 2024]
    • OpenHands: OpenHands (formerly OpenDevin), a platform for software development agents [Mar 2024]
    • MLAB ResearchAgent: Evaluating Language Agents on Machine Learning Experimentation [Aug 2023]

Agent Application

1. KHOJ: Open-source, personal AI agents. Cloud or Self-Host, Multiple Interfaces. Python Django based [Aug 2021]
2. PR-Agent: Efficient code review and handle pull requests, by providing AI feedbacks and suggestions [Jan 2023]
3. Auto-GPT: Most popular [Mar 2023]
4. CAMEL: CAMEL: Communicative Agents for “Mind” Exploration of Large Scale Language Model Society [Mar 2023]
5. babyagi: Simplest implementation - Coworking of 4 agents [Apr 2023]
6. SuperAGI: GUI for agent settings [May 2023]
7. AgentGPT: Assemble, configure, and deploy autonomous AI agents in your browser [Apr 2023]
8. GPT Pilot: The first real AI developer. Dev tool that writes scalable apps from scratch while the developer oversees the implementation [Jul 2023]
9. OpenDAN : Your Personal AIOS: OpenDAN, an open-source Personal AI OS consolidating various AI modules in one place [May 2023]
10. RasaGPT: Built with Rasa, FastAPI, Langchain, and LlamaIndex [Apr 2023]
11. ChatDev: Virtual software company. Create Customized Software using LLM-powered Multi-Agent Collaboration [Sep 2023]
12. skyagi: Simulating believable human behaviors. Role playing [Apr 2023]
13. AIOS: LLM Agent Operating System [Jan 2024]
14. SeeAct: GPT-4V(ision) is a Generalist Web Agent, if Grounded git [Jan 2024]
15. WrenAI: Open-source SQL AI Agent for Text-to-SQL [Mar 2024]
16. Devon: An open-source pair programmer. [Mar 2024]
17. Project Astra: Google DeepMind, A universal AI agent that is helpful in everyday life [14 May 2024]
18. SakanaAI AI-Scientist: Towards Fully Automated Open-Ended Scientific Discovery [Aug 2024]
19. Integuru: An AI agent that generates integration code by reverse-engineering platforms' internal APIs. [Oct 2024]
20. skyvern: Automate browser-based workflows with LLMs and Computer Vision [Feb 2024]
21. LaVague: Automate automation with Large Action Model framework. Generate Selenium code. [Feb 2024]
22. Mobile-Agent: The Powerful Mobile Device Operation Assistant Family. [Jan 2024]
23. AppAgent-TencentQQGYLab: Multimodal Agents as Smartphone Users. [Dec 2023]
24. TEN Agent: The world’s first real-time multimodal agent integrated with the OpenAI Realtime API. [Jun 2024]
25. Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks [Nov 2024]

OSS Alternatives for OpenAI Code Interpreter (aka. Advanced Data Analytics)

1. OpenAI Code Interpreter Integration with Sandboxed python execution environment [23 Mar 2023]
   • We provide our models with a working Python interpreter in a sandboxed, firewalled execution environment, along with some ephemeral disk space.
2. SlashGPT The tool integrated with "jupyter" agent [Apr 2023]
3. gpt-code-ui An open source implementation of OpenAI's ChatGPT Code interpreter. [May 2023]
4. OSS Code Interpreter A LangChain implementation of the ChatGPT Code Interpreter. [Jul 2023]
5. Open Interpreter:💡Let language models run code on your computer. [Jul 2023]

Domain-specific

1. 5 Top AI Agents for Earth Snapshots VLMs and LLMs for Geospatial Intelligent Analysis: GeoChat | GEOBench-VLM | RS5M | VHM | EarthGPT
2. DrugAgent: Automating AI-aided Drug Discovery [24 Nov 2024]
3. FinRobot: AI Agent for Equity Research and Valuation [13 Nov 2024]
4. An LLM Agent for Automatic Geospatial Data Analysis [24 Oct 2024]