Langchain shows that AI agents are not yet a human level because they are overwhelmed by tools

As soon as AI agents have shown promising, organizations had to deal with whether a single agent was sufficient or whether they should invest in the establishment of a broader Multi-agent network That touches more points in your organization.

Orchestration frame company Praise tried to get closer to an answer to this question. It maintained an AI agent of several experiments in which individual agents have a context limit and tools their performance begins to worsen. These experiments could lead to a better understanding of the architecture, which is required to maintain agents and multi-agent systems.

In A Blog postLangchain detailed one sentence of experiments that it carried out with a single react agent and subjected its performance. The main question that Langchain hoped to answer: “At what point will a single react agent be overloaded with instructions and tools and then sees the drop in performance?”

Langchain decided to use the React Agent Framework Because it is “one of the most basic agent architectures”.

During the benchmarking agent performance can often lead to misleading resultsLangchain decided to limit the test to two easily quantifiable tasks of an agent: answering questions and planning meetings.

“There are many existing benchmarks for tool use and tool calling, but for the purposes of this experiment we wanted to evaluate a practical agent that we actually use,” wrote Langchain. “This agent is our internal e -mail assistant, which is responsible for two main domains of work -to answer meetings and support customers with their questions.”

Parameter of Langchain’s experiment

Langchain mainly used prefabricated reactive via its long-graph platform. These active ingredients showed tool calculation models (LLMS) that became part of the benchmark test. These LLMs included Anthropics Claude 3.5 sonet, Meta-Lama-3,3-70B and a trio of models from Openai, GPT-4O, O1 and O3 mini.

The company tested the exam to better evaluate the performance of the E -Mail assistant for the two tasks and to create a list of steps to follow. It began with the customer support functions of the E -Mail assistant, in which it deals with how the agent accepts an e -mail from a customer and responds with an answer.

Langchain first rated the tool -Calling flight or the tools that an agent takes up. If the agent followed the correct order, he passed the test. Next, the researchers asked the assistant to respond to an e -mail and to assess his performance with an LLM.

For the second work domain, calendar planning, Langchain focused on the ability of the agent to follow instructions.

“In other words, the agent must remember certain instructions that are specified exactly, e.g.

Overloaded by the agent

As soon as they defined parameters, Langchain set the burden and overwhelmed the e -mail assistant.

It set 30 tasks for calendar planning and customer support. These were carried out three times (for a total of 90 runs). The researchers have created a calendar planning agent and customer support agent in order to better evaluate the tasks.

“The calendar planning agent only has access to the calendar planning domain, and customer support agency only has access to customer support domains,” explained Langchain.

The researchers then added more domain tasks and instruments to the agents to increase the number of responsibilities. These could range from the HR department to technical quality assurance, legal and compliance and a variety of other areas.

An agent instructions deterioration

After Langchain had carried out the reviews, he found that individual agents were often too overwhelmed if they were supposed to do too many things. They began to forget to call tools or could not react to tasks if they received more instructions and contexts.

Langchain found that calendar production agents with GPT-4O “do worse than Claude 3.5-sun, O1 and O3 over the different context sizes and the performance dropped more than the other models than a larger context was provided.” The performance From GPT-4O calendar planners fell to 2%when the domains rose to at least seven.

Other models didn’t cut much better. Lama-3.3-70b forgot to call the “Send_email” tool.

Only Claude-3.5-Sonnet, O1 and O3-MINI remembered the tool, but Claude-3.5-sun came worse than the other two Openai models. However, the O3-Mini performance worsens as soon as irrelevant domains are added to the planning instructions.

The customer support agent can call more tools, but for this test Langchain said that Claude-3.5-mini was done as well as O3-Mini and O1. It also presented a flatter drop in performance when more domains were added. However, if the context window extends, the Claude model works worse.

GPT-4O also carried out the worst among the models tested.

“We saw that the instruction was worse with more context. Some of our tasks were designed in such a way that they follow niche -specific instructions (e.g. no specific action for customers in EU), ”said Langchain. “We found that these instructions were successfully followed by agents with fewer domains, but increased as the number of domains, these instructions were more often forgotten, and the tasks then failed.”

The company said it examined how multi-agent architectures are evaluated with the same domain overload method.

Langchain has already been invested in the performance of active ingredients, as it introduced Concept of “environmental agents“Or agents who run in the background and are triggered by certain events. These experiments could make it easier to find out how the agents’ performance can best be guaranteed.