Ideally, you’ll be able to consider agentic purposes whilst you’re growing them, as a substitute of analysis being an afterthought. For this to work, although, you want to have the ability to mock each inner and exterior dependencies of the agent you’re growing. I’m extraordinarily excited by PydanticAI as a result of it helps dependency injection from the bottom up. It’s the first framework that has allowed me to construct agentic purposes in an evaluation-driven method.
On this article, I’ll discuss in regards to the core challenges and show growing a easy agent in an evaluation-driven approach utilizing PydanticAI.
Challenges when growing GenAI purposes
Like many GenAI builders, I’ve been ready for an agentic framework that helps the complete improvement lifecycle. Every time a brand new framework comes alongside, I attempt it out hoping that this would be the One — see, for instance, my articles about DSPy, Langchain, LangGraph, and Autogen.
I discover that there are core challenges {that a} software program developer faces when growing an LLM-based software. These challenges are sometimes not blockers in case you are constructing a easy PoC with GenAI, however they are going to come to chunk you in case you are constructing LLM-powered purposes in manufacturing.
What challenges?
(1) Non-determinism: In contrast to most software program APIs, calls to an LLM with the very same enter may return totally different outputs every time. How do you even start to check such an software?
(2) LLM limitations: Foundational fashions like GPT-4, Claude, and Gemini are restricted by their coaching information (e.g., no entry to enterprise confidential data), functionality (e.g., you can’t invoke enterprise APIs and databases), and cannot plan/motive.
(3) LLM flexibility: Even in the event you resolve to stay to LLMs from a single supplier akin to Anthropic, you could discover that you just want a special LLM for every step — maybe one step of your workflow wants a low-latency small language mannequin (Haiku), one other requires nice code-generation functionality (Sonnet), and a 3rd step requires glorious contextual consciousness (Opus).
(4) Charge of Change: GenAI applied sciences are transferring quick. Lately, most of the enhancements have come about in foundational mannequin capabilities. Not are the foundational fashions simply producing textual content primarily based on consumer prompts. They’re now multimodal, can generate structured outputs, and may have reminiscence. But, in the event you attempt to construct in an LLM-agnostic approach, you typically lose the low-level API entry that can activate these options.
To assist deal with the primary drawback, of non-determinism, your software program testing wants to include an analysis framework. You’ll by no means have software program that works 100%; as a substitute, you have to to have the ability to design round software program that’s x% appropriate, construct guardrails and human oversight to catch the exceptions, and monitor the system in real-time to catch regressions. Key to this functionality is evaluation-driven improvement (my time period), an extension of test-driven improvement in software program.
The present workaround for all of the LLM limitations in Problem #2 is to make use of agentic architectures like RAG, present the LLM entry to instruments, and make use of patterns like Reflection, ReACT and Chain of Thought. So, your framework might want to have the flexibility to orchestrate brokers. Nevertheless, evaluating brokers that may name exterior instruments is tough. You want to have the ability to inject proxies for these exterior dependencies so that you could check them individually, and consider as you construct.
To deal with problem #3, an agent wants to have the ability to invoke the capabilities of several types of foundational fashions. Your agent framework must be LLM-agnostic on the granularity of a single step of an agentic workflow. To handle the speed of change consideration (problem #4), you wish to retain the flexibility to make low-level entry to the foundational mannequin APIs and to strip out sections of your codebase which are now not obligatory.
Is there a framework that meets all these standards? For the longest time, the reply was no. The closest I may get was to make use of Langchain, pytest’s dependency injection, and deepeval with one thing like this (full instance is right here):
from unittest.mock import patch, Mock
from deepeval.metrics import GEvalllm_as_judge = GEval(
title="Correctness",
standards="Decide whether or not the precise output is factually appropriate primarily based on the anticipated output.",
evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
mannequin='gpt-3.5-turbo'
)
@patch('lg_weather_agent.retrieve_weather_data', Mock(return_value=chicago_weather))
def eval_query_rain_today():
input_query = "Is it raining in Chicago?"
expected_output = "No, it's not raining in Chicago proper now."
outcome = lg_weather_agent.run_query(app, input_query)
actual_output = outcome[-1]
print(f"Precise: {actual_output} Anticipated: {expected_output}")
test_case = LLMTestCase(
enter=input_query,
actual_output=actual_output,
expected_output=expected_output
)
llm_as_judge.measure(test_case)
print(llm_as_judge.rating)
Basically, I’d assemble a Mock object (chicago_weather within the above instance) for each LLM name and patch the decision to the LLM (retrieve_weather_data within the above instance) with the hardcoded object each time I wanted to mock that a part of the agentic workflow. The dependency injection is all over, you want a bunch of hardcoded objects, and the calling workflow turns into extraordinarily arduous to comply with. Word that in the event you don’t have dependency injection, there isn’t any approach to check a operate like this: clearly, the exterior service will return the present climate and there’s no approach to decide what the proper reply is for a query akin to whether or not or not it’s raining proper now.
So … is there an agent framework that helps dependency injection, is Pythonic, offers low-level entry to LLMs, is model-agnostic, helps constructing it one eval-at-a-time, and is simple to make use of and comply with?
Nearly. PydanticAI meets the primary 3 necessities; the fourth (low-level LLM entry) shouldn’t be doable, however the design doesn’t preclude it. In the remainder of this text, I’ll present you learn how to use it to develop an agentic software in an evaluation-driven approach.
1. Your first PydanticAI Utility
Let’s begin out by constructing a easy PydanticAI software. This can use an LLM to reply questions on mountains:
agent = llm_utils.agent()
query = "What's the tallest mountain in British Columbia?"
print(">> ", query)
reply = agent.run_sync(query)
print(reply.information)
Within the code above, I’m creating an agent (I’ll present you the way, shortly) after which calling run_sync passing within the consumer immediate, and getting again the LLM’s response. run_sync is a approach to have the agent invoke the LLM and look ahead to the response. Different methods are to run the question asynchronously, or to stream its response. (Full code is right here if you wish to comply with alongside).
Run the code above, and you’ll get one thing like:
>> What's the tallest mountain in British Columbia?
The tallest mountain in British Columbia is **Mount Robson**, at 3,954 metres (12,972 ft).
To create the agent, create a mannequin after which inform the agent to make use of that Mannequin for all its steps.
import pydantic_ai
from pydantic_ai.fashions.gemini import GeminiModeldef default_model() -> pydantic_ai.fashions.Mannequin:
mannequin = GeminiModel('gemini-1.5-flash', api_key=os.getenv('GOOGLE_API_KEY'))
return mannequin
def agent() -> pydantic_ai.Agent:
return pydantic_ai.Agent(default_model())
The concept behind default_model() is to make use of a comparatively cheap however quick mannequin like Gemini Flash because the default. You may then change the mannequin utilized in particular steps as obligatory by passing in a special mannequin to run_sync()
PydanticAI mannequin help seems to be sparse, however essentially the most generally used fashions — the present frontier ones from OpenAI, Groq, Gemini, Mistral, Ollama, and Anthropic — are all supported. By Ollama, you will get entry to Llama3, Starcoder2, Gemma2, and Phi3. Nothing vital appears to be lacking.
2. Pydantic with structured outputs
The instance within the earlier part returned free-form textual content. In most agentic workflows, you’ll need the LLM to return structured information so that you could use it straight in packages.
Contemplating that this API is from Pydantic, returning structured output is kind of simple. Simply outline the specified output as a dataclass (full code is right here):
from dataclasses import dataclass@dataclass
class Mountain:
title: str
location: str
peak: float
If you create the Agent, inform it the specified output sort:
agent = Agent(llm_utils.default_model(),
result_type=Mountain,
system_prompt=(
"You're a mountaineering information, who offers correct data to most of the people.",
"Present all distances and heights in meters",
"Present location as distance and course from nearest large metropolis",
))
Word additionally using the system immediate to specify items and so forth.
Working this on three questions, we get:
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(title='Mount Robson', location='130km North of Vancouver', peak=3999.0)
>> Is Mt. Hood straightforward to climb?
Mountain(title='Mt. Hood', location='60 km east of Portland', peak=3429.0)
>> What is the tallest peak within the Enchantments?
Mountain(title='Mount Stuart', location='100 km east of Seattle', peak=3000.0)
However how good is that this agent? Is the peak of Mt. Robson appropriate? Is Mt. Stuart actually the tallest peak within the Enchantments? All of this data may have been hallucinated!
There isn’t a approach so that you can understand how good an agentic software is except you consider the agent towards reference solutions. You cannot simply “eyeball it”. Sadly, that is the place lots of LLM frameworks fall quick — they make it actually arduous to guage as you develop the LLM software.
3. Consider towards reference solutions
It’s if you begin to consider towards reference solutions that PydanticAI begins to point out its strengths. All the things is kind of Pythonic, so you’ll be able to construct customized analysis metrics fairly merely.
For instance, that is how we are going to consider a returned Mountain object on three standards and create a composite rating (full code is right here):
def consider(reply: Mountain, reference_answer: Mountain) -> Tuple[float, str]:
rating = 0
motive = []
if reference_answer.title in reply.title:
rating += 0.5
motive.append("Right mountain recognized")
if reference_answer.location in reply.location:
rating += 0.25
motive.append("Right metropolis recognized")
height_error = abs(reference_answer.peak - reply.peak)
if height_error < 10:
rating += 0.25 * (10 - height_error)/10.0
motive.append(f"Top was {height_error}m off. Right reply is {reference_answer.peak}")
else:
motive.append(f"Improper mountain recognized. Right reply is {reference_answer.title}")return rating, ';'.be a part of(motive)
Now, we will run this on a dataset of questions and reference solutions:
questions = [
"Tell me about the tallest mountain in British Columbia?",
"Is Mt. Hood easy to climb?",
"What's the tallest peak in the Enchantments?"
]reference_answers = [
Mountain("Robson", "Vancouver", 3954),
Mountain("Hood", "Portland", 3429),
Mountain("Dragontail", "Seattle", 2690)
]
total_score = 0
for l_question, l_reference_answer in zip(questions, reference_answers):
print(">> ", l_question)
l_answer = agent.run_sync(l_question)
print(l_answer.information)
l_score, l_reason = consider(l_answer.information, l_reference_answer)
print(l_score, ":", l_reason)
total_score += l_score
avg_score = total_score / len(questions)
Working this, we get:
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(title='Mount Robson', location='130 km North-East of Vancouver', peak=3999.0)
0.75 : Right mountain recognized;Right metropolis recognized;Top was 45.0m off. Right reply is 3954
>> Is Mt. Hood straightforward to climb?
Mountain(title='Mt. Hood', location='60 km east of Portland, OR', peak=3429.0)
1.0 : Right mountain recognized;Right metropolis recognized;Top was 0.0m off. Right reply is 3429
>> What is the tallest peak within the Enchantments?
Mountain(title='Dragontail Peak', location='14 km east of Leavenworth, WA', peak=3008.0)
0.5 : Right mountain recognized;Top was 318.0m off. Right reply is 2690
Common rating: 0.75
Mt. Robson’s peak is 45m off; Dragontail peak’s peak was 318m off. How would you repair this?
That’s proper. You’d use a RAG structure or arm the agent with a instrument that gives the proper peak data. Let’s use the latter strategy and see learn how to do it with Pydantic.
Word how evaluation-driven improvement reveals us the trail ahead to enhance our agentic software.
4a. Utilizing a instrument
PydanticAI helps a number of methods to supply instruments to an agent. Right here, I annotate a operate to be referred to as each time it wants the peak of a mountain (full code right here):
agent = Agent(llm_utils.default_model(),
result_type=Mountain,
system_prompt=(
"You're a mountaineering information, who offers correct data to most of the people.",
"Use the offered instrument to search for the elevation of many mountains."
"Present all distances and heights in meters",
"Present location as distance and course from nearest large metropolis",
))
@agent.instrument
def get_height_of_mountain(ctx: RunContext[Tools], mountain_name: str) -> str:
return ctx.deps.elev_wiki.snippet(mountain_name)
The operate, although, does one thing unusual. It pulls an object referred to as elev_wiki out of the run-time context of the agent. This object is handed in once we name run_sync:
class Instruments:
elev_wiki: wikipedia_tool.WikipediaContent
def __init__(self):
self.elev_wiki = OnlineWikipediaContent("Listing of mountains by elevation")instruments = Instruments() # Instruments or FakeTools
l_answer = agent.run_sync(l_question, deps=instruments) # word how we're capable of inject
As a result of the Runtime context might be handed into each agent invocation or instrument name , we will use it to do dependency injection in PydanticAI. You’ll see this within the subsequent part.
The wiki itself simply queries Wikipedia on-line (code right here) and extracts the contents of the web page and passes the suitable mountain data to the agent:
import wikipediaclass OnlineWikipediaContent(WikipediaContent):
def __init__(self, matter: str):
print(f"Will question on-line Wikipedia for data on {matter}")
self.web page = wikipedia.web page(matter)
def url(self) -> str:
return self.web page.url
def html(self) -> str:
return self.web page.html()
Certainly, once we run it, we get appropriate heights now:
Will question on-line Wikipedia for data on Listing of mountains by elevation
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(title='Mount Robson', location='100 km west of Jasper', peak=3954.0)
0.75 : Right mountain recognized;Top was 0.0m off. Right reply is 3954
>> Is Mt. Hood straightforward to climb?
Mountain(title='Mt. Hood', location='50 km ESE of Portland, OR', peak=3429.0)
1.0 : Right mountain recognized;Right metropolis recognized;Top was 0.0m off. Right reply is 3429
>> What is the tallest peak within the Enchantments?
Mountain(title='Mount Stuart', location='Cascades, Washington, US', peak=2869.0)
0 : Improper mountain recognized. Right reply is Dragontail
Common rating: 0.58
4b. Dependency injecting a mock service
Ready for the API name to Wikipedia every time throughout improvement or testing is a foul thought. As an alternative, we are going to wish to mock the Wikipedia response in order that we will develop rapidly and be assured of the outcome we’re going to get.
Doing that may be very easy. We create a Pretend counterpart to the Wikipedia service:
class FakeWikipediaContent(WikipediaContent):
def __init__(self, matter: str):
if matter == "Listing of mountains by elevation":
print(f"Will used cached Wikipedia data on {matter}")
self.url_ = "https://en.wikipedia.org/wiki/List_of_mountains_by_elevation"
with open("mountains.html", "rb") as ifp:
self.html_ = ifp.learn().decode("utf-8")def url(self) -> str:
return self.url_
def html(self) -> str:
return self.html_
Then, inject this faux object into the runtime context of the agent throughout improvement:
class FakeTools:
elev_wiki: wikipedia_tool.WikipediaContent
def __init__(self):
self.elev_wiki = FakeWikipediaContent("Listing of mountains by elevation")instruments = FakeTools() # Instruments or FakeTools
l_answer = agent.run_sync(l_question, deps=instruments) # word how we're capable of inject
This time once we run, the analysis makes use of the cached wikipedia content material:
Will used cached Wikipedia data on Listing of mountains by elevation
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(title='Mount Robson', location='100 km west of Jasper', peak=3954.0)
0.75 : Right mountain recognized;Top was 0.0m off. Right reply is 3954
>> Is Mt. Hood straightforward to climb?
Mountain(title='Mt. Hood', location='50 km ESE of Portland, OR', peak=3429.0)
1.0 : Right mountain recognized;Right metropolis recognized;Top was 0.0m off. Right reply is 3429
>> What is the tallest peak within the Enchantments?
Mountain(title='Mount Stuart', location='Cascades, Washington, US', peak=2869.0)
0 : Improper mountain recognized. Right reply is Dragontail
Common rating: 0.58
Look rigorously on the above output — there are totally different errors from the zero-shot instance. In Part #2, the LLM picked Vancouver because the closest metropolis to Mt. Robson and Dragontail because the tallest peak within the Enchantments. These solutions occurred to be appropriate. Now, it picks Jasper and Mt. Stuart. We have to do extra work to repair these errors — however evaluation-driven improvement at the very least provides us a course of journey.
Present Limitations
PydanticAI may be very new. There are a few locations the place it could possibly be improved:
- There isn’t a low-level entry to the mannequin itself. For instance, totally different foundational fashions help context caching, immediate caching, and so forth. The mannequin abstraction in PydanticAI doesn’t present a approach to set these on the mannequin. Ideally, we will determine a kwargs approach of doing such settings.
- The necessity to create two variations of agent dependencies, one actual and one faux, is kind of frequent. It will be good if we had been capable of annoate a instrument or present a easy approach to swap between the 2 sorts of companies throughout the board.
- Throughout improvement, you don’t want logging as a lot. However if you go to run the agent, you’ll normally wish to log the prompts and responses. Generally, it would be best to log the intermediate responses. The best way to do that appears to be a business product referred to as Logfire. An OSS, cloud-agnostic logging framework that integrates with the PydanticAI library can be supreme.
It’s doable that these exist already and I missed them, or maybe they are going to have been carried out by the point you’re studying this text. In both case, depart a remark for future readers.
Total, I like PydanticAI — it provides a really clear and Pythonic approach to construct agentic purposes in an evaluation-driven method.
Steered subsequent steps:
- That is a type of weblog posts the place you’ll profit from really operating the examples as a result of it describes a means of improvement in addition to a brand new library. This GitHub repo comprises the PydanticAI instance I walked by on this put up: https://github.com/lakshmanok/lakblogs/tree/predominant/pydantic_ai_mountains Comply with the directions within the README to attempt it out.
- Pydantic AI documentation: https://ai.pydantic.dev/
- Patching a Langchain workflow with Mock objects. My “earlier than” resolution: https://github.com/lakshmanok/lakblogs/blob/predominant/genai_agents/eval_weather_agent.py
