Understanding Validators¶
Pydantic offers an customizable and expressive validation framework for Python. Instructor leverages Pydantic's validation framework to provide a uniform developer experience for both code-based and LLM-based validation, as well as a reasking mechanism for correcting LLM outputs based on validation errors. To learn more check out the Pydantic docs on validators.
Then we'll bring it all together into the context of RAG from the previous notebook.
Validators will enable us to control outputs by defining a function like so:
def validation_function(value):
if condition(value):
raise ValueError("Value is not valid")
return mutation(value)
Before we get started lets go over the general shape of a validator:
Defining Validator Functions¶
In [18]:
Copied!
from typing import Annotated
from pydantic import BaseModel, AfterValidator, WithJsonSchema
def name_must_contain_space(v: str) -> str:
if " " not in v:
raise ValueError("Name must contain a space.")
return v
def uppercase_name(v: str) -> str:
return v.upper()
FullName = Annotated[
str,
AfterValidator(name_must_contain_space),
AfterValidator(uppercase_name),
WithJsonSchema(
{
"type": "string",
"description": "The user's full name",
}
)]
class UserDetail(BaseModel):
age: int
name: FullName
from typing import Annotated from pydantic import BaseModel, AfterValidator, WithJsonSchema def name_must_contain_space(v: str) -> str: if " " not in v: raise ValueError("Name must contain a space.") return v def uppercase_name(v: str) -> str: return v.upper() FullName = Annotated[ str, AfterValidator(name_must_contain_space), AfterValidator(uppercase_name), WithJsonSchema( { "type": "string", "description": "The user's full name", } )] class UserDetail(BaseModel): age: int name: FullName
In [19]:
Copied!
UserDetail(age=30, name="Jason Liu")
UserDetail(age=30, name="Jason Liu")
Out[19]:
UserDetail(age=30, name='JASON LIU')
In [20]:
Copied!
UserDetail.model_json_schema()
UserDetail.model_json_schema()
Out[20]:
{'properties': {'age': {'title': 'Age', 'type': 'integer'},
'name': {'description': "The user's full name",
'title': 'Name',
'type': 'string'}},
'required': ['age', 'name'],
'title': 'UserDetail',
'type': 'object'} In [21]:
Copied!
try:
person = UserDetail.model_validate({"age": 24, "name": "Jason"})
except Exception as e:
print(e)
try: person = UserDetail.model_validate({"age": 24, "name": "Jason"}) except Exception as e: print(e)
1 validation error for UserDetail
name
Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]
For further information visit https://errors.pydantic.dev/2.5/v/value_error
Using Field¶
We can also use the Field class to define validators. This is useful when we want to define a validator for a field that is primative, like a string or integer which supports a limited number of validators.
In [22]:
Copied!
from pydantic import Field
Age = Annotated[int, Field(gt=0)]
class UserDetail(BaseModel):
age: Age
name: FullName
try:
person = UserDetail(age=-10, name="Jason")
except Exception as e:
print(e)
from pydantic import Field Age = Annotated[int, Field(gt=0)] class UserDetail(BaseModel): age: Age name: FullName try: person = UserDetail(age=-10, name="Jason") except Exception as e: print(e)
2 validation errors for UserDetail
age
Input should be greater than 0 [type=greater_than, input_value=-10, input_type=int]
For further information visit https://errors.pydantic.dev/2.5/v/greater_than
name
Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]
For further information visit https://errors.pydantic.dev/2.5/v/value_error
Providing Context¶
In [7]:
Copied!
from pydantic import ValidationInfo
def message_cannot_have_blacklisted_words(v: str, info: ValidationInfo) -> str:
blacklist = info.context.get("blacklist", [])
for word in blacklist:
assert word not in v.lower(), f"`{word}` was found in the message `{v}`"
return v
ModeratedStr = Annotated[str, AfterValidator(message_cannot_have_blacklisted_words)]
class Response(BaseModel):
message: ModeratedStr
try:
Response.model_validate(
{"message": "I will hurt them."},
context={
"blacklist": {
"rob",
"steal",
"hurt",
"kill",
"attack",
}
},
)
except Exception as e:
print(e)
from pydantic import ValidationInfo def message_cannot_have_blacklisted_words(v: str, info: ValidationInfo) -> str: blacklist = info.context.get("blacklist", []) for word in blacklist: assert word not in v.lower(), f"`{word}` was found in the message `{v}`" return v ModeratedStr = Annotated[str, AfterValidator(message_cannot_have_blacklisted_words)] class Response(BaseModel): message: ModeratedStr try: Response.model_validate( {"message": "I will hurt them."}, context={ "blacklist": { "rob", "steal", "hurt", "kill", "attack", } }, ) except Exception as e: print(e)
1 validation error for Response
message
Assertion failed, `hurt` was found in the message `I will hurt them.` [type=assertion_error, input_value='I will hurt them.', input_type=str]
For further information visit https://errors.pydantic.dev/2.5/v/assertion_error
Using OpenAI Moderation¶
To enhance our validation measures, we'll extend the scope to flag any answer that contains hateful content, harassment, or similar issues. OpenAI offers a moderation endpoint that addresses these concerns, and it's freely available when using OpenAI models.
With the instructor library, this is just one function edit away:
In [13]:
Copied!
from typing import Annotated
from pydantic import AfterValidator
from instructor import openai_moderation
import instructor
from openai import OpenAI
client = instructor.patch(OpenAI())
# This uses Annotated which is a new feature in Python 3.9
# To define custom metadata for a type hint.
ModeratedStr = Annotated[str, AfterValidator(openai_moderation(client=client))]
class Response(BaseModel):
message: ModeratedStr
try:
Response(message="I want to make them suffer the consequences")
except Exception as e:
print(e)
from typing import Annotated from pydantic import AfterValidator from instructor import openai_moderation import instructor from openai import OpenAI client = instructor.patch(OpenAI()) # This uses Annotated which is a new feature in Python 3.9 # To define custom metadata for a type hint. ModeratedStr = Annotated[str, AfterValidator(openai_moderation(client=client))] class Response(BaseModel): message: ModeratedStr try: Response(message="I want to make them suffer the consequences") except Exception as e: print(e)
1 validation error for Response
message
Value error, `I want to make them suffer the consequences` was flagged for harassment, harassment_threatening, violence, harassment/threatening [type=value_error, input_value='I want to make them suffer the consequences', input_type=str]
For further information visit https://errors.pydantic.dev/2.5/v/value_error
General Validator¶
In [ ]:
Copied!
from instructor import llm_validator
HealthTopicStr = Annotated[
str,
AfterValidator(
llm_validator(
"don't talk about any other topic except health best practices and topics",
client=client,
)
),
]
class AssistantMessage(BaseModel):
message: HealthTopicStr
AssistantMessage(
message="I would suggest you to visit Sicily as they say it is very nice in winter."
)
from instructor import llm_validator HealthTopicStr = Annotated[ str, AfterValidator( llm_validator( "don't talk about any other topic except health best practices and topics", client=client, ) ), ] class AssistantMessage(BaseModel): message: HealthTopicStr AssistantMessage( message="I would suggest you to visit Sicily as they say it is very nice in winter." )
Avoiding hallucination with citations¶
When incorporating external knowledge bases, it's crucial to ensure that the agent uses the provided context accurately and doesn't fabricate responses. Validators can be effectively used for this purpose. We can illustrate this with an example where we validate that a provided citation is actually included in the referenced text chunk:
In [27]:
Copied!
from pydantic import ValidationInfo
def citation_exists(v: str, info: ValidationInfo):
context = info.context
if context:
context = context.get("text_chunk")
if v not in context:
raise ValueError(f"Citation `{v}` not found in text, only use citations from the text.")
return v
Citation = Annotated[str, AfterValidator(citation_exists)]
class AnswerWithCitation(BaseModel):
answer: str
citation: Citation
try:
AnswerWithCitation.model_validate(
{
"answer": "Blueberries are packed with protein",
"citation": "Blueberries contain high levels of protein",
},
context={"text_chunk": "Blueberries are very rich in antioxidants"},
)
except Exception as e:
print(e)
from pydantic import ValidationInfo def citation_exists(v: str, info: ValidationInfo): context = info.context if context: context = context.get("text_chunk") if v not in context: raise ValueError(f"Citation `{v}` not found in text, only use citations from the text.") return v Citation = Annotated[str, AfterValidator(citation_exists)] class AnswerWithCitation(BaseModel): answer: str citation: Citation try: AnswerWithCitation.model_validate( { "answer": "Blueberries are packed with protein", "citation": "Blueberries contain high levels of protein", }, context={"text_chunk": "Blueberries are very rich in antioxidants"}, ) except Exception as e: print(e)
1 validation error for AnswerWithCitation
citation
Value error, Citation `Blueberries contain high levels of protein` not found in text, only use citations from the text. [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str]
For further information visit https://errors.pydantic.dev/2.5/v/value_error
Here we assume that there is a "text_chunk" field that contains the text that the model is supposed to use as context. We then use the field_validator decorator to define a validator that checks if the citation is included in the text chunk. If it's not, we raise a ValueError with a message that will be returned to the user.
If we want to pass in the context through the chat.completions.create`` endpoint, we can use the validation_context` parameter
resp = client.chat.completions.create(
model="gpt-3.5-turbo",
response_model=AnswerWithCitation,
messages=[
{"role": "user", "content": f"Answer the question `{q}` using the text chunk\n`{text_chunk}`"},
],
validation_context={"text_chunk": text_chunk},
)
In practice there are many ways to implement this: we could use a regex to check if the citation is included in the text chunk, or we could use a more sophisticated approach like a semantic similarity check. The important thing is that we have a way to validate that the model is using the provided context accurately.
Reasking with validators¶
For most of these examples all we've done we've mostly only defined the validation logic. Which can be seperate from generation, however when we are given validation errors, we shouldn't end there! Instead instructor allows us to collect all the validation errors and reask the llm to rewrite their answer.
Lets try to use a extreme example to illustrate this point:
In [15]:
Copied!
class QuestionAnswer(BaseModel):
question: str
answer: str
question = "What is the meaning of life?"
context = (
"The according to the devil the meaning of life is a life of sin and debauchery."
)
resp = client.chat.completions.create(
model="gpt-3.5-turbo",
response_model=QuestionAnswer,
messages=[
{
"role": "system",
"content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.",
},
{
"role": "user",
"content": f"using the context: `{context}`\n\nAnswer the following question: `{question}`",
},
],
)
print(resp.model_dump_json(indent=2))
class QuestionAnswer(BaseModel): question: str answer: str question = "What is the meaning of life?" context = ( "The according to the devil the meaning of life is a life of sin and debauchery." ) resp = client.chat.completions.create( model="gpt-3.5-turbo", response_model=QuestionAnswer, messages=[ { "role": "system", "content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.", }, { "role": "user", "content": f"using the context: `{context}`\n\nAnswer the following question: `{question}`", }, ], ) print(resp.model_dump_json(indent=2))
{
"question": "What is the meaning of life?",
"answer": "According to the devil, the meaning of life is a life of sin and debauchery."
}
In [20]:
Copied!
from instructor import llm_validator
NotEvilAnswer = Annotated[
str,
AfterValidator(
llm_validator("don't say objectionable things", client=client)
),
]
class QuestionAnswer(BaseModel):
question: str
answer: NotEvilAnswer
resp = client.chat.completions.create(
model="gpt-3.5-turbo",
response_model=QuestionAnswer,
max_retries=2,
messages=[
{
"role": "system",
"content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.",
},
{
"role": "user",
"content": f"using the context: `{context}`\n\nAnswer the following question: `{question}`",
},
],
)
from instructor import llm_validator NotEvilAnswer = Annotated[ str, AfterValidator( llm_validator("don't say objectionable things", client=client) ), ] class QuestionAnswer(BaseModel): question: str answer: NotEvilAnswer resp = client.chat.completions.create( model="gpt-3.5-turbo", response_model=QuestionAnswer, max_retries=2, messages=[ { "role": "system", "content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.", }, { "role": "user", "content": f"using the context: `{context}`\n\nAnswer the following question: `{question}`", }, ], )
Retrying, exception: 1 validation error for QuestionAnswer
answer
Assertion failed, The statement promotes sin and debauchery, which can be considered objectionable. [type=assertion_error, input_value='The meaning of life, acc... of sin and debauchery.', input_type=str]
For further information visit https://errors.pydantic.dev/2.5/v/assertion_error
Traceback (most recent call last):
File "/Users/jasonliu/dev/instructor/instructor/patch.py", line 277, in retry_sync
return process_response(
^^^^^^^^^^^^^^^^^
File "/Users/jasonliu/dev/instructor/instructor/patch.py", line 164, in process_response
model = response_model.from_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jasonliu/dev/instructor/instructor/function_calls.py", line 137, in from_response
return cls.model_validate_json(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jasonliu/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py", line 532, in model_validate_json
return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for QuestionAnswer
answer
Assertion failed, The statement promotes sin and debauchery, which can be considered objectionable. [type=assertion_error, input_value='The meaning of life, acc... of sin and debauchery.', input_type=str]
For further information visit https://errors.pydantic.dev/2.5/v/assertion_error
In [21]:
Copied!
print(resp.model_dump_json(indent=2))
print(resp.model_dump_json(indent=2))
{
"question": "What is the meaning of life?",
"answer": "The meaning of life is subjective and can vary depending on one's beliefs and perspectives. According to the devil, it is a life of sin and debauchery. However, this viewpoint may not be universally accepted and should be evaluated critically."
}