Skip to content

2023

  • 7 min read

Introduction to Caching in Python

Instructor makes working with language models easy, but they are still computationally expensive.

Today, we're diving into optimizing instructor code while maintaining the excellent DX offered by Pydantic models. We'll tackle the challenges of caching Pydantic models, typically incompatible with pickle, and explore solutions that use decorators like functools.cache. Then, we'll craft custom decorators with diskcache and redis to support persistent caching and distributed systems.

  • in Python
  • 6 min read

Generators and LLM Streaming

Latency is crucial, especially in eCommerce and newer chat applications like ChatGPT. Streaming is the solution that enables us to enhance the user experience without the need for faster response times.

And what makes streaming possible? Generators!

  • 4 min read

Verifying LLM Citations with Pydantic

Ensuring the accuracy of information is crucial. This blog post explores how Pydantic's powerful and flexible validators can enhance data accuracy through citation verification.

We'll start with using a simple substring check to verify citations. Then we'll use instructor itself to power an LLM to verify citations and align answers with the given citations. Finally, we'll explore how we can use these techniques to generate a dataset of accurate responses.

  • in Python
  • 6 min read

Async Processing OpenAI using asyncio and Instructor with Python

Today, I will introduce you to various approaches for using asyncio in Python. We will apply this to batch process data using instructor and learn how to use asyncio.gather and asyncio.as_completed for concurrent data processing. Additionally, we will explore how to limit the number of concurrent requests to a server using asyncio.Semaphore.

  • 15 min read

Smarter Summaries w/ Finetuning GPT-3.5 and Chain of Density

Discover how to distil an iterative method like Chain Of Density into a single finetuned model using Instructor

In this article, we'll guide you through implementing the original Chain of Density method using Instructor, then show how to distile a GPT 3.5 model to match GPT-4's iterative summarization capabilities. Using these methods were able to decrease latency by 20x, reduce costs by 50x and maintain entity density.

By the end you'll end up with a GPT 3.5 model, (fine-tuned using Instructor's great tooling), capable of producing summaries that rival the effectiveness of Chain of Density [Adams et al. (2023)]. As always, all code is readily available in our examples/chain-of-density folder in our repo for your reference.

Good LLM Validation is Just Good Validation

What if your validation logic could learn and adapt like a human, but operate at the speed of software? This is the future of validation and it's already here.

Validation is the backbone of reliable software. But traditional methods are static, rule-based, and can't adapt to new challenges. This post looks at how to bring dynamic, machine learning-driven validation into your software stack using Python libraries like Pydantic and Instructor. We validate these outputs using a validation function which conforms to the structure seen below.

def validation_function(value):
    if condition(value):
        raise ValueError("Value is not valid")
    return mutation(value)

Enhancing Python Functions with Instructor: A Guide to Fine-Tuning and Distillation

Introduction

Get ready to dive deep into the world of fine-tuning task specific language models with Python functions. We'll explore how the instructor.instructions streamlines this process, making the task you want to distil more efficient and powerful while preserving its original functionality and backwards compatibility.

If you want to see the full example checkout examples/distillation

  • in RAG
  • 7 min read

With the advent of large language models (LLM), retrieval augmented generation (RAG) has become a hot topic. However throughout the past year of helping startups integrate LLMs into their stack I've noticed that the pattern of taking user queries, embedding them, and directly searching a vector store is effectively demoware.

What is RAG?

Retrieval augmented generation (RAG) is a technique that uses an LLM to generate responses, but uses a search backend to augment the generation. In the past year using text embeddings with a vector databases has been the most popular approach I've seen being socialized.

RAG

Simple RAG that embedded the user query and makes a search.

So let's kick things off by examining what I like to call the 'Dumb' RAG Model—a basic setup that's more common than you'd think.