Building a search bar with Sanity Embeddings Index API & Next.js

Introduction

We've been wanting to tinker with Sanity's Embeddings Index API for a long time. We've also been toying with the idea of using it as a search bar too. So guess what we did.

Sanity Embedding is essentially a pretty powerful, AI powered semantic search for your content. It leverages embeddings, which are simplified vector representations of your documents, and can retrieve content based on meaning and context rather than just keywords.

So with that out the way, let's walk through how to create a search bar in a Next.js application that utilizes Sanity's Embeddings Index API. We'll cover the key steps including:

• Setting up an embeddings index for your Sanity dataset
• Configuring the Embeddings Index API in your Sanity project
• Building a search bar component in Next.js that queries the embeddings index
• Displaying the relevant search results to users

By the end, you'll have a fully functional semantic search experience that allows users to find the most relevant documents based on their search queries.

Whether you want to provide an intelligent search interface for your blog, documentation, or any other type of content, Sanity's Embeddings Index API makes it straightforward to implement. Let's dive in and see how it works!

Setting up an embeddings index

To set up an embeddings index in Sanity, you have two main options:

1. Using the Sanity Studio UI:
   • Go to your project's Sanity Studio.
   • Navigate to the "Embeddings" section.
   • Click on "Create new index".
   • Configure the index by specifying the dataset, projection, and other settings.
   • Save the index configuration.
2. Using the Sanity CLI:
   • Install the Sanity CLI if you haven't already.
   • Run the command sanity init to initialise a new Sanity project or you can use an existing one
   • Use the sanity embeddings create command to create a new embeddings index.
   • Provide the necessary configuration options such as dataset, projection, etc.
   • The CLI will create the embeddings index based on your configuration.

For detailed instructions on setting up an embeddings index using the CLI, you can refer to the Sanity documentation.

Play

Once you have created the embeddings index using either method, Sanity will process your documents and generate the embeddings. The index will be ready to use for semantic search and other applications.

In the next section, we will focus on configuring the Embeddings Index API in your Sanity project to enable querying the index from your Next.js application.

Adding Search to Next.js

To add search functionality to your Next.js application using Sanity's embeddings index API, you need to implement three main components:

1. Creating a search box.
2. Creating an API endpoint to handle the query.
3. Showing results to the client.

Search Box

To create a search box component in your Next.js application, you can use a simple input field. Here's an example of how you can implement a basic search box component:

// ~/components/search
export const SearchBox: FC = () => {
  const [query, setQuery] = useState("");
  return (
    <div className="ml-auto w-full md:w-auto">
      <div className="relative flex-1 flex-grow">
        <Search className="absolute left-2.5 top-2.5 h-4 w-4 text-muted-foreground" />
        <Input
          type="search"
          placeholder="Search..."
          onChange={(e) => setQuery(e.target.value)}
          className="w-full rounded-lg bg-background pl-8 md:w-[336px]"
        />
      </div>
    </div>
  );
};

API Endpoint

To create an API endpoint you can start by creating /api/search/route.ts file and add business logic in it. The logic for the search endpoint consists of two main components:

1. Getting embeddings results from Sanity that match the vector of the search query and return the corresponding document IDs.
2. Retrieving the actual documents from Sanity using the obtained document IDs.

Here is the code snippets for both functions:

const getEmbeddingResult = async (query: string, token:string) => {
  const response = await fetch(SANITY.SANITY_EMBEDDINGS_INDEX_URL, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    Authorization: `Bearer ${token}`,
  },
  body: JSON.stringify({
    query,
    maxResults: 5,
    filter: {},
  }),
});

  if (!response.ok) {
    throw new Error(response?.statusText);
  }

  const data = (await response.json()) as EmbeddingIndexResponse[] | undefined;

  if (!Array.isArray(data)) return [];

  return data.map((item) => item?.value?.documentId).filter(Boolean);
};




const getSanityDocuments = async (ids: string[], token:string) => {
  const groqQuery = `*[_type == 'article' && _id in ['${ids.join("', '")}']] {
    _id,
    title,
    "slug":slug.current,
    date
  }`;

  const queryUrl = `${
    SANITY.SANITY_DOCUMENT_QUERY_URL
  }?query=${encodeURIComponent(groqQuery)}&returnQuery=false`;

  const sanityDocs = await fetch(queryUrl, {
    method: 'GET',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${token}`,
    },
  });

  if (!sanityDocs.ok) {
    throw new Error(sanityDocs?.statusText);
  }
  const sanityDocsData = (await sanityDocs.json()) as
    | SanityDocumentResponse
    | undefined;

  if (!Array.isArray(sanityDocsData?.result)) return [];

  const _docs = sanityDocsData?.result ?? [];

  // preserving order of the embeddings
  const documentMap = new Map<string, SearchEmbeddingResponse>();

  _docs.forEach((doc) => documentMap.set(doc._id, doc));

  return ids
    .map((item) => documentMap.get(item))
    .filter(Boolean);
};

Showing it to the user

To display the search results to users, in a user-friendly manner, we can use React Query. React Query simplifies the process of fetching, caching, and managing the state of the search results. It also takes care of the heavy lifting, allowing you to focus on presenting the search results in an intuitive and visually appealing manner.

When the search results are fetched from the API, React Query automatically caches them on the client-side. This means that if the user performs the same search again, React Query can serve the cached results instantly, providing a faster and more responsive user experience.

Here's a custom hook we made using react query

const fetchArticleSearch = async (q: string) => {
  const response = await fetch('/api/search', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ query: q }),
  });
  if (!response.ok) throw new Error(response?.statusText);
  return ((await response.json()) as SearchEmbeddingResponse[]) ?? [];
};

type UseArticleSearchProps = {
  query: string;
};

export const useArticleSearch = ({ query }: UseArticleSearchProps) => {
  const debouncedQuery = useDebounce(query);
  const result = useQuery({
    queryKey: ['some-unique-key', debouncedQuery],
    queryFn: () => fetchArticleSearch(debouncedQuery),
    retry: 0,
    enabled: !!debouncedQuery,
    refetchOnMount: false,
    refetchOnWindowFocus: false,
  });
  return {
    ...result,
    debouncedQuery,
  };
};

Finally we can use data returned from useArticleSearch on our search result.

Search In Action

Play