How we gave superpowers to our search with ChatGPT 💪

In late 2022, ChatGPT surprised everyone, including us. Elium is built around knowledge, so we had to find out what a model this good at language could do for the way our users work with content. Here is what we built, and how it works.

Hackathon

We started with a two-day hackathon, open to the whole company. The goal was simple: build proof-of-concept (POC) projects that put AI inside our product.

Two days produced more than 30 ideas. We examined each one for feasibility and for what it would change for our users, and kept two: a podcast generated automatically from key content, and a question-and-answer feature.

The creation of Answer

The product team took both POCs to users and asked for feedback. The answer was clear: people wanted the question-and-answer feature. It promised faster and better responses in natural language. It also changed the interaction between the people who produce content and the people who consume it, which leads to a more correct and precise knowledge base.

So we built it, and called it Answer.

How does Answer work under the hood

This is the whole life cycle of Answer. It has two parts: index time and query time.

Index time

When content is saved in Elium, we index it semantically, to capture its meaning rather than just its words.

We divide the content into chunks of about 300 words, keeping sentence boundaries where we can. For each chunk, we ask a large language model (LLM) to produce an embedding: a vector that represents the meaning of the chunk.

You can picture an embedding as a position on a graph, with one axis running from formal to informal, another from positive to negative, and many more. Context decides where the text lands.

We store the embedding in the index, together with the chunk text and metadata such as the content ID, its title and its space ID.

Query time

When someone asks a question, we compute the embedding of the query with the same LLM. Simple mathematics, the cosine similarity between the query embedding and each stored embedding, tells us which chunks sit closest to the question, across one or several pieces of content.

We could stop there and return those chunks. That would be cheap. Instead, we pass them to ChatGPT and ask it to answer the user’s question from them.

Here is a simplified example of the prompt we use.

Finally, we return the answer from ChatGPT, together with the list of content the chunks came from, so the reader can check the sources.

What we learned

The model impressed us. The bigger discovery was how much the quality of an answer depends on the quality of the content behind it. ChatGPT answers well when the content it quotes is well written, organised and current. That work is still done by people, and Answer makes it more visible, not less.