How does generative AI work? And why doesn't it work all the time?

How does Generative AI work?

Generative AI is something that is very complex in how it’s built, but not that complex in how it works.

Generative AI is typically a Large Language Model, or LLM. How LLMs work boil down to three main points:

1. They are trained on billions of documents, from websites to books to software code.
2. They build a model that connects all of the words and phrases in these to each other.
3. Given that model and a text input (a “prompt”), it outputs what its model says is most likely to come next.

So if you give an LLM text like “The cat sat on the”, it will finish that sentence with “mat”. Not because it has any strong idea of cats or mats or rhyming, but because most of the time it sees “The cat sat on the mat” in the data, rather than variants like “The cat sat on the dog.”

On the other hand, if the LLM is trained on exclusively Chinese-language data, giving it “The cat sat on the mat” will probably produce something like “我不明白”, because it doesn’t have “The cat sat on the mat” in its data, so it’s giving what is most likely to come next given what data it’s trained on.

That’s most of what an LLM is — a very powerful, very well-sourced, fill-in-the-blank.

What it can do

Because LLMs are trained on trillions of human conversations, they’re able to imitate the style, grammar, and meaning of human conversations in a way that seems very similar to humans. And because it’s designed to fill in the blanks in a conversation given what came before, it’s able to improvise on the fly, producing a system of talk-and-response that feels for a lot of people like they’re communicating with an actual person.

All of that data also means that, unlike traditional software, it’s less likely to get tripped up by ambiguity or minor errors in input. So if you mispell a request or mean one thing but say another, it’s more likely to pick up what you meant and not break.

The large amount of data also means it has a lot of collective human knowledge at its disposal, which makes it very useful for asking questions with a lot of data supporting them like “What are the main components of a bicycle”.

This is part of why we often see LLMs deployed as chatbots or as coding agents — they work much better than traditional software at responding to natural human questions with reasonable (or at least reasonable-sounding) responses.

So why don’t we just use them all the time? Well, if you’re someone who’s tried, you’ve probably come across one particularly troublesome issue with generative AI.

LLMs get things wrong

Back in 2024, when LLMs were just starting to enter broader use, Google’s “AI Overview” suggested that users should use “non-toxic glue” to make cheese better stick to pizza.

Google’s AI Overview no longer has this issue, at least with respect to pizza. But why did the AI, which was powered by an LLM, suggest this? Did it accidentally pull data that was a joke? Did it see “stick to” and connect that with “glue”? Is there a group of “innovative” chefs who are looking into using glue to meet their culinary needs?

The answer is that nobody knows. In fact, nobody knows precisely why an LLM makes any answer.

The model connecting all the data in an LLM is so large and complex that nobody is actually able to say why a specific input triggered a specific output. We know it’s dependent on the data, the algorithm, and the prompt, but as to why your specific input created that specific response, the answer is “The math worked out that way”.

That poses a problem when people aren’t just looking for an answer that seems plausible, but is correct. An LLM can’t verify that what it’s saying is true. It’s entirely dependent on its data, its algorithm, and how both interact with its prompt.

Nor can we always predict when an LLM will go wrong and when it won’t. If you ask the same question to an LLM twice, you will often get two different answers. This is partly because LLMs, being based on billions of connections between words and phrases, are very sensitive to even slight changes in a prompt.

Even if you ask an LLM why it made a decision, it won’t necessarily give you an accurate explanation. Given what it already wrote and how you phrased the question, it will come up with what its system determines is the most likely response to the question. Which is not how it actually came to the answer it gave.

We might not be able to solve this

AI service providers have come up with a variety of ways to reduce this rate of error, or “hallucination” as it’s commonly called. This includes refining the prompts that go into the LLM, adjusting what data is used to build the LLM, using more powerful models, and adding additional systems to check the outputs.

And yet, most generative AI services still include a fine-print caveat that “Responses may be inaccurate”. Because while these may reduce the rate of hallucination, they haven’t eliminated it. In fact, there may be no way to fully eliminate hallucinations from these systems — they are just too complex to account for every possible wrong response.

Depending on the scenario, that might be fine. I use generative AI every day, and it sometimes makes mistakes. But over time I’ve learned when its outputs are good enough that I can use them comfortably, when I need to double check its results, and when I need to not use generative AI at all. That’s a larger conversation for a later blog post.