🤖⚡ Ask Mauro: Why I Went Smaller, Not Bigger

I got tired of people browsing my resume. So I made myself queryable. 🤖

Meet Ask Mauro: 👉 ai.maurogioberti.com
Ask it about my experience, projects, talks, AI work, architecture, leadership — pretty much anything I've made public about my professional life. The same AI also lives as a popup inside this site.

That's the product. The part worth arguing about is the architecture behind it.

The rules I care about

• 🧠 Answers are grounded in my own public knowledge base — not in whatever the model absorbed about someone with a similar name.
• 🔎 Retrieval checks the actual evidence before answering.
• 🔗 If a link isn't supported by that evidence, it doesn't get to invent one.
• ⚡ Response behavior and latency are measured, so "it feels faster now" is never the justification for a change.

None of those four is a model capability. They are all architecture.
A bigger model doesn't give you any of them for free, and a smaller one doesn't take them away.

The first version worked. It was also unusable 🐌

That fourth rule is the one that rewrote the whole project.

The first working version did what I designed it to do. Grounded in my knowledge base, retrieval before answering, no invented links. On the rules I care about, it was the product I wanted.

It averaged around 68 seconds per response.

68 seconds is not "a bit slow". It's a different product.
Nobody sits in front of a chat box for over a minute to find out whether I've spoken at a conference. They type, they watch a spinner, they decide the site is broken, and they leave.

A correct answer nobody waits for is not an answer. It's a log entry.

And here's the uncomfortable part: this was not the kind of slow you profile your way out of. The system was doing exactly what I designed it to do — with a model far too heavy for the machine it lived on.

The hardware 🖥️

All of this runs on a single server: 8 vCPU, 24 GB RAM.
No cluster, no autoscaling group, no "we'll figure out the bill later" scaling plan.

That constraint is deliberate. A personal AI that costs me real money every time a stranger gets curious about me is a project I quietly turn off in three months. If I want it to stay online, it has to be cheap and boring to run.

Model size stops being capability and becomes a latency tax

General engineering reasoning here, not something I measured axis by axis — but it explains the shape of the result:

• Generation is sequential. Tokens come out one at a time, each one requiring a full pass over the weights. Bigger model, more work per token, and the user feels every single one.
• Context isn't free. Everything you stuff into the prompt gets processed before the first output token appears. Dumping every plausible chunk into the prompt "just in case" is a latency decision disguised as a retrieval decision.
• Concurrency multiplies it. On a modest box, two people asking at the same time do not get the experience one person alone gets.

So the honest question becomes: what am I actually buying with those extra parameters?

My problem is bounded and closed. Every legitimate question about me has its answer sitting somewhere in my own public knowledge base. Nothing outside that corpus is a valid answer — if the evidence isn't there, the correct output is "I don't have that", not a plausible guess.

When the corpus is closed, the model doesn't need to know things.
It needs to read the evidence in front of it and not go beyond it.

Retrieval already did the hard part. What's left is reading comprehension and faithful writing — closer to a formatting job than a frontier reasoning task. Those extra parameters are buying world knowledge I explicitly don't want the system to use, and charging me seconds per answer for it.

That's not capability. That's a tax.

So I went smaller 📦

The reflex when an AI feature disappoints is to reach for a bigger model. I went the other way: Qwen3 0.6B, a small language model, tiny compared with the ones we usually talk about in AI.

The work didn't disappear, though. It moved. Every bit of slack I took out of the model, I had to put back into retrieval, grounding, and the answer paths around it.

What you have to fix when the model gets small 🔧

Swapping the model is a config change. Making the swap survivable is the actual project.

• Precision over recall. With a big model, extra context is mostly harmless noise. With a small one, irrelevant passages actively compete for attention. Deciding what not to send became a first-class design step.
• Self-contained chunks. A small model won't reconstruct missing context from background knowledge it doesn't have. If a chunk needs its neighbour to make sense, it fails.
• Short, unambiguous instructions. Elaborate multi-clause system prompts full of conditional rules are a big-model luxury — and every extra token in the prompt is paid for before the first word comes out.
• An explicit "I don't know" path. If the evidence doesn't cover the question, saying so is the correct output. Left unhandled, that gap is precisely where a model starts improvising, and improvisation about my career is the one failure mode I refuse to ship.
• Answer shape. Short, direct, evidence-first. Generated tokens are literally what the user is waiting on.

A small model is a forcing function

This is the part I didn't expect. A large model is forgiving. Feed it mediocre chunks with irrelevant ones mixed in, and it will usually still find the thread. That forgiveness feels like quality. It's actually insulation — it hides how bad your retrieval is, and you pay for that insulation on every single request, forever.

A small model has no such generosity. Give it noisy context and the answer falls apart immediately and visibly. Every retrieval weakness stops being a slow quality tax and becomes a bug you can see and fix.

A big model lets you ship bad retrieval. A small one refuses to.

The link rule 🔗

A hallucinated sentence is bad. A hallucinated link is worse.

A wrong sentence is an opinion you might question. A wrong link is a promise the reader can click — and when it 404s, nobody concludes the model was uncertain. They conclude I was careless. My name is on the domain.

Which is why this is a constraint, not a request. "Please only cite sources you actually retrieved" written into a prompt is a probability, and models comply with requests most of the time. The rare fabricated URL is exactly the one that costs you trust, and exactly the one you won't catch in casual testing.

So the rule I hold the pipeline to is blunt:

    question
      → retrieve   the evidence for it
      → ground     the answer in that evidence
      → answer     only what the evidence supports
      → links      only what the evidence supports
  

If the evidence doesn't carry the link, the answer doesn't get to.

The payoff: 68s → 17s ⚡

Moving from the larger model I started with to the SLM took the average response time from roughly 68 seconds down to roughly 17 seconds. The rules around it didn't change. The model did.

I didn't make the server bigger. I made the work smaller.

What that number is and isn't: it's an average response time, on my corpus, with my retrieval setup. It is not a claim that 0.6B models are four times faster than large ones for everybody, and it says nothing about answer quality that I measured. What it is: the evidence that turned an architectural opinion into a decision I can defend.

Without that instrumentation I'd have had a hunch, a migration, and no way to know whether I'd made things better or just different.

Where a big model is still the right call

To be fair, there are problems where reaching for the largest model available is correct engineering, not laziness:

• Open-domain reasoning — when the answer isn't in any corpus you own, retrieval has nothing to retrieve. Capacity is the product.
• Long agentic chains — small per-step error rates compound into large end-to-end failure rates, and raw capability buys real reliability.
• Hard code generation — holding a big mental model of unfamiliar code and producing something that actually compiles is where large models still separate themselves.
• Synthesis across many documents — when the task is "read all of this and find the tension between them", no single chunk contains the answer.
• You haven't built retrieval yet — a big model is a legitimate way to validate that a feature is worth building at all. The mistake isn't starting there. It's staying there and calling it an architecture.

Ask Mauro is none of those. Bounded corpus, bounded intents, and a hard requirement not to invent things. That's a retrieval-shaped problem, and it deserved a retrieval-shaped solution.

Before your next model upgrade

• When the system is wrong, was the correct evidence in the retrieved context? If it wasn't, this is not a model problem and the upgrade won't fix it.
• If you paste the perfect context in by hand, does a small model answer correctly? If yes, you have a retrieval problem.
• Can your system refuse? A pipeline that can't say "I don't have that" will hallucinate at every model size.
• Are you measuring latency the way you measure quality, or is it in a dashboard nobody opens?

If most of your failures are retrieval failures, a bigger model is buying silence, not correctness. It just makes the wrong answers sound better.

Now go break it 😄

• 🤨 Ask weird questions.
• 👻 Try to make it hallucinate.
• 🎯 Find something it gets wrong.
• 🔗 See if you can get it to hand you a link it shouldn't have.

👉 ai.maurogioberti.com
Or just open the popup on this site. Same AI.

If you catch it inventing something, contradicting itself, or confidently answering a question it has no evidence for, I genuinely want to hear about it. 💬
A failure you can reproduce is worth more to me than a compliment — and it tells me whether the weakness is in retrieval, in grounding, or actually in the model.

I stopped asking people to read about me. Now they can just ask. 🚀

#AI #SLM #RAG #LLM #Qwen3 #SoftwareArchitecture