Lectorium Devlog #3: Our Model Learned Estonian in 41 Minutes - and Immediately Started Making Things Up

What we actually did

A quick recap of the ingredients, in plain terms.

First, we taught the model how to read Estonian - we built a fresh tokenizer, the component that decides how text gets chopped into pieces, trained on Estonian itself so it reads our language in whole, natural chunks rather than spelling it out.

Then we gave it something to read: the entire Estonian Wikipedia - around 240,000 articles - converted into a form the model could learn from.

Then we built the model. Deliberately small: 33 million "parameters," which sounds like a lot but is tiny by the standards of the AI you read about in the news - thousands of times smaller than the big commercial models. Small is the point. It's cheap to train, cheap to run, and possible for a small team to actually understand end to end.

And then we rented a single second-hand graphics card by the hour and let it learn.

The part that surprised us: speed and price

We had braced ourselves for a training run of a couple of hours. It finished in 41 minutes. The rented card churned through the whole of Estonian Wikipedia several times over before we'd finished making coffee.

The total bill for the entire session - setup, downloading the data, the training itself - came to $0.24. Twenty-four cents. We keep repeating that number because it still feels wrong to write, and because it's the whole point: building a real language model for your own language is not the preserve of giants with warehouses of hardware. It's genuinely within reach.

It speaks Estonian now

Here is the before-and-after that made our week. It's the same prompt - "Tallinn on…" ("Tallinn is…") - given to our tiny practice model from an earlier post, and to this one.

The old practice run gave us charming baby-talk:

Tallinn on ta, aga ta ta ta ta aga neid see ta ta…

This new model gave us this:

Tallinn on väljamõeldud piirkond Põhja-Makedooniast, mis hõlmab Lõuna-Makedooniast, Kreekani poolsaarest ja selle lääneosast.

That is fluent, grammatically correct Estonian. Real sentence structure, proper endings, the works. Elsewhere it wrote coherent passages of political news, correctly using the names of actual former Estonian prime ministers. For a model this small, trained for the price of a snack, we were genuinely delighted.

…and it immediately started making things up

You may have noticed something about that lovely Estonian sentence above. Translated, it says:

"Tallinn is a fictional region of North Macedonia, encompassing South Macedonia, the Greek peninsula and its western part."

Which is, of course, complete nonsense. Tallinn is the capital of Estonia and has never been anywhere near North Macedonia. The grammar is flawless; the facts are invented out of thin air.

And this - we cannot stress this enough - is not a bug we need to fix. It is the single best argument for Lectorium's entire design, delivered by our own creation, unprompted.

A language model's talent is language. It learns how words fit together, how sentences flow, how a paragraph sounds. What it does not reliably learn is facts - it will happily produce a beautiful, confident, wrong answer. Now imagine that tendency pointed at the law. A model that recites statutes from memory will invent paragraph numbers and misquote articles with total conviction. For legal questions, that's not just unhelpful - it's dangerous.

Which is exactly why Lectorium never trusts the model's memory. It retrieves the relevant law first, and only then forms an answer, always citing the source so you can read it yourself. The model supplies the fluent Estonian; the actual legal text supplies the truth. Our fact-inventing little model just proved, in one sentence, why that split matters.

Where things honestly stand

So we ran the whole system together, end to end, for the first time with a real model - and the result is exactly as instructive as the North Macedonia incident.

The retrieval half works. We asked it a real employment-law question - how long is annual leave in Estonia - and it correctly surfaced the right article, with its citation. That part is trustworthy today.

The answering half is not there yet. Our model, so far, has only learned to continue Estonian text, not to follow instructions like "read these articles and answer the question." Hand it the law and a question, and it doesn't yet know it's supposed to answer - it just keeps writing. Teaching it that skill is a well-understood next step (a further round of training focused on question-and-answer), and it's the next chapter of this project.

We're telling you this plainly because that's the whole spirit of building in the open: the retrieval is solid, the Estonian is fluent, and the bridge between them is the work in front of us. And crucially - because Lectorium is built retrieval-first - even with an unpolished answer, the system still hands you the correct, cited article rather than a confident fabrication. It fails safely. That was always the design, and it held.

The bigger picture

Step back and here's what we have: a language model that speaks Estonian, trained from scratch, that we own outright - no subscription, no foreign dependency, no per-question fee. And a pipeline that can produce another one, bigger or better, for pocket change, whenever we like.

That's not just a component of one product. It's a small, reusable Estonian-language capability - and the fact that it cost twenty-four cents to make means we can keep improving it without ever needing a giant's budget.

Next time: teaching it to actually answer. That's the fun one.

More soon, from the Crowned Phoenix workshop.