Lectorium Devlog #1: Our Model's First Words in Estonian

We've spent the last stretch actually building the thing rather than talking about it, and we have our first real results. The headline, if we're honest, is that we now have a small AI model that has started to babble in Estonian - in the most endearing way imaginable. More on that in a moment.

For anyone just joining: Lectorium is our exploration into a small, purpose-built AI that understands Estonian and can eventually answer questions about Estonian law with real, clickable references to the source. Built by us, from scratch, deliberately small. If that sounds interesting, our first post sets the scene.

Reading Estonian in words, not letters

Here's something we suspected going in and have now confirmed with numbers: most big AI models are, at heart, English machines. When they meet Estonian, they cope by chopping our words into tiny fragments - almost spelling them out, piece by piece. That's wasteful. The model spends its energy reassembling Estonian from crumbs instead of reading it whole.

So one of the first things we did was build our own Estonian tokenizer - the component that decides how text gets broken into pieces the model can read - and we trained it on Estonian text itself. The result reads the language in far larger, more natural chunks: roughly a third fewer pieces to represent the same sentence compared to a standard English-first model.

That sounds like plumbing, and it is. But it's also the whole thesis of Lectorium in miniature: when you treat Estonian as the starting point rather than an afterthought, the machine simply fits the language better. Fewer pieces means the model sees more context at once, learns faster, and wastes less effort - all from a decision made on day one.

From noise to first words

We kept this first model deliberately tiny - small enough to train on an ordinary computer, with no specialised hardware, in about two hours. It is very much a warm-up, not the finished article. But watching it learn was genuinely moving.

It begins as pure noise: random output, no structure, no words. Then, step by step, patterns start to surface. And after a couple of hours, when we asked it to continue the phrase "Tallinn on" ("Tallinn is"), it gave us this:

Tallinn on ta, aga ta ta ta ta aga neid see ta ta kui ta oma ta oli ka tema…

It's babbling. Ta - he, she, it. Aga - but. Kui - if. Oli - was. The handful of little words that hold every Estonian sentence together, repeated like a toddler who has just discovered them and can't quite stop. It isn't saying anything yet. But it is unmistakably trying to say it in Estonian - and we found that oddly delightful.

And every so often, it startles you. Elsewhere in the same session, out of the same fog, came a clean, grammatical fragment:

…1991. aastal kuulus ta Eesti Vabariigi koosseisu.
("…in 1991 it became part of the Republic of Estonia.")

A real sentence, correctly formed, appearing out of nowhere. That's the moment you realise the thing is genuinely learning - not memorising, learning - and you start to imagine what it will sound like after a proper, full-scale training run.

Grounded, or nothing at all

The other half of Lectorium is the part that matters most for law: it should never make things up. Our design principle has been blunt from the start - an answer without a citation is not an answer worth giving.

So we built the retrieval-and-citation machinery: the system finds the relevant legal text first, and only then forms a response, always pointing back to the source. Crucially, if it can't find anything relevant enough, it says so instead of guessing. We've tested this end to end and it behaves exactly as intended - ask it something off-topic and it politely declines rather than inventing a paragraph.

One honest caveat: for now, we've tested this on a small, hand-made sample of employment-law text - not the real statute book. Wiring Lectorium up to live Riigi Teataja is one of our next steps. We're being careful here on purpose: the whole point is trustworthiness, and we'd rather show you a modest thing that works than an impressive thing that doesn't.

What we're still figuring out

We're early, and we're happy to say so. The open questions from our first post are still open - but we now have first answers to some of them:

• How much does an Estonian-first tokenizer matter? More than we expected. That "reads in words, not letters" gain is real and measurable.
• How small can the model be and still produce coherent Estonian? Our tiny warm-up babbles charmingly but isn't coherent yet - that takes a bigger model and a longer training run, which is exactly where we're headed next.
• Can retrieval keep it honest? So far, yes - the refuse-when-unsure behaviour works. The real test comes when we point it at the full body of Estonian legislation.

The next milestone is a proper training run on real hardware, and connecting the system to live Riigi Teataja so it reads actual law rather than our practice text.

A few things under the hood

For the curious, in plain terms:

• From scratch. We're not fine-tuning someone else's model - we're training our own, so we understand every part of it.
• Small on purpose. This first model is a fraction of the size of the giants in the headlines, and it ran on a normal computer. Small is a design choice, not a limitation.
• Estonian-first. Our own tokenizer, trained on Estonian, built around the language rather than bolted onto an English core.
• Cite, then answer. Retrieval comes before generation, and sources come with every answer - by design, not as an afterthought.

Where we go next

This was the "does the whole machine actually run?" milestone, and the answer is yes - end to end, on real Estonian text, with our own tokenizer, our own model, and a citation-first design. What we have is not yet a tool you'd hand to a lawyer. It's the foundation you build one on, and it's standing.

We'll keep sharing as we go - the good runs and the bad, the things that surprise us, and hopefully, before long, a model that graduates from babbling to something you'd actually want to read.

More soon, from the Crowned Phoenix workshop.