Lectorium Devlog #2: Renting a Brain by the Hour

So we set out to train the model properly. And here's the thing that surprised us - the hard part wasn't the training itself. It was the deciding. Where to train, on what, for how much, and what could go wrong. We're not seasoned AI engineers; we're a small team learning this in the open. So in the spirit of teaching what we learn as we learn it, here are the questions we found ourselves having to answer - the ones we think anyone standing where we're standing should think about too.

You (probably) don't need a supercomputer

The headlines about AI are all about scale - vast data centres, thousands of chips, budgets with a lot of zeros. It's easy to assume you need a slice of that to train anything at all.

You don't. The single most useful realisation we had is that you match the machine to the model, not to the hype. Our model is deliberately small, so it needs a modest amount of power - the equivalent of one second-hand gaming graphics card, not a room full of the latest hardware. When we went shopping, the temptation was to reach for the most powerful (and most expensive) option available. The right move was almost the opposite: work out what the job actually needs, and rent exactly that. Nothing more.

Rent, don't buy

You don't have to own the hardware. There's a whole market for renting computing power by the hour - you spin up a machine somewhere in the world, use it for an afternoon, and give it back.

This changes the economics completely. Instead of buying an expensive computer that sits idle most of the time, you pay only for the few hours you're actually training. For us, that turned a scary-sounding "train your own AI" into something that costs - genuinely - about the price of a cup of coffee. Under a dollar for a full training run.

That number matters beyond our own budget. It's proof of something we believe strongly: building AI for a small language and a specific purpose is far more accessible than the big-tech narrative suggests. You don't need to be a giant. You need to be deliberate.

Your own computer works - but "slow" is the whole problem

We could, in theory, have just kept training on our own machine. It would get there eventually. And that "eventually" is exactly the trap.

Here's the subtle bit we had to internalise: a regular computer isn't incapable of training a model - it's just slow, and slowness compounds in a way that's easy to underestimate. Training a model is never one clean run. It's try something, look at the result, adjust, try again - over and over. On a slow machine, each of those loops takes days, so you get only a handful of attempts. On the right hardware, each loop takes a couple of hours, so you can actually learn from your experiments. The real cost of underpowered training isn't the final run - it's that you never get to iterate. That insight alone was worth the whole exercise.

Check that you can actually reach your ingredients

This one caught us off guard, and it's the kind of thing nobody warns you about.

A model learns from data - in our case, a large collection of Estonian text that lives on the internet and has to be downloaded to wherever you're training. When we compared rental options, one was noticeably cheaper - but it sat in a part of the world where the place our data lives isn't reliably reachable. A perfect machine is useless if it can't get to the ingredients. So we paid a few cents more for a location where the download would simply work.

The general lesson: before you fall in love with a price, make sure the machine can actually reach everything it needs. The cheapest option that doesn't work isn't cheap.

Assume something will go wrong - and save your work

Rented machines can hiccup. Connections drop. Occasionally an instance gets reclaimed mid-run. If a multi-hour job vanishes with no trace, that's a genuinely bad afternoon.

So we built in the boring safety net from the start: the training saves its progress regularly and can pick up exactly where it left off. It's the digital equivalent of saving your document often. Unglamorous, and you're deeply grateful for it the first time something interrupts a run.

Remember that a model is more than one file

The last thing we had to plan for is almost philosophical: you train the model in one place, but you want to use it in another. So you have to bring it home - and make sure it still works when it gets there.

A trained model isn't a single magic file. It comes with companions - in our case, the small "dictionary" that teaches it how to read Estonian in the first place. Bring back the model but forget its dictionary, and it's gibberish. Learning to think of a model as a small bundle of things that belong together, rather than one file, saved us a headache we'd otherwise have discovered the hard way.

What's next

We've done all the thinking. The plan is set, the machine is chosen, the safety nets are in place. What we haven't done yet is pull the trigger - and that's deliberate, because the next post is the fun one: how it actually went. The real cost, the honest results, the loss curves, and whether our carefully-reasoned plan survived contact with reality. (In our experience so far, plans rarely survive entirely intact - that's usually where the good stories are.)

If there's one thing we'd pass on from this stage, it's this: training your own AI is less about raw power and more about a handful of sensible decisions made calmly up front. We're still learning - but that part, at least, we're now fairly sure of.

More soon, from the Crowned Phoenix workshop.