I let an AI
improve itself
overnight.
An idea that came out of a project by Andrej Karpathy (a former OpenAI and Tesla guy, one of the most respected researchers in the field): you can ask an AI to edit its own work, test whether it's better, keep what works, throw out what fails. And start over. All night long. I tried it at home on a script that was a pain to tune. Here's what happened.
What you'll understand
- Karpathy's wild idea: let an AI agent edit its own work, test it, keep what works, throw out the rest. On a loop, without you, for hours.
- My real experience: I let Claude work on my email-sorting script from 10 p.m. to 7 a.m. When I woke up, it had tried 14 versions and found a recipe that missed fewer important emails.
- The result in numbers: 23 % fewer important emails missed, €8 in AI-token cost, zero input from me overnight.
- Why it's worth it even if you're not a dev: it works just as well on newsletter subject lines, product pages, prompts — anything you can test and score.
- My blunt advice: start with a case that isn't critical, that's easy to measure, and set yourself a budget in euros before you launch. No magic, just automated patience.
- Transparency: this article was written with Claude (Anthropic), reviewed and approved by me. The overnight experiment itself ran on Claude. If you spot a mistake, write to me — I'll fix it.
The idea that kept me up at night.
It was a Sunday evening in March 2026. I was poking around on GitHub (the platform where developers share their code, kind of the YouTube of software) and I stumbled on a project by Andrej Karpathy called autoresearch. I won't pretend half the project isn't gibberish to me — there are charts, researcher terms, metrics that mean nothing to me at all. But the core idea hit me right between the eyes.
The idea fits in one sentence: you put an AI to work on a problem, you ask it to edit its own work, you let it test whether the new version is better, and if it is, it keeps going from there — otherwise it rolls back and tries something else. And you repeat. All night long. Without you.
What made it click for me was that I happened to have one thing that had been driving me crazy for three months: my email-sorting script. Not some sophisticated program, just a little script Claude had written for me (Claude is Anthropic's AI, one of the big conversational models like ChatGPT) to scan my inbox in the morning and surface the 10 most important emails to deal with first. The problem: it kept missing some. A quote request lost in the newsletters, an email from my accountant buried in the LinkedIn ads. Not catastrophic, but annoying.
And then I think: "What if I let the AI improve its own script? On its own, while I sleep?" I close the laptop. I sleep badly. The next morning I get up, make the coffee, and decide to give it a shot that very evening.
A quick honesty note
I'm not a dev. I never studied computer science. Everything I did in this article, I did by asking Claude to explain each step like I was a beginner, and by having it write the code for me. If I can do it, you can do it too.
My night with Claude at the wheel.
Monday night. 9:45 p.m. My son is asleep. My wife is watching a show. I settle in at my desk. I open Claude Code (a tool that lets you chat with Claude right inside your terminal — I've got a dedicated tutorial on it). I explain to it in plain English, no jargon, what I want to do:
My brief, written in plain English
"I've got a little program that looks at my morning emails and pulls out the 10 most important ones. It misses too many. I've prepared a list of 200 emails from last week where I marked by hand which ones really mattered. I want you to edit the program, test it against my list, measure how many important emails it dropped, and start over trying other approaches. You've got all night. Keep the version that misses the fewest emails. In the morning, show me the top 3 versions and explain what you changed."
I prepared my list of 200 emails in a plain spreadsheet, with an "important: yes/no" column. Three hours of work on Sunday morning to build it — the least fun part of the whole thing, but that's where everything else is decided. Without that reference list, the AI can't measure anything.
10:12 p.m. I launch Claude. I tell it "go ahead." It tells me it'll start by reading the current program, that it'll propose a first change, then ping me if anything trips it up. I set a cost cap at €15 in the settings (more on that later) and I go to bed.
6:47 a.m. the next morning. My alarm goes off. I dash into the office before even the coffee. The screen is waiting for me. Here's what I find.
What the AI changed (that I'd never have found)
Here's what really surprised me. The changes Claude kept were three things I'd never thought of:
- It looked at the conversation history. If someone had already replied to me three times that week, it figured a new email from that person was more likely to be important. Obvious once it's said. Not obvious to code yourself.
- It learned to recognize professional attachments. A 2 MB PDF in a short email is often a quote or a contract — so it's important. A meme image in a long email is rarely urgent.
- It learned to be wary of "urgent" subject lines. Statistically in my list, emails screaming "URGENT!!!" in the subject were actually 4 times less important than emails that calmly announced a topic in two words. Marketing taught me to distrust the shouters.
Honestly? It would have taken me three months to find those three tweaks on my own, tuning my program bit by bit, every time an email slipped through. Claude did it in 9 hours while I slept.
The funniest part of the story: the final version is simpler than the original. Not longer, not more complicated — smarter. That's exactly what Karpathy said in his project: good improvements often remove code, they don't add it.
An AI that cooks, tastes, repeats.
Now that we've seen my concrete case, let's take two minutes to understand the idea behind it. It's the simplest idea in the world dressed up in a fancy name ("autoresearch," automatic research). Here's the analogy that made it click for me.
Picture a cook who wants to perfect his pizza recipe. He has a base recipe. He tries a variant: a bit more salt. He has 10 people taste it. If more people prefer it, he keeps it. If not, he goes back to the previous recipe. Then he tries something else: less cheese. He has them taste again. He keeps it or tosses it. And so on, 50 times, 100 times.
The AI does exactly the same thing, except it doesn't cook pizzas — it cooks code, prompts, email subject lines, product pages. And it doesn't have 10 people taste it — it tests against a reference list you've prepared in advance (my 200 emails scored by hand, for example).
The loop, in four beats
The AI proposes a change
It looks at the current version of whatever it's working on (your script, your prompt, your email subject line) and proposes a specific change. Something like: "What if we added a rule saying that heavy PDF attachments are a sign of importance?"
It tests the new version against your reference
It takes your reference list (the 200 scored emails, or the 50 newsletter subject lines with their open rates, whatever) and runs the new version on it. It counts how many times it works.
It compares against the best version so far
If the new version does better (or just as well) as the previous best, it keeps it and that becomes the new starting point. If it does worse, it goes back to the previous best and tries something else. That's it. No magic.
It starts over — as many times as you tell it to
The loop runs until you say stop, or until your AI-token budget runs out. In the morning, it shows you the best version it found and the list of everything it tried.
That's it. Seriously, that's it. And that's what makes the idea so powerful: you can apply it to anything you can measure. As long as you can say "this version is better than that one because…", the AI can test on a loop for you.
The pitfall Karpathy himself warns about
The AI will sometimes find "improvements" that only work on your reference list and not in real life. It's a classic pitfall. To avoid it, always keep a second control list the AI has never seen — like a pop quiz. If the final version works just as well on your secret list, you're good. If not, it just learned to pass the test.
Five steps to set up your first try.
If you want to give it a go this weekend, here's how to do it. I'll describe what I did, without assuming you've already installed anything. Budget two to three hours the first time, mostly to prepare the reference list.
Pick something to improve that is NOT critical
For your first try, definitely not a system that sends your invoices or touches money. Pick something where a mistake has no consequence: a prompt you use often, the subject line of your prospecting emails, a small personal script. My advice: start with a project that only you use, not one for a client.
Prepare a reference list by hand
This is the longest step but it's the most important one. You need between 50 and 200 examples with, for each, what the "right" answer would be. My 200 emails labeled "important / not important", for example. Without this list, the AI has no way to measure whether it's improving or not. You can put it together in a plain Google Sheets or Excel spreadsheet.
Install Claude Code (or Codex CLI)
Claude Code is the tool I used. It's an app that lets you chat with Claude directly in a window where you can ask it to edit files and test things. Codex CLI (OpenAI's competitor) does the same. I've got a dedicated tutorial for Claude Code if you're starting from scratch. 30 minutes of setup the first time, then you don't touch it again.
Give the brief in plain English, set a budget
You write out what you want in plain words: here's my current program, here's my reference list, here's the metric ("count how many important emails you miss"), here's how much I'm willing to spend. The AI-token budget is crucial: without a cap, the AI can run for days and cost you a fortune. Put €10–15 for your first try — that's plenty to find out whether it works.
Launch it, go to sleep, check in the morning
Start it in the evening, let it run. In the morning, look at what the AI suggests. Important: never ship straight to production. Read what the AI changed, understand why, test it yourself on 5 or 10 new examples the AI has never seen. If it holds up, you can replace your program. If not, you open up another loop.
The trick that changes everything for non-devs
Don't fight with the code. Ask Claude to explain in plain English what it changed, and why. If you don't understand an explanation, ask it to simplify it even more. If I can keep up, anyone can. The golden rule: until you've understood what the AI changed, don't ship it to production.
Four concrete cases that don't require being a coder.
Email sorting is my case. But the idea transfers anywhere you can measure whether one version is better than another. Here are four examples I've tested or that entrepreneurs around me have tested since.
Finding the best email subject lines
You've got 50 newsletters sent in the past with their open rates. You ask the AI to propose 30 new subject lines for your next email, to score them based on the rules it infers from your best past performances, and to keep the top 3. Typical cost: €3 to €5. Typical gain observed with two contacts: between 8 and 18 % more opens over the following weeks.
For: every newsletter creatorTesting 50 variants of your first message
You've got a list of 100 prospects. You prepare 50 variants of your first message. You send 2 to each prospect (alternating). The AI measures the reply rates, identifies what works best (length, tone, hook), proposes 10 new variants inspired by the winners, and you start over. Very powerful, but respect the anti-spam rules — don't send just anything to just anyone.
For: salespeople, freelancers, foundersImproving the clarity of your writing
You give the AI your 20 best past articles (the ones that were shared or commented on the most). You tell it: "Rewrite my new draft in the style of those 20. Measure readability, sentence length, the hook. Keep the clearest version." It's not exactly the same scheme as Karpathy's, but the idea is the same: test, measure, keep, repeat.
For: writers, freelancers, content creatorsOptimizing your sales pages
You've got 30 product pages on your online store with their conversion rates. The AI infers the rules that set the best ones apart (length, photos, text structure, price placement) and proposes rewrites. You approve, you publish, you measure. It's A/B testing (the practice of comparing two versions of a page to see which works best), but driven by the AI instead of you.
For: e-commerce sellers, SaaS, service sitesThe common thread across these four cases: you can measure the result. Without an objective measure, the Karpathy idea doesn't work. That's the one limit, but it's a real one. If you can't say "this version is measurably better than that one," switch methods.
Four gaps in the net to know about.
I'd rather tell you now than have you discover them while losing €50. Here they are, the four things that can frustrate you if you don't know them from the start.
Pitfall 1 — Launching with no cost cap
An AI running on a loop can burn through a lot of tokens. With no limit, your planned €5 can turn into €80 in one night (it happened to me once on another test). Always set a cap before you launch. For Claude Code, you set it in the settings in two clicks. For other tools, ask the AI itself how to do it — it'll walk you through it.
Pitfall 2 — Not keeping a secret list
The AI can "memorize" your reference list and hand you a version that works great on it but flops in real life. To avoid this: prepare a second list of 30–50 examples you don't give the AI. In the morning, test the final version on it. If it works just as well, you're good. If it falls apart, the AI cheated — start over with a tighter brief.
Pitfall 3 — Letting it run on anything critical
Never on your client CRM. Never on your billing. Never on your live site. Always on a copy. The AI can easily break things while trying to improve them. For your first try, pick a case where the worst thing that can happen is that you laugh. Not that you lose a client.
Pitfall 4 — Thinking more = better
More versions tested doesn't necessarily mean a better result. Beyond 20–30 changes, the marginal gain becomes tiny, but the cost keeps climbing. My experience after a few tries: stop at the first version that satisfies you. You can always relaunch a loop later if you need to. Perfect is the enemy of good — AI included.
And a fifth, sneakier one: not understanding what the AI changed. If you ship a version you haven't reviewed, you're going to get surprises. Always ask Claude to summarize its changes in plain English, and always read that summary before adopting the new version.
The promises to unplug right now.
I see a lot of hype about self-improving agents right now. "AI will develop itself, we won't need devs anymore." Easy there. Here are the real ballpark figures, after two months of testing at home.
| The marketing promise | The reality on my test bench |
|---|---|
| "AI improves itself without you" | The AI improves one specific thing that you defined, on examples that you prepared. Without that human prep, it can't improve anything. |
| "Gains of 200, 300 %" | 10 to 30 % improvement per loop, in my tests. Sometimes more, never 300 %. And it plateaus fast: the second loop gains less than the first. |
| "It replaces devs" | It helps devs iterate faster and lets non-devs like me improve small bits of software. Nobody gets replaced; some people get upgraded. |
| "Soon AI will build its own successors" | Karpathy himself dials it back: we're talking micro-improvements on narrow problems. Not an AI that redesigns itself from A to Z. That may come, but we're not there. |
My honest take: it's powerful, it's real, it's usable today for specific cases. But it's not the revolution they're selling you on LinkedIn. It's a new productivity lever, the way the arrival of Excel was for accountants or search engines were for journalists. Important. Not magic.
My real advice after two months of testing
Don't invest three weekends building your first test bench thinking you'll discover the secret to business. Invest three hours, run a try on a harmless case (a newsletter subject line, for example), see if you enjoy it, and if you do, apply it to a more useful case the following week. Gradual learning beats initial enthusiasm every single time.
Who it's useful for, and who it isn't.
After these two months and a good dozen tries (successful and failed), here's how I sort it out.
It'll help you if you're…
- An entrepreneur or freelancer who already has automated routines (sending emails, sorting leads, generating quotes) and would like to make them 20–30 % more accurate without hiring someone.
- A content creator with enough measurable history (opens, clicks, conversions) for the AI to have material to analyze.
- Curious about AI and already dabbling with Claude Code or a similar tool, wanting to take the next step. If that's you, it's the logical step after my tutorial on loops and my tutorial on agents.
It will NOT help you if you're…
- A total AI beginner who hasn't installed Claude Code or an equivalent yet. Start with my beginner guide first. Without the basics, it'll be pure frustration.
- Someone with nothing to measure. If you can't objectively say "this version is better," it won't work. No big deal, plenty of other AI uses work without a measure.
- In a hurry to see a return on investment. Count on two to five tries before you really save time. The first one is always for learning.
What's going to happen in 2026–2028
Three things, very likely, that will change the landscape:
- The tools will get simpler. Today you need Claude Code and a bit of patience to set up a loop. Within 18 months, it'll be an "optimize this script overnight" button inside tools like Notion, Zapier or Make. No need to understand the terminal.
- The cost will drop by 80 %. AI tokens roughly halve in price every year. My tries that cost €8 today will cost €1 or €2 in two years.
- New uses will appear. Especially in website maintenance, sales-funnel optimization, customer-support improvement. The companies that get on it now will pull ahead.
What's not going to happen, in my view: a fully autonomous AI that rebuilds itself from scratch, with no human in the loop. We're a long way from that, and probably for good reasons that the researchers themselves explain to us.
Three leads to dig into this weekend.
If you want to test now
- Pick the least risky topic possible. Not your CRM. Not your billing. More like a newsletter subject line, a prompt you use often, or a small never-critical script.
- Prepare a reference list of at least 50 examples. Without it, the AI has nothing to measure. It's the most tedious part, and it's the one that makes all the difference.
- Set a cap at €10. Enough to find out whether it works, not enough to ruin you if it goes off the rails.
- Launch it on an evening when you have nothing urgent the next morning. Just in case you have to start over from scratch.
- Note your takeaways. Three lines in a notebook: what worked, what failed, what I'd do differently. That's the real value of your first try.
Three articles to push further
If this article spoke to you, here are the next three to read in this order:
- Claude Code loops, explained — the technical basics to understand how Claude can run on a loop. Essential before you try your own experiment.
- Building a Hermes Agent step by step — how to set up an agent that does specific tasks for you. The step before self-improvement.
- Karpathy's work, made simple — if you want to understand who the guy is and why his ideas matter in the AI world.
- The LLM Wiki, your second brain — the other Karpathy project that complements autoresearch · a local encyclopedia that learns from your feedback.
- My Claude Code takeaways after 6 months — the daily routine that makes this kind of overnight experiment possible without breaking everything in the morning.
And if you've never touched Claude or ChatGPT, don't skip steps. Go read my beginner guide first. Everything else will be easier after that.
If you want me to send you more stories like this one (what I test, what works, what fails), sign up for AI Playbook — it's my weekly watch, I share the same thing with you that I keep for myself. And if you launch your first try and you're struggling, or you think I'm talking nonsense somewhere, write to me. I read everything, I don't take it badly, and I often learn more from your feedback than from my own tests.
Autoresearch FAQ.
What is autoresearch according to Karpathy?
The core idea behind Andrej Karpathy's autoresearch project: you put an AI to work on a problem, you ask it to edit its own work, it tests whether the new version is better, keeps it if so, throws it out if not, and loops. It's the simplest idea in the world dressed up in a fancy name.
How do you let an AI improve its own code overnight?
You prepare a reference list (200 examples scored by hand), you hand the current program to Claude Code with a plain-English brief, you set a cap on AI-token cost, and you launch it in the evening. In the morning, the AI shows you the best version it found and the list of everything it tried.
How much does it cost to test 14 versions of an agent overnight?
My overnight experiment on email sorting cost me €8.40 in AI tokens for 9 hours of work and 14 versions tested, against a cap set at €15. Budget €10 to €15 for a first try — that's plenty to find out whether it works.
What are some non-dev use cases for autoresearch?
Four concrete cases tested: finding the best newsletter subject lines (8 to 18 % more opens), testing 50 cold-email variants, improving the clarity of blog posts by learning from your best past ones, and optimizing e-commerce product pages. The common thread: you have to be able to objectively measure whether a version is better.
What pitfalls should you avoid before launching a self-improvement loop?
Four main pitfalls: launching without a cost cap (a planned €5 can turn into €80), not keeping a secret control list (the AI memorizes your reference list), letting it run on anything critical (never your CRM or your billing), and believing that more versions tested necessarily means a better result (the gains plateau fast after 20–30 changes).
What tools let you actually do autoresearch?
Claude Code (the app that lets you chat with Claude in a terminal and have it edit files) or OpenAI's Codex CLI are the two main tools today. Budget 30 minutes of setup the first time. Within 18 months, this will probably be a built-in button inside Notion, Zapier or Make.
Do you need to be a developer to do AI autoresearch?
No. I'm not a dev and I never studied computer science. The whole experiment was done by asking Claude to explain each step like I was a beginner and having it write the code. The golden rule: until you've understood what the AI changed, don't ship it to production.
What realistic results can you expect from a self-improvement loop?
10 to 30 % improvement per loop in my tests, with a quick plateau (the second loop gains less than the first). On the email sorting, I got 23 % fewer important emails missed. The marketing promises of 200 or 300 % are noise: Karpathy himself dials it back and talks about micro-improvements on narrow problems.
What's the difference between autoresearch and classic fine-tuning?
Fine-tuning changes the weights of the AI model itself with a dataset; it takes infrastructure and data. Autoresearch changes the code, the prompt or the configuration around the model, without touching the model. It's more accessible, cheaper, and applicable to anything you can measure.
How long does it take to get a first usable result?
Budget two to three hours the first time, mostly to prepare the reference list (the longest but most important step). The loop itself runs overnight while you sleep. In the morning, you've got your best version. Then count on two to five tries before you really save time: the first one is always for learning.
Shall we keep going?
I test AI for real and share what works, with no jargon and no hype. If this article helped you, the easiest way to never miss a thing is my Friday letter. And if you have a question or a doubt: reply to me, I read everything.