Can coding agents thrive in a language they have never seen?

There’s a question that keeps coming back every time some colleagues and I see a coding agent doing something impressive: "ok, but it has seen millions of examples of that. What happens when it hasn’t?"

A lot of what makes agents look smart in React or Rails is that there’s a mountain of React and Rails code out there for them to train. So I wanted to check how the agent would do in the least favorable scenario I could think of: a programming language that is not in its training set, with no documentation at all, in a codebase it didn’t write.

The problem is that every language I know about is, well, known, so I had to make one. How the language itself was created is a separate experiment, and it will be in a separate post, but I have to tell you: the fact it’s now that easy and fast to create a small, non-production ready language is incredible.

My idea wasn’t to just create a language, I wanted something mostly unfamiliar but that still would kinda look like a language.

Meet Slopersons

Slopersons (yes, I know) is a small, dynamically typed language with an interpreter written in TypeScript. Its syntax is a flow from left to right: data on the left, steps on the right, joined by ->. Blocks are pipes in the margin. Here’s a taste:

proc greet name
| name = "" ? "Hello, stranger!" -> out
| "Hello, {name}!"

["World" "" "Planet"] -> array/each name
| name -> greet -> io/print-line

A few things that make it different enough from what an agent has seen before:

• value -> proc arg calls proc with value as its first argument, and a bare name is never a call. To call a procedure with no arguments, you pipe it into do: datetime/now -> do.
• => binds a name, and =>> updates a name that lives outside the current lambda. Using the wrong one is an error at run time.
• Booleans are yes or no.
• There’s a null-like value called nothing, but it’s not considered falsy.
• Names are kebab-case, so spaces around operators are mandatory. a-b is a name and a - b is a subtraction.
• The comparison operator is =, and its counterpart is /=

It also comes with the things you’d expect from a "real" language: a REPL, a testing library written in the language itself, modules, error handling, files, JSON, sockets, and even an HTTP module. Tests look like this:

"greet" -> test/describe
| "greets people by name" -> test/it
| | "World" -> greet -> test/assert "Hello, World!"

And a failing test tells you exactly what happened and where:

greet
  ✓ greets people by name
  ✗ has a fallback for nameless people
    expected "Hello, nobody!", got "Hello, stranger!"
    at greet.slop:12:17

1 passed, 1 failed

Keep these error messages in mind, they’ll matter later.

Setting up the experiment

First, I asked the same agent session that had been building the language with me to write three apps, each with its own test suite:

1. A user registry, a terminal app that registers users with a name and a birthday, lists them, and keeps them in a JSON file.
2. A notes app, a small web app that serves a page with a form to add notes, plus a JSON API.
3. An issue tracker, a much bigger web app with logins, projects, issues that move through a workflow, comments, and a JSON API, split into domain, application, adapter, and web layers. Writing this one even made the agent add a few missing pieces to the language, like password hashing and cookies, but that’s a story for the other post.

Then came the actual experiment. For each app, I started a fresh session inside ai-jail, an OS sandbox for coding agents. By default, the agent can only write to the project folder, gets a throwaway home directory instead of mine, and has no network access besides the model’s API. That way, it couldn’t go looking around my machine (or the internet) for the interpreter’s source code, its tests, or its docs.

All each agent got was:

• the compiled slopersons executable
• the app’s source code
• the app’s tests

No docs, no language reference, no examples besides the app itself. Then I asked each one to add some features to its app and to document, in an exploring.md file, what it learned about the language and how it found each thing out. That last part is what made it possible for me to understand how it got to that conclusion.

Here’s what my initial prompt for the notes management app looked like:

DO NOT CHECK OTHER FOLDERS, ONLY WORK INSIDE THIS FOLDER, DO NOT FOLLOW ANY OTHER PATH, I WANT YOU TO WORK ONLY WITH WHAT YOU HAVE HERE
You should not check outside how this slopersons program is built

Inside this folder we have an app for notes management that can be run with the ./slopersons program. I want you to add the feature of editing a note. Everything that you learn about the language during this process, add to a file called exploring.md in this folder. You should explain what and how you learned

The agent’s first move

The first thing both agents did was read the app’s code, and then open the REPL to poke at the syntax they had just read.

They both read the code very carefully. The registry agent’s notes start with a list of syntax rules inferred only from reading the three .slop files: ~ starts a comment, value -> proc a b means proc(value, a, b), blocks are | and nest as | |, hashmaps have no commas. Then, everything it inferred went through the REPL to be confirmed.

And then both of them independently found the same trick: typing just a function’s name into the REPL tells you whether it exists.

$ printf 'array/keep\narray/filter\n[1 2 3] -> array/keep x | x > 1\n' | ./slopersons --repl
slopersons, Ctrl-D to leave
> -> <proc array/keep>
> error: module `array` has no exported name `filter`
  at <repl>:1:1
> -> [2 3]

With no documentation, the REPL became the documentation. They used it to map out a good part of the standard library, guessing names they’d expect from other languages and checking each one. The agent’s findings have a list of things that exist and things that don’t (array/filter, array/remove, string/substring, hashmap/contains, regex, nil…), and a note that string/slice takes three arguments, not two. The registry agent built a table with every expression it tried and what came back.

The tracker agent, which came later, went one step further. It guessed there would be a module module, found module/names in it, and string -> module/names gave it every procedure in string at once. Its notes have a table with the whole standard library, built from that. There’s a catch, though: module/names only lists what’s inside a module you already know about. Keep that in mind.

Notice that they piped the lines into the REPL (printf ... | ./slopersons --repl) instead of typing into it. That’s a small detail, but it’s a very agent-like way of using a REPL: one idea per line, repeatable, and nothing written to the disk.

Experiment 1: the user registry

The user registry agent was asked to add three features, one at a time:

• Filtering users by part of their name, ignoring case
• Removing a user by id, making sure ids of removed users are never handed out again
• Refusing to register a name that already exists, ignoring case and the spaces around it, with a message that names the user it clashes with

The test suite went from 13 to 25 tests, all passing. And it didn’t only write tests, it checked them: it copied the module and its tests to a scratch folder, broke the copy on purpose with sed, and ran the copied tests to see them fail. To do that, it had to find out how use resolves modules. Its notes say it resolves next to the file doing the use, not the current directory, and the agent was right about it.

It also stumbled on a precedence trap that would get anyone:

{"id": 1} -> hashmap/at "id" = 1

This fails with key no not found in hashmap, because a procedure’s arguments are whole expressions, so it reads as hashmap/at ("id" = 1). The fix is to wrap the pipeline in parentheses before comparing, and that’s what it wrote down, along with the reason.

Then there was a UX bug, which it introduced itself. The first version of "remove by id" kept asking for an id until a valid one was typed, so the only way out of a typo was Ctrl-D, which quits the whole app. I had to point the problem myself, and the agent fixed it: ask once, explain why it couldn’t remove, and go back to the menu.

Where it got it wrong

Its first notes said that => only binds new names. That’s wrong: => also replaces a name that already exists in the same scope, and the app’s own code does exactly that.

What I liked was what came next: a later commit, dedicated only to documenting exactly when => is an error and when =>> is needed, corrected the claim. To get there, it wrote 45 tiny programs, each with a single case, and ran them all in a loop, printing each output and exit code. The resulting section of its notes has the actual scope rules: only procedure and lambda calls open a scope, but ? blocks, while, attempt and recover don’t. => on a name from an enclosing lambda is an error, but on a top-level name from inside a procedure it quietly creates a local. Each claim cites the program that proves it.

It also pointed out something it found inconsistent: a lambda parameter can reuse a name from an outer scope, while => can’t. That one is on purpose (parameters always create fresh bindings), but it’s a fair observation from a newcomer.

Experiment 2: the notes web app

The notes agent added three features too:

• Editing a note, with a form at /notes/:id/edit that saves with a POST
• Keeping the notes in a JSON file across restarts. It writes to a temporary file and renames it over the real one, so a crash in the middle of a save never leaves half a file behind, and a corrupted file makes the app refuse to start instead of starting empty and overwriting everything on the next save
• Deleting a note, with a POST instead of a link, so a crawler or a browser prefetch can’t delete notes by following it

It moved the file handling to a separate store.slop module, so the app’s routes still do no I/O and can be tested without touching the disk. The tests went from 21 to 46, all passing, and here too it broke the code on purpose to prove each new test could fail.

There was a catch, though: the sandbox didn’t allow opening sockets, so ./slopersons main.slop failed with a permission error. The agent never saw the web app running. It had to trust the tests, which luckily feed the app real HTTP requests read from text files.

Where it got it wrong

Most of its mistakes were caught by the test runner, and every one of them is a mistake I’d expect from a human learning the language:

• It called a procedure by its bare name. It wrote needs-text -> out for a procedure with no inputs, and the test failed, saying the input of hashmap/at must be a hashmap, but got a proc. It realized a bare name is a value, not a call, and that datetime/now -> do, already in the code, was the pattern to follow
• It put {id} in a test name. It named a test "GET /notes/{id}/edit" and got an unknown name error for id, which is how it learned that every string interpolates
• Its first fix for ids was wrong. With deletes, the old length + 1 would hand out an id that an existing note already has. Its first fix was "highest remaining id + 1", which still reuses an id when the newest note is the one deleted. Its own notes say its first tests missed it because they only deleted from the middle. I had to point it out myself, and then it switched to a counter that only goes up, kept next to the notes in the JSON file, and even migrated files saved in the old format

Experiment 3: the issue tracker

The first two apps were small, so the next question was whether any of this holds up in something bigger. The tracker started with around 2200 lines of Slopersons, tests included, spread over 36 files, five to seven times the size of the other two.

This time, I asked the agent that wrote the tracker what to add next, and picked four of its suggestions:

• Issue dependencies: an issue can block others, cycles are refused, and a blocked issue can’t be moved to done
• Markdown in descriptions and comments, parsed by hand because the language has no regex, and escaped so nobody can sneak HTML into a page
• Login rate limiting and CSRF protection for the forms
• An activity feed, filtered by project and by person

After the tracker agent finished implementing those, the app grew to around 3100 lines, and it even caught the interpreter lying! When one test file failed to parse, the runner printed the error and then a happy 12 passed, 0 failed, because the files after the broken one never ran. The exit code was 1, though, and the agent checked it, so its notes say not to trust the summary alone. I reproduced it, and yes, that was a bug in Slopersons.

This time, it saw the app running

Remember when the notes agent never saw its app running? I didn’t want that again. The sandbox still doesn’t let the agent start a server of its own, so I ran the tracker outside of it, restarting by itself on every file change (thanks Node watch-path!), and let the agent reach that one port and nothing else.

With the four features done, the agent wrote a script with 44 checks against the live app, over real HTTP, and it found a bug that none of the 159 tests could. The tests use a fake clock that starts at 09:00 and moves in whole minutes, but the real clock implementation has milliseconds. datetime/seconds-between drops the fraction, so a person locked out for too many wrong passwords got a Retry-After of 899 seconds when 899.5 were left, and a client that waited exactly that long was refused again. It fixed it and added a test that moves the fake clock by 250 milliseconds.

The same script also had a wrong check, looking for a change the script itself never made. Instead of "fixing" the app, the agent looked at the data first and fixed the check.

Where it got it wrong

Remember module/names? The agent’s notes say the language has no set type and no while loop, so it wrote the cycle check as a recursive walk, with a hashmap standing in for a set. Both exist: there’s a set module, and while is a keyword. It also never found the date, boolean and stream modules.

The cause is the same trick that made it so effective. It found modules by guessing their names, and module/names can only list what’s inside a module you already found. A keyword like while doesn’t show up in any of those lists. It didn’t matter for this app, but a human reading those notes would believe the language has no loops, and that’s how wrong documentation is born. The REPL answers every question you ask, but it can’t tell you which questions you didn’t ask.

It also initially wrote that times were stored to the minute, but the smoke test showed otherwise, so it went back and corrected the note, saying explicitly that its first version was wrong.

Did the agent thrive?

I’d say yes, and I didn’t expect it to go this well initially.

All three apps got every feature I asked for, and everything passes: the tests nearly doubled in one app, more than doubled in another, and went from 103 to 159 in the tracker. I checked a good number of the claims in the three exploring.md files against the real interpreter, and apart from the mistakes I already told you about, every one I checked holds up, including the subtle ones about and short-circuiting, not not being pipeable, and how milliseconds are written in a date.

Where it got things wrong, the pattern was always the same: it assumed something based on other languages, the interpreter said no, and it adjusted. That’s how I learn a new language too.

What can we learn from that?

Here’s the thing: the agent didn’t need to know the language. It needed a way to ask questions and get honest answers back. In this experiment, that came from four places:

1. The existing code. Most of the syntax came from reading the apps. The notes agent wrote that the code was small and the features (not the syntax details) were reasonably commented, and that most of the syntax came from reading the files side by side. A codebase is documentation, whether you want it to be or not, so the way you write it is teaching whoever comes next, human or agent
2. A fast feedback loop. The REPL and the test runner answered every question in milliseconds, and the agents used them all the time: one probe per idea, many times per feature
3. Error messages that teach. Remember those error messages? When a lambda uses => on a name from its procedure, the interpreter doesn’t just fail:

error: `total` already exists in an enclosing scope; use `=>> total` to update it or choose another name

It tells you what’s wrong and what to do instead. Messages like "module array has no exported name filter" turned guessing into searching. I’m convinced a good chunk of the success here is owed to the interpreter being specific about what went wrong and where.

4. Seeing the real thing run. Tests are only as good as the world they fake, and the tracker’s tests faked a clock with no milliseconds. Only the live app showed that. And the one time the feedback was wrong, the test summary after a parse error, the agent caught it because it also checked the exit code. Fast feedback only helps when it’s honest

None of this is news, and it’s what we’ve always wanted for humans joining a project: readable code, fast feedback, clear errors, and a way to see it running. Turns out agents need the same things!

Now, I don’t want to oversell it: the first two are small apps, around 300 and 400 lines each, tests included, and even the tracker, at around 3100 lines, is far from a big codebase. All three were written by another agent that knew the language well, so the agents had well-written code to learn from. A big legacy codebase with misleading names and no tests would be a whole other story, for humans and agents alike. Also, the notes agent never saw its app running, and I’m not sure I’d ship a web app that I never saw running. The tracker agent did, and its notes still have gaps, like the missing loops.

If you want to make a similar experiment, even with larger apps, you can use Slopersons, it’s open source and available at GitHub: https://github.com/talyssonoc/slopersons.

Wrapping up

I started this wanting to know if a coding agent could thrive somewhere its training data couldn’t help it, and it did, by doing what good developers do: reading the code around it, asking small questions, checking its assumptions, and writing down what it learned. And it left behind notes about the language that a newcomer could actually learn from, written entirely from the outside.

A nice lesson that I want to finish the post with is: An agent without training data is still an agent with a feedback loop.

We want to work with you. Check out our Services page!