Can a coding agent give a brand-new language a decent editor?

At the end of my last post, I listed what Slopersons doesn’t have compared to a real language: a package manager, a debugger, decent performance, and editor support. Writing Slopersons in VS Code meant plain white text, and finding out you misspelled a name only when the program crashed on it.

So I wanted to know: how far can a coding agent take editor support for a language with no training data and no grammar on the Marketplace to copy from? I wanted a good DX, with highlighting, errors as you type, go to definition, completion, hover with docs, rename across files. The result was good, but the interesting parts, again, were the decisions, and the moments the agent was confidently wrong.

If you haven’t read the previous posts (how I built the language and how agents learned it from scratch): Slopersons is a small experimental language I built with a coding agent, where data flows from left to right and blocks are pipes in the margin:

~ Load a module from another file, then flow values through procedures
"./lib/geometry" -> use => geometry

2.0 -> geometry/circle-area -> io/print-line
[1 2 3] -> array/each n
| n -> io/print-line

The brief

This time, there was no big prompt, I just started with:

I want to plan developing a VS Code extension for the language. Let’s start with syntax highlighting

After each step, the agent asked me what to do next as a multiple-choice question, with a recommendation and the trade-offs of each option, the same way it did while designing the language. I picked one, it planned, built, and tested it, and then I tried it in VS Code before saying "commit. Next!". The whole thing took about seven hours, from the first message to the last commit, and that includes the time I spent testing each step myself.

Take 1: a grammar and its limitations

The first step was a TextMate grammar, the regex-based format VS Code uses for syntax highlighting, plus some editor behavior: ~ toggles comments, brackets auto-close, and pressing Enter on a | | x line starts the next one with the same bars. That one is small, but in a language where blocks are made of margin bars, it’s the difference between nice and annoying.

What I liked the most was how it tested the extension. Since it lives inside the interpreter’s repository, the tests can compare the grammar against the real thing:

• the grammar’s keyword list must match the lexer’s, and its builtin module list must match what the interpreter registers, so adding a keyword or a module without highlighting it fails pnpm test;
• every token the real lexer reads in all 36 .slop files of the repository must get a matching color in the grammar.

Then it told me what a grammar can’t do:

[1 2 3] -> array/each n | n -> io/print-line

[1 2 3] -> array/each n
| n -> io/print-line

(x > 1 ? "many" : "one") => label
{"name": "Ada"  "born": 1815} => person

The n on the first line is colored as a parameter, but the one on the third line isn’t. A TextMate grammar works one line at a time, and what makes that n a parameter is the block on the next line. From that line alone, it could just as well be an argument. And both : look the same, even though one is the "else" of a condition and the other separates a key from its value in a hashmap, and telling them apart requires a parser.

Take 2: let’s build a language server

To tackle this, the agent recommended building a language server as a slopersons --lsp flag on the executable you already have to install to run programs anyway. That way the editor uses the very same parser as the interpreter, and they can’t drift apart.

The agent is overruled once again

Before building the server, it asked three questions. I followed its recommendation on two of them: start with parse errors only, and recover from errors per top-level statement, so one broken procedure doesn’t hide the errors in the others.

On the third one, I went against it. The agent recommended writing the protocol by hand, about 200 lines, so the interpreter would keep its zero runtime dependencies and it could write it very quickly. I chose vscode-languageserver because I didn’t want to recreate the protocol VS Code expects. VS Code’s own client is tested against that library, and every new feature would add more protocol to get right by hand.

That choice had consequences:

• The library calls process.exit itself when the editor closes, and the project has a rule against that. Instead of hiding it, the agent documented it as the one exception — in AGENTS.md and in the ADR
• The standalone executable crashed on --lsp with exit code 70. The library loads Node modules in a way the bundle didn’t support, and the existing bundle tests only ran programs, so they never noticed. The agent found it because it ran the same protocol session against the checkout, the bundle, and the executable

One step at a time

After that, each step was one commit, one ADR, and one round of me trying it out:

1. Semantic tokens from the parser, fixing the two colors the grammar got wrong
2. A name resolver, for "go to definition", "find references" and highlighting the other uses of a name
3. Warnings for unknown names, as you type
4. Completion
5. Following use into module files, so F12 on geometry/circle-area opens the module at proc circle-area
6. Hover and signature help, with builtin docs
7. An outline and rename

I’ll give more detail about some of them:

The resolver followed the wrong rule

To jump to a definition, the server has to know what each name refers to without running the program. The agent’s first resolver let every name see every declaration in its scope, in any order. Then it broke the code on purpose to check its tests (the mutation testing habit mentioned in the last post), and found that no test noticed the difference.

Digging into it, it found the rule itself was wrong: inside a procedure, a read that comes before a local => x would point at that local, while the program actually reads the outer x at that point. So it switched to resolving names in statement order, the way the program runs them, and added tests for both behaviors.

Statement order is still an approximation, though: a loop can read a name its own body binds later, which works from the second time around. That’s why unknown names became warnings instead of errors: the editor can be wrong, and an error in valid code is worse than a missed warning. They also use the exact same messages the interpreter prints when the program runs.

What?!

During the warning step, the agent’s summary said every program in the repository gets zero warnings, "except totl in the golden program that expects it", and I had to ask what on earth that meant.

I’m sure you’ve been through this kinda confusing wording before as well, the agent was trying to say something that made no sense and the context was super confusing. After some frustrating seconds, it explained that using a variable called "totl" inside a file where it’s never created would show the error. Was I even expected to understand that?

Completing a line that doesn’t parse

Completion has the same problem as everything else in an editor: the line you’re typing usually doesn’t parse, and the parser throws away the whole statement, often the whole procedure around your cursor. The agent’s trick was to replace the word being typed with a placeholder name and ask the resolver what’s visible there. If that still doesn’t parse, it tries again without the rest of the line, then with only the margin bars, then at the end of the file. With that, when typing io/, it lists the module’s members straight from the interpreter, so a new builtin shows up in the editor without touching the editor code.

Renaming symbols

In the first version of the renaming feature, it refused to rename exported names because other files use them, but I wanted to rename exports as well! The implementation now scans the .slop files in the workspace and renames the usage in every file whose use points at that module.

The moments I had to step in

At the end, I asked the agent to check if all the docs were up to date. Every single step had "updated the docs", and still it found a lot of problems in multiple docs. docs/cli.md said that the members of qualified names have no definition, a few lines after saying F12 on geometry/area opens the module. AGENTS.md called the server "the only runtime dependency", when there were two, and didn’t list the eight editor modules added after it. The extension’s README listed only the first five features.

Each step updated the paragraph about its own feature, and nobody reread the whole thing, so I added a rule to AGENTS.md: a change isn’t done until every doc that describes what it touched says what the code now does, in the same commit, with the list of docs to check.

And the ADRs did their job again: at one point, the agent edited an older ADR to say a limitation was gone, then reverted it on its own, because AGENTS.md says existing ADRs only change their status.

So, can a coding agent give a brand-new language a decent editor?

I’d say yes! In some hours, Slopersons went from plain text to highlighting, errors and warnings as you type, go to definition, find references, completion, hover with docs, signature help, an outline, and rename across files. That’s 11 commits, around 1,400 new lines in the interpreter source, 1,500 lines of tests, 9 ADRs, and a test suite that ended the day at 1022 passing tests.

What can we learn from that?

1. Check what you’re actually choosing between. Before building the language server, I had to think about a choice: "semantic tokens or a language server?". Turns out that wasn’t a real choice, which I only understood after the agent explained why. The best answers I got were the ones that questioned the question
2. Test against the source of truth. The grammar is checked against the lexer, the server is the interpreter’s own parser, warnings use the interpreter’s messages, and hover reads the real docs. Every one of these turned drift into a failing test
3. Small steps, each one tried for real. One feature, one commit, one round of me using it. The agent can test the server, but someone has to open the editor
4. Read the docs as a whole once in a while. Docs updated one paragraph at a time are each correct and, together, contradict each other. Put the rule where the agent will read it

None of this is new, right? Tests against the real thing, small steps and docs that match the code are what we’ve always wanted for humans, and the agent did better wherever the project made them checkable.

But notice, it’s still a small and limited test: I tested the extension by hand in VS Code, on my machine, with small programs. The extension itself has no automated test inside VS Code, it’s not on the Marketplace, and even though the server speaks LSP, I haven’t tried it in any other editor. The resolver works by statement order, which is an approximation of how the program runs, so some warnings will be wrong.

If you want to try it, the extension lives in the language’s repository, in editors/vscode available on GitHub.

Wrapping up

I started questioning how a coding agent would do with an unknown language, and as the agent let me dare higher and higher, I could go further than I could have imagined at the beginning.

Thank you for joining me in this whole adventure, and see you some other time!

We want to work with you. Check out our Services page!