AI Coding Assistants: What They Do Well and Badly

AI coding assistants have settled into a strange middle ground. They are neither the replacement for programmers that some marketing implied, nor the useless toy that early sceptics predicted. They are power tools: fast, occasionally dangerous, and heavily dependent on the skill of the person holding them. Understanding the shape of their competence, rather than their average quality, is what separates people who get real leverage from people who spend their afternoons untangling plausible-looking nonsense.

This piece describes that shape in general terms, because the specifics change constantly. The categories, though, have been remarkably stable.

Where AI assistants genuinely earn their keep

The common thread in the successes is work that is well specified, locally scoped, and verifiable. When you can state what you want in a sentence or two, when the answer touches a small area of the codebase, and when you can immediately tell whether the result is right, assistants perform well.

Boilerplate and mechanical transformation

Converting a data structure from one format to another, writing the fortieth API client method that looks like the previous thirty-nine, generating test fixtures, adding logging to a set of functions, converting a class to a different pattern. This work is tedious, low-judgement, and easy to check. An assistant will often do it in seconds, and errors surface immediately.

Getting unstuck in an unfamiliar area

Assistants are excellent at answering "roughly how do people do this?" in a language, framework or domain you do not know well. The answer may be imperfect, but it gives you vocabulary and a starting point — as long as you then confirm the details against the real documentation.

Explaining code you did not write

Paste an unfamiliar function and ask what it does. This is one of the safest uses, because you can validate the explanation against the code in front of you. It is particularly valuable for regular expressions, dense SQL, build configuration, and old code with no comments.

First-draft tests and edge-case brainstorming

Asking "what could go wrong with this function?" reliably produces a list containing several things you had not considered, alongside several that do not apply. Even a 50 percent hit rate is useful when the cost of reading the list is thirty seconds.

Reducing friction on small chores

• Writing a shell command or script you would otherwise look up
• Drafting commit messages and pull-request descriptions from a diff
• Renaming and restructuring within a single file
• Producing a quick throwaway script to inspect some data
• Turning rough notes into readable documentation

Where they fail, and why the failures are sneaky

The failure modes matter more than the successes, because AI mistakes do not look like human mistakes. Human errors tend to be visibly incomplete or obviously confused. AI errors are fluent, well formatted, confidently commented, and wrong. That fluency defeats the pattern-matching most reviewers rely on.

Invented interfaces

An assistant may call a function, flag or configuration key that does not exist, because something very like it exists in a neighbouring library or an older version. The code reads perfectly. It simply does not run. This is the single most common failure and the easiest to catch, since the program breaks immediately.

Code that runs but is subtly wrong

Far more dangerous. Off-by-one boundaries, incorrect handling of empty inputs, time zones treated as if they were not a problem, floating-point money, silently swallowed errors, race conditions in concurrent code. The tests pass because the assistant wrote tests that match its own misunderstanding.

Missing the context it was never given

An assistant sees the code you show it, not your architecture, your deployment constraints, your team conventions, or the reason a previous engineer wrote something in an odd way. It will happily "simplify" a workaround that exists for a real reason, or introduce a dependency your organisation does not permit.

Security and correctness under adversarial conditions

Generated code tends toward the common case. Input validation is often thin, permission checks are sometimes assumed rather than written, and secrets management is frequently naive. This reflects the average of the code such systems learned from, and the average of public code is not secure code.

Large, cross-cutting changes

Ask for a refactor that spans many files and interacting concerns and the quality drops sharply. The assistant loses track of what it changed, produces inconsistent naming, and quietly drops functionality. Big changes need to be decomposed by a human into small ones.

The confidence problem

Assistants rarely say "I do not know." They produce an answer of consistent tone regardless of whether they are on firm ground, so there is no signal telling a solid answer from a guess. External verification is not optional.

Review discipline that actually works

The productivity gain from an assistant is real only if it survives review. Treat generated code as a submission from a fast, well-read contributor with no knowledge of your system and no accountability for the result.

Never accept code you cannot explain

This is the core rule and it subsumes most of the others. If you cannot walk a colleague through why each line is there, you are not reviewing the code; you are hoping. Code you do not understand becomes code nobody can maintain, and the moment it breaks in production you have no mental model to debug it with.

Keep the units small

Request changes in pieces you can review in a few minutes. A twenty-line diff gets read properly. A four-hundred-line diff gets skimmed, and skimming is where fluent errors slip through.

Verify the interface, not just the logic

Check that every function, method, flag and configuration key actually exists, in the version you are actually using. Documentation and type checkers do this faster than you can.

Write or check the tests yourself

If the same system writes the code and the tests, both encode the same misunderstanding. At minimum, decide the test cases yourself and let the assistant fill in the mechanics.

Apply extra scrutiny in specific zones

• Anything touching money, dates, or units — classic sources of silent error
• Authentication, authorisation and input handling — where the cost of a mistake is highest
• Concurrency and shared state — where bugs hide from tests
• Deletion, migration and anything irreversible — where you cannot undo the lesson
• Error handling paths — usually the least-tested code and the least-attended-to by generation

Run it before you believe it

Static reading is not enough. Execute the code against real inputs, including empty, malformed and boundary cases.

How to evaluate a tool for yourself

Benchmarks and demonstrations are poor predictors of whether a tool suits your work. A better approach is a small structured trial. Pick three or four tasks representative of your actual week — one boilerplate chore, one unfamiliar-domain question, one bug in your own code, one small feature — and run them through any tool you are considering. Then ask:

• How much of the output needed correcting, and how long did correcting it take?
• Did it understand your codebase's conventions, or fight them?
• How gracefully did it handle being told it was wrong?
• Where does your code go, and does that satisfy your organisation's policy?
• Does it fit your existing workflow, or demand you rebuild your workflow around it?

The last two questions decide more real-world outcomes than raw model quality does.

A practical takeaway

Use AI assistants for the parts of programming that are typing, and keep the parts that are thinking. Delegate the mechanical and the well specified; retain architecture, judgement calls, security-critical logic, and anything where being wrong is expensive. Ask for changes small enough to review properly, verify that every referenced interface exists, run the code against awkward inputs, and refuse to commit anything you could not explain from memory. Done this way, an assistant removes a layer of tedium from your day. Done carelessly, it converts an hour of typing into an afternoon of debugging.