AI generated code using TDD

The Plan

I have not been massively impressed with the code generated by AI tools in my experiments. As mentioned in my Digital Playground post I have been experimenting with using AI tools like Copilot and ChatGPT, in my previous post I mentioned trying to get copilot to generate unit tests for a repo and being very disappointed with the results. The tests were mostly testing language features over the code that I had written, although one file was tested to a degree.

Given how popular AI generated code is using systems like Claude Code and Copilot I knew that there was more to it than my little experiments so I decided to try another experiment from the other end, how well would an AI coding agent implement so code that I had written tests in a TDD style for?

Test App

As a small application not needing to be production level I decided to do this in my playground (see previous post). In keeping with the theme I made an express server that would call out to the PokéAPI and store the results in a MongoDB instance. I wrote simple services for the database and PokéAPI calls and then implemented the routes leaving the logic of detecting whether the value passed in was an id or a name, the error handling, and the aggregation of the API calls into the correct data shape for the response to be generated by an AI.

My Version

I was not trying to write perfect code for this, just something simple that would work. It is definitely not perfect and in a production environment I would probably have done some things differently. I wanted to keep the code simple and not over-engineer anything.

Code

import { Ability } from "pokenode-ts";
import { pokemonService } from "./pokemon-service";
import { storage } from "./storage";
import { PokeboxEntry } from ".";

export const getPokemon = async (
  idOrString: number | string,
): Promise<PokeboxEntry> => {
  let pokeboxEntry: PokeboxEntry | null;
  if (!isNaN(Number(idOrString))) {
    pokeboxEntry = await storage.getPokemonById(Number(idOrString));
  } else {
    pokeboxEntry = await storage.getPokemonByName(String(idOrString));
  }

  if (!pokeboxEntry) {
    try {
      const pokemon = await pokemonService.getPokemon(idOrString);
      const abilities: Ability[] = await pokemonService.getAbilities(pokemon);
      if (!abilities) throw new Error("Unable to retrieve abilities");
      const species = await pokemonService.getSpecies(pokemon);
      const species_description = species.flavor_text_entries.find(
        (entry) => entry.language.name === "en",
      )?.flavor_text;
      if (!species_description)
        throw new Error("Unable to retrieve species description");

      const newPokeboxEntry: PokeboxEntry = {
        id: pokemon.id,
        name: pokemon.name,
        species_description,
        types: pokemon.types.map(({ type }) => type.name),
        sprites: pokemon.sprites,
        abilities: abilities.map(
          ({ name, flavor_text_entries, effect_entries }) => ({
            name,
            flavour_text:
              flavor_text_entries.find(({ language }) => language.name === "en")
                ?.flavor_text || "",
            effect:
              effect_entries.find(({ language }) => language.name === "en")
                ?.effect || "",
          }),
        ),
      };

      await storage.storePokemon(newPokeboxEntry);

      return newPokeboxEntry;
    } catch (error) {
      console.log(`Could not find Pokemon with id: ${idOrString}: ${error}`);
      throw new Error(
        `Could not find Pokemon with id: ${idOrString}: ${error}`,
      );
    }
  } else {
    return pokeboxEntry;
  }
};

export const getPokedex = async (): Promise<{ id: number; name: string }[]> => {
  return await storage.getPokedex();
};

I did not want to spend much time writing this and it took me a few minutes to write. I did take some elements from another server to speed things up as I would when normally coding. As you can see it is functional but there are many improvements I would make normally like better error handling.

AI

The plan was to use different coding agents to generate different results and compare them, however not wanting to spend money on tokens I was limited to the free Copilot tier, I tried to get Claude Code working with a local LLM but was limited by my system (AMD 5900X, 32GB DDR4, Powercolor Radeon RX7900XT Hellhound GPU) so I was only able to get small models working and even then I was running out of memory before they could complete a task.

Copilot

I started with Github Copilot as it is built in to VSCode and has a free amount of tokens usable each month. The integration into VSCode is unsurprisingly excellent and I was able to quickly delete my implementation and with a single prompt have it create the correct file and populate it.

Prompt

Prompt writing is not my specialty as I have not used AI tools extensively but this was my first attempt and it seemed to be fine.

using tdd and the tests inside pokebox.test.ts please implement the pokebox.ts file so that it passes the all of the tests

Generated Code

import type { PokemonSprites } from "pokenode-ts";
import type { PokeboxEntry } from ".";
import { pokemonService } from "./pokemon-service";
import { storage } from "./storage";

const isNumeric = (value: number | string): boolean => {
  return (
    typeof value === "number" ||
    (!Number.isNaN(Number(value)) && String(value).trim() !== "")
  );
};

const getStoredPokemon = async (
  idOrName: number | string,
): Promise<PokeboxEntry | null> => {
  if (isNumeric(idOrName)) {
    return await storage.getPokemonById(Number(idOrName));
  }

  return await storage.getPokemonByName(String(idOrName));
};

const findEnglishText = <T extends { language?: { name?: string } }>(
  entries: Array<T> | undefined,
  key: keyof Omit<T, "language">,
): string | null => {
  if (!Array.isArray(entries)) {
    return null;
  }

  const entry = entries.find(
    (item) => item?.language?.name === "en" && typeof item[key] === "string",
  );
  return entry ? String(entry[key]) : null;
};

const normalizeSprites = (sprites: any): PokemonSprites => ({
  front_default: sprites?.front_default ?? null,
  front_shiny: sprites?.front_shiny ?? null,
  front_female: sprites?.front_female ?? null,
  front_shiny_female: sprites?.front_shiny_female ?? null,
  back_default: sprites?.back_default ?? null,
  back_shiny: sprites?.back_shiny ?? null,
  back_female: sprites?.back_female ?? null,
  back_shiny_female: sprites?.back_shiny_female ?? null,
  versions: sprites?.versions ?? {},
});

export const getPokemon = async (
  idOrName: number | string,
): Promise<PokeboxEntry> => {
  const storedPokemon = await getStoredPokemon(idOrName);
  if (storedPokemon) {
    return storedPokemon;
  }

  let pokemon;
  try {
    pokemon = await pokemonService.getPokemon(idOrName);
  } catch (error) {
    throw new Error(`Could not find Pokemon with id: ${idOrName}: ${error}`);
  }

  if (!pokemon) {
    throw new Error(
      `Could not find Pokemon with id: ${idOrName}: Error: Pokemon not found`,
    );
  }

  const abilities = await pokemonService.getAbilities(pokemon);
  if (!abilities) {
    throw new Error(
      `Could not find Pokemon with id: ${idOrName}: Error: Unable to retrieve abilities`,
    );
  }

  const species = await pokemonService.getSpecies(pokemon);
  const speciesDescription = findEnglishText(
    species?.flavor_text_entries,
    "flavor_text",
  );
  if (!speciesDescription) {
    throw new Error(
      `Could not find Pokemon with id: ${idOrName}: Error: Unable to retrieve species description`,
    );
  }

  const pokeboxEntry: PokeboxEntry = {
    id: pokemon.id,
    name: pokemon.name,
    species_description: speciesDescription,
    types: Array.isArray(pokemon.types)
      ? pokemon.types.map((typeInfo) => typeInfo?.type?.name).filter(Boolean)
      : [],
    sprites: normalizeSprites(pokemon.sprites),
    abilities: Array.isArray(abilities)
      ? abilities.map((ability) => ({
          name: ability.name,
          flavour_text:
            findEnglishText(ability.flavor_text_entries, "flavor_text") ?? "",
          effect: findEnglishText(ability.effect_entries, "effect") ?? "",
        }))
      : [],
  };

  await storage.storePokemon(pokeboxEntry);
  return pokeboxEntry;
};

export const getPokedex = async (): Promise<{ id: number; name: string }[]> => {
  const pokedex = await storage.getPokedex();
  return pokedex ?? [];
};

The code it generated is interesting. I can see where it made assumptions about the return of some things like the storage.getPokedex() function, this will always return an array unless the database call throws an exception (the handling of which was missing from both the tests and my code ╮(︶︿︶)╭). I was interested to see the addition of a bunch of functions, the getStoredPokemon and isNumeric functions feel over-engineered to me. If those calls were being repeated throughout the codebase or the file then I feel like they are a worthy addition but as they are only used in one place it feels like overkill to me. It was interesting to see that it made the same choice of the error handling for speciesDescription that I did, I think this was because I realised I had missed it after I wrote my code and added it after in a rush and did not notice I was testing the description but had no error handling for the call failing other than the general try catch block. The last thing that surprised me was the normalizeSprites function, I feel that this is not needed as the sprites will be returned by the API and it should return an error if it is unable to populate the field.

Conclusion

I was very impressed with the results, even with my above thoughts. Compared to getting Copilot to write tests for a repo it is a night and day difference and somewhat reinforces my belief that AI tools should be used carefully and with a good amount of checks and more importantly a very thorough and comprehensive test suite that has been written or at least audited by a human. The code took about 2 minutes to generate which is faster than what I wrote but I would not submit this code and would make some changes to it to ensure it matched my standards and coding style. This would probably add at least 5 minutes to the time it took to implement the file this way making it slightly slower than my own. I will admit however, that this is a simple and small example and with a larger amount of code to test it would probably be faster but the time to correct and audit it would go up. My last thought on the code was that there was a noticeably American spelling of normalize in normalizeSprites and yes I know that the z is technically allowed as a British spelling it would still annoy me enough to have to go through and correct the spelling in the generated code, possibly this can be altered in the prompt to ensure that it uses British spellings, I don’t know.