Good Start: Test Your Getting-Started Docs

An open-source project’s “Getting Started” section is load-bearing for user success, tedious to test and maintain, and among the least interesting parts to the builders. In the best cases, a lot of effort is invested to get the prose and example code perfectly detailed, and then the code moves on without it. A package gets renamed, the minimum Python version goes up, a dependency starts needing a system library.

Of course, the literal, technical setup and installation process can be automated in bash scripts or CI/CD pipelines, which can reveal breakage to developers. But great intro documentation also entails explanation to users, especially when setup deviates or is complicated by system details, version mismatches, or other edge cases, and that is harder to test in scripts.

Good Start is a simple idea for addressing this using coding agents. Let the instructions speak for themselves by having an agent test the setup process. If the agent gets the project installed and working, the docs pass. If not, it returns an error along with a message detailing what went wrong. This effort fits into my general interest in “plugging” agents into existing toolchains, like little reasoning modules.

Installation and more info are available in the docs and Github repo.

Docs as a test

My favorite part is that it runs as a pytest plugin. Installing the package registers a good_start fixture, so a documentation check lives in your test suite next to everything else:

# tests/test_docs.py

def test_readme(good_start):
    result = good_start("README.md")
    assert result.passed, result.details

result.passed is the verdict and result.details is the agent’s report provided right in the CI output.

Agent runs are slow and cost API calls, so these tests are marked automatically, which makes them easy to keep out of the fast loop:

uv run pytest -m "not good_start"   # everything else
uv run pytest -m good_start -v      # just the docs

For pipelines that don’t use pytest, there’s also a CLI, good-start check ., which exits non-zero on failure.

Output schema

While the input task for the agent is inherently open-ended, the output can be structured. The agent provides the information to fulfill the required output schema, such as the boolean result, a detailed text summary, a log of steps it took along the way, and, optionally, a verification command it ran to confirm the successful result.

from pydantic import BaseModel


class AgentStep(BaseModel):
    tool: str        # Bash, Read, Grep, Glob
    input: str       # the command or argument passed to the tool
    output: str      # the tool's result, truncated if very long
    is_error: bool = False


class AgentFindings(BaseModel):
    passed: bool     # result status, success or failure
    details: str     # summary, with constructive feedback on failure
    steps: list[AgentStep] = []
    verification_command: str | None = None  # e.g., python -c "import good_start"

The agent reports back in the above object, which makes the output much like a traditional program at that point. By default, the agent runs in a container with the project mounted read-only, so test runs don’t impact the local machine. After adding containerization, I now understand the recent focus on making startups much faster and more ephemeral.

Here’s an example of a failed result, from a project whose package was renamed from text_stats to textstats without the README catching up:

{
  "passed": false,
  "details": "`pip install .` works and installs `textstats` 0.2.0, but the README's check command fails. It runs `import text_stats`, which gives `ModuleNotFoundError: No module named 'text_stats'`. The package is actually at src/textstats/ and imports as `textstats` (no underscore). `python -c \"import textstats; print(textstats.__version__)\"` prints 0.2.0. To fix, change the README check command to `import textstats`, or rename the package to text_stats. The project name is also inconsistent: the README heading says \"text_stats\" while pyproject.toml names it \"textstats\".",
  "steps": [
    {
      "tool": "Bash",
      "input": "ls; cat README.md",
      "output": "README: pip install . then python -c \"import text_stats; print(text_stats.__version__)\"",
      "is_error": false
    },
    {
      "tool": "Bash",
      "input": "cat pyproject.toml; find src; python --version; pip install .; python -c \"import text_stats; print(text_stats.__version__)\"",
      "output": "name=textstats; src/textstats/__init__.py; Python 3.12.13; Successfully installed textstats-0.2.0; ModuleNotFoundError: No module named 'text_stats'",
      "is_error": true
    },
    {
      "tool": "Bash",
      "input": "python -c \"import textstats; print(textstats.__version__)\"",
      "output": "0.2.0",
      "is_error": false
    }
  ],
  "verification_command": "python -c \"import text_stats; print(text_stats.__version__)\""
}

Notes on next steps

This project was primarily a first experiment for me to build a tool around agents.

For now, I’ve only implemented this to use the Claude Agent SDK. I’d like to incorporate other agents as well to make this more general purpose.

Considering the work isn’t too complicated, it would make sense to use simpler, more affordable models, which in turn would allow this to be part of routine automated testing and CI/CD.

I also could see building a custom GitHub Actions workflow within the repo to make CI/CD integration even easier.