Writing tests: what they prove, and what they don't

Program testing can be used to show the presence of bugs, but never to show their absence!

Edsger W. Dijkstra, Notes on Structured Programming, 1970

Dijkstra was right, and it’s worth starting there, because it sets honest expectations. A test can’t prove your code is correct. What it can prove is that the code does what you expected, for the cases you thought to check, every time you run it. That turns out to be enormously valuable.

What tests prove, and what they don’t

A passing test proves:

• This behavior works today. For these inputs, the code gives these outputs.
• It still works tomorrow. Run the suite after every change and you’ll know the moment something breaks.
• What the code is supposed to do. Tests are documentation that can’t drift out of date, because it fails when it’s wrong.
• You can refactor safely. Restructure freely; the tests tell you if behavior changed.

A passing test doesn’t prove:

• That there are no bugs. Only that the cases you wrote down behave as expected.
• That your expectations were right. If you misunderstood the requirement, the test will happily confirm the misunderstanding.
• That untested inputs work. And there are always more inputs than tests.

A real example: green tests, real bug

For this post we wrote the same small function in all five languages. slugify turns a title into a URL slug, so “Hello, World!” becomes “hello-world”. Every version passed every test below. Then we tried an input nobody had written a test for:

Language	slugify("Café Crème")
Go	café-crème
Rust	café-crème
Python	caf-cr-me
C++	caf-cr-me
PHP	caf-cr-me

The Python, C++ and PHP versions only recognize ASCII letters, so the accented letters vanish. The Python version even had 100% code coverage:

Name      Stmts   Miss  Cover   Missing
---------------------------------------
slug.py       3      0   100%

Every line ran; the bug survived anyway. Coverage tells you which code your tests executed, not which behaviors they checked. Is dropping accents a bug? That depends on what the function is supposed to do, and that’s exactly the point. Until someone writes the test, nobody has decided.

What makes a good test

• Fast. Fast tests get run constantly; slow tests get skipped.
• Independent. Each test sets up its own data and can run alone, in any order.
• Repeatable. The same result every time: no real clocks, random numbers or network calls unless you control them.
• One behavior per test. When it fails, the name tells you what broke.
• Named for the behavior. test_empty_input_returns_no_rows beats test_load_3.
• Arrange, act, assert. Set up, do one thing, check the result.
• Public behavior, not internals. Test what the code does, not how, so refactoring doesn’t break the tests.
• Red first. Watch a new test fail before you make it pass. A test you’ve never seen fail might not be testing anything.

Kinds of tests

• Unit tests check one function or class in isolation, in milliseconds. Most of your tests should be these.
• Integration tests check that pieces work together: your code plus a real database, file system or HTTP server.
• End-to-end tests drive the whole application like a user would. They’re valuable but slow and brittle, so keep them few.
• Property-based and fuzz tests generate hundreds of thousands of inputs and check rules that should always hold. More on these below.

Testing in each language

Each example tests the same slugify function, so you can compare them side by side.

Go: the built-in testing package

Go needs nothing extra. Test files end in _test.go, test functions start with Test, and go test finds and runs them. The idiomatic style is a table-driven test with a subtest per case:

package slug

import "testing"

func TestSlugify(t *testing.T) {
	tests := []struct {
		name, in, want string
	}{
		{"simple", "Hello World", "hello-world"},
		{"punctuation", "Hello, World!", "hello-world"},
		{"extra spaces", "  red   green  ", "red-green"},
		{"empty", "", ""},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if got := Slugify(tt.in); got != tt.want {
				t.Errorf("Slugify(%q) = %q, want %q", tt.in, got, tt.want)
			}
		})
	}
}

$ go test -v
=== RUN   TestSlugify
=== RUN   TestSlugify/simple
=== RUN   TestSlugify/punctuation
=== RUN   TestSlugify/extra_spaces
=== RUN   TestSlugify/empty
--- PASS: TestSlugify (0.00s)
    --- PASS: TestSlugify/simple (0.00s)
    --- PASS: TestSlugify/punctuation (0.00s)
    --- PASS: TestSlugify/extra_spaces (0.00s)
    --- PASS: TestSlugify/empty (0.00s)
PASS
ok  	example.com/slug	0.002s

Useful flags:

go test ./...                 # every package in the module
go test -run Slugify/empty    # one test or subtest
go test -race ./...           # detect data races
go test -cover ./...          # coverage percentage
go test -bench=. ./...        # run benchmarks

Many teams add testify for shorter assertions, but the standard library is enough to start.

Python: pytest

Python ships with unittest, but pytest is the community standard. Tests are plain functions with plain assert, and parametrize runs one test over many inputs:

import pytest

from slug import slugify


@pytest.mark.parametrize(
    "title, expected",
    [
        ("Hello World", "hello-world"),
        ("Hello, World!", "hello-world"),
        ("  red   green  ", "red-green"),
        ("", ""),
    ],
)
def test_slugify(title, expected):
    assert slugify(title) == expected

$ pytest -v
test_slug.py::test_slugify[Hello World-hello-world] PASSED               [ 25%]
test_slug.py::test_slugify[Hello, World!-hello-world] PASSED             [ 50%]
test_slug.py::test_slugify[  red   green  -red-green] PASSED             [ 75%]
test_slug.py::test_slugify[-] PASSED                                     [100%]

============================== 4 passed in 0.02s ===============================

Fixtures handle setup and teardown, pytest.raises checks for exceptions, and plugins add coverage (pytest-cov) and parallel runs (pytest-xdist). Run it with uv run pytest; see our uv guide.

C++: GoogleTest

C++ has no built-in test framework, and GoogleTest (gtest) is the most widely used. It’s mature, works with every major compiler, and comes with gMock for mock objects.

Setting it up with CMake

The simplest setup lets CMake download GoogleTest for you, so there’s nothing to install system-wide. CMakeLists.txt:

cmake_minimum_required(VERSION 3.20)
project(slug LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

include(FetchContent)
FetchContent_Declare(
  googletest
  URL https://github.com/google/googletest/archive/refs/tags/v1.18.0.tar.gz
)
FetchContent_MakeAvailable(googletest)

add_library(slug slug.cpp)
target_include_directories(slug PUBLIC .)

enable_testing()
add_executable(slug_test slug_test.cpp)
target_link_libraries(slug_test PRIVATE slug GTest::gtest_main)

include(GoogleTest)
gtest_discover_tests(slug_test)

Linking GTest::gtest_main gives you a ready-made main(), and gtest_discover_tests registers every test with CTest.

Writing tests

slug_test.cpp:

#include "slug.h"

#include <gtest/gtest.h>

TEST(Slugify, Simple) {
    EXPECT_EQ(slugify("Hello World"), "hello-world");
}

TEST(Slugify, PunctuationAndSpaces) {
    EXPECT_EQ(slugify("Hello, World!"), "hello-world");
    EXPECT_EQ(slugify("  red   green  "), "red-green");
}

TEST(Slugify, Empty) {
    EXPECT_TRUE(slugify("").empty());
}

// One test body, many inputs.
class SlugifyCases
    : public ::testing::TestWithParam<std::pair<std::string, std::string>> {};

TEST_P(SlugifyCases, MatchesExpected) {
    const auto& [title, expected] = GetParam();
    EXPECT_EQ(slugify(title), expected) << "title: \"" << title << "\"";
}

INSTANTIATE_TEST_SUITE_P(
    Titles, SlugifyCases,
    ::testing::Values(std::pair{"C++ Tips", "c-tips"},
                      std::pair{"Version 2.0", "version-2-0"},
                      std::pair{"--dashes--", "dashes"}));

• TEST(Suite, Name) defines a test. Group related tests under one suite name.
• EXPECT_* records a failure and keeps going, so you see every problem in one run.
• ASSERT_* stops the test immediately. Use it when continuing makes no sense, such as after a null pointer check.
• << after an assertion adds your own message to the failure output.
• TEST_P and INSTANTIATE_TEST_SUITE_P run one test body over a list of parameters, like pytest’s parametrize.
• TEST_F uses a fixture class, whose SetUp() and TearDown() run around each test.

Other useful assertions include EXPECT_NE, EXPECT_LT, EXPECT_NEAR for floating point, EXPECT_STREQ for C strings, and EXPECT_THROW for exceptions.

Building and running

cmake -S . -B build
cmake --build build
./build/slug_test                              # run directly
ctest --test-dir build                         # or through CTest
./build/slug_test --gtest_filter='Slugify.*'   # just one suite

[==========] Running 6 tests from 2 test suites.
[----------] 3 tests from Slugify
[ RUN      ] Slugify.Simple
[       OK ] Slugify.Simple (0 ms)
[ RUN      ] Slugify.PunctuationAndSpaces
[       OK ] Slugify.PunctuationAndSpaces (0 ms)
[ RUN      ] Slugify.Empty
[       OK ] Slugify.Empty (0 ms)
[----------] 3 tests from Titles/SlugifyCases
[ RUN      ] Titles/SlugifyCases.MatchesExpected/0
[       OK ] Titles/SlugifyCases.MatchesExpected/0 (0 ms)
...
[==========] 6 tests from 2 test suites ran. (0 ms total)
[  PASSED  ] 6 tests.

What a failure looks like

Here’s the accented-letters test from earlier, in GoogleTest:

TEST(Slugify, KeepsAccentedLetters) {
    EXPECT_EQ(slugify("Café Crème"), "café-crème");
}

[ RUN      ] Slugify.KeepsAccentedLetters
slug_test.cpp:34: Failure
Expected equality of these values:
  slugify("Café Crème")
    Which is: "caf-cr-me"
  "café-crème"
    Which is: "caf\xC3\xA9-cr\xC3\xA8me"
    As Text: "café-crème"

[  FAILED  ] Slugify.KeepsAccentedLetters (0 ms)

GoogleTest shows both expressions, both values, and even the raw UTF-8 bytes. That’s a red test worth having: now the bug has a name, and you’ll know when it’s fixed.

Popular alternatives are Catch2 and doctest, both lighter-weight, and Boost.Test if you already use Boost.

Rust: built into the language

Rust has testing built into the compiler and Cargo. Unit tests usually live in the same file as the code, in a module that’s only compiled for tests:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn simple() {
        assert_eq!(slugify("Hello World"), "hello-world");
    }

    #[test]
    fn punctuation_and_spaces() {
        assert_eq!(slugify("Hello, World!"), "hello-world");
        assert_eq!(slugify("  red   green  "), "red-green");
    }

    #[test]
    fn empty() {
        assert_eq!(slugify(""), "");
    }
}

Integration tests go in a tests/ folder and use your crate exactly like a user would, through its public API. tests/public_api.rs:

// Integration tests see the crate only through its public API.
use slug::slugify;

#[test]
fn keeps_accented_letters() {
    assert_eq!(slugify("Café Crème"), "café-crème");
}

And examples in documentation comments are compiled and run as doc tests, so your docs can never show code that doesn’t work:

/// Turn a title into a URL slug.
///
/// ```
/// assert_eq!(slug::slugify("Hello, World!"), "hello-world");
/// ```
pub fn slugify(title: &str) -> String {

cargo test runs all three kinds:

$ cargo test
running 3 tests
test tests::empty ... ok
test tests::punctuation_and_spaces ... ok
test tests::simple ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

running 1 test
test keeps_accented_letters ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests slug
test src/lib.rs - slugify (line 3) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Add #[should_panic] to test for panics. Worth knowing: proptest for property-based tests and cargo-nextest, a faster test runner.

PHP: PHPUnit

PHPUnit is PHP’s standard test framework. The current release, PHPUnit 13, needs PHP 8.4 or newer. Install it per project with Composer:

composer require --dev phpunit/phpunit

Map your namespaces in composer.json so tests can find your classes:

{
    "autoload": { "psr-4": { "App\\": "src/" } },
    "autoload-dev": { "psr-4": { "App\\Tests\\": "tests/" } }
}

tests/SlugTest.php:

<?php

declare(strict_types=1);

namespace App\Tests;

use App\Slug;
use PHPUnit\Framework\Attributes\DataProvider;
use PHPUnit\Framework\TestCase;

final class SlugTest extends TestCase
{
    public function testSimple(): void
    {
        $this->assertSame('hello-world', Slug::slugify('Hello World'));
    }

    #[DataProvider('titles')]
    public function testTitles(string $title, string $expected): void
    {
        $this->assertSame($expected, Slug::slugify($title));
    }

    public static function titles(): array
    {
        return [
            'punctuation' => ['Hello, World!', 'hello-world'],
            'extra spaces' => ['  red   green  ', 'red-green'],
            'empty' => ['', ''],
        ];
    }
}

$ vendor/bin/phpunit --testdox tests
PHPUnit 13.4.6 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.5.12

Slug (App\Tests\Slug)
 ✔ Simple
 ✔ Titles with data set "punctuation"
 ✔ Titles with data set "extra spaces"
 ✔ Titles with data set "empty"

OK (4 tests, 4 assertions)

Leave off --testdox for the compact dotted output.

Use assertSame rather than assertEquals: it checks the type too, so "1" won’t quietly equal 1. Data providers are PHPUnit’s version of parametrized tests, and modern PHPUnit configures them with attributes such as #[DataProvider] rather than docblock comments.

Many Laravel projects use Pest, which runs on top of PHPUnit with a shorter style:

composer require --dev --with-all-dependencies pestphp/pest

tests/SlugTest.php:

<?php

use App\Slug;

it('slugifies titles', function (string $title, string $expected) {
    expect(Slug::slugify($title))->toBe($expected);
})->with([
    ['Hello, World!', 'hello-world'],
    ['  red   green  ', 'red-green'],
]);

$ vendor/bin/pest

  PASS  Tests\SlugTest
  ✓ it slugifies titles with ('Hello, World!', 'hello-world')
  ✓ it slugifies titles with (' red green ', 'red-green')

  Tests:    2 passed (2 assertions)
  Duration: 0.07s

At a glance

Language	Built in	Popular choice	Run with
Go	testing package	Standard library, plus testify	go test ./...
Python	unittest	pytest	pytest
C++	Nothing	GoogleTest, Catch2, doctest	ctest or the test binary
Rust	#[test], doc tests	Built-in, plus proptest	cargo test
PHP	Nothing	PHPUnit, Pest	vendor/bin/phpunit

Finding the cases you didn’t think of

Example-based tests only check the inputs you wrote down. Property-based tests and fuzzing flip this around: you state a rule that should hold for every input, and the tool generates inputs trying to break it.

Python’s Hypothesis:

import re

from hypothesis import given
from hypothesis import strategies as st

from slug import slugify


@given(st.text())
def test_slug_is_url_safe(title):
    assert re.fullmatch(r"[a-z0-9]+(-[a-z0-9]+)*|", slugify(title))


@given(st.text())
def test_slugify_is_idempotent(title):
    assert slugify(slugify(title)) == slugify(title)


@given(st.text(alphabet=st.characters(categories=["L"]), min_size=1))
def test_letters_are_never_dropped(title):
    assert slugify(title) != ""

The first two properties held. The third found the accent bug on its own, and shrank the failure down to a single character:

E       AssertionError: assert '' != ''
E        +  where '' = slugify('ª')
E       Failing test case: test_letters_are_never_dropped(
E           title='ª',
E       )
1 failed, 2 passed in 1.98s

Go has fuzzing built in:

func FuzzSlugify(f *testing.F) {
	f.Add("Hello, World!")
	f.Add("  red   green  ")
	f.Fuzz(func(t *testing.T, title string) {
		s := Slugify(title)
		if strings.HasPrefix(s, "-") || strings.HasSuffix(s, "-") {
			t.Errorf("Slugify(%q) = %q: starts or ends with a dash", title, s)
		}
		if strings.Contains(s, "--") {
			t.Errorf("Slugify(%q) = %q: contains a double dash", title, s)
		}
		if Slugify(s) != s {
			t.Errorf("Slugify is not idempotent for %q", title)
		}
	})
}

$ go test -fuzz=FuzzSlugify -fuzztime=10s
fuzz: elapsed: 11s, execs: 334048 (0/sec), new interesting: 73 (total: 75)
PASS

Over 330,000 generated inputs in about ten seconds, and the rules held. That still isn’t proof of correctness, but it’s far more evidence than four hand-written cases. Rust has proptest and cargo fuzz; C++ has libFuzzer and RapidCheck.

Coverage: useful, not a goal

Coverage tools show which lines your tests ran:

go test -cover ./...
pytest --cov=myproject --cov-report=term-missing
cargo llvm-cov                       # via the cargo-llvm-cov tool
vendor/bin/phpunit --coverage-text   # needs Xdebug or PCOV

For C++, compile with --coverage and use gcov, lcov or gcovr.

Low coverage is a useful warning: that code has never been run by a test. High coverage proves much less. As the Python example showed, 100% coverage happily coexists with a real bug. Use coverage to find what you’ve missed, not as a score to hit.

The bottom line

So write the test first, watch it fail, make it pass, and keep it forever. When a bug gets through, as some always will, start the fix with a failing test that reproduces it. Then that bug can never come back unnoticed. For how that fits into tracking issues, see Issue tracking: from bug report to commit.