Testing and Debugging

A program that compiles is not yet a program that is right. A mistake in a program is called a bug. Testing is how you find out whether a program has bugs. Debugging is finding the line that is wrong and repairing it.

Until now the lab's checks did the testing for you. This lesson shows how to do it yourself.

Three ways to be wrong

A program can be wrong in three ways. They differ in who notices.

The first way you know from module 1: a compile error. The compiler does not accept the program, and nothing runs.

#include <iostream>

int main() {
    int total = 7 + 8;
    std::cout << "Total: " << totl << "\n";
}

The name totl is misspelt. The compiler reports the line, and you repair it. This is the easiest kind of bug, because the compiler finds it for you.

The second way is a run-time error: the program compiles and starts, and then stops before it has finished its work. This program shares sweets among some children.

#include <iostream>

int main() {
    int sweets = 12;
    int children = 0;
    std::cin >> children;
    std::cout << "Each child gets " << sweets / children << "\n";
    std::cout << "Done\n";
}

Each child gets 4
Done

With the input 3 it works. With the input 0 it cannot: 12 divided by 0 has no answer. C++ does not say what a program does then. In the lab the program is stopped at the line with the division, and Done is never printed.

Notice that the compiler had nothing to complain about. sweets / children is correct C++. Whether it works depends on a value that exists only while the program runs.

The third way is a wrong answer: the program compiles, runs to its end, and what it prints is not true.

#include <iostream>

int main() {
    int first = 7;
    int second = 8;
    double average = (first + second) / 2;
    std::cout << "Average: " << average << "\n";
}

Average: 7

The average of 7 and 8 is 7.5. Both sides of the / are int, so the fraction was dropped before the result reached average. Nothing stopped and nothing was reported.

Kind	What you see	Who notices
compile error	a message from the compiler; nothing runs	the compiler
run-time error	the program starts, then stops early	the lab, at the moment it happens
wrong answer	a normal run, with false output	only you

The third kind is the dangerous one. No tool will tell you that 7 is the wrong average. You have to know what the right answer is, and compare.

Decide the answer first

Here is a function with a rule that is easy to state: a ticket costs 5 for a child under 12 and 9 for anyone else.

To test it, choose an input, and work out from the rule what the function should return. Do this before you run anything. An input together with the output you expect is a test case.

Input: age	Expected price	Why
8	5	8 is under 12
30	9	30 is not under 12

Then run the function with each input and compare.

#include <iostream>

// Returns the price of a ticket: 5 for a child under 12, 9 for anyone else.
int ticket_price(int age) {
    if (age < 12) {
        return 5;
    }
    return 9;
}

int main() {
    std::cout << ticket_price(8) << "\n";
    std::cout << ticket_price(30) << "\n";
}

5
9

Both lines match the table, so both test cases pass. A test case whose output differs from what you expected fails.

The order matters. If you run the program first and then ask yourself whether 5 looks right, you will nearly always say yes. An expected answer copied from the program's output can never disagree with the program, so it tests nothing. The expected answer must come from the rule.

Where programs break

The next version of ticket_price has a bug. Read its condition.

#include <iostream>

// Returns the price of a ticket: 5 for a child under 12, 9 for anyone else.
int ticket_price(int age) {
    if (age <= 12) {
        return 5;
    }
    return 9;
}

int main() {
    std::cout << ticket_price(8) << "\n";
    std::cout << ticket_price(30) << "\n";
    std::cout << ticket_price(11) << "\n";
    std::cout << ticket_price(12) << "\n";
    std::cout << ticket_price(13) << "\n";
}

The two test cases from the last section, 8 and 30, still pass. So do 11 and 13. Only the input 12 gives a wrong price: the rule says 9, and the function returns 5.

Of all the ages there are, exactly one exposes this bug, and it is the one where the rule changes from one price to the other. An input at such a place is an edge case. Bugs collect at the edges, because that is where < and <=, or 0 and 1, make a difference. In the middle of a range, a right version and a wrong version usually agree.

When you choose test cases, look for the edges first.

Each boundary in the rule. For "under 12", test 11, 12 and 13: one below, on, and one above.
Zero and one. A loop that runs zero times or once, a count of 0, an empty input.
The smallest and largest inputs the function allows.
Equal values, when the function compares its inputs with each other.
Every listed case. If a rule names several special values, test each of them, not only the first.

Add one ordinary input from the middle of each range as well. Two or three of those are enough.

assert

Comparing printed numbers with a table by eye is slow, and you will stop doing it. C++ can do the comparing for you.

#include <cassert>
#include <iostream>

// Returns the price of a ticket: 5 for a child under 12, 9 for anyone else.
int ticket_price(int age) {
    if (age < 12) {
        return 5;
    }
    return 9;
}

int main() {
    assert(ticket_price(11) == 5);
    assert(ticket_price(12) == 9);
    std::cout << "Both held\n";
}

Both held

assert(condition); states that the condition is true at this line. It comes from the header <cassert>.

If the condition is true, nothing happens. Nothing is printed, and the program carries on with the next statement.
If the condition is false, the program stops at that line. No later statement runs.

Each assert here is one test case: the input is inside the call, and the expected output is on the right of ==.

Now the same main with the buggy function from the last section.

#include <cassert>
#include <iostream>

// Returns the price of a ticket: 5 for a child under 12, 9 for anyone else.
int ticket_price(int age) {
    if (age <= 12) {
        return 5;
    }
    return 9;
}

int main() {
    assert(ticket_price(11) == 5);
    assert(ticket_price(12) == 9);
    std::cout << "Both held\n";
}

The first assert is true, and nothing happens. The second is false: ticket_price(12) is 5, not 9. The program stops there, and Both held is never printed. The lab's status line then shows a message that names the line of the failed assert, here line 14, and repeats its condition.

That is a run-time error, and one you asked for. An assert turns a wrong answer, which nobody notices, into a stop at a known line, which nobody can miss.

A program in which every assert is true prints nothing extra. Silence means that everything you stated was true.

A function that tests a function

Test cases belong together, in a function of their own. A test function calls one function with several inputs and asserts each result. By custom its name is test_ followed by the name of the function it tests.

#include <cassert>
#include <iostream>

// Returns the price of a ticket: 5 for a child under 12, 9 for anyone else.
int ticket_price(int age) {
    if (age < 12) {
        return 5;
    }
    return 9;
}

void test_ticket_price() {
    assert(ticket_price(0) == 5);     // the smallest age
    assert(ticket_price(8) == 5);     // an ordinary child
    assert(ticket_price(11) == 5);    // one below the boundary
    assert(ticket_price(12) == 9);    // on the boundary
    assert(ticket_price(13) == 9);    // one above the boundary
    assert(ticket_price(30) == 9);    // an ordinary adult
}

int main() {
    test_ticket_price();
    std::cout << "All tests passed\n";
}

All tests passed

main calls the test function first. If every assert is true, the call returns and main prints its line. If one is false, the program stops inside test_ticket_price, and the message names the line of the test case that failed.

Three things make a test function worth keeping.

It can be run again. After every change to ticket_price, one run tells you whether the old cases still pass.
It says what the function should do, not how. Nothing in test_ticket_price depends on how ticket_price is written. The same six lines would test any version of it.
A good one tells versions apart. These tests pass with the correct function and stop with the <= version, because of the case for 12. A test function with only the cases 8 and 30 would pass with both, and so would tell you nothing about the boundary.

When a test fails, there are two places to look: the function, and the test. Check first that the expected value in the assert really follows from the rule.

Two bugs you cannot see

Two bugs compile, run, and look correct on the screen. Both are about which statements an if or a loop controls.

The first is a semicolon in the wrong place.

#include <iostream>

int main() {
    int temperature = 20;
    if (temperature < 0); {
        std::cout << "Freezing\n";
    }
    for (int i = 0; i < 3; ++i); {
        std::cout << "row\n";
    }
}

Freezing
row

It is not freezing at 20 degrees, and row should appear three times.

Look at the ; straight after if (temperature < 0). A semicolon on its own is a complete statement that does nothing. So the if controls that empty statement, and the if ends there. The braces below are then an ordinary group of statements, and they run every time.

The for line has the same semicolon. The loop does run three times, and each time it runs the empty statement. Then the loop is over, and the braces below run once.

The rule: no semicolon between the ) of an if, for or while and its {.

The second bug is braces that are not there.

#include <iostream>

int main() {
    int coins = 3;
    if (coins >= 10)
        std::cout << "Enough for a ticket\n";
        std::cout << "Printing the ticket\n";
    std::cout << "Goodbye\n";
}

Printing the ticket
Goodbye

Three coins are not enough, and still the ticket is printed.

C++ allows an if or a loop without braces. Then it controls exactly one statement: the next one. Here that is the first std::cout line. The second std::cout line is indented as though it belonged to the if, and it does not. The compiler ignores indentation completely.

The rule: always write the braces, even around one statement. Every program in this course does.

Looking inside a running program

A test tells you that a function is wrong. It does not tell you which line. To find the line you need to see the values the variables hold while the program runs. The lab gives you two tools for that.

The first is watch. The call watch("name", value); records one value under a name. It comes from scimigo.h, like the drawing calls.

#include "scimigo.h"
#include <iostream>
using namespace scimigo;

int main() {
    int power = 1;
    for (int i = 0; i < 5; ++i) {
        power = power * 2;
        watch("power", power);
    }
    std::cout << power << "\n";
}

The watch line runs five times, once on each turn of the loop. The lab's Watch panel then lists five values for power, in the order they were recorded: 2, 4, 8, 16 and 32. The output panel is unchanged. watch prints nothing.

Use watch when you suspect one variable. Before you run, write down the values you expect it to take. Then compare your list with the panel. The first value that differs tells you on which turn things went wrong.

The second tool is the Step through button. It runs the program one statement at a time. After each statement it shows the statement that will run next, and the value of every variable at that moment.

Use stepping when you do not know which variable to suspect. Work like this.

Before each step, say what the next statement will change, and to what value.
Press the button and compare.
Stop at the first statement where what you see differs from what you said. Either that statement is the bug, or the statement did not run when you thought it would, and the condition that guards it is the bug.

Both tools are for looking. They do not repair anything, and a watch line can be deleted when the bug is found.

What a function promises

Every test case needs an expected answer, and the expected answer comes from knowing exactly what the function is supposed to do. The place to write that down is a comment above the function. Such a comment is called the function's contract. It has two parts.

Needs: what must be true of the values the caller gives.
Returns: what the function gives back, if the needs are met.

#include <cassert>
#include <iostream>

// Needs: whole is greater than 0, and part is 0 or more.
// Returns: part as a percentage of whole, with the fraction dropped.
int percent(int part, int whole) {
    assert(whole > 0);
    assert(part >= 0);
    return part * 100 / whole;
}

int main() {
    std::cout << percent(1, 4) << "\n";
    std::cout << percent(2, 3) << "\n";
}

25
66

A contract divides the work between two people, even when both are you.

The caller must meet the needs. A call such as percent(1, 0) breaks the contract, and that is a bug in the caller, not in percent. The function has promised nothing for it.
The function must keep the promise for every input that meets the needs. percent(2, 3) must be 66, because the contract says the fraction is dropped.

The two assert lines at the top of percent check the needs. If a caller breaks the contract, the program stops there, at once, instead of carrying on with a meaningless number.

The contract also tells you what to test. Each sentence of "Returns" gives you test cases with their expected answers. "Needs" tells you which inputs are not test cases at all: there is no right answer for percent(1, 0), so there is nothing to assert.

A good contract passes this test: someone who has never seen the body of the function could write the body from the comment alone.