d9-testing.tex

%
% TODO: do some slides or maybe add to notes with 
%
% - how the testing works, i.e. BeforeClass, (Before, Test, After),
% (Before, Test, After), (Before, Test, After), (Before, Test, After),
% (Before, Test, After), (Before, Test, After), (Before, Test, After),
% AfterClass, next class, repeat.
%

\section{Software testing}
\label{sec:software-testing}

I know your code has bugs. If it is a small project it will have a few
bugs. If it is a big project it will have a lot of bugs. I know.

How do I know? Because I know you are a human being, and human beings
make mistakes. When a programmer is writing code there are too
many things to keep in the short-term memory at the same time: data
structures, communication between different parts of the program,
flows of information\ldots ~ Every program except the most trivial ones is
too complex for the human mind to see in its entirety. This is why we
use methods to isolate operations (using methods), and
why we use classes to group different behaviours and states in our program
around some conceptual ideas that we can grasp (e.g.~a supermarket has
queues, a queue has people, a person has a name and knows who is
behind in the queue; a company has products and employees, employees
have names and take money from the company, products give money to the
company). 

Structured and object-oriented programming are ways in which
programmers can reduce the complexity of their programs to levels that
are (more or less) manageable for their human minds, 
but it is still impossible to write
programs that do not contain bugs. Humans, even the best programmers,
have a tendency to forget some details of the program. That is the way
the human mind works: it concentrates on the most fundamental aspect
and forgets the details until needed\ldots which in computing means
when the program is already executing and nothing can be done about
it. 

Long story short, as long as programming is performed by humans,
programs will have bugs. The compiler can make sure that a program is
syntactically correct but it cannot tell if  it makes sense or
even whether it will do what the programmer expects. 

That is why software testing is important. Testing a program is a way
to ensure that the program does what it should. There are two types of
testing: manual and automatic (Table~\ref{tab:test}). You are already
familiar with the former type: you have been doing it for weeks, for
those programs you have been writing until now. Now we will learn how to
do things properly in an automated way. 

\begin{table}[htbp]
  \centering
  \begin{tabular}{|p{2.5cm}|p{5.8cm}|p{5.5cm}|}
    \hline
    & Manual & Automated \\
    \hline
    Speed & Very slow & Very fast \\
    Focus & Most important bugs known & Every single bug known \\
    Thoroughness & Sometimes forgets to test some things & Tests everything always \\
    It feels\ldots & Very repetitive and boring & Nothing: work is done by computer \\
    \hline
  \end{tabular}
  \caption{Manual testing vs automated testing}
  \label{tab:test}
\end{table}

\subsection{Automated testing}
\label{sec:automated-testing}

The idea behind automated testing is very simple: make a ``testing'' program that
verifies whether your ``main'' program behaves as expected. You can execute
this ``testing'' program every day as your original program grows to
make sure that everything works as it
should. Figures~\ref{fig:sdddfsdsrrers} and~\ref{fig:sdfsdsers} provide
two examples from real life when programs did not behave as expected. 

\begin{figure}[tbp]
\begin{verbatim}
    Programmer 1: The program has messed-up and now we have lost data
        from our clients!
    Programmer 2: How is that possible?
    Programmer 1: The function returnPastPurchases() returned null,
        the system got a NullPointerException and crashed. 
    Programmer 2: Oh! It returned null because there were no past
        purchases. 
    Programmer 1: But it has always returned an empty array! 
    Programmer 2: But I thought that null was more elegant than an
        empty array. If there is nothing to return, why not return
        null? 
    Programmer 1: What about... because the rest of the program
        expects an empty array! Now we face our clients sueing us!
    Programmer 2: I am sorry...
\end{verbatim}  
  \caption{Scenario 1: unmet expectations}
  \label{fig:sdddfsdsrrers}
\end{figure}

\begin{figure}[tbp]
\begin{verbatim}
    Programmer 1: Good, now we have found the bug. Method 
        getTaxReturns() had problems when it received a null
        TaxPayer. We have fixed the method that was returning 
        a null TaxPayer. Let's create a test for it to make sure it
        does not happen again. 
    Programmer 2: No need to. It will never happen again. I will never
        forget this week working 20h a day! I promise you no method
        will return a null TaxPayer ever again. 
    (...one year later...)
    Programmer 1: There is a serious bug in the system!
    Programmer 2: I have found it! Someone returned a null TaxPayer. 
    Programmer 1: What? A year ago, you promised me that it was not
        going to happen again!
    Programmer 2: It is not my fault! Someone else must have made
        changes to the core classes!
\end{verbatim}  
  \caption{Scenario 2: Star Bugs: Return of the Bug}
  \label{fig:sdfsdsers}
\end{figure}

Programs can behave badly for an infinite number of reasons,
including: 

\begin{itemize}
\item bad programmers writing bad code.
\item lack of communication between programmers in a big project.
\item programs without documentation that have to be modified, usually
  by a programmer that has nothing to do with the original
  programmers.
\item programs with \emph{obsolete} documentation that have to be modified, usually
  by a programmer that has nothing to do with the original
  programmers.
\item version changes in external libraries (or even the programming
  language) that are not backwards-compatible.
\item changes in one part of the program that affect some other part of
  the program, apparently unrelated. 
\end{itemize}

The last reason is particularly important because it is the most
common and it is quite difficult to fight. Big programs are usually
written by many programmers, and none of them knows everything about
the code. Sometimes a change in one region of the code breaks something
else that had been working seamlessly until then. The problem becomes worse as
programmers leave the project and other programmers join in with no
previous knowledge about the program. 
This results in programs that
are more and more difficult to modify and adapt as they evolve over
time. This is know as \emph{code viscosity}, and it is bad. Automated
testing, by means of regression tests, is a way of fighting code
viscosity: when the automated tests verify that everything 
is still working ---even those parts of the program unknown to the
programmer---, one can be confident that the last change did
not brake anything. 

\subsection{Creating tests}
\label{sec:creating-tests}

How do you write your own tests for your Java programs? You write them
as Java programs, using a special library called JUnit. There are
other ways, but JUnit is very simple, powerful, and well-known. JUnit
does not come with Java by default, but you can easily install it from
the web as it is free software\footnote{\emph{Free} as in free speech,
not free beer.}. 

An automated test is just a collection of methods that will be
executed by JUnit. These methods will test the methods in your
``main'' classes, and check that they return the right values. If they
do not, for any reason, JUnit will raise a flag so that you can look
into your code and find out why it is not behaving as expected. 

Let's see an example. Imagine that we have 
a method that returns the initials of a name
in a class called \verb+Person+: 

\begin{verbatim}
    public String getInitials(String fullName) {
        String result = "";
        String[] words = fullName.split(" ");
        for (int i = 0; i < words.length; i++) {
             String nextInitial = "" + words[i].charAt(0);
             result = result + nextInitial.toUpperCase();
        }
        return result;
    }
\end{verbatim}

A test program would be a Java class that could look like this: 

\begin{verbatim}
    import org.junit.*;
    import static org.junit.Assert.*;
    
    public class PersonTest {
       @Test
       public void testsNormalName() {
            Person p = new Person();
            String input = "Dereck Robert Yssirt";
            String output = p.getInitials(input);
            String expected = "DRY";
            assertEquals(expected, output);
       }
       // In a "real" testing class, there are usually
       // more methods here, each of them annotated with 
       // the annotation @Test, each of them testing
       // a different aspect of the code
    }
\end{verbatim}

As you can see, there are several things that make a test class (using
JUnit) different from a normal Java class. First of all, there is no
main method. JUnit tests are not executed directly, they are executed
by JUnit, so there is no need for a main method. 

A second important difference are those \verb+import+ statements at
the beginning, before the class definition. These  statements
are the way in which a programmer can tell Java to use classes that
are not in the current folder (they must be in the \emph{classpath},
though). 

If you want to import just some \emph{static methods} from some
class, you can use \emph{static imports}. In this example, the method
\verb+assertEquals(String,String)+ is statically imported from class
\verb+Assert+. This is only a convenience so that you do not need to
write \verb+Assert.assertEquals(....)+ every time you need it, 
only the method's name is enough. 

The third difference is the most important one. There is a special
kind of word before the method definition, a word starting with an at
symbol: \verb+@Test+. These at-words are called \emph{annotations} in
Java, and have a special meaning for the compiler. In this case, the
annotation \verb+@Test+ tells JUnit that the method
\verb+testsNormalName()+ is a test to be run by JUnit. This is the way
JUnit knows which methods to run and which methods to ignore. (We will
learn more about annotations in the future). In short, you must
write \verb+@Test+ before each of your test methods. 

You can also appreciate some conventions in the code. The test class
for class \verb+Person+ is called \verb+PersonTest+ (some people
prefer \verb+TestPerson+). The method's name tells the story of
the test: it can be read as ``this method''\ldots \verb+testsNormalName()+:
this method tests a normal name. You can also appreciate some of the
variable names: \verb+input+, \verb+output+, \verb+expected+ (for
expected output). 

The tests ends with the method \verb+assertEquals(String,String)+,
statically imported from class \verb+org.junit.Assert+. This method performs an
\emph{assertion}, a test that the parameters verify some condition
(equality in this case). If that is not true, the method will throw an
exception and JUnit will report a failure in your test. Assertion can
be made on equality, difference, nullity, and truth value. The most
common methods are \verb+assertEquals()+, \verb+assertTrue()+, and
\verb+assertFalse()+. The full list
of methods of \verb+Assert+ can be found in the documentation of the
class\footnote{http://junit.sourceforge.net/javadoc/org/junit/Assert.html}. 

\subsection{Running tests}
\label{sec:running-tests}

To run your tests, you must run JUnit from the command
line\footnote{In the future, we will learn how to do this
  automatically from an IDE like Eclipse, but for now we will make the
effort of understanding each step by itself.}. In order
to do so, you must bear in mind some facts: 

\begin{itemize}
\item JUnit is not part of core Java, so it must be added to the
  \emph{classpath} either manually or by modifying the environment
  variable CLASSPATH.
\item When you modify the CLASSPATH, you must remember to add the
  current folder (or Java will not find your class!).
\item The core class in JUnit is called, perhaps unsurprisingly,
  \verb+JUnitCore+.
\item From version 4.11 on (or, in other words, since 2013) JUnit uses
  a pattern-matching library called HamCrest. 
\end{itemize}

With this in mind, you are now able to run the following command
(please note that the JAR files may have a version number in their
names):

\begin{verbatim}
    > javac -cp .:junit.jar:hamcrest-core.jar PersonTest.java
\end{verbatim}

This will compile your test class(es). The \verb+-cp+ parameter is the
way you can modify the classpath when executing \verb+javac+ or
\verb+java+. Remember that elements in the classpath are separated by
colons~(:) on Linux and Mac, but they are separated by semicolons~(;) on
Windows. To run the tests, you must execute the main method in class
\verb+JUnitCore+, which is inside package \verb+org.junit.runner+,
passing all your test classes as parameters: 

\begin{verbatim}
    > java -cp .:junit.jar:hamcrest-core.jar org.junit.runner.JUnitCore PersonTest
\end{verbatim}

You can read that as ``execute on the Java Virtual Machine'', using as
classpath both the current folder~(.) and the JAR file of
JUnit\footnote{If you have left the JAR file, your classpath will look
different. You must write the full path to your JAR file in the
classpath.}, the
main method of class \verb+org.junit.runner.JUnitCore+ and pass as
argument the string ``PersonTest''. This will make JUnit run all the
tests inside the class PersonTest (as marked by the \verb+@Test+
annotation). 

If all tests run successfully, JUnit will congratulate you. If one or
more of the tests fail, JUnit will inform you of which tests failed
and why. 

\subsection{Designing tests}
\label{sec:designing-tests}

So far we have seen how to create automated tests, and how to execute
them in order to find bugs in our \emph{production code} (i.e.~our
``main code'', the program that we are creating to solve a
problem). Programmers usually create a lot of tests to verify the
behaviour of their classes (i.e.~as defined by its public methods, its
interface). 

But there is something that we need to bear in mind at all
times: tests do \textbf{not} prove that your code does not contain
bugs. No matter how many tests you have written, no matter how many
cases you cover with them, there is no way to prove that a code does
not contain bugs. We can only try to push the chance of bugs as low as
possible, but it is never zero.

You can think of tests as torches in a cavern, where the cavern
represents your code. The more
torches you place in the cavern, the more light there will be, so there
will be fewer shadows where bugs can hide. But you must place your
torches carefully. If you just put a lot of torches in one place, most
of the cavern will be in shadows, and your bugs will lurk in there until
the time is right to appear (usually to make your company look bad or
lose money or both). If you place your torches carefully and
illuminate as many corners as possible, your bugs will find it more
difficult to hide undetected. How can we ensure that we keep our
cavern as illuminated as possible? There are three basic strategies.

\subsubsection{Testing basic functionality}
\label{sec:test-basic-funct}

The first tests that you must write, the first torches you have to
place in your cavern, are those that shed light on the main features
of your code: if you have written a class with a method that returns
the number of colours in an image, write a test that provides an image
and checks that the right answer is returned; if you have written a
method that returns the initials in a name, write a test that checks
that the right initials for a name are returned. 

This is the simplest strategy, and everybody has it in mind, so there
is no need to hammer it anymore. The next two strategies are usually
forgotten by many programmers, so it is important that you think about
them carefully. 

\subsubsection{Testing border cases}
\label{sec:testing-border-cases}

Good code is simple, clear, and general. Programmers create programs
that repeat the same operations most of the time: checking the
salaries of all employees in a company, verifying the medicine doses
of all patients in a hospital, calculating all stress tensors for
every point on the wing of a plane, etc. Usually, this is performed
correctly in most programs. If it is not, the error tends to be quite
evident and is detected with the simple
``basic'' tests (or even at compile time).

Basic functionality is not the main source of bugs. 
Programs have a tendency to break around the \emph{borders}. Using the
metaphor of the cave, you can think that corners are usually darker
than the rest of a cave room, so they are good hiding places for
bugs. Border cases (the first of something, the last of something, null
cases, empty structures, etc) are usually slightly
different from the general case and your implementation may break
there: a list may not remove the first element correctly, or the
last; a loop may not check the last element in a list, or go out of
bounds; a method may break when it is given an empty list as a
parameter (or null). 

Border cases must always be checked in your tests. \textbf{This is very
important}. Leaving corners and borders in the shadows is just a
disaster waiting to happen. Many programmers forget to test border
cases and are later surprised by bugs in a code that seemed to be
``working fine''.

\subsubsection{Find bugs once}
\label{sec:find-bugs-once}

This is the most important rule in software testing, yet it is usually
forgotten by many programmers (especially lazy programmers). Whenever
you find a bug in a code ---and you will find many in your life as a
programmer--- you have to follow this simple algorithm: 

\begin{enumerate}
\item First of all, do \emph{not} fix the bug yet. 
\item Find the bug again. Repeat the same steps until you know exactly
  how and when the bug appears. This is called
  \emph{reproducing the bug} and it is not always easy in large,
  complicated programs.
\item Once you have been able to reliably reproduce the bug manually,
  write a test that reproduces the bug programmatically. Check that
  the bug is always fired when you run your test with JUnit. Make the
  test as simple as possible.
\item Once you have written the simplest test that reproduces the bug
  ---\textbf{and only then}--- fix the bug. Verify that the tests passes. 
\end{enumerate}

Following this simple algorithm will ensure that the same bug does not
appear again and you never find yourself in the situation described in
Figure~\ref{fig:sdfsdsers}. You are a human being and you make mistakes. No
matter how careful you are, your code will have some bugs. Most of the
time you will find bugs in code that was written by other programmers
that were not as careful as you. Following the ``find bugs once''
algorithm will at least ensure that you do not have to track the same bug
twice and repeat work you have already done. This is a algorithm to make
sure your projects only move forward, never backwards, and you do not
waste your precious time. 

\subsection*{Final note: Running tests overnight}
\label{sec:runn-tests-overn}

The good thing about automated tests is that you do not need to run
the tests yourself over and over again. You write the tests and then
tell JUnit to run them. Every day, you can tell JUnit to run every
single test that you have written so far. Serious professional
projects have thousands or millions of tests: no human being can do
all of that without dying of boredom. That is why automated testing is
important. 

Testing everything frequently ensures that the code does not go
backwards, e.g.~that introducing new features does not break other
features that were already working. This why it is sometimes called
\emph{regression testing}: it makes sure that the code does not
regress, i.e.~it does not go backwards. 

It is common for most professional projects to run \emph{all tests}
on a daily basis, usually overnight when the programmers involved are
sleeping\footnote{Well, most of them anyway.}. This is done by means
of an automatic script that retrieves the sources from a public
repository (e.g.~on GitHub), compiles everything, and then runs all
the regression tests. If anything is wrong, the programmers can fix it
as soon as they are back to their computers. Usually programmers run
a subset of all tests before pushing their changes, but a serious
project can have millions of tests that take several hours to run;
therefore, a full comprehensive test run is necessary every day to
keep projects on track. 

% How to design your tests
%   - check basic functionality
%   - check border cases
%   - Find bugs once
%     - the second test
%  @Test
%    public void trySG_S()
%      {
%         T t = new T();
%         String input = "Dereck Robert  Yssirt";
%         String output = t.getInitials(input);
%         String expectedOutput = "DRY";
%         assertEquals(output, expectedOutput);
%      }


%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End: