Open Source.NET

Continuous Testing: Think Different

• 09/07/2011

What we are really talking about in continuous testing is applying a lean workflow. By looking at how we can apply lean principles to our daily workflow, we can push the boundaries of what we're capable of. We constantly seek to:

1. Eliminate waste
2. Amplify learning
3. Deliver as fast as possible
4. Build in integrity
5. See the whole

Subverting the Dominant Paradigm

As programmers, we're constantly battling against interruption, trying to stay in a groove. Meetings, office chatter, bureaucracy and more all stand in our way. Many of us have done Test-Driven Development (TDD) or something close to it in an effort to be more productive. However, most of us, in our daily workflow, have missed the elephant in the room. That elephant is described in this quote from Steve Jobs:

"Your time is limited, so don't waste it living someone else's life. Don't be trapped by dogma — which is living with the results of other people's thinking. Don't let the noise of others' opinions drown out your own inner voice. And most important, have the courage to follow your heart and intuition. They somehow already know what you truly want to become. Everything else is secondary."

Whether by conditioning or obedience to dogma, our workflows will blind us if we don't challenge them. What's proposed here is very different from what most .NET developers are used to -- but it's worth exploring the benefits to see if it can change your thinking and productivity.

Continuous Testing

When applying TDD, in whichever fashion, we're continually compiling and writing tests. We've accepted the overhead of compiling and running our tests and get into a cycle of code-->build-->test.

To streamline this process, there are (at least) three OSS projects for .NET developers that are worth taking a look at.

1. Giles
2. SpecWatchr
3. AutoTest.Net

Each of these tools takes a different approach, and each is valuable. Each watches your code for changes and then compiles and runs your tests automatically. To provide feedback, they all hook into Growl, a notification system.

Giles

Giles is a .NET command-line tool created by JeffSchumacher. It's very opinionated about how it works, but the defaults can be overridden if needed. You can download Giles from Nuget:

>nuget install Giles
>Install-Package Giles

Out of the box, Giles supports xUnit, NUnit, and Machine.Specifications (MSpec).

Opening the Power Shell console, you can execute giles.ps1 and it will try to use the name of the current directory to find the default solution. If you use a different convention, you can specify the relative path to the solution file as well.

>.\giles.ps1
or
>.\giles.ps1 -s src\YourSolutionName.sln

At which point Giles will greet you and offer you some options:

Giles - your own personal watcher
"I'd like to test that theory..."

Interactive Console Options:
? = Display options
C = Clear the window
I = Show current configuration
R = Run build & tests now
V = Display last test run messages
B = Set Build Delay
Q = Quit

Giles will prompt you for a command to run, or automatically build and try to run your tests when you save. You can press “R” to start a build/run task. If everything is configured correctly, you should see MSBuild kick off and run any tests you have in your solution.

You can change the delay between saving and building by using “B” and then entering the delay in milliseconds. Remember that you need to press “Q” to quit, as Ctrl+C is ignored.

You can create a simple set of specs and watch Giles run them for you very quickly.

class EmptyListSpecs

  {

  static List<int> list;

  Establish context = () => list = new  List<int>();

  It should_have_a_count_of_zero = () =>  list.Count.ShouldEqual( 0 );

  It should_return_minus_one_when_getting_the_index_of_an_item  = () => list.IndexOf( 5 ).ShouldEqual( -1 );

  }

And Giles should print out:

Building...
Build complete in 0.4159754 seconds. Result: Success

=======  TEST RUNNER RESULTS =======
MSPEC Results: Passed: 2, Failed: 0, Ignored: 0
Total Passed: 2, Failed: 0, Ignored: 0

One thing you should notice is how fast this happens. From saving the code to receiving feedback, you'll begin to change how you code. Remember to not wait for the compiler or results; just keep coding. Pause when you see a test brake, then just fix the test and continue.(If you start watching the output window and waiting for results, you'll end up hurting your workflow.)

SpecWatchr

SpecWatchr is a set of Ruby scripts written by AmirRajan and MattFlorence that leverages the Watchr Gem to track changes to your code, triggering builds and automated tests.

There are a few dependencies you'll have to make sure are installed, since this is a Ruby package:

1. Install Ruby 1.9.2 forWindows.
2. Install the watchr ruby gem (gem install watchr).

You can install SpecWatchr via NuGet:

>Install-Package SpecWatchr

The SpecWatchr will add a few files to your solution directory. To configure, open dotnet.watchr.rb. The configuration file will guide you through the various sections. Out of the box, it supports MSBuild and Rake for compiling and NSpec, NUnit, and MSTest for unit testing.

SpecWatchr takes an opinionated stance on convention and will guide you depending on which builders and runners you choose. Once configured, open a console window and enter:

>watchr dotnet.watchr.rb

SpecWatchr should now be up and running. If you're not seeing your tests running, read the output messages; they're very helpful. One difference you'll see between Giles and SpecWatchr is that SpecWatchr tries very hard to only run the affected tests, and it determines which files trigger test runs via convention. Every time you start SpecWatchr, it reminds you of the convention used for your chosen test runner.

You can create a simple set of NSpec specifications and watch SpecWatchr execute them:

namespace DataStuctures

  {

  public class Deque

  {

  public int Count { get { return 0; } }

  public bool IsEmpty { get { return Count ==  0; } }

  }

  }

namespace  DataStructures.Tests.describe_Deque

  {

  class DequeSpecs : nspec

  {

  private Deque deque;

  void given_a_new_deque()

  {

  before = () => deque = new Deque();

  it["count should be 0"] = ()  => deque.Count.should_be( 0 );

  it["isempty should be true"] =  () => deque.IsEmpty.should_be_true();

  }

  }

  }

Save the file and SpecWatchr outputs the results:
====================== changed: DataStuctures/Deque.cs ====================
"C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" "./DataStuctures.sln" /verbosity:quiet /nologo

=========== running spec: Deque =====
"./packages/nspec.0.9.50/tools/NSpecRunner.exe" "DataStuctures/bin/Debug/DataStuctures.dll" "Deque"

DequeSpecs
given a new deque
count should be 0
isempty should be true

2 Examples, 0 Failed, 0 Pending

Head on over to the SpecWatchr Web site for more great walk-throughs and video .

AutoTest.Net Originally started by JamesAvery, and picked up by SveinArneAckenhausen, AutoTest.Net (Figure 1) is based on the Ruby tool autotest. It brings the same type of functionality to .NET.

If you're craving the fully-integrated setup, you can explore AutoTest.Net. Head over to the download page and run the latest installer. AutoTest.Net has the widest test framework support with NUnit, xUnit, MSpec, MbUnit, and MSTest. If you want to stay away from Visual Studio integration, you can also run it from the command line.

An easy way to get started with AutoTest.Net is to launch AutoTest.WinForms.exe. From there, specify the directory with your solution and click OK. It will now monitor the directory for changes and run your tests automatically.

If you have Growl installed, it will display progress notifications as is detects changes and runs your tests. If a test fails, it will show up in a list. You can see the details by selecting the error and pressing 'i'.

[Click on image for larger view.]

Figure 1. AutoTest.Net

To use AutoTest.Net in the console, you can run a command-line like this (from the Power Shell Console):

> & .\packages\AutoTest.Net\AutoTest.Console.exe "$pwd"
[Info] 'Default' Starting up AutoTester
[Info] Tracker type: file change tracking
[Info] Starting AutoTest.Net and watching "C:\dev\DataStructures" and all subdirectories.
[Info] Ignore patterns:
[Info] 'AutoTest.Console.ConsoleApplication'
[Info] Preparing build(s) and test run(s)
[Info] Ran 1 build(s) (1 succeeded, 0 failed) and 2 test(s) (1 passed, 1 failed, 0 ignored)
[Info] Test(s) failed for assembly DataStructures.Tests.dll
[Info]     Failed -> DataStructures.Tests.EmptyListSpecs.should_have_a_count_of_zero:   Expected: [1]
But was:  [0]
[Info] 'AutoTest.Console.ConsoleApplication'
[Info] Preparing build(s) and test run(s)
[Info] Ran 1 build(s) (1 succeeded, 0 failed) and 2 test(s) (2 passed, 0 failed, 0 ignored)

Either way, getting up and running is very simple.

To summarize this approach of continuous testing: Discard dogma, think big, act small, fail fast, learn rapidly.