The Power to Shell

James Bach has written a blog post recently advocating for thorough software logging in order to help exploratory testing. In the same vein, Michael Bolton says in his blog that exploratory testing is accountable, if done properly.

With no doubt, proper logging helps. But what if we have to test completed, non-modifiable systems or if the development team can not or would not inject such kind of tracing in the code? A solution against such inflexibility is to resort to record-and-play tools that, if used judiciously, may make up an adequate framework for good, reproducible exploratory testing.

However, what if the system under test has no GUI elements (as is the case of autonomous libraries, services or drivers) and by consequence it does not support record-and-play tools? Can we still have exploratory testing for internal components?

My past experience says that exploratory testing is possible for internal components (even services or drivers) provided we have the right tools to do it. Those tools don’t have to be fancy or expensive, as shown below.

In this article I present my background on the problem and I suggest a way to do simple and elegant exploratory testing on non-GUI .NET components.

A little bit of history

When I was doing testing on CLFS (Common Log File System) at Microsoft, I was writing my tests in C/C++ as most testers in my group did. Quite frustrated by the fact that I had to write a complete C/C++ program every single time, even when all I wanted to do was to try some ad-hoc scenario, I asked myself: “isn’t there a better way?”.

It’s quite normal to ask such a question. Put plain and simple, you cannot do exploratory testing in C++, with its write-compile-run cycle. By the time you have the chance to “explore”, your mind is already emptied by making the code to pass compilation (if you are able to write error-free C++ code in one shot, this article isn’t for you). The result: in C++, spontaneity in testing is dead and dead is exploratory testing, too.

That was my situation: I wanted a tool that would allow me to exercise the system under test freely as the ideas were popping up in my head. But C/C++ was not such tool. The solution? Scripting.

So, I wrote a little COM Automation library as a wrapper over the CLFS APIs and, lo and behold, I instantly had the component’s functionality at my fingertips with all the freedom to try quirky scenarios right on the spot.

Obviously, not every kind of functionality was accessible from scripting. For instance, I couldn’t do multi-threaded testing upon the component (which was necessary for certain scenarios). But nearly 90% of the component’s functionality was perfectly accessible through scripting.

Moreover, as a bonus, my tests were truly open (being written in VBScript instead of being C++ executables) so when my developer tried to reproduce a bug by running one of my tests he was able to see what my test was doing and possibly add minor test cases before declaring the bug fixed.

I presented my little library and my script-based approach to my colleagues (testers of system components like me) and I encouraged them to “downgrade” their tools to a simpler programming language that would cover most of their testing needs. Some of them were intrigued, others were outright amused: “so, going back to Basic?”. “For 90% of the time, yes”, I said.

Let’s admit it: trying to make a C++ programmer write anything in Basic is blasphemous, almost like a personal insult. Yet, there are deep reasons to adopt such an approach in the case of test development.

Automated translators, information theory and software testing

Let us think a little bit of automated translators that accept sequential inputs and produce an output which is (information-wise) equivalent to the input (such as compilers). In all these cases, the input language is much more simple than the translator itself - even for very simple languages. See the Brainfuck language and imagine writing a translator for it.

The cause of this discrepancy resides in the fact that, in the case of sequential input, complexity manifests in length whereas in the translator’s case complexity manifests in states.

It’s basically the same information: but one is stateless and infinite whereas the other is state-full and finite (in terms of size). The potentially infinite complexity manifests in:

• a potentially infinite length, in the case of sequential input.
• a potentially infinite state space, in the case of the translator’s execution.

Due to the equivalence of these two forms of the same information, we don’t need a rich set of input words to challenge the inner execution of a translator. It is the quality and the length of the input sequences that count and not the number of keywords that the translator has to accept.

Let us move now from automated translators to other kinds of software systems. We can conceptualize most software systems as being large, unconventional translators: they accept commands (the “sequential inputs”) and translate those commands into actions, data transfers and so on (the "output”). By the same line of reasoning as above, the input of a system is much simpler (in terms of individual commands) than the system itself. Just think of Microsoft Word, whose inputs consists in keyboard strokes, mouse clicks and not much more – therefore are much simpler than the data structures and the code within Word.

Applying this view to software testing, it turns out that the commands used in testing can be (and should be) much simpler than the kind of processing happening in the system under test: it is not the fanciness of the commands or the price of the tools the we use in testing that matter. It is the quality and length of inputs sequences (i.e. test scenarios) that make all the difference.

Coming back to my CLFS testing, this means that you don’t need C++ to do high quality system testing upon components written in C++. In fact, a simpler programming language (like VBScript) that lets you quickly try things is preferable in most cases, being more productive and more fun.

The advent of .NET

Assuming I’ve made a strong case for test languages as simple as possible for testing, even when testing very complex systems, let us take a look at how the situation has changed with the advent of .NET.

In the pre-.NET era there was COM Automation as the natural gateway from simple to complex (that’s why I ended up writing my CLFS tests in VBScript). Once .NET has arrived both COM and Automation start to fade away (without disappearing completely) and as a sure sign of that Microsoft hasn’t produced a truly .NET-friendly variant of VBScript so far (registering .NET classes as COM interfaces and accessing them from VBScript doesn’t count, sorry). The path is closed, no questions asked.

Is there a viable alternative? Do we have in .NET something flexible, powerful yet easy to use as VBScript?

We have. It’s called PowerShell.

The power to shell: PowerShell as the test language for .NET

  The administrators of the Windows operating system have long envied their colleagues from the Unix/Linux world for the powerful shell environments those platforms enjoy. Being able to write functions in scripts, having pipelines and a myriad of tools to do any thing imaginable were features putting cmd.exe and its reduced set of commands truly to shame.

But Windows eventually did retaliate. Strongly.

PowerShell has many characteristics in common with the Unix shells: functions, pipelines and the like. But besides those, it has two huge advantages:

1. the commands are not standalone programs but embedded little modules named cmdlets, highly uniform in terms of parameters and very predictable in terms of behavior.
2. the commands do not work with textual information but with structured information - that is, they work directly with .NET objects (even when piping).

It is the second characteristic that can be fully exploited by .NET testing because one can instantiate from PowerShell, with no restriction whatsoever, any .NET class and can call any method while gathering any result. Hey, but this not any kind of scripting, this is truly test scripting for free!

So, what I propose in this article is to use, as extensively as possible, Windows PowerShell as the standard test language for .NET components.

PowerBlend: the tester and the admin are one

Using PowerShell as a standard testing language for .NET brings another benefit: the tester can leverage the huge potential represented by the administrative power of the cmdlets in order to configure various execution environments for particular test runs.

Imagine you have to test a little module that creates a certain kind of file within a certain directory. Writing tests that verify the file’s content is not enough. What if the user doesn’t have the right to write into that directory? What if the user doesn’t have the right to go at all into that directory? What if the directory is actually a network path that needs to be created on the fly? What if the user has lower priviledges? What if … ?

Without the administrative power of PowerShell, the test developer is left with two options:

• to hard-code the configuration sequence into the test itself (within the test body or as a Setup/Teardown sequence). This makes difficult to switch to another configuration with ease, if necessary.
• to run the whole test harness under a different configuration. This is very problematic, since it may break the test harness itself!

There is another reason to use PowerShell for such configurations: performing admin tasks from C++ or C# (i.e. hard-coding the admin part into the test) is much more difficult than doing them with PowerShell commands. PowerShell was created with system administration in mind right from the beginning. So it is the natural choice when thinking about altering the environment to run tests under unusual conditions. Why wouldn’t it be the natural choice to write the .NET tests themselves?

Conclusions

Because complexity is represented differently in the inputs of a software system compared to the system itself, it makes a lot of sense to use a programming language as simple as possible when developing tests - even for the most complex systems. In this respect, scripting languages bring the advantage of simplicity while allowing to do exploratory testing even in the absence of a GUI interface.

With the advent of .NET, the Windows shell becomes a fully fledged, powerful, flexible environment that can access, manipulate, call and pipe .NET objects natively. All these qualities make PowerShell an excellent candidate for testing .NET components with the added benefit that test administration and test execution may be handled by the same language under the same hood.

This article presented some arguments in favor of using PowerShell in .NET testing. Future postswill bring more details on how .NET testing can be achieved in PowerShell.