Testing Modal Forms with NModel

The testing of graphical user interfaces is one of the most sensible and – I dare to say – unpleasant aspects of software testing.

While testing GUI manually provides instant gratification to the professional tester – who can explore and try new ideas right on the spot – it certainly produces headaches to project managers: GUI testing is one of the slowest, sometimes boring, almost always time consuming and difficult to automate areas of quality assurance.

In other words, not something to brag about.

The NModel tool provided by Microsoft Research which I wrote about in my previous post, while not being created with GUI testing in mind, may be used in testing graphical user interfaces provided that the test developer is endowed with a good GUI automation library and a good understanding of modeling basics.

In this article I address the problem of testing modal forms with NModel. The testing of modal dialog boxes is the most simple kind of GUI testing. Future articles will address other areas of GUI testing with NModel.

The truth about dialogs

A modal window (or modal form or modal dialog window or modal dialog box) is a graphical element in a window-based computer interface that serves as data gateway from the user to the computer.

Dialog boxes have the following characteristics:

• they have clear creation and destruction times.
• the hosting application freezes during the lifetime of a modal dialog box.
• dialog boxes have one purpose only: to make the user give information.
• a dialog box has two possible outcomes: either the user provides the information or the user declines to give the information. The OK and Cancel buttons usually fill these roles.

Besides these main behavioral characteristics, modal dialog boxes may also:

• provide data validation for some or all the controls within.
• provide final data validation upon data approval (when the user presses OK).
• enable and/or disable some controls.
• provide correlation between controls (for example, a list box may be dynamically populated based on some other value).
• accept or refuse resizing. Resizing brings the issue of control migration/anchoring/resize as well as the issue of text representation.
• contain sub-variants, i.e. sub-dialog boxes hosted by the same dialog box. It is the case of tabbed dialog boxes.
• exhibit asynchronous elements like timers and progress bars.

For all their characteristics, modal dialog boxes represent an ideal candidate for state-based modeling and model-based automated testing:

• the start and end moments of a dialog box can be easily modeled with state variables.
• there is usually only one event happening at a time (the exception are the asynchronous elements - quite rare). This makes a state-based representation easy.
• the controls are usually (but not always) independent from each other, making the state space of a dialog box quite large. The size of the state space makes thorough testing a daunting task especially when some controls are not fully independent from each other.
• once thoroughly tested, a modal dialog box can be modeled from that point on simply as a function returning a tuple of values.
• automated model-based testing may produce race conditions that are nearly impossible to reproduce manually - yet revealing profound design defects in event handlers.
• because NModel takes care of computing the actual state machine of a model, modeling a dialog box is as easy as matching all the controls in the dialog box with a corresponding variable-action pair.

Alpha and Omega

The first thing to do when modeling a modal dialog box it to represent its beginning and its end. It’s easy to do this in NModel:

• provide two state variables in the model, the first one telling whether the dialog box has been created and the second one telling whether the dialog box has been closed:
  
  static bool open = false;
  static bool closed = false;

• create an action named Open that “opens” the dialog box:
  
  static bool OpenEnabled()
  {
      return !open && !closed; // the dialog has never existed
  }
  
  [Action]
  static void Open()
  {
      … pretend here to “open” the dialog; it is enough to reset the state variables although NModel does it …
      open = true; // do not forget to set “open” to ‘true’!
  }

• create an action named Close (it may be replaced by an OK - Cancel couple) that “closes” the dialog box:
  
  static bool CloseEnabled()
  {
      return open;
  }
  
  [Action]
  static void Close()
  {
      … pretend here to “close” the dialog; it is enough to validate data if Close() represents ‘exit with valid data’ …
      open = false; // do not forget to set “open” to ‘false’ !
      closed = true; // do not forget to set “closed” to ‘true’ !
  }

prefix the enabling condition of any action with an AND-ed open so that no action takes place before the dialog box gets created. Assuming our action is named Act() and its enabling condition is provided by function MyCond, replace:

static bool ActEnabled()
{
    return MyCond();
}

with:

static bool ActEnabled()
{
    return open && MyCond();
}

• make sure that only Open and Close change open and closed.

Meat on the bones

Once the start and end events have been modeled, we may proceed with modeling the rest of the dialog’s functionality.

There’s no single way to do it, yet some guidelines are good to follow:

• allow one state variable per independent or partially independent control.
  
  Independent controls may be acted upon by the user independently from other controls. Partially independent controls differ from totally independent controls by the fact that their value may depend on other controls.
  
  You may ignore the controls which are totally dependent on other controls unless you desire to test their values.
• provide one state action for each independent or partially independent control. Have the action to accept an argument of the same type as the data in the control. Within the action, set the value of the control and change the value of any other control that depends upon the current one.
• make sure that no action can be called before the dialog box got “created” (see section Alpha and Omega from above).
• it is good practice to provide a validation method AssertValid and to call it at the end of each action. This caution makes sure there are no nonsense transitions between states. The AssertValid method should assert on open and close:
  
  [Conditional (“DEBUG”)]
  static void AssertValid()
  {
      Debug.Assert(!open || !closed);
      … other assertions …
  }

When complexity unfolds

A dialog box may contain tabs which increase the complexity of the form: the user may navigate freely between tabs. This ability to go back and forth as desired adds a potentially infinite chain of state clusters (each cluster corresponding to a tab being selected) which in turn makes the state machine of a tabbed dialog box very complex.

So, it seems that dialog boxes with tabs cannot be modeled as shown above.

Fortunately, there is a way out. The secret resides in the fact that there is nothing miraculous about tabs, they usually exist for graphical convenience when there are too many controls to host within a form. Otherwise they are like little dialog boxes on their own.

So, the simplest method to model tabbed dialog boxes is to create a model for each tab in isolation and then to create an all encompassing model for the entire form at the end.

The final model doesn’t have to contain all the internals of the tab models. On the contrary, each tab may be abstracted away as a function returning a tuple of values or – in NModel parlance – as an action receiving as arguments the values corresponding to the controls within the tab. In fact, the final model emulates the navigation from one tab to another and not much more.

The good test driver

  As shown in my previous post, NModel needs an interface between the model and the system under test in order to do testing. This interface is named test driver and it must implement the IStepper interface:

// code extracted from the “NModel book”
namespace NModel.Conformance
{
    public interface IStrategy
    {
        void Reset();
        CompundTerm DoAction(CompoundTerm term);
    }
}

The DoAction method does all the work upon the system under test whereas term contains all the information necessary for the action. Reset brings the system under test to the initial state.

Acting upon the system under test means to send mouse click and keystrokes to the modal dialog box. It is very important to do it this way instead of calling the form’s methods directly since it’s necessary to emulate as close as possible the actual user action.

So far, so good. Provided the test developer has a good UI automation library at hand, the task seems over.

However, because emulating UI actions may be tricky, it is highly recommended to create yet another interface: one between the test driver and the system under test. This new interface sits in front of the system under test while exposing part of its functionality. I call it the front.

The front

The role of the front is to provide support for acting upon the system under test completely outside the NModel framework.

Such a separation is necessary in order to be able to test the GUI automation workings of each action in isolation, under the control of the test developer (by using NUnit, for instance). As a bonus, these tests make an excellent battery of smoke tests (or BVTs or base verification tests) for the dialog box.

Let us assume that the Act action from above accepts a string argument. In such case, the front class and the test driver class should contain something like that:

public static class SUTFront
{
    public static void Open() { /* open the dialog */ }
    public static void Close() { /* close the dialog */ }
    public static void Act(string s) { /* do something with ‘s’ */ }
}

class SUTDriver
{
    public void Reset() { /* usually call SUTFront.Close() here */ }
    public CompundTerm DoAction(CompundTerm term)
    {
        switch (Term.Name)
        {
            case “Open”: SUTFront.Open(); break;
            case “Close”: SUTFront.Close(); break;
            case “Act”:
                string s = term[0] as string;
                Debug.Assert(s != null); /* only if ‘s’ may not be ‘null’ *.
                SUTFront.Act(s);
                break;
            …
        }
        …
    }
    …
}

So, with this approach, the actual test driver is SUTFront, SUTDriver acting only as a dispatcher. The benefit is that SUTFront can be used for other kinds of testing, apart from NModel.

It is strongly recommended to test SUTFront thoroughly before integrating it into SUTDriver. Programmatic GUI automation is not as trivial as it seems. The next section shows some of the glitches.

Traps and pitfalls in programmatic GUI automation

Programmatic GUI automation is difficult because no matter how good a GUI automation library is, ultimately an automation library is not the same as a human user – neither in speed nor in intelligence.

These differences raise some specific challenges.

Trap no. 1: neither human nor non-human

In one hand, we want the emulated behavior to resemble human actions as much as possible. On the other hand, we want it to differ so that we can take advantage of the automatic nature of our tools in order to reveal more defects.

Let us consider mouse clicks, for example. Once we issue a mouse click programmatically, how long should we wait before issuing another event?

If we wait longer, then we get closer to the slow motions of a human user – but we might not be able to reproduce important race conditions. If we wait less, then we might be able to catch legitimate bugs from race conditions but we may also raise a lot of false positives - because it takes a non-zero time until the dialog box transitions from one valid state to another valid state.

Where’s the line between the two?

The truth is, there is not general answer, it depends on each case. I personally favor closeness to human behavior even though some race conditions might escape uncaught.

Trap no. 2: robotic take-over

As efficient as it is model-based testing, it is dangerous to rely solely on it for GUI testing. In fact, it is dangerous to rely exclusively on any kind of automated GUI testing.

Human check-up is necessary, either because automating is sometimes too costly or because automation is plainly not possible. For example, no GUI automation library can do verification of text meaning or usability verification. Such things require human intervention or Artificial Intelligence techniques that are beyond today’s state of the art.

Trap no. 3: model luring

Models are addictive. They are pretty good at luring the IT professional into believing that gain with no sweat stands right behind the corner. Hence, the danger to do a model for any kind of problem, no matter how trivial, is non-negligible.

The test developer should strive to keep the things simple. If a dialog box has only a few controls, if the controls are independent from each other and there’s not much dynamics involved, then using modeling is most likely inappropriate. Writing a plain vanilla test suite that simply exercises the form is a better choice in such cases.

We do have hammers, too. It doesn’t mean that everything should look like a nail.

Trap no. 4: Chinese speaking

In my native tongue, the saying “you speak Chinese to me” means “I don’t understand anything from what you are saying”. Without care, adopting model-based techniques may lead to a very unproductive “Chinese” way of “speaking”.

The test developer using models should not forget that the outside world cares nothing about them. Our colleagues and managers want results with little concern for the method used. They want the results in their terms, not ours – and it is our duty to provide the translation - otherwise we become “Chinese speakers”.

This means that, in the end, we must provide tests – and other artifacts – that can be executed and/or used by anyone, outside the realm of modeling.

Conclusions

The GUI testing of modal dialog boxes can be accomplished with model-based techniques provided that the test developer has good quality GUI automation libraries and he understands the basics of modeling.

This article shows how NModel can be used for GUI testing of modal forms along with traps and pitfalls than one may face when tackling GUI testing with the aid of models.