“Inversion of control” and “dependency injection” are the 2 terms you can research for more information on this topic. I am going to talk in layman terms, only, as to why I think software test automation developers (or all developers) should be aware of this techniques. It is another very useful tool for your toolbox.

My first experience with IOC and dependency injection was with java, using the Spring framework a while back. More recently I have used Spring.NET, the .NET version of Spring. I am currently using Spring.NET and Unity, although there are many flavours of IOC containers out there.

For those of you who don’t have a clue what I am talking about, imagine a washing machine and a dishwasher. They both do the same job, washing, but go about it in very different ways. Although the idea of a combo washer/dishwasher has been patented, it is highly likely that for the foreseeable future we require 2 machines. They even have very similar user instructions,

1. Load machine
2. Insert detergent
3. Select program
4. Press Start

Imagine that you indeed had 1 machine that was a washer / dishwasher combo. Imagine that the hardware for the machine was sufficient for all types of washing. Imagine the machine had a card slot that accepted a card that had the software to execute for the type of washing you wanted to do. You inserted the “Dishwasher” card into the slot and the machine washed dishes. You didn’t care how it worked or what was under the shell, you just washed clothes and dishes according to the card you put in the machine. In a years time a new card came out for your machine that allowed it to wash small pets; No more taking your Chihuahua to the grooming parlour once a month. Put the new card in the machine, pop your dog in and hey presto, your beautifully clean, although mentally scarred, best friend appears 30 minutes later. That is inversion of control. The machine supplies the rules of how to wash and any implementation that follows the machines rules may be implemented as a new “card”. You configure the machine (put the card in) and it goes off and gets the implementation to execute.

So IOC, in my mind, is the idea of decoupling rules from implementation. Dependency injection provides one way to enable this. Hopefully you can already see how this may be useful to you. Let me now give you a more concrete example.

In my series Create Test Automation Framework I show how to create an automated test framework from the ground up. Currently it uses QTP as the test tool, or “engine”. In a future series I am going to modify the test application I provided for the above series to allow users to run the same tests against different “engines”.

Say your company had API, UI and database tests. These tests may target the same functionality, but the type of testing is different for each.

If you adopted a test language for your company then you could use the same language to execute tests using different engines,

The above steps all use QTP (4th column). So all steps are executed as UI test steps. If you changed column 4 in step 2 and 3 to API then these steps would be run as API test steps. We use the same language, but our configuration (API, UI, DB) tells the application what “engine” to use. IOC and dependency injection make this easy.

There are various types of dependency injection and I tend to prefer interface injection. It lends itself very well to unit testing and mocks in particular.

So the test language defines the rules by way of providing interfaces. Each engine that comes along implements functionality according to the provided interfaces. This would also mean you are not tied to a specific implementation. If your company suddenly decided that QTP was no longer the UI tool of choice, you would create a new assembly with your newly adopted UI tool, update the configuration files to point to the new assembly and away you go.

Hopefully that's enough to illustrate that these techniques are worth knowing and can provide huge benefit to not only software development projects, but software test automation efforts as well.

If you use Visual Studio 2003, 2005, 2008 or 2010 then you have to use resharper. It’s my favourite plug-in. You can find out more about it here on Wikipedia. It does require a licence, but I think its worth it.

A quick summary from Wikipedia,

The following is an incomplete list of ReSharper features and feature groups:^[1]

• Over 750 on-the-fly code inspections that can be displayed as errors, warnings, suggestions, or hints^[2] (including compiler errors and warnings)
• 550 quick-fixes - automated corrections to resolve errors and optimize code
• Navigation and search: view file structure; search for any code symbol, type or file in solution; search for symbol usages; quick navigation to usage, declaration, implementation, inheritors, base types etc.
• Code Completion and Parameter Info that extends Visual Studio's built-in IntelliSense
• 35 solution-wide code refactorings
• Over 150 context actions - "local refactorings"
• Code generation: generate code from usage; generate common type members, including properties, constructors, interface implementation etc.
• Support for unit testing using NUnit and MSTest

There are personal and commercial reasons why I like it.

I love keeping up to date with the latest technology, .NET Framework updates and C# updates. This can be time-consuming and actually is quite hard to keep up with. Especially if you are committed to project work. So the refactoring help Resharper provides is great. You can write some code in the way that you normally would and Resharper will offer to re-factor it if required, using the latest techniques available. There are loads of examples on the net, so your favourite search engine will reveal many examples.

On a commercial level it is a perfect solution to enforcing code standards. Configure resharper to suit your teams coding standards. Your developers then load the standardised resharper configuration. They code away and resharper keeps them on track to make sure that they adhere to specified standards.

Note: I am not affiliated with Jetbrains and do not receive any money for affiliate links or advertising.

Part 1bdetailed tests, keywords and showed that the Test Editor is essentially a GUI for a file structure. To create keywords to be used in Test Editor, simply create a keyword file and put it in the keywords folder. When loaded Test Editor will load all keywords it finds at the specified path. We learnt the structure of Tests objects, how they are stored and how they are loaded into to the test grid. Now, I would like to digress with some theory.

If you are not interested in test automation project management and are only interested in doing the practical steps in this series, then please continue to the next post. This post is aimed at new test automation projects, or QA developers new to test automation. However, if you can add anything then please don’t hesitate.

If you’ve spent any time reading up on test automation, either in books or on the internet, you will have seen the phrase,

“proper test automation is NOT record-playback. data-driven test frameworks are the way to go!”

Very true, but why exactly? Record-playback produces brittle tests, yes, but i doubt any failed record-playback test project carried out any sort of upfront analysis before recording a single test. I believe that this is the main reason why any test automation project may fail. A test automation project has to be managed correctly. Progress should be transparent and the test automation team should be just as accountable as any other team in the software development lifecycle.

Here’s a high level list of the minimum activities that I believe should be undertaken in any TRADITIONAL test automation project,

1. Create test automation project plan – High level
   • Identify the scope of test automation
   • Identify business processes
2. Create keyword files
3. Update project plan
4. Develop business process functions & utility functions
5. Create tests
6. Execute tests
7. Review results
8. test framework, test maintenance

An agile test automation effort is unlikely to follow the above. It is more likely that in an agile team, a QA Developer will work closely with the testers to help identify areas where test automation can gain quick wins using any method that is, easily available, quick to implement and as cheap as possible. The test automation efforts are done in short repeatable bursts. Also, a lot of agile test automation efforts relate to low-level testing like TDD, automated unit / integration tests and tests that will be integrated into Continuous Integration(CI). The traditional test automation project usually targets integration / regression cycles. In any type of project CI and running tests after a nightly build should be a top priority.

The activities may overlap. The degree of overlap is proportional to the size of the team. For a 1 person team, there may be no overlap. As the team increases in size, it may be that activities are staggered; Business process functions may be developed whilst keyword files are still being created. You may also decide to repeat the above steps for each business process area one at a time, or simultaneously, if you have the resource.

1. Create Test Automation Project Plan

What we want to do here is breakdown the areas of functionality in your target application. These are your business process areas. Next, for each business area we want to identify suitable tests to be automated. This can be achieved using any or all of the following,

• Analysing a set of manual tests
• Analysing requirements specification documents
• Analysing the target application
• Being proactive and talking to BAs and manual testers

Once we have a list of tests for each business process area, we need to evaluate each tests automation suitability. You’re looking to automate tests that are repetitive, reusable and target core functionality. These 3 things alone can help you prioritise the order of test creation.  Actually, this same process can be applied to prioritising keyword creation. Think of rating each test / keyword / business process area out of ten for each category (Let’s call this a PRIORITY INDEX),

Keyword	Repetitive	Repeatable	Core	Total
Fight_Login	10	10	10	30
Flight_FaxOrder	2	8	4	14
Flight_AddOrder	8	9	10	27

You get the idea. We do Login first. The general theme I adhere to when performing analysis is to break everything down into their smallest component parts. This helps especially with complex applications that I may not initially understand.

Tests that may not be suitable for test automation may include tests that are not performed very often, hard to maintain and/or very complex. Let automated tests do the boring, easy, repetitive stuff and leave the interesting stuff for the manual testers. 100% test automation coverage sounds cool, but breath easy manual testers. You will never be replaced.

So your initial project plan should contain level 1 estimates for work to be carried out. As you can see above the plan may take days to do, or even weeks. Spend time here. Your project needs to start on a solid footing. Before you go near any test tool you should be very clear on what it is you and your team will be working towards. Working towards, because you will be working to a plan that you can track, update and broadcast progress of.

Doing this analysis will also highlight how work may be divided up when it comes to delegating work to test automation team members.

Questions you might like to ask at this point,

1. Does the project have CI in place? If not get the ball rolling to get it in place.
2. Your assets need to be stored in source control. Do you have access? if not, get it.
3. Ask your developers about including your tests in a nightly build script
4. Do you have access to a test lab to execute your tests? If not request the required hardware, VMware now as it may take some time to organise.
5. Do you have access to Defect Management software? if not get your project set up in your companies tool of choice.
6. Do you have the required licences for any test management / tool software that you have chosen?

2. Create Keyword Files

By the time you get to this stage you should have a decent plan that lists all required keywords. Each keyword required should come with the required parameters it needs to perform it’s function. These parameters can be changed in this phase, but remember to update the plan.

A keyword file for the flight application login keyword may look like this,

<?xml version="1.0"?>
<Keyword>
    <Name>Flight_LO_Login</Name>
    <Description>Logs into QTP sample flight application</Description>
    <ReturnValue>None</ReturnValue>
    <Category>Flight</Category>
    <Param>
        <name>Username</name>
        <description>Username</description>
        <type>String</type>
        <defaultValues>admin</defaultValues>
    </Param>
    <Param>
        <name>Password</name>
        <description>Firstname</description>
        <type>String</type>
        <defaultValues>MERCURY</defaultValues>
    </Param>
</Keyword>

We haven’t touched on the QTP side of things yet. Look at the Name of this keyword; Flight_LO_Login. What’s going on? The naming conventions for keywords and tests is something you need to consider. In this example, I’ll break down this naming convention,

Flight_LO_Login

Flight	Target Application Identifier (maps to target application assets location)
LO	Object Repository Identifier (object repositories may be loaded dynamically
Login	VBScript Identifier (Uses whole name to work out location)

So when this Keyword is encountered by the QTP driver script it knows what assets to load and what to execute by parsing the keyword name.

There are numerous approaches to naming conventions. This is just the one I’ve chosen for this project.

3. Update Project Plan

By rights you should already have updated the plan by the time you get to this step. When you initially created the plan you may have included estimates for how long this work will take. You should always keep an eye on the accuracy of your estimates. You started creating keywords in order of priority and after creating say ten, you will have some actual times to compare against your estimates. maybe you were able to create keyword files quicker than expected or maybe the initial parameters you gave to some keywords needed more analysis and so you took longer. if it took you longer then adjust your plan. This may be adjusting estimates, extending milestones or adding extra resource.

4. Develop Business Process Functions & Utility functions

Right, lets get coding!

You will now have in front of you a list of keywords that require implementing. Remember,

Keyword = Business Process Function = A specific action that applies to your target applications functionality

These keywords will have been prioritised using any method you choose, or using the priority index I showed above. The repeatable rating is useful in that if we do more of those early on then this will give us early quick wins. Some people say that test automation can be a huge upfront effort and takes a long time before you start to see any ROI. Well, if you prioritise correctly then you can start to see results with as little as 10% coverage. Do the easy, repeatable BPFs first.

You will find that your BPFs may rely heavily on utility functions. It is likely that as you start to create BPFs you will require a piece of functionality that helps your BPF do its job. Think of utility functions as private methods and your BPF as the public method. You will require navigation utilities, error handling, reporting, formatting, logic utilities. These need to be created either upfront in a prioritised way, or in tandem with creating BPFs. Whichever suits you best. Probably the most important utilities will be the data utilities. These utilities will fetch the lists that you would like to reference in your keyword parameters (see previous post).

5. Create Tests

At this point you have created keyword files and the corresponding VBScript to execute when the keyword is parsed by the driver script. The keyword files have been stored in the keywords repository known by Test Editor and so are available to be used during test creation. You have a prioritised list of tests to create. If possible, start by trying to create a set of tests that may be used as a sanity pack (smoke tests); a small set of tests that touch the core parts of your target application. Once you have this you can start executing it after builds. The beginning of your ROI.

6. Execute Tests

The exciting bit!

At this stage you might start becoming familiar with the quirks of your target application. Problems with tests not doing what they are supposed to, may cause you to do quite a bit of investigation and perhaps identifying issues with BPFs or identifying new utility functions that may be required. Or, your tests will run beautifully and you will be filled with a warm glow.

7. Review Results

Check failures to see if defects are as a result of the target application or your tests. If they are a real defect then raise a bug. When you log your bug note the automated test that should be executed when the bug is fixed. If you can automate the process of executing automated tests when a bug is fixed, then even better.

After a while managers may (will!) start becoming interested in metrics. So be prepared to write utilities to extract information they require, if metrics is something that cannot be easily extracted from your test management or defect management software.

8. test framework, test maintenance

You will always find ways of enhancing your test automation framework. Create a backlog and prioritise work accordingly.

Test maintenance is a funny one to me. Maybe I’ve been lucky or maybe, just maybe, it’s because I invest a reasonable amount of time in carrying out thorough analysis upfront that results in me not having to spend a lot of time performing test maintenance. I read a lot about people saying that test maintenance is extremely labour intensive and I can see why. Tests may be brittle due to too much static data, references to hard-coded data that is dynamic, keywords not aligned with business process etc.

Ideally, if you are using your automated tests for regression tests, they should not change. Ensure your tests are version controlled.

Ok that’s a very high level view on some of the things I think may be beneficial when getting involved in a software test automation effort. It doesn’t list all techniques, but may point you in what I think is the right direction.

Next we’ll start looking at the test tool side of things.

Create Test Automation Framework – Part 1b

In Part 1a, I mentioned that the Test Editor had 3 inputs and 1 output for a given application under test (AUT). Let me explain using code. This is how I load Test Editor,

private void Form1_Load(object sender, EventArgs e)
{
    library1.ActualRootPath = @"c:\TestEditor";
    var keywordPath = library1.ActualRootPath + @"\keywords\";
    var testsPath = library1.ActualRootPath + @"\Tests\";
    var resourcesPath = library1.ActualRootPath + @"\Resources\";
            
    library1.LoadTestCategories(testsPath);
            
    library1.Grid.TestRootActual = testsPath;
    library1.Grid.LoadKeywords(keywordPath);
}

Test Editor is a blank form that contains 1 user control, library1. This control contains standard controls such as Treeview, ListView, DatagridView as well as another custom control, which is the Grid where you edit test steps and add keywords. So this could be used standalone, if required.

The first thing I do is tell TestEditor where the physical root path for my application(s) assets are stored. This path maps to the “All Plans” root node in the Test Categories treeview in Test Editor. So all tests within sub-folders of the root path are loaded into Test Editor.

Next, I specify the locations for the 3 inputs. These are currently hard-coded, but I will move it all out into config files. Then I won’t be forced to store multiple application assets under the root path. It is highly likely that you would want to work with multiple application test files that are stored in various locations. For example, API tests will not be stored in the same place as QTP tests, but you may wish to create a test that uses keywords from both frameworks.

Finally we populate the Tree with test categories and load the Grid with keywords, filtered by Application. I’ll discuss Resources in a bit, when we talk about keyword files.

Tests

When you select a test category in the tree the tests that belong to that category are displayed in the list. When you select a test in the list, Test Editor will work out where the test lives and will de-serialise the test file, which has a .test extension. Or in simpler terms, it will load the .test file into the Grid. The test is de-serialisedinto a Test object that has these properties,

public string Name { get; set; }
public string Author { get; set; }
public string Description { get; set; }
public string Path { get; set; }
public List<TestStep> Steps { get; set; }

Pretty basic. A TestStep is another object that a test may have many of. It is also basic and contains only a few properties,

public int Index { get; set; }
public int LoopCount { get; set; }
public string Engine { get; set; }
public Keyword Action { get; set; }

A test step only has 1 keyword,

public string Name { get; set; }
public List<KeywordParameter> Parameters { get; set; }

Name is the name of the VBScript method and the keyword may contain many parameters.

So to load tests I de-serialise the test into a Test object that is then bound to the Grid. To save a test I serialise the Test object into a .test file. The Test object will be created on the fly using whatever data currently exists in the Grid.

Keywords

Clearly without keywords you cannot have tests. Keyword files are .xml files that may look something like this,

<?xml version="1.0"?>
<Keyword>
    <Name>flight_AddOrder</Name>
    <Description>Adds order successfully</Description>
    <ReturnValue>None</ReturnValue>
    <Category>OR</Category>
    <Param>
        <Name>Date (Format: mm/dd/yy)</Name>
        <Description>Date of Flight</Description>
        <type>String</type>
        <defaultValues>today+1</defaultValues>
    </Param>
    <Param>
        <Name>FlyFrom</Name>
        <Description>Fly From</Description>
        <type>List</type>
        <defaultValues>Denver, Frankfurt, London, Los Angeles, 
    Paris, Portland, San Francisco, Seattle, Sydney, Zurich
    </defaultValues>
    </Param>
    <Param>
        <Name>FlyTo</Name>
        <Description>Fly To</Description>
        <type>List</type>
        <defaultValues>Denver, Frankfurt, London, Los Angeles, 
    Paris, Portland, San Francisco, Seattle, Sydney, Zurich
    </defaultValues>
    </Param>
    <Param>
        <Name>Flights</Name>
        <Description>Flights</Description>
        <type>String</type>
        <defaultValues>1</defaultValues>
    </Param>
    <Param>
        <Name>Name</Name>
        <Description>Name</Description>
        <type>String</type>
        <defaultValues>Shawn Barrett</defaultValues>
    </Param>
    <Param>
        <Name>Tickets</Name>
        <Description>Tickets</Description>
        <type>Integer</type>
        <defaultValues>1</defaultValues>
    </Param>
    <Param>
        <Name>Class</Name>
        <Description>Class</Description>
        <type>List</type>
        <defaultValues>First, Business, Economy</defaultValues>
    </Param>
</Keyword>

A very basic schema.

• Name: is the VBScript method name.
• Description: This is displayed as a tooltip when the keyword is added to the Grid
• Type: This tells the Grid what kind of control to add to the cell in the DataGrid when added to the Grid.
• DefaultValues: This functionality allows you to specify default values to add to the cell when added to the Grid. These values could be static string values like the Flights parameter above. They could be dynamic values like the Date parameter above; today is a special value that will generate the value of today at runtime. This makes tests less brittle. So you could have values like today+1 or today-5. You may also choose to display a list of values as your default value, as in the FlyTo parameter above. All of these make the creation of tests a lot easier and more intuitive.

Let’s talk about Resources. Default value lists are a great way to help users create tests. But we don’t really want to hard-code these values in keyword files as they may change. Take the FlyTo parameter,

<Param>
    <Name>FlyTo</Name>
    <Description>Fly To</Description>
    <type>List</type>
    <defaultValues>Denver, Frankfurt, London, Los Angeles, 
    Paris, Portland, San Francisco, Seattle, Sydney, Zurich
    </defaultValues>
</Param>

It would be better to store the list of destinations in a separate file and reference it in the keyword file instead,

<Param>
    <Name>FlyTo</Name>
    <Description>Fly To</Description>
    <type>List</type>
    <defaultValues>Resources/Destinations</defaultValues>
</Param>

Destinations would be Destinations.xml stored in the Resources folder whose path was defined in the Form.Load event,

var resourcesPath = library1.ActualRootPath + @"\Resources\";

These Resources could be cached or generated when required. Logic may be applied to data to filter on options chosen at runtime. Say you had a VBScript keyword called ClickField(screenid,fieldid)

When added to the Grid the screenid parameter cell would be populated with a list of all screens retrieved from Resources/Screens.xml. We couldn’t populate fieldid until we had selected a screen. So the screenid combobox would fire an event when a screen was selected by the user. This would cause the fieldid list to be populated by fields that belonged to the chosen screen. The fieldid parameter may look something like,

<Param>
    <Name>Fields</Name>
    <Description>Fields for a selected screen</Description>
    <type>List</type>
    <defaultValues>
      Resources/Fields where screenid = [cell-1]
    </defaultValues>
  </Param>

This says go and get all fields where (screenid = the current cell columnid – 1). It would get the screenid from the cell in the previous column and find a file called [screenid].Fields.xml. For example AddOrder.Fields.xml

Ok, next we’ll continue with some analysis theory and try and understand why we’re using a UI like this instead of traditional spreadsheets.

Create Test Automation Framework – Part1a          Create Test Automation Framework – Part 1c

If you are following my Create Test Automation Framework series you will have downloaded and installed the Test Editor application. If not it can be downloaded from here

I have not yet supplied source code for this application and some users may want to look at the code, either out of curiosity or wanting to expand on the codebase for their own use. They would definitely not simply be wanting to copy and publish the code as their own work :)

So for those who are not already aware, let me introduce Reflector

It is a free download and it allows you look at the code for any given .net assembly. I encourage you to download it and use it to look at my Test Editor application. The important bit is that the code you will see is unlikely to be the code I wrote when developing the application. The C# compiler will have optimised my code.

It is a great way to understand what the compiler is doing and also to look at code for assemblies of interest to you.

Important: I have given express permission to dissemble my application, but that may not always be the case. Please check out the licence terms of any software you would like to dissemble before proceeding. If the licence prohibits it then it is illegal.

 

In Part 1 we'll detail the User Interface for creating tests. At this point forget QTP, or any other test tool. The creation of an application like this, is independent of and does not rely upon any test-tool-specific functionality. An application, or UI, for automated tests has inputs and outputs that should be configurable. The main input is the target application inputs. There are 3. These are the location of existing tests, the location of keyword files to be used in tests and any assets (data, object metadata) for a specific application. The main output is the created tests. In our case this is xml. The schema of resultant xml files may be application specific, but in our case it should be a fairly basic schema that could be applied to most, if not all applications. When a test tool is introduced, the framework that drives the test tool has to comply with the xml schema output by the test editor UI.

There are 2 main components to Part 1. Firstly, I’ll walk you through the Test Editor sample application and explain how to perform core functions. Next I would like to touch on test automation project design and analysis. It may seem logical to get all the technical stuff out of the way first and then talk about how to use this newly created test framework. The reason I choose to do this upfront is that once you understand what I’m showing you in this part, you can start creating tests straight away. It will also add perspective to the entire project. This may be beneficial to users new to test automation project design and management.

Prerequisites

1. Test Editor sample application. This comes with a few sample files
    based on the sample Flight application that comes with QTP

2. Please visit the HP website and install an evaluation copy
    of QTP, if required. We won’t be running tests, but you
    may need QTPs sample flight application. If you don’t
    want to waste evaluation time (currently 14 days) then
    don’t run QTP until we need to.

After installing Test Editor it should open automatically. If it is closed you can open it by clicking

Start -> All Programs -> Shawn Barrett -> Test Editor.

A standard flow for using Test Editor may be,

1. Create / Open test category in tree on the left hand side. All tests within selected test category
    are displayed in the right hand top list

2. Select a test to load test details into test grid below for editing. To create a new test either, enter
    new details into blank test grid and click save or, load an existing test, change the test name, edit details
    and click save.

3. Test steps may be added, removed, moved up and down by using the toolbar buttons.
    You can cut, copy, paste test steps between tests or within the same test.
    a. Adding Test Steps
        The list on the bottom right contains keywords to be used by test steps. Keywords
        are application specific and can be filtered using the drop down above the list. Currently
        we have QTP keywords. Each keyword in the list has a help balloon to the left of the
        keyword name. This shows information about the keyword when you hover your mouse
        over it. To add a keyword to your test select the row in the grid where you would like to
        insert the test step (keyword) and either double-click the keyword or use the toolbar buttons
        to the right of the test grid.
    b. Removing Test Steps
        Delete one test step by clicking on the relevant test step and clicking the cut toolbar button
        next to the test grid.
        Delete multiple test steps by clicking the first step to delete, dragging your mouse to the
        last step to delete, before letting your mouse go. Select the Remove toolbar button.
        Delete a selection of test steps by holding down your Ctrl key and clicking the test
        steps to delete. Click Cut.
    c. Editing Test Steps
        Single or multiple steps may be selected and moved up or down using the toolbar
        buttons next to the test grid. Inspect the context menu that appears when you right-click
        a selection of test steps.

Test step columns are made of mandatory fields and dynamic fields. So looking at the image above,

Column 1: Test Step index
Column 2: Loop times. How many times you would like to execute this step
Column 3: Execute button. This is a placeholder for future functionality we’ll add to execute test
                steps directly from the test editor
Column 4: This is a placeholder for future functionality we’ll add to specify which test framework we
                would like to use to execute the current step. It is highly conceivable that you may want to execute
                some API test steps before running a QTP step to verify UI state.
Column 5: Keyword name. This name will exist in the list on the bottom right. It maps to the VBScript
                method that will be executed when this test step is parsed by the QTP framework.
Column 6 to n: A keyword may require any number of parameters in order to carry out its action. 
                Therefore the number of parameters is not fixed. “Login” has 2 parameters, but “AddOrder” has 7. 
                So the given parameters for each keyword are supplied to the keyword VBScript method .

4. Add test details and save edited or newly created tests.

That is a brief overview of how the application may be used. Next we’ll dive deeper and I’ll show how we
create the keywords and make them available to the Test Editor. We’ll examine the structure of tests, keyword
files and I’ll explain what exactly is under the UI of the sample application. I’ll also provide a brief comparison between this user interface and traditional spreadsheet usage.

Please e-mail me if you have any questions, issues installing or you receive any errors when using the application.

 

 

Create Test Automation Framework – Introduction         Create Test Automation Framework – Part 1b

1. Create tests
2. Add test definitions to QC automatically.
3. Schedule and execute tests in QC
4. QTP is invoked and runs specified tests
5. Test results are uploaded to QC

The above diagram outlines the high-level architecture for the test automation framework we will create. I have chosen to use QTP as the underlying test automation "engine". Why? Typically keyword / data driven QTP frameworks are pretty similar, but in many cases the front end, to create the tests, is excel. That in itself is not necessarily a bad thing, but test templates may not be intelligent. Standards in test creation may not exist or test steps require manual entry. Experienced users of such environments rely on copy and paste when creating new tests. Data to populate the test step parameters may not be data driven or difficult to maintain. Using XML as a test format gives us some nice advantages. We save more than half on disk space (I use compression and serialisation in this project). We can diff files in source control (This project will deal with source control later) and more. Converting from XML to XLS(X) is very easy and this can be done if you have an existing QTP test schema that you use. The front end is written in C# (.NET 3.5) and I use Visual Studio 2008.

We will make Quality Centre (QC) the prime candidate for scheduling test execution. This will give you an idea of how to use the QC API. I will also show how we can run tests directly from the test editor (1). This will detail QTP API usage.

We will make use of VBScript to create the QTP framework. This will contain a driver script that will retrieve tests created in the Test Editor when a run is triggered in QC. Each test will be parsed and for each row in the test a VBScript call will be made to execute a VBScript that may be interspersed with QTP functionality. For example,

Step 3, when executed, will cause the driver script in the QTP framework to call a VBScript method "flight_AddOrder" and it will pass the subsequent fields as parameters,

Flight_AddOrder("17-03-2010","31-03-2010","London","Denver")

Notice the drop downs in the test step. These are dynamically populated when you add the test step to the grid.

Notes:

This is the outline for this project. When complete, I will expand this to include other test automation "engines"; Rational Robot, Selenium2, API integration/ Unit tests. I am also interested in showing you how to bypass VBScript for keyword code creation. So we'll create a C# engine that will drive QTP (or any other test automation tool).

Stay tuned...

Create-Test-Automation-Framework - Part-1a

Hello and welcome! This blog will provide information and examples on automated software testing solutions. Primarily the focus will be on creating custom hand-crafted solutions using .NET rather than how to use off-the-shelf tools. Although, I may provide code examples of frameworks I have developed in the past for tools like Mercury QTP, Rational Robot etc. Initially my posts may be quite focussed on UI automation (UIA), an accessibility framework for Microsoft Windows, as that is what I am currently working with. I have started recording a video series on various aspects of UIA and so will begin uploading that soon. A mid to long term project that will be documented is the development of a fully functioning test automation tool developed in .NET Framework 3.5+ that you may want to follow. All code will be provided along with video tutorials. This tool should work on any 3rd party application and so UIA may not be enough on it's own. We may have to fall back on MSAA support along with some useful Win32 API methods. The idea is that by providing concrete examples and links to other good stuff that I have found, anybody new to UIA or custom test automation development may get up and running a little quicker. Hopefully techniques presented here will be helpful and also invite healthy discussion. Shawn Barrett 

Post archived - Go to home page

UIAutomation(UIA) is commonly known to be natively supported by WPF. This has been in place
since .NET 3.0. It can easily be assumed that if you are developing a test automation tool
using the System.Windows.Automation namespace, which targets a WPF application, test automation
should work by default. This is not the case.

If all controls in the application under test (AUT) are standard controls, or derive from
standard control templates, then test automation will be that much easier. Any custom control
that derives from a low-level control that has no standard template will require UIA server
provider support. This means that the developer has to override the custom control’s
OnCreateAutomationPeer method and return an instance of the AutomationPeer for the control. I
posted a reply herein the UIA support forum (I am svb70) that shows an example of how to
do this, even though the example derives from a standard control, the concept is the same.
The interesting thing here is that because the custom control derives from a checkbox you
could cheat and simply call base.Focus() in the custom control’s OnMouseLeftButtonDown method.
You couldn’t do this in a custom control that derives from a low level control.
Ok, so now you have implemented UIA server providers for all your custom controls, job done!
not quite.

Test automation using UIA is a lot more robust and easy to work with when all
controls have a unique AutomationID. The compiler sets this based on the controls name. So you
can name all controls, ensuring names are unique. This could be done in code or
XAML. You can also set the AutomationID using AutomationProperties.AutomationID. But what if the
AUT dynamically creates controls based on custom controls? where do you set the AutomationID and
how do you ensure it is unique? You could say that the developer that creates a control that
derives from the custom control has the responsibility to build in accessibility. Or not, as
this potentially creates a lot of duplication and each developer has to learn UIA.
One approach is to create the AutomationPeer in the base control and define the AutomationID
(and any other required UNIQUE properties) in the base control itself. I say unique as it is ok
to have static values supplied in the AutomationPeer, but any unique values should be set in
the control. Why?
If you look at the custom controls OnCreateAutomationPeer() method,

protected override AutomationPeer OnCreateAutomationPeer()
{
      return new CustomAutomationPeer(this);
}

Every time a UIA client (test tool, screen reader) interacts with the custom control, a new
instance of the AutomationPeer is returned. So you can imagine many un-necessary calls for
unique values will be made. It is up to the developer to set the AutomationID based on the AUT.
For example, if your application allows gadgets to be added to a canvas and the gadgets are
based on a custom control, you could set the AutomationID of the control based on the
child control count of the custom control’s parent. This may require a recursive method to also
uniquely name all child controls of the custom control if necessary. This is because you can not
be sure that the root element that has the unique name will receive the user interaction.
This is not to say that AutomationIDs are a must, but with a bit of upfront work they will
save a lot of effort.

We have now dealt with potential UIElement identification issues. It should be said that
although the UIA appears to have a steep learning curve, once you have done one implementation
it becomes very easy. For newbie / intermediate developers or testers new to UIA and WPF,
it may be a lack of experience with interface development that makes it difficult to get your
head around. If this is you and you aim to work with WPF in the future, go and learn WCF. I
think it is a great way of understanding contracts and their implementation, whilst learning
a new skill.

So the entire AUT now has full UIA identification support. We can make an observation. This
test tool is only ever going to work with this UAT. If we introduce a new AUT we have to do the
above all over again for your new AUT. This may be acceptable. Say a developer comes along
and introduces a new toolbar where all of the toolbar buttons have the same name.
You record a new session and click the third button. On playback the first button will be
clicked as it is the first item found with the specified name. The tester has to wait for the
developer to implement UIA support. You could and probably will use conditions to locate
UIElements. The point I am making is that you cannot rely on UIA alone and these quirks may
have to be dealt with in the test tool rather than in the UAT. In my next post I will outline
an algorithm that you can implement to get around a lack or duplication of AutomationIDs.
The next hurdle the developer and tester face is that by default UIA does not support actions
like double-click, drag, right-click and text insertion in certain circumstances. The
implemented UIA control pattern will perform whatever the default action for that control is.
So the developer either has to implement this custom functionality in the UAT or the
QA developer uses Win32 API calls within the test tool. I am in favour of the later. A future
post (soon) will show a working example of how to do this.