Automated Functional Testing Tools – Evaluation Guide

Automated Testing

The lure of automated testing in software design is obvious if you’ve ever done any testing. With automated tools, you can create dozens of test scenarios and run them as many times as required by basically just pushing a button (as long as you’ve set everything up right 🙂 ). A good tool will document the results of each step of the test, frequently with screen captures. What would have taken hours or days can occur in minutes or less, or even be scheduled to run at night and be waiting in your inbox the next morning.

This is especially valuable if you are dealing with systems that have lengthy business processes with lots of interdependencies such as ERP’s. Minor changes to code or configurations can bring a business to its knees. In some cases, testing can be over half of the development cycle. Creating and executing 50 different scenarios of your Order-to-Cash process is not nearly the daunting task with an automation tool as it is with a bunch of black binders that you pass around from one team to the next.

The great thing with automation is that you build a testing repository of unit tests so that next time, you can drag-and-drop the units to build your integrated scenarios, or even use prior integration tests cases. It’s not completely without maintenance, but it’s significantly better than starting from scratch if done right.

The dark side of automated testing:

  • Potentially expensive…the tools are costly. If you are doing rigorous testing on your software development, the time saved can easily show a quick break even and positive ROI. If you are not doing rigorous testing, it is an opportunity to reduce your risk of business interruption.
  • Test cases and unit tests must be maintained to ensure they account for your system changes. It’s a pain today that saves you an amputation tomorrow.
  • Previous automated testing products have not always lived up to expectations…but then again, the Wright brothers didn’t exactly circumnavigate the globe. It is generally agreed that the current generation of tools is getting there.

The following items relate to my evaluation of testing tools for an SAP environment. Most of the tools can be used to test a variety of platforms such as SAP, Oracle, Internet sites, .Net apps, etc. This post has an SAP slant, but is generic enough to be applied to other platforms as well.

Considerations in Evaluating Automated Testing Tools

  • Standard considerations around the prospective vendors:
    • reputation
    • financial stability
    • likelihood of being acquired by someone who will squash the product
    • etc.
  • Technical footprint:
    • dedicated servers vs. PC based
    • web based vs. fat client
    • databases supported
    • integration with the version of the software being tested (e.g. – does it work with R/3 4.6C, or just myERP 2005?)
    • time and resources required for implementation
  • User impact:
    • type of internal personnel required to build test cases
      • testing engineers vs. functional team members
    • ease of maintaining test scenarios
    • require coding vs. click and record
    • user friendliness of tool
    • training
  • Licensing & costs:
    • named users vs. concurrent users
    • annual maintenance fees
    • purchase price
    • consulting costs for implementation
    • costs for training
    • costs for future consulting (upgrades, etc)
  • Further reading on general considerations:

Products and Resources

Mercury Interactive

Mercury Interactive’s Quality Center is acknowledged by Forrester and Gartner as the Cadillac of the automated testing tools on the market today. (You can purchase Forrester’s research on the topic here (not inexpensive). Their synopsis indicates that Mercury’s Quick Test and IBM’s Rational are the leaders in this relatively small race.) But that doesn’t make it an automatic purchase, there are plenty of consideration.

Overview & Links:

  • Product brands: Quality Center, Test Director, and QuickTest Professional (also have testing solutions)
  • Web-based system with dashboard-like interface
  • They offer trial versions, but those might not be practical if running against an ERP…definitely want to talk with them before trying this

IBM Rational Functional Tester

According to Gartner and Forrester, IBM’s Rational comes in a close second in the industrial strength automated testing tools. That research was from 2006, so it’s getting to be about a year old. Check out the IBM testing goodnes here.

WorkSoft Certify

WorkSoft’s Certify was recently certified by SAP against the Netweaver platform. WorkSoft’s offering is not as feature-rich as Mercury’s or IBM’s, but their price isn’t either. This is certainly a product worth evaluating if your budget, time, and resources do not fit with the big boys. Check out their webinars and whitepapers here.

One Response to Automated Functional Testing Tools – Evaluation Guide

  1. Murali January 28, 2009 at 7:57 am #

    I’m new to bloggong & My blog is about basics of Software Testing,Manual Testing,SDLC,Testing Techniques,Levels of Testing,Types of Testing,Test Planning,Test Execution,Test Development,Bug Tracking,Result Analysis,Test Design Techniques and QTP. so I write about that which I know.Give it a visit if you get a chance..
    feel to free to visit:

Powered by WordPress. Designed by Woo Themes