Automated Testing SAP with Worksoft Certify – Reviewed

Replicator image by Peter from Iceland on PicassaLooking for the holy grail in testing SAP? I am. We’ve got an upgrade from 4.6C to ERP 6 looming on the horizon, and a little testing magic would go a long ways. Wouldn’t it be cool to say, “Replicator, one sales order, 20 line items.” And…presto! “Replicator, pick and delivery on last Sales Order.” Sha-blam! All set. “Replicator, one purchase order with 300 line items, and tell me if it meets this set of criteria.” Dare I dream? Oh, yes, just like butter. Usually, testing feels more like a stint as the Black Knight from Monty Python’s Holy Grail than quality time spent with a Star Trek replicator. The good news is that the SAP world is moving towards that magical replicator experience in automated testing. Albeit slowly, but good progress all the same. Enter Worksoft Certify, HP’s Mercury, et al. This post is an introduction to Worksoft Certify.

What is Certify?

In the 2006 to 2007 timeframe, Worksoft made the strategic decision to tailor its automated software testing platform for the SAP experience. This well may be the tipping point for this relatively small company. They are gaining a lot of traction with some pretty major SAP customers. Here in St. Louis, there are already several Certify customers (here’s hoping a Worksoft Certify User’s Group will be in the making).

Among the testing tools that we looked at, Certify most closely matched our testing strategy – at least what we hoped our strategy would be going forward. Certify came across as having the advantage of allowing your business process experts (BPX’s) create and maintain the test cases for their areas of responsibility. Other testing platforms talked about Quality and Testing Engineers, and other dedicated resources that we were not prepared to secure.

Has Certify lived up to the ease of use we expected? Not quite. Let me qualify that. This is certainly a tool that the average BPX can become proficient in, however, there is a learning curve, and this bike requires a lot of practice to get good at riding it. We are finding that as BPX’s use the system more regularly, they are finding more uses for it in their daily work. They are dreaming up additional test cases and variations that give us much broader testing coverage than we’ve had in the past. Some of them actually like it 🙂 . Certify is a complex and quirky tool that is made all the more complex because of the complexity of the patient – SAP. If SAP weren’t so complex, Certify would seem a lot simpler.

Does Certify Deliver the Replicator Experience?

Yes…once you’ve got the test cases built. That is the mountain you must climb to reach Valhalla. Our BPX’s are able to do Sales Tax testing on 50 states in a tiny fraction of the time that it was taking them in the past. We have people creating 300 line item PO’s with the push of a button. This ties up the computer for 20 minutes, by this was an hour+ of manual work before, and heaven help the person who flubbed up the downstream testing and requested another 300 line item PO.

The creation of test cases is not to be taken lightly, especially if you’re running some bolt-ons in your SAP environment. Hopefully we are blazing the trail for future Certify customers (thank you cards are accepted 😉 ) in the area of bolt-ons. We are running a number of them such as GuiXT, and it seems that these frequently have quirks that Certify is not prepared for. Worksoft has been very responsive in applying resources to help us through these problems. These usually result in a work around or in some cases a hot fix that gets applied to your clients.

Gaining user acceptance is a major feat, but it is definitely a battle worth fighting, no matter which testing platform you go with. The payoff is huge for your testing strategy. We have had the most success by dedicating 2 resources to the test case ramp-up phase. All of our BPX’s are busy with daily life in SAP, so adding the Certify learning curve and test case creation to their plates was not going to really work. We’ve had our BPX’s identify test cases to be created. Our Certify experts are scheduling time to work directly with the BPX’s to get these created. We’ve seen a complete turn around once we provided these resources to our BPX’s…knock on wood 😉

A List of Issues to Consider

Here are some items we are/have been dealing with:

  1. General testing strategy – what, when, where, why, and how to test our system…and how to document that
  2. User acceptance
  3. Test data management – no testing system will work with bad or unpredictable data
  4. Test case maintenance
  5. Possible Integration with SAP Solution Manager – Worksoft is building in Solution Manager integration…we are considering what that looks like and how we would use it, but haven’t gotten very far on this item
  6. Tracking test case creation – I created a web application that lets us track the status of our test cases (are they built yet? do they work? are they scheduled to be created? when?). I’ve talked with Worksoft about the need for something like this. Whether you have a dedicated department of testers doing the creation or whether you have distributed that responsibility to your BPX’s, you will need some mechanism to track and manage your progress.
  7. Integrating test results with your Support/Development Ticket system – whether you use Remedy or some other ticket system, it would be great to be able to document the Certify testing in that system…automagically

That’s the short list…

Final Words

Should you use Certify? Well, that’s begging for the age old consultant answer…I won’t even say it. Suffice it to say that Certify is a solid product, gaining some real traction among SAP users, and it’s getting better and better every day. The leap from Certify 7 to Certify 8 was like Windows 3.1 to Windows 95 (maybe that’s a bad analogy…forgive me, but you get the point). You should seriously consider how you want to test your system. Look at the alternatives on the market, and pick the best for you. We’re giving Certify a tentative 2 thumbs up. We’re still early on in our use, so anything else would be rash. Worksoft are interested in being a successful company, and they realize that there is really only one path to that…making their customers successful. We can attest to that.

Cheers!

Byron

, ,

11 Responses to Automated Testing SAP with Worksoft Certify – Reviewed

  1. Bill Simpson November 24, 2008 at 2:07 pm #

    There is another product/service in the marketplace that blows certify away in the SAP space and Worksoft is in the acquistion stage of this company in order to take them out of the marketplace. Their recording technology is awesome and we have used them numerous times in the past at my company.

    Bill Simpson

  2. Byron Bennett November 25, 2008 at 9:18 am #

    Bill,

    Thanks for the tip on the Test Factory purchase. I had heard that WorkSoft were working on adding performance testing to their repertoire, but I didn’t know the details. From the press release, it would appear that Test Factory’s technology will be the vehicle for that.

    If, as you say, the Test Factory technology is superior to Certify, we can only hope that those advances will not be completely removed from the market, but rather find their way into WorkSoft’s core product.

    Certify is certainly not a record/replay technology, and you seem to indicate that Test Factory is. This gives you a lot of flexibility with Certify, though it makes test case building more tedious than Record & Replay. In truth, there is very little in the way of looping and basic if/then functionality that you cannot do with Certify. Some of the members of our team have described building a Certify test case as “programming by point-and-click.” The upside is flexibility, the downside is that flexibility increases complexity.

    In a perfect world, and Certify may get there one day, we could Record an SAP test through Certify and then go back and add in the variables, recordsets, if/then forks, etc. That would let you quickly “scaffold” a test case and then go back and make it robust.

    On another note, we are half-way through our ECC6 upgrade and things have been going very well using Certify through this process. I intend on giving a post mortem here on WhyPad when it’s all said and done. Knock on wood! 😉

    Byron

  3. Randy June 12, 2009 at 10:47 am #

    Hey Byron,

    This is a fantastic review, as one of the founders of Worksoft I can verify that you hit the nail right on the head!

    CapCal is like a “sibling” of Worksoft – both of them emerged from AutoTester but took different directions.

    It might interest you to see what we are doing with SAP and the Amazon Cloud. We also have an integration with Certify that will soon be announced. Check out our blog at http://capcalblog.blogspot.com.

    thanks and keep up the good work!

    Randy Hayes
    CTO
    Capacity Calibration

  4. Byron Bennett June 12, 2009 at 12:45 pm #

    Hi Randy,

    Thanks for dropping by! And thanks for pointing me to CapCal. I’ll be interested to check it out.

    Cheers,
    Byron

  5. cgault August 18, 2009 at 11:28 am #

    Hi Byron, my company too purchased Certify and we’ve been at it for about a year now. We’re converting our manual regression test cases (located in Quality Center) to BPT’s in Certify. Our automaters are located offshore. We use Sharepoint for communication and collaboration and that’s turned out to work well. Your comment about building a web app. to keep up with status gives me an idea about interfacing between Sharepoint and Certify. I’d rather Worksoft put some energy into the Reporting module and add filters to the queries so that we could at least get execution results out in a reportable fashion. We’re in pretty close contact with the support engineers, and I’ll keep making my pitch.

    It’s been pretty slow going, but we now have over 800 BPT’s. We don’t consider them ready for execution until the BP experts (BA’s) execute them from Certify and enter in comments that we’ve added as attributes in Certify. The validation flow has turned into a bottleneck, but we’re trying to work through that and make automation a part of the BA’s regular activiteis so that validation and/or “scrubbing” manual test cases for automation becomes part of what they do everyday.

    We’ve been working on our test data strategy and we’re building queries to our production systems (R3, SCM, SRM, BI, HR) to load our recordsets. We’ve started with SRM. Certify doesn’t really have any SQL tools to help us here. Can you share anything about your test data strategy, or what you’ve built to load data for test execution?

    I found a Certify Yahoo User Group but am not sure how well known or frequented it is. Sharing information like test data strategy and tips and tricks in Certify would be of value to us.

  6. Byron Bennett August 18, 2009 at 11:41 am #

    Hi cgault,

    Glad you dropped by! This post is over a year old, and we’ve come a long way since this, so I really need to put out an update on what we’re doing.

    The short answer to your data question is: We purchased a tool from Hayes Technology called Gold Client. There are a number of tools out ther (Applimation, et al) that do this, but Hayes is the one we bought. Gold Client lets us pull down data from our production system (or in our case, a production-like system). We pulled all master data and 1 weeks worth of transactional data. You get to determine what you want to treat like master vs. transactional in a lot of cases.

    So, in our QA system, we have a client that we don’t touch, it’s loaded up with that subset of production data…it’s our gold client. We use it to refresh the other 2 clients on that QA box by doing a Client Copy (clopy in our tech groups lingo). We haven’t gotten into a rhythm of doing the refreshes yet, so I’m not really sure how frequent that will be. It takes most of a weekend to do the client copies, so we can’t do it too often.

    We have about 1400 test cases that we’ve built. We’re in the process of putting them into regression test batches that can be run overnight. We just got our first 150 test case batch to execute with only 5 failures last weekend. So, we’ve still got some work to do there.

    I’m planning to do a new post on this in the near future if you’re interest.

    Cheers,
    Byron

  7. cgault August 18, 2009 at 12:19 pm #

    Hi again Byron. We too have Hayes’ Gold Client. We are sharing our QA test client with our manual testers and normally refresh our SAP test clients as of a certain date. We have been using flash copies to get a full copy of production, but we have been told that we can no longer afford to load this much test data — too much DASD is being consumed. Thus, the Gold Client will become one of our best friends.

    What I’m chiefly interested in learning is whether you’ve been able to write queries to find the data that your BPT’s need (in QA or production) and then either populate a table or one or more Excel spreadsheets that then get imported into Certify for test execution? Does/Can the Gold Client play a part in building your Certify recordsets?

  8. Siba Ram Baral October 1, 2009 at 2:29 am #

    Hello Byron,

    I am a SAP senior manager for British Petroleum product, wish to know the following things about your product. Would appreciate if receive your response by today EOD.

    1. A brief about your “Certify” product capabilities ( compatibilities with SAP and add ins )
    2. How relevant are they in testing space
    3. Pricing details
    4. What are the quantitative and qualitative benefits of your product ?

    Regards
    Siba

  9. Byron Bennett October 1, 2009 at 7:58 am #

    Hi Siba,

    1) Certify is compatible with SAP…we’ve used it on 4.6C and now ECC 6.0. We also use it on various add-ins like Vistek’s products and GUiXT. We use it against our SAP Portal and Web Commerce sites from the internet side.

    2) Automated testing is a relatively small space today. Worksoft are very relevant in my opinion.

    3) You’d have to contact Worksoft directly for pricing details. We found them competitive.

    4) Quantitative: I don’t have hard numbers on the increase in productivity, but I can say that in our ECC 6 upgrade, we ran > 1,300 test cases in < 5 days. In 2 separate testing cycles. Qualitative: for us, better test case documentation; better testing coverage; automation permits us to run a broader and deeper testing of our changes - that is we are able to test a broader range of SAP functions with more variations of each function (deeper) when we test our changes now. I would qualify this with the statement that you don't flip a switch and you get all this goodness. It requires a significant commitment to stay the course when you're moving to automation. The creation of your test cases is, alas, not automated...maybe some day 😉 BB

  10. David Milette October 1, 2009 at 4:14 pm #

    Siba-

    In response to your queries above please see my response.

    Worksoft offers innovative and comprehensive solutions to optimize lifecycle management for SAP® applications. Worksoft solutions are specifically designed to meet the unique architectural needs of SAP application environments and the specific needs of both business and technical users.

    Worksoft customers typically realize faster deployment times for SAP upgrades, gain better visibility into the impact of business process changes and reduce their risk of performance and availability issues.

    The Worksoft lifecycle management suite delivers a comprehensive and integrated solution for managing the impact and risk of recurring changes in your SAP environment by providing:

    * Lifecycle management solutions that enable both business process experts and technical SAP users to assess, analyze, validate and document the impact from business or environmental changes.

    * An object-oriented approach to testing resulting in a code-less solution that is more functional and more reusable than traditional automated testing products.

    * Cross-platform business process validation that spans multiple application platforms including SAP, HTML, Java, .NET, Windows, Unix, iSeries, Mainframe, Peoplesoft and other packaged ERP applications.

    * A comprehensive solution portfolio that spans the entire SAP change lifecycle including: impact analysis, functional testing, performance testing and auditing.

    More specifically, Worksoft Certify fully eliminates custom coding and programming, a requirement of most legacy test automation products, making it fast and easy to implement and maintain. Using an object-driven approach rather than generating scripts or code, Worksoft Certify validates business process workflows using a data model of fields, screens, and transactions making it easy to keep pace with dynamic changes.

    Worksoft Certify supports a vast array of applications that span these platforms and interfaces: All SAP interfaces, web-browser, web services, Java, Microsoft .NET, mainframe, Visual Basic and more.

    In response to your question on Worsoft’s relevance in the testing space, Gartner places Worksoft as a visionary for delivering leading-edge, script-free test automation tools and our leading support for testing SAP applications. Although we have the ability to
    test general applications, our focus during the past two years has shifted to SAP testing, and users of the product are finding that they can drive very high test automation coverage and have
    maintainable scripts. The toolset has integration to both SAP Solution Manager and the
    IntelliCorp suite, enabling an integrated change impact analysis and validation loop.

    Our pricing is very competitive and you will find that the ROI of our solution is unequaled in the market today. I would be very happy to discuss this further with you if you so wish. Feel free to contact me at the coordinates below.

    David Milette
    VP International Sales, Worksoft

Powered by WordPress. Designed by Woo Themes