Introduction:
Software project managers, software developers and testers building today’s applications face the challenge of doing so within an eversible schedule with minimal resources. As part of their attempt to do more with less, organizations want to test software adequately, but as quickly and soundly as possible. To accomplish this goal, organizations are turning to automated testing. Automation enables you to control the functionality of an application programmatically.
The blog provides a five-step process for comparing, evaluating and finally choosing the right tool for your organization.
Step 1 : Define Initial Requirements.
Before you begin looking at the tools on the market, it’s important to create a list of the initial requirements. What problems will the tool you choose help you solve? What capabilities will the tool need in order to be effective in your environment? What constraints, budgetary or otherwise, will the tool need to meet? The initial requirements give you a starting point.
Compatibility Issues: Any tool you choose will need to be compatible with:
For example, if your application runs on both Mac and Windows, you may want to find a tool that supports both platforms. If your application is written in Java, the tool you choose will need to support Java.
Tool Audience: In defining the initial list of requirements, keep the audience for the tool in mind. Ask questions such as:
The skill level of the people who will be using the tool will play a part in determining the best tool for your organization. If your organization expects to put the tool into the hands of extremely technical, senior staffers, then power and flexibility will be more important than ease of use. If junior testers or non-technical staff members will be using the tool, then ease of use becomes a critical consideration.
Budget Constraints: You’ll also want to determine the budget for the tool before you purchase it. Cost is likely to be a factor, and how much your organization is willing to spend could have a significant impact on your choice of tools.
As an example of an implementation cost, consider a test automation tool. If the tool will be used for unattended test execution, the budget will need to include funds for additional machines on which to run the completed scripts. Initially, scripts won’t take long to run. However, when you have enough scripts to run for 15 hours, you won’t want to tie up the computer on which you do most of your work while the scripts execute.
When you and your management set the initial automated test tool budget, consider these questions:
How much you can afford is a different question from how much a tool is worth. If the future of the company rides on the ability of a given application to perform well under load, then a good load testing tool may be worth a tremendous amount to the company. If the company may have a large tools budget, but does not see worth in a given type of tool because the development group has already created a custom tool that does 50% of the job at 5% of the cost.
You will also want to take following points into consideration:
Addressing the benefits, as well as the cost, in the budgeting process enables you to discuss preliminary ROI (return on investment) projections. Even if you aren’t thinking about ROI, your executives probably are. You’ll want to be able to quantify the expected return on investment to ensure executive-level support.
While considering the benefits of the tool, you’ll need to determine how your organization will measure its success with the tool. Will you measure the number of hours that the tool saves? The number of defects the tool finds? Improvements in quality?
Business Requirements: If you work in a large company or regulated industry, your organization may have additional requirements that tool vendors must meet in order to become “approved suppliers.” This is the time to determine those requirements, so you can eliminate unqualified vendors before you spend much time evaluating their offerings.
Step 2 : Define Initial Requirements.
When you are investigating your options, you want to spread your search net as far and wide as possible to find the tool that’s right for your organization. Read newsgroups, search the Web, visit websites, attend tradeshows and conferences, ask questions on related mailing lists, check out ads in industry trade magazines, ask co-workers about tools they’ve seen or heard about, and call vendors to get them to send you product literature.
In this stage, you’ll want to talk with the tool sales people. It is to your advantage to give the sales person as much information as possible about your decision criteria. Be very open with them about your requirements and about their competition.
Ask the sales representatives up front if their tool meets the requirements you have gathered so far. It’s very likely that the sales person will respond positively.
It is also reasonable to ask the sales person how their tool compares to their competition:
The sales representative may pressure you at this stage to do an evaluation or to let them come in to do a demo. Resist this pressure. During this stage, you’re gathering information that will allow you to decide which products make sense to bring in for a demo. Demos and evaluations are time-consuming. Unless you have a lot of time to spare, you want to avoid doing demos on products that don’t come close to meeting your needs. You want to avoid evaluating products that don’t come close to meeting your needs.
Step 3 : Refine Your Requirements.
While investigating the tools on the market, you probably discovered additional requirements that you either didn’t know about when you created your initial list or forgot to include. Add them now.
During this step, you need to communicate the effects of implementing these tools to everyone affected by them. Don’t just talk to those who will be using the tools on a day-to-day basis. There are two key reasons to involve everyone affected proactively: it gives them a chance to discuss their concerns about how the tool will change their job, and their questions may imply additional requirements.
Some people may have concerns that seem to have nothing to do with the tool. For example, “If we automate everything, will I still have a job?” If people are very concerned about or even threatened by change, then your organization may have an additional requirement that the tool not necessitate a drastic change in the way the department does business now. You may also choose to address these concerns through training and communication.
Other questions imply functional requirements. For example, a manual tester may ask, “How will I know which of my tests have been automated?” In this example, the manual tester needs to know which tests have been automated so he or she knows which tests not to execute anymore. The tool needs to provide a method of tracking the manual tests represented by each automated script.
In your initial investigation of the tools on the market, you may have found that the budget you set in Step 1 is unrealistic. The tools that meet your functional requirements may not meet your budget requirement. If that is the case, now is the time to make the business case for increasing the budget. If management is unwilling to invest more than originally budgeted, you may need to pare down the functional requirements.
Ranking Priorities: If you can’t find a tool that meets all your requirements, you will need to prioritize the requirements so that you can choose the tool that is the best fit.
A face-to-face meeting is a good forum for prioritizing.
Step 4 : Narrow The List.
Now it’s time to purchase the tool. It is certainly possible that there is only one tool on the market that meets your requirements. If that is the case, this step is easy. (You will still do the next step: even with a single candidate, you’ll want to make sure that the product can do what you need it to do before investing). It is more likely that there are four or five real possibilities and several more partial solutions.
Use your list of requirements to rank the tools you investigated earlier. The top two or three from this ranking will be the final candidates that you bring in for a hands-on evaluation. You may find that none of the tools meet all of your requirements, but several of the tools meet some percentage of them. In this case, it is a good idea to get input from others in your organization on prioritizing the requirements so you can rank the possible tools definitively.
Step 5 : Evaluate The Final Tools/Vendors.
The first step in evaluating the finalists is to identify a representative set of activities to accomplish with the tool during the evaluation. You’ll want to choose one or more tests that pose interesting challenges for the tools.
The next step is to contact the sales representative for each product. Ask for an evaluation copy of the tool. You’ll want to have access to each tool for at least a week, and preferably a month, to evaluate its capabilities thoroughly. Gaping holes in functionality show up right away, but little annoyances that will drive you crazy over the long haul show up only after extended use. Most of the tool vendors will allow you to evaluate their product for a limited time (30 days or so). It is unlikely that the tool vendor will be unwilling to let you evaluate the tool, particularly if you indicate that an on-site evaluation is a requirement for the sale.
Some of the tool vendors require you to allow one of their staff to participate in the evaluation. Vendor representatives may suggest that they can take care of the setup and write a representative script to perform a test on your application for you. If the vendor insists on doing the work for you, it is worth your time to watch over his shoulder so that you understand the complexity involved in setting up the tool and how to use the tool effectively.
During the evaluation, it is a good idea to model the entire process that your group intends to follow. For a GUI automation tool, that means you should:
Imagine doing the above with not one, but hundreds of tests. Does the process scale? If possible, perform these steps several times, taking note of the things you have to repeat. Then imagine how you would feel repeating those things 100 times.
Once you have decided to buy a tool, whether you or your management negotiates the final deal with the tool vendor, remember that many vendors offer quantity discounts. Consider options such as rollout plans in which your organization commits to purchasing a given number of licenses over a specified period of time.
When rolling out the new tool to your organization, remember to:
Anyone who has contemplated the implementation of an automated test tool has quickly realized the wide variety of options on the market in terms of both the kinds of test tools being offered and the number of vendors. The best tool for any particular situation depends on the system engineering environment that applies and the testing methodology that will be used, which in turn will dictate how automation will be invoked to support the process.
References:
Du musst angemeldet sein, um einen Kommentar abzugeben.
