Enterprise Applications

Issue Date: September 1997
Issue Number: 8.04
Category: SOFTWARE

Nobody argues against software quality, but elevating it to a specialty and buying tools often runs up against organizational inertia. This report examines our findings, discussing the costs, the benefits, and the means for selling automated software quality to internal sponsors.

Like any tool, automated testing tools require additional up-front investment, and theoretically pay themselves back with better productivity. Whats brought the issue of automating the testing of client-server applications to the fore is that the architectural complexity provides so many more targets to test: the user interface, business logic, network support, database connectivity, and overall performance, to name a few factors.

Last month, we opened the topic by describing the QA process and the automated solutions on the marketplace. This month, we complete our research. We surveyed 15 sites (12 of which were using automated software quality assurance tools) and examined the selling of the testing process, the professional makeup and backgrounds of automated testers, the resource requirements and labor costs associated with automated testing, and the benefits that accrued.

TEST TOOL MARKET BACKGROUND
As we mentioned last month, there are different types of testing, including logic testing (debugging), GUI testing, load testing, coverage testing, and acceptance testing. Not all of these stages can be automated; for instance, it is impossible to take the human element out of source code debugging (although some partial automation is possible), or the determination that a delivered software module meets the users requirements.

But in between, there is plenty of mechanical testing that can be, and is, automated. The testing of visual controls to ensure that they work and function consistently, the testing of database connectivity, and of application, database, and network performance are highly amenable to automation. A market has emerged, so far with a handful of players who until now have carved out distinct niches (e.g. AutoTester for SAP R/3 applications, Mercury Interactive for load testing, SQA for GUI testing of Windows applications, Segue for multi-platform GUI and Web application testing, and Softbridge for large, server-centric applications).

But the market is fast-growing. Industry analysts generally place the market at roughly $500 million by 1999, although that is likely to include tools that package vendors (e.g. SAP) bundle themselves, and established mainframe tools, such as Compuwares Hiperstation.

Of the five principal testing players now in the market, we estimate that overall revenues are currently less than $100 million (if revenues from subsidiaries, such as Rationals SQA are broken out). New players such as Compuware and CenterLine are diving in for what should be an expanding opportunity.

Part of that opportunity is the $4 billion (according to Meta Group) ERP marketplace. Automated tools have until now been primarily aimed at homegrown applications, since commercial packaged applications are theoretically supposed to be bug free. Testing is not a user issue when it comes to shrink-wrapped applications such as desktop productivity suites.

However, enterprise packages (e.g. ERP), which are heavily reconfigured and customized by end users or integrators, is another story. Introduce a single change even as trivial as a field length expansion, and the application could easily bomb downstream.

Currently, some ERP vendors (e.g. SAP) offer their own tools for developing test scripts; however, the dominant trend is for third parties to develop templates that integrate to the APIs of specific ERP packages; currently, Mercury Interactive and AutoTester have developed versions of their products for R/3, while SQA offers test tools for PeopleSoft. We expect that Mercury will add extensions for other ERP packages such as Oracle in the near future.

ORIGINS OF OUR RESEARCH SAMPLE
We contacted the market leaders in automated client-server testing, including AutoTester, Mercury Interactive, Rational/SQA, Segue, and Softbridge. We also contacted ERP vendors SAP, Oracle, Baan, QAD, and PeopleSoft. The sample was almost equally split between internally developed applications and ERP projects.

Because users were provided by vendors, there was an imbalance in the sample when it came to the question of whether a site used automated test tools. With the ERP group, slightly less than half the respondents did not use an automated test tool; however, since test tool vendors provided the internal development sites, none of them lacked testing tools. Because our research was qualitative, we do not believe that this imbalance skewed the results.

The data provided by ERP users who did not use test tools was not necessarily ERP-specific. For instance, the comment of one ERP non-user, We didnt use automated tools because we werent aware of any tools that could help us, could have easily applied to internal development teams.

Our sample (see Table 1) included a mix of users across financial and manufacturing industries; the actual mix was over 60% from the banking, insurance, and securities investment sectors (they accounted for all of the internal development sites). This reflects the fact that the financial industry was one of the first, and for several vendors (e.g. SQA, Mercury-Interactive) remains the largest industry segment for automated software quality (ASQ) tools. The cost of downtime there is tremendous, said Bill Hayduk, a New York-based SQA consultant, who added, The brokerage industry is not shy about spending money.

Organizations interviewed ranged in size from less than $10 million overall to multi-billion dollar business units of the Fortune 10. QA staffs ranged from external consultants on retainer to internal consultants available to internal IT groups, to testing personnel who were part of a specific IT development organization. (Note: In one case, we interviewed an application development group manager who recently switched organizations. The results he gave pertained to his previous organization; since switching organizations, he is currently involved in an uphill battle justifying investment in QA teams and tools.)

Applications involved ranged from enterprise to business unit and workgroup (which we define as a subset of business unit). Most (but not all) of the enterprise-level applications studied were ERP packages, reflecting a trend away from custom development, toward enterprise-level offerings. According to Meta Group, by last year, over half of the Global 2,000 were either implementing or evaluating off-the-shelf products for manufacturing, financials, customer service, and human resources.

Our questions were open-ended. Among the major issues examined were:

* Did the organization use an automated testing tool? Why or why not?

* Was there internal resistance to automated testing, and from which groups?

* Was there a formal testing process in place, and if so, what was it?

* Who did the testing, developers or dedicated staff?

* If specific personnel were dedicated to testing, what types of professional backgrounds did they have? What types of qualities were preferable? What was the ideal ratio of testers to developers?

* How did payscales of testing staff compare to developers?

* What proportion of project time was devoted to testing?

* What were the benefits of automated testing, and could they be quantified?

SELLING AUTOMATED TESTING

Strategic Sell: Nobody argues against testing new software, but the eternal question is whether to throw special resources against it, or to just allow some extra time at the end of a project for developers to manually run a few tests.

A number of the organizations contacted indicated that testing was embraced because the applications in question were strategic.

For instance, a west coast bank is striving to become the leader in on-line banking, a market where functionality, security, and application robustness are clear competitive drivers. It has developed a Java and HTML-based Internet application which uses Segues SILK Web page testing tool. Automated testing was promoted as a means for providing the bank peace of mind that its leading-edge reputation would remain intact.

According to John Seiberling, an independent QA consultant who is working with the on-line banking application, another justification for automated testing could be quantitative: if applications are turned out with fewer defects, it gives management the idea as to how rapidly they could add new features to their on-line banking product. But the bank isnt keeping any numbers; it is focused on enhancing its on-line banking to stay ahead of its rivals.

The case was similar with a New York-based securities trading firm. Its rationale for automated testing epitomizes the securities industry, where having information sooner is equated with competitiveness. Cost benefit analysis of using SQA Suite was never an issue, according to the QA project manager. Top management is a strong believer in information technology, said the QA manager, and added that the firm has the financial results - and deep pockets - to sustain such a strategy.

Securities firms cannot afford to deploy applications, such as portfolio management, that might be found to spew out answers redolent of first-generation Pentiums. The application portfolio, written in a combination of Powerbuilder and C++, primarily consists of large quantities of small applications (about 10-15 windows apiece) that can be developed and tested quickly.

The QA manager at the securities firm added that the lack of cost justification requirement does not equate to a blank check. He is developing metrics on the QA process to make it more efficient. Some of the factors will include what types of applications best lend themselves to automated testing, how much time is spent planning and running the tests, how much time is spent conferring with developers and users, and how much time is spent resolving problems that are not software-related.

Productivity: At a mid-Atlantic region securities dealer, the decision to use automated QA tools (Segues QA Partner) is up to the development group. In this organization, a new QA group was set up this year, and currently supports three projects teams.

The QA staff has sold the idea based on productivity savings. This was considered a hard sell, since most IT teams took testing for granted, and never considered that it merited special talents or tools.

Mr. Fix-It: At a large, Indiana-based insurance company, a QA evangelist was recruited to develop a testing mentality. Without someone at enterprise level pushing the envelope to set a direction, things bounce around in limbo, said Curt Hansen, director of enterprise testing for the firm.

Hansen spent 20 years in data processing with the US Marines, and among his experience, had taught software testing at a Marine Corps school. He credits his Marine Corps days with instilling the type of strict discipline necessary for software testing, and it is that sense of mission that he is trying to instill within the company.

The QA lead confessed that a lot of the justification was intangible, but he promotes software testing as cost avoidance strategy.

His function is to set up a methodology, recruit and train developers to become test tool experts. With over 170 major application projects (including maintenance and new development) currently active, Hansen characterizes his QA challenge as a target-rich environment.

The organization recruited him because software bugs were seriously impacting business efficiency. Evidently, software bugs were just the tip of the iceberg with the organization; this fall, IBM Global Services will take over IT as part of a multi-year outsourcing agreement. It will be interesting to see if the newfound spirit will be maintained under new management.

The process is being repeated by an application development group manager who was just brought in by a Toronto-based securities trading organization to reengineer its development processes.

The incoming manager is still at the missionary education stage. He is trying to convince top management of the benefits of QA investment through pilot efforts with small projects. He admits that its an uphill battle, because management regards the notion of recruiting QA staff and buying QA tools as just another cost center.

Automated Testing is Overkill: At several ERP sites, testing remained strictly manual. The obvious rationale, that packaged applications have already been tested by vendors, was only cited by one of the organizations. Its rationale was that, since it was implementing the Oracle package plain vanilla, without changes or customization, the only testing that should be necessary would be acceptance testing (where the end user certifies that the application meets his or her functional requirements).

It helped that the organization was small (the smallest one we sampled, at under $10 million), and therefore could more readily adopt such a minimalist implementation and test strategy. (The whole Oracle project was finished in 70 days, ahead of schedule and below the $900,000 budget.)

The other two ERP sites not using testing claimed ignorance: they werent aware that tools existed to automate the testing process (one was considering using SAPs native CAT tools, which require significant programming skills).

In both cases, testing was still considered a major portion of software implementation. For instance, one site estimated that half of the 160 staff months required for implementing Oracle Manufacturing and Financials was devoted to testing.

DETAILS OF THE TESTING PROCESS

Life Cycle: Ideally, application development teams should draw testing staff into the loop at the design and specifications stage, and if the application is developed in an object-oriented manner, hopefully that will increase reuse of test cases.

Consultant Seiberling suggests another advantage. If you can go right from the software specification to generating a test plan, you could save a lot of development time and money, especially from resolving ambiguities in the spec, he said.

There is another rationale for getting QA involved early - that a defect caught early in the process will be cheaper to fix than if it festers until later on, since it might be harder to spot. Unfortunately, we have not yet found any studies that quantify that conclusion.

We found a couple instances where QA was engaged up front. For an R/3 human resources system implementation at a global telecommunications firm, the project team used the concept of quality gates, the minimum level of quality necessary to advance the project through each phase in the development cycle (see Table 2). Test case planning began at the conclusion of the analysis and design phase, when the screens and transactions were planned.

We found a similar result at a New York securities firm which develops a library of Powerbuilder-based portfolio management applications.

Another variation is bringing in QA staff, not at the design phase, but just before each increment of an application was completed. This is a form of concurrent engineering (a concept borrowed from manufacturing), where QA personnel work side-by-side with developers as they fine tune their code.

However, conventional practice is more reactive, according to Mary Ann Fitzsimmons, who heads the consulting firm Visual Systems Toronto area QA practice. Testing is a change management issue that often is relegated too late in the project, she said.

Conventional practice is that test scripts and test cases are not generally defined until after the first application screens and database links are developed.

Most of the blame is cultural: testers are usually regarded as the people that developers inevitably must deal with at the end of the line. But part of the blame is also the lack of integration between testing and modeling tools.

That was one of the goals of Rationals SQA acquisition; although an interface is now in place between the Rational ROSE object modeling tool and SQA Suite, SQA was unable identify any users yet who are using both tools.

Triage: Testing takes a large chunk of project time. In our previous reports on Year 2000 project testing, we found that the testing portion accounted for 50-60% of overall project time. For an ERP project, at a midwestern manufacturing company, which conducted its testing manually, the proportion was similar.

Admittedly, for client-server projects, which involve more development of new code, testing proportions are likely to be smaller. A common level was 25% for a workgroup application that involved a team of 12 developers and three QA staff.

But, adds a QA manager for another organization, the proportion of project time varies widely, and is often dependent on how much testing (and testing staff) a project is willing to budget.

It is generally not economical to test everything thats developed. For instance, systems integrator IMI Systems uses a risk analysis approach, classifying applications by their importance to the organization.

The highest risk category is reserved for mission-critical applications, where software bugs would cause core business processes to halt. There is a moderate risk category, which amounts to a miscellaneous category of applications that carry some risk if problems occur, while at the far end of the spectrum are peripheral applications whose malfunction would cause negligible risk to the conduct of business.

For instance, a retail banking application, where malfunctions would prevent the bank from opening new checking accounts for its customers, would be considered high risk, whereas a malfunction of a Java applet that displays an advertising slogan on a Web page would be considered low risk.

This approach is used, not only to identify which applications to quality test, but which portions of applications as well. For instance, in a retail banking application, a checking account transaction screen would accord higher risk than a screen that simply displays balances.

We have found that most users adopt a less formal version of this methodology, called the 80/20 rule. At a regional power utility system which was implementing SAPs R/3, eight core transactions were identified for testing.

The same was true for a 120-window, Powerbuilder portfolio management application developed by a Canadian securities firm. According to the project manager, a two stage process was used: first make sure that all of the windows worked, then develop test scripts for the most critical transactions. As for occasional transactions, they would be manually tested as project schedules permitted.

Lead Time: Automating testing requires additional planning up front compared to manual efforts, which are one-shot deals that may or may not be documented. You have to have process and methodology in place, said Tracy Morgan, a QA business systems analyst for a mid-Atlantic region securities firm.

For instance, just like there are programming standards for application development, there should be standards for developing and documenting test scripts and specifying test cases, she says.

And there is the learning curve associated with any new tool; test tools that rely on scripting are like learning a 3GL or 4GL (GUI test tools, which can record and playback screen sessions, can be augmented with scripting as well that rerun specific interactions based on conditional logic).

The goal, says Morgan, is achieving a form of reuse. In the future, when you have large numbers of teams, anybody should be able to pick up where people leave off, she said. Configuration management tools (e.g. PVCS, Clear:CASE, CCC:Harvest) in conjunction with test tool repositories make this feasible.

Reuse, which is normally associated with savings in development efforts, can save money with testing as well. But reuse has not yet surfaced as an issue for most QA shops, because most are in the preliminary stages of implementing a testing process. Also, as we mentioned earlier, the technology to enable test reuse is still being developed.

There are tools for configuration management and object modeling, but as yet, little if any integration with the repositories that accompany testing tools. Additionally, it requires an object-oriented development mentality: for instance, to reuse a GUI test, the GUI itself must be designed in an OO manner for reuse.

We see some evidence that test considerations are impacting version control practices of development teams. The notion is that tests should not have to be rewritten every time something as trivial as a screen color changes.

At the Indiana insurance company, developers have been asked to keep the control IDs - a method of identifying a screen - constant unless the screen is radically changed. By maintaining such discipline (a form of configuration management), test scripts can be reused unless the actual objects being tested change.

Admittedly, the control ID strategy isnt foolproof, since developers might not necessarily be aware of what modifications to a screen or application impact the design of the test case.

Anecdotally, we know that it takes longer to develop a test plan, generate, and run automated test scripts than it takes to simply run manual tests on a one-shot basis.

We have not seen any objective data, but some subjective data points include:

* A director of the QA program for a large Minneapolis-based banking institution estimated that test preparation took 25% additional time upfront for automated testing compared to manual.

* When a test script is created, you often have to run a manual test on the same application to ensure that the test script itself is working properly.

* The testing cycle for a Toronto area securities firm included 3-5 day increments for each of the following steps: test planning, preparation, and running. That might imply that manual testing would save one third of the time (test planning), but in reality, results are not necessarily so linear.

SALARIES AND STAFFING ISSUES

Recruitment: Historically, QA personnel were at the bottom of the IT totem pole. Since the mainframe era QA has traditionally been considered a stepping stone to development for entry level personnel, according to Alex Marchicelli, Practice Director for Testing at national consulting firm IMI Systems.

Testing was far less complex and far more tedious. There were no connectivity, visual controls, and relatively few network issues. Most developers would not be caught dead doing testing because it wasnt creative.

That explains some of the results that QA evangelist Hansen encountered when he first tried recruiting developers into the task.

On one project team numbering a dozen developers, only one person stuck with QA after learning the Softbridge ATF testing tool for a year.

Hansen realized that it would take more active mentoring to increase the QA population there. He worked with the vendor, Softbridge, and ran a Fast Track orientation which included a week of training and project kickoff, under the supervision of a Softbridge consultant. After the week one kickoff, he acted as part-time consultant to the project team over the next 6-8 month period.

Who Does The Testing? Most QA managers and application project team leaders whom we interviewed stated that developers were not the right targets for recruitment; in a few cases, they urged that developers or technical writers be cross-trained in QA tools.

In summary, QA and developers have different mindsets: the QA persons goal is figuring out how to break something, while a developer builds something.

As for end users, most project and QA managers were leery about the idea of unleashing end users on testing tools, especially tools that record and playback their keystrokes in front of the GUI. If you get this [tool] in the wrong hands, it will run away from you, said a maintenance engineer, who led the manufacturing implementation team for R/3 at a midwestern utility system. If it doesnt find a certain screen, it will keep hitting enter and put information into the wrong fields, he added.

At minimum, the result is wasted effort; at worst, it could be corrupted or erased data.

A minority view was expressed by a senior software engineer, who managed the testing effort for another R/3 implementation at a PC manufacturer. Testers were part of a cross-functional implementation team of developers, analysts, and representative end users. In fact, there wasnt a QA professional in sight; the more technical members of the team assumed the lions share of the testing load, but end user representatives also performed some GUI testing. We would have to assume that, as members of the project team, they were akin to power users.

Nonetheless, getting the less technical members of the project team initiated with automated testing proved challenging, the project leader admitted. For some users, it took a while for the light bulb to go on. Most of them initially regarded automating test screens as work.

So who should do testing? A sampling of opinions included the following:

* Independent QA consultant. Knowing the development language is a necessity. The consultants firm, Aegis Software, is involved in developing multi-platform applications for Wall Street firms which may involve mainframe or Unix/NT servers. Clients range from all flavors of Windows to Motif (some brokerage houses still maintain Unix clients), with Web clients on the horizon.

Most of the applications are written in C or C++, with anticipated future demand for Java. I personally believe that QA software should also be used by developers, said Joseph Horowitz, QA consultant. That in turn means that as they update applications, they should be responsible for updating tests scripts as well.

* Telecommunications firm. We look for someone with a testing background, not a programmer/analyst. On a project team, experience levels (e.g. number of years with a tool or platform) should be similar.

* Minneapolis banking institution. We hired several people who were doing technical jobs, but they werent programmers. They were analysts who developed functional requirements.

The test tool and the ability to automate the process made it attractive for them.

The QA manager added that the shift from analyst to QA was now considered a lateral move. In the early 90s at his previous company - prior to the use of automated test tools - testing was considered almost a clerical job. There, testers were simply reviewing and executing checklists, and his old company for a time tried making testing an hourly (non-exempt) position, akin to clerical staff.

* Baltimore insurance company. We look for someone who has excellent test design skills. That includes the ability to understand application functionality and all of the different forms of testing.

Strong analytical skills are desired. Although program design skills are not important, the ability to use a scripting language is.

A talent for diplomacy is the major obstacle to finding people. You cant find people who really want this job because [they know that] programming staff do not want to hear bad news. She added, Henry Kissinger would have made a very good tester.

* New York brokerage. We need someone who understands testing methodology, has worked in a client-server environment, and has basic programming skills, like Visual Basic, so they can write SQA scripts. The QA manager adds that no one with SQA skills wants to work as an employee because the market [for consultants] is so hot. The organization also hopes to cross-train developers and technical writers to shoulder some of the testing load.

* Toronto brokerage. Structured analysis skills are necessary to take apart a business application and build test cases. That places a lot in common with software engineers, but not necessarily developers.

Then there is the part about enforcing quality standards; a background in law enforcement might not hurt, the development manager said facetiously.

Staffing Ratios: While having a 1:1 developer/QA ratio is a pipe dream, in some cases, projects have been able to achieve rations of 4:1.

This was the case at the two projects - the Toronto brokerage Powerbuilder application (a 6-month effort) and the telecom SAP human resources effort (a multi-year effort). Coincidentally, those were the same projects cited earlier for placing QA staff early in the development process.

According to the Toronto project leader, 4:1 was the optimal ratio. Our consultant had never seen a project where the QA staff was able to respond so rapidly to defects. This allowed QA staff to work almost as a buddy team with developers in the concurrent engineering fashion described earlier; as developers finished increments of an application, the QA specialist would test them, with the developer using the test results to refine the application as it was being built.

For the telecom SAP project, a cross-functional team of 300 included about 40 QA specialists; the project itself was divided into smaller modules, each of which usually maintained a 5:1 developer/QA staff ratio.

Elsewhere in the real world, staffing levels are related to the degree that QA services can be sold to development teams. For the Baltimore insurer, ratios of 15:1 are not uncommon. At the Minneapolis bank, four QA staff maintain a specific set of applications that are modified three times per month, for an organization numbering 4,800 people (admittedly, their efforts do not cover all application development within the organization).

Payscale: Reflecting their modest heritage, QA salaries have always been dwarfed by developers, since QA staff performed tedious functions and did not have to do any programming. QA people are finally getting some respect, said consultant Hayduk.

Today, the QA tools themselves have become a form of programming, and in a minority of cases, development organizations may also require their QA staff to know the 4GL development environment as well.

With the emergence of QA tools, QA skills have become rare and precious.

In general, in-house QA consultants are earning about 90-95% of the salaries of their developer counterparts, while in the consulting side, in growing cases, QA specialists are charging at a premium.

We have found, informally, that in areas such as Boston and New York, it is hard to find Powerbuilder developers - and even harder to find their SQA counterparts. Our observations were confirmed with the organizations in our sample. In New York, we found SQA consultants billing at $800-$900/day, compared to prevailing salary levels of $75,000-$85,000. Hayduk has heard of some QA professionals earning in six figures, plus bonuses.

Salary levels always vary by region, but the scarcity of proffessionals doesnt.

A Cleveland ERP site reported extreme difficulty finding personnel who knew Mercury Interactives LoadRunner. In Toronto, salaries ranged from $45,000-$80,000. In the midwest, respondents indicated paying QA professionals from $45,000-$65,000, which was about 90-95% of that received by developers; however, the Indiana insurer, which has an active program to beef up QA, is willing to pay up to 10% premiums for QA compared to developers (developers averaging $50,000, while QA averages $55,000 salaries).

Just about the only place where demand did not exceed supply was in Utah; the development center for a global mutual funds giant pays QA professionals with 3-4 years experience between $40,000-$55,000, which is much less than developers. Novell and WordPerfect have had major layoffs, so the market isnt as tight here, said the QA manager.

In general, consulting firms are often charging premiums for QA tools specialists, compared to 4GL developers.

We heard all sorts of rates; in the northeast, consulting firms contacted pointed to premiums ranging from 15-50% for QA specialists; in the Toronto area, we heard 30% premiums quoted.

WHATS THE ASQ TOOLS PAYBACK?
Now well cut to the chase. Yes, its costing more and more to recruit QA personnel, test tools (like any other software tool) take time to learn, and it takes longer to prepare and write a test script, if the tests are to be performed properly.

As for tool costs, the amounts vary depending on whether the tools are for testing clients or servers. Generally, front end GUI testing tools tend to be more modestly priced, up to $2,000 per seat, whereas server load testing is generally priced in the $30,000-$50,000 range for a single server license. Mercury Interactive, which sells a mix of server load testing, front-end testing, and test management tools reports typical sales averaging $50,000-$100,000. Training often costs about $1,000-$2,000/day.

We have so far found that few if any organizations have documented the benefits of automated testing. But we know that many organizations are looking for answers. One user (not part of the sample) remarked, Our QA center (which is in its infancy as a formally recognized organization), has been meeting with several other Canadian financial services companies to find out how they organize their QA efforts (staffing, tester-to-developer ratios, process improvement approaches), and how they conduct testing and in what sorts of technical environments.

Productivity: This is the obvious payoff. A test tool that has the capability to record and playback screen sessions eliminates the need to rewrite the routine when a new version of the same software is released. Similarly, a test tool with a test repository and management module can provide a degree of automated documentation. A test tool with scripting capabilities allows more complex tests to be devised; for instance, it can add conditional logic to routines that simply record keystrokes.

Ultimately, if the test tool scripting language is object-oriented, or if the test tool is integrated with an object modeling tool, there is the possibility that test scripts or modules could be reused in different applications.

The most common scenario was the reuse of test scripts for new releases of the same application. A typical example, at the Indiana insurance company, computed a rough payback of less than a year, if the cost of the tool (and associated training) is excluded.

The equation for one project was:

* Test script development: 32 hours (about 2-3x the amount of time for writing up a manual test).

* Run test script: 1.5 hours.

* Test new releases of software: 4x annually.

The payoffs increase with the frequency of testing. The Minneapolis bank conducts weekly software updates for its 40 service regions, where the four-person QA staff run 17,000 test cases in 20 minutes.

The west coast bank developing on-line banking adds changes weekly. Each release requires thorough regression testing. The consultant estimates that testing the updates manually would require at least 20-25 people, whereas with the automated tool, testing can be performed by six people on a part-time basis over a two-day period; he estimates the labor savings are 75%.

Intangible Savings: Several respondents pointed to the advantage that automated testing is more comprehensive. With the use of conditional test scripts, users can exercise more what-ifs and test applications more thoroughly.

There is also the notion that problems nipped in the bud prevent problems from snowballing. If you do a better job of finding software problems up front, this will reduce other costs such as help desk charges down the road, said Hansen.

Author Barry Boehm said long ago that a defect that takes only 1 hour to correct at specification time can snowball into at least 100 hours once the system enters production - after accounting for all the problems, delays, and the necessary effort to search for the bug, which by that time is buried under thousands of lines of code.

Significantly, we havent located any studies formally documenting that claim.

Automated tools can be used to build, enforce, or reinforce discipline, especially with the management capabilities (e.g., automated documentation and results tracking). One user noted that the use of ASQ tools forced us to get our act together in developing a structured approach to building test requirements.

Like any tool, automated software quality assurance does not guarantee quality. Faulty tests are simply conducted faster if not properly planned.

Furthermore, automated tools cannot test everything. Developers still need to check their logic manually before handing their code off to QA. The same goes for end user acceptance testing, and for testing out weird cases on a random basis.

For large applications, we found that over half the testing process is not automated. Automating the testing process will never replace the dose of diplomacy that is necessary when informing developers that there are defects in their code.

 

User sector	Application type	Application source	Application vendor or tool	Scope	Test tool vendor
Educational institution	ERP	Vendor	SAP	Enterprise	None (SAP tools under consideration)
Electric utility	ERP	Vendor	SAP	Enterprise	AutoTester
Financial (banking)	ERP (HR only)	Vendor	PeopleSoft	Enterprise	Mercury Interactive
Financial (banking)	On-line banking	Internal	Java, HTML	Business unit	Segue
Financial (insurance)	Customer information	Internal	C++	Enterprise	Softbridge
Financial (insurance)	Various	Internal	Various	Enterprise, workgroup	AutoTester
Financial (insurance)	Various	Internal	Various	Enterprise, workgroup	AutoTester
Financial (securities trading)	Portfolio management	Internal	Powerbuilder	Workgroup	Rational SQA
Financial (securities trading)	Trading, research	Internal	C++, Powerbuiler	Business unit	Rational SQA
Financial (securities trading)	Various	Internal	VB, C++, Delphi, Entera middleware	Enterprise, workgroup	Segue
Financial	Workflow	Internal	C	Business unit	AutoTester
Manufacturing (computer peripherals)	ERP	Vendor	Oracle	Enterprise (single site)	None
Manufacturing (computer)	ERP	Vendor	SAP	Business unit	AutoTester
Manufacturing (electric motors)	ERP	Vendor	Oracle	Enterprise	None
Telecom provider	ERP (HR only)	Vendor	SAP	Enterprise	Mercury Interactive