Y2K Final Report: Ending With A Whimper, Whimper, Not A Bang

Issue Date: 02/01/2000
Issue Number: 10.09
Category: Y2K

The IT world has just gotten over the big one - the fixing of aging systems to make them Year 2000-compliant. It was huge project that snowballed from a long string of small decisions years ago to abbreviate year designations to two-digits. The $300 - $600 billion estimates on Y2K expenditures worldwide from Gartner Group have become modern IT folklore. Nonetheless, a lot of time and effort was spent.

Y2K was the full Employment Act for IT. According to Leon Kappelman, a co-chair of the Society for Information Management's Year 2000 Working Group, and a professor at North Texas State University, the effort involved from 500,000 to 2 million professionals over the past 2 -3 years. Roughly $25 - 30 billion was spent on Y2K work over the past year in the US alone, according to Meta Group research fellow Howard Rubin. And, according to urban legend, next to nothing was supposedly spent in Italy (we'll discuss that below). Nonetheless, in both countries, the outcome was pretty similar: civilization survived.

We've been tracking end user Y2K preparedness since late 1997 with periodic reports (see Computer Finance, October 1997, May 1998, March 1999, August 1999). With this report, we look back and see how everybody did. We've polled 19 organizations of all sizes across a wide range of industries, and we've consulted the experts. This report examines how much we've all spent, how we fixed Y2K problems, and how Y2K efforts have impacted IT development.

SILENT NIGHT

While the rest of the world was gazing at tribal rites in the western Pacific, floral fireworks displays gracing the Eiffel Tower, abortive attempts for setting the Thames afire, or the dropping of the ball at Times Square, many organizations were augmenting their skeletal graveyard shift staffs with additional sober eyes. In most cases, nothing happened. The lights stayed on at midnight and IT systems kept functioning. 'We had lots of buffet food and a champagne toast, and yawned through it all,' noted Larry Bloom, Y2K coordinator for Mercator Bank.

The few glitches that did arise tended to be almost laughable. For instance, at one company, the voice mail system gave the right dates on the phone, but erred on the screen displays, showing the year 19100. According to Gartner, less than a dozen computer viruses materialized, and none caused any damage. Most embedded systems continued operating, with the biggest 'problems' being the need to hit reset buttons.

However, according to Gartner, fewer than 10% of all Y2K faults would hit during the first two weeks of January. Some faults occurred previously during FY 2000 rollovers the previous six months. Gartner predicts that 55% of all failures likely to hit between now and September 1, 2000, mostly due to corrupted data which won't necessarily cause any applications to crash. For instance, the 'failure' of many small retailers to update their credit card terminals means that for a few months, some unlucky consumers might get hit with bogus double charges that they will have to get corrected. In all, Y2K glitches should be transient and readily fixable. And then there's the question of February 29, which in most centuries - except of course, Year 2000 - doesn't occur.

Arguably, Y2K money was well spent. For some organizations, major projects such as ERP migrations were accelerated during the 1996 - 98 timeframe, enabling these organizations a stronger platform from which they could speed up their supply chains or build e commerce strategies. For others, it was the excuse to rummage through ancient code, take stock of it, fix up the parts worth keeping and eliminating the dormant function points. In so doing, organizations had to beef up their change management, project management, and asset tracking processes if they were to successfully get their code cleaned up and tested before the witching hour.

Rick Ridge, who coordinated Y2K efforts for the Orlando Regional Healthcare System, summed up the hopes of many, when it came to lasting improvements from the effort. 'I hope [in the future] we will be more diligent in staying current on application system releases and track more closely the management of systems not supported by IS,' he said.

That's the silver lining of the cloud. On the downside, many projects were delayed. For evidence, just look at the recent stock prices of your favorite ERP vendor. Additionally, while the heightened sense of urgency that yielded accomplishments such as improved processes, contingency planning, and more accurate IT asset inventories, it's human nature to sit back on one's laurels and let things slip. Maintaining the improved asset databases, stringent change management processes, and commitments to quality could pose an even steeper challenge going forward.

RESEARCH SAMPLE

We surveyed a cross section of organizations, ranging in size from a regional realtor with a single-person IT staff to global enterprises whose IT staffs numbered in the thousands. Our sample also cut across industries, from financial and services to retail, manufacturing, healthcare, government, and academia. A description of the study sample is shown in Table 1. Additionally, we validated our results with the Arizona Millennium Group, a regional Y2K user organization, and checked with several authorities on the topic, Howard Rubin, who has conducted quarterly surveys of 147 Fortune 500 companies for Cap Gemini, and Leon Kappelman.

Table 1. Research Sample (Source: Computer Finance).

Company	Org. Type	Org. Size	Project Scope	Project Start	Percent work performed Internally	Application Inventory	Y2K Project Strategy	Y2K Impact on IT Projects
Choice Hotels	Hotel Franchiser	Small	$300,000	1998	100%	Mostly homegrown; Most apps. recent and already Y2K-compliant	Date Expansion where needed	IT projects delayed 90 days; Accelerated replacement of legacy property management system
Dow Chemical	Chemicals	Large	$78 million + $5 million (new systems)	1997	25%*	70% packaged applications; Remainder is homegrown or heavily modified	Mostly windowing	Many IT projects delayed
Ezralow Co.	Regional Realtor	Small	$5,000 (our estimate)	1999	100%	Mostly packaged applications	Installing vendor patches	IT projects delayed 6 months
Home Depot	National Retailer	Large	$10.3 million	1997	100%	Mostly homegrown; Most systems built after 1991 and already Y2K compliant	Date Expansion where needed	Few IT project delays; Future software releases to be bundled
Lockheed Martin Tactical Aircraft Systems Co.	Aerospace	Large	Over $10 million	1998	100%	Mixed: Homegrown mainframe applications; Homegrown and packaged UNIX/PC apps.	Upgrade hardware; Remediate software (mostly windowing)	Many IT projects delayed; Improved IT project management and more disciplined AD life cycle processes
MedStar Health	Regional Healthcare Provider	Midsize	$20 million	1998	30%	Mostly packaged applications	Windowing	Few IT projects delayed; Improved IT asset tracking system; Implemented new desktop software distribution system
Mercantile Bank	Banking	Midsize	$9.3 million (not incl. internal labor)	1996	Mostly internal	Mostly homegrown	Windowing; Date Expansion where needed	Spurred new IT projects; Improved IT asset and license tracking
North Pacific Insurance	Regional Insurer (recently acquired)	Midsize	$1.1 million (plus $600,000 for new hard- ware.)	1996	25%	Mostly homegrown	Date Expansion	Few IT projects delayed; Improved existing IT asset tracking system; More detailed contingency plans
Oce Printing	Printing Services	Midsize	$500,000	1997	100%	Mostly packaged applications implemented after 1994	Date Expansion	No impact on IT projects; More detailed contingency plans
Orlando Regional Healthcare System	Regional Healthcare Provider	Midsize	$4.2 million	1997	95%	Mostly packaged applications	Upgrading and testing vendor patches	Few project delays; Improve release/version control for IT and embedded systems
Premera Blue Cross	Regional Healthcare Insurer	Midsize	$8 million (our estimate)	1997	25%	Multiple homegrown systems + some packages; Most needing remediation	Date Expansion	Accelerated system consolidation projects; Some IT projects delayed; More disciplined project mgmt. processes
Presbyter- ian Health System	Regional Healthcare Provider	Midsize	$8 million (does not include new systems)	1996	60%	Mostly packaged applications	Installing vendor patches; Replacing dated systems	Spurred new IT projects; Improved existing asset tracking system; Improved project mgmt. processes
Province of British Columbia	Government	Large	$80 million (Can- adian)	1996	Significant outside contracting	Mostly homegrown	Varied by agency	Many IT projects delayed; More disciplined AD life cycle processes; More complete contingency plans
Prudential Fox & Roach	Regional Realtor	Small	$10,000 (our estimate)	1999	100%	Mostly packaged applications	Installing vendor patches	IT projects delayed 90 days
SF News-papers	Publishing	Midsize	$3 million	1997	98%	Mostly packaged applications; Several homegrown systems	Windowing; Replacing legacy systems	Many IT projects accelerated; New IT asset tracking and help desk systems; Improved project mgmt. processes
Texas Instruments	Electronics	Large	$65 million	1995	100%	Mostly homegrown; Implementing many packages	Windowing; Installing vendor patches	Few IT project delays; New IT asset tracking system; Improved project mgmt. processes; More detailed contingency plans
Univ. of Maryland/Office of Telecom. Services	Telecom Services Dept.	Small	$300,000 (our estimate)	1999	100%	Many telecom switches replaced; Software upgraded	Replaced 10% of equipment; Upgraded software	Scattered telecom project delays; Improved IT asset tracking
Univ. of Penn. Health System	Regional Healthcare Provider	Midsize	$18 million	1997	50%	Mostly packaged applications	Installing vendor patches; Replacing dated systems	Spurred new IT projects
US Cellular	National Wireless Telecom	Midsize	$8.5 million	1996	70%	Homegrown billing system; Most other apps. are packages	Installing vendor patches; Windowing where needed	Did not delay IT projects; Accelerated replacement of legacy billing system; Developed more comprehensive, end-to-end testing processes

*At Dow Chemical, consultants were part of a long-term outsourcing agreement that company already had with Andersen Consulting for detailed design, programming and much of the testing. For this analysis, we classified that as internal staff, since the Andersen consultants were not hired specifically for Y2K work.

We asked our respondents about:

* The duration of their Y2K projects, and when they started and ended.
* How did they staff their Y2K efforts. Did they rely on internal staff, consultants, or both (and if so, what roles staff and consultants played)?
* What was their technical approach: date expansion or windowing?
* How much (staffing, resources, funding) was invested?
* How did Y2K efforts impact other IT projects?
* What were the biggest surprises and lessons learned?
* Have Y2K efforts changed internal IT processes or conveyed any benefits going forward?

FINDINGS

Project Duration. Some early birds began exploring the problem four years ago. At Texas Instruments, a small task force convened in late 1995; otherwise, the earliest efforts from our sample dated from the following year. For instance, US Cellular was driven by the need to make 3-year customer contracts Y2K compliant as of 1997, while North Pacific Insurance, a property and casualty company, promptly began its assessments in January 1996. North Pacific's goal was to complete assessment within six months, and have all remediation finished within 18 months after that (by year end, 1997).

Several companies in our sample benefited, either from foresight, or their status as young companies with relatively new, well-documented code bases. 'Y2K was not a big issue for us,' noted Gary Thomson, Vice President of information systems for Choice Hotels, which had a policy of using four-digit year fields since the mid 1990s, and whose application portfolio was relatively new. The situation was similar at Home Depot, which also began using four-digit year fields beginning in 1991. Therefore, although the retailer had a large 20 million LOC code base, much of it was already compliant before the Y2K project started. At Prudential Fox & Roach, the Philadelphia area operation of a national realty organization, most hardware and software assets were two years old or young, according to Dean Panos, who manages IT systems.

Reflecting the fact that many people did not take the Y2K issue seriously until 1998, several respondents indicated that assessments begun as early as 1996 - 97 were initially not taken that seriously. 'We spent about six months trying to persuade people of the importance of Y2K,' said Russell Opland, Y2K program manager at the University of Pennsylvania Health System (Penn), who noted that the original goal was to have departments conduct their own assessment. In the end, Penn contracted that part of the job out at a cost of about $3 million.

In most cases, work was not judged complete until the last minute. For instance, MedStar Health, which began its effort in 1997, did not finish most testing until October 1999, a result that was fairly typical.

Budgeting. We did not get scientific data on how large the spending was as a chunk of the total IT budget, but anecdotal evidence ranged from miniscule to up to 50% of all expenditures during 1998 - 99. Respondents reported spending from a few thousand dolars to tens of millions of dollars. For instance, several organizations polled by the Arizona Millennium Group, ranging from 500 - 10,000 employees in size, reported spending from $2 million to $11 million, according to Michael Sperling, who headed the group.

Rubin's more scientific sample revealed consistently that two thirds of his sample spent 21% - 30% of the IT budget on Y2K. In our sample, Premera Blue Cross, a Pacific Northwest regional healthcare insurer, was right on target, spending about 17% of its IT budget on Y2K during its 1997 - 99 project run.

Organizations with modern IT infrastructures got away easy. For instance, US Cellular, whose product is driven by technology, consistently spends $40 million annually on IT, making its $8.5 million, four-year Y2K effort appear trivial by comparison. The same was true at Choice Hotels, which spent only $300,000 on Y2K out of a total annual IT budget of $16 million.

Complicating the analysis was the fact that several companies in our sample underwent mergers and acquisitions during this period. That proved the case at North Pacific Insurance and MedStar Health; however, their Y2K efforts were not impacted by merger and acquisitions, as the first priority was to get things cleaned up before IT consolidation entered the agenda. Conversely, Premera used the occasion of Y2K to accelerate transition of systems from a newly merged unit.

How accurate were the estimates? In most cases, not very, according to Rubin's sample, where 80% admitted their original assessments did not reflect the full cost. That was not the case at North Pacific Insurance, whose detailed tally of its three million LOC (lines of code) base was in place before the Y2K project began back in 1996, making the company confident in its $1.1 million remediation estimates (the company also spent about $600,000 to replace hardware, which in most cases was anticipated).

Where did the money come from? The obvious source was the IT budget. For instance, that was the case with the Province of British Columbia. The story was similar at Dow Chemical. 'We completed the Y2K from existing [IT] resources and other projects were delayed,' said Doug Snoddy, Dow's Year 2000 program director.

However, in most cases, money came from IT and supplemental funding sources; well over 80% of Rubin's sample reported funding coming from multiple sources. For instance, in our sample, the $18 million spent on Y2K by Penn was supplemental to its normal IT budget. The same was true at Presbyterian Health System (PHS), a New Mexico-based healthcare network, which used IT and supplemental funds to cover its $6 million effort. At TI, the IT budget was increased or Y2K in order not to impact other projects. At San Francisco Newspaper Association, the joint publisher of the city's two dailies, the $3 million budget was mostly in addition to normal IT funding.

Staffing. Y2K projects managers are certainly having the last laugh here. Remember those dire prognostications of Y2K consultants getting more and more expensive as time marched closer to the millennium? In our first report (see Computer Finance, October 1997), we reported Meta Group's predictions that Y2K service costs would rise 50% annually through 1998 - 99.

In the next report in our series (see Computer Finance, May 1998), we reported that an ITAA (International Technology Association of America) study reported plenty of excess Y2K consulting capacity. And we surveyed regional Y2K user group leaders who indicated that the myth of the $100,000 COBOL programmer proved to be just that. 'We were pretty much able to keep consulting costs down to $50 - 54 per hour,' said Alan Wold, applications manager at North Pacific Insurance. 'We never had any problems finding consultants,' added Paul Shapin, who helped coordinate the Y2K program for MedStar Health.

Some organizations have long-term outsourcing relationships where in-house consultants performed the Y2K work. For instance, at Dow, which outsources application development and testing to Andersen Consulting, Andersen did the Y2K remediation and testing, while Dow IT staff retained project management responsibilities.

The reality was that most organizations primarily relied on internal staffing. A popular reason, as Texas Instruments explained, was that the job was best suited for professionals who knew the code base. Since TI had relatively little turnover in its IT staff, it decided to keep the task in-house. At San Francisco Newspapers, consultants were only called in at the end to check internal remediation work.

Rubin's figures show that most of the consulting was performed in 1998. At the peak (March 1998), Rubin's sample reported allocating 49% of their Y2K budgets for consulting.

That percentage declined to 25% a year later, and only 8% by December 1999. By comparison, Kappelman's research found that only about 5% of Y2K work was outsourced. By contrast, only a third of our sample reported outsourcing more than half their Y2K work, while another third kept the job entirely in-house.

In many cases, outsourcing was selective. At Home Depot, Cap Gemini was hired to do code scans because they had an automated tool. Penn brought in consultants to do assessments because of staff inertia. On the other hand, Premera chose to have internal staff do the assessment, while leaving consultants to perform the code fixes and testing. Perhaps the driver was that, having chosen the more labor-intensive strategy of date expansion, Premera did not want to tie up all its IT resources.

Remediation Approach. In the case of packaged applications, users were at the vendor's mercy. 'When we looked at off-the-shelf software, in many cases, we could not rely on the vendor's documentation alone,' noted Jack Guthrie, Year 2000 program manager for Lockheed Martin Tactical Aircraft Systems Co. In some cases, software vendors kept customers waiting until the last minute. For instance, SMS didn't deliver its final Y2K patches to Penn until December 6, 1999, according to Penn's Opland. For homegrown code, there was a choice of two strategies, including:

Date Expansion, where original two-digit year fields were converted to four digits; and

Windowing, where original data is left alone, but manipulated with external logic to represent either the 20th or 21st centuries. A popular option was to treat years 65 - 99 as twentieth century dates, and 00 - 64 as 21st century.

Windowing proved the more popular technique because it allowed source systems to be left largely intact. According to Kappelman's survey of large organizations, 46% stated that they did not use date expansion techniques at all, while 32% agreed that it was their primary remediation policy (the survey was conducted using a 1 - 9 scale, of disagree to agree).

The obvious drawback of windowing is that it isn't a permanent fix. The betting is, obviously, that the programs in question would be replaced well before the chosen rollover date; however, that was the same logic used for justifying two-digit year fields in the very programs that were just fixed. A less obvious drawback of windowing, added Premera CIO Alan Smit, is that developing and applying windowing logic requires some knowledge of the business, making the task more difficult to outsource.

In our sample, for organizations with large inventories of homegrown code, there was a surprisingly even breakdown between windowing and date expansion, but there were also some extenuating circumstances that may have explained this anomaly. For one thing, since our survey was voluntary, our sample may have been skewed towards organizations that took more aggressive Y2K strategies. For instance, two respondents|Home Depot and Choice Hotels - each adapted four-digit year fields as their development standards way back in the early 1990s.

Another respondent, Oce Printing, actually made four-digit year date fields one of its criteria when it chose SAP in 1994.

Were these strategies admirable? Yes. Typical? Probably not.

Tools. In previous reports, we examined the usefulness of automated tools for assessing and testing Y2K-sensitive code (see Computer Finance, March 1997). We found them useful for objective tasks, such as identifying date field occurrences and most date dependencies, but obviously no substitute for human judgment on whether something should be fixed.

Most of the larger organizations in our sample reported using tools to varying degrees. For instance, Home Depot, Penn, and MedStar Health each hired consultants who used automated tools for assessments. Others, such as Mercantile Bank, used tools for assessment, testing, and documentation, finding the latter feature especially useful when dealing with state and federal regulators. For its part, US Cellular actually built its own assessment tool.

Y2K Problem Incidence. Not surprisingly, mainframe programs accounted for the brunt of remediation work. At Home Depot, it was the systems developed before 1991, which were mostly mainframe-based. A similar case was true at US Cellular, where the problems were in older COBOL programs.

According to Rubin's figures, mainframe code dominated the Y2K spending picture, accounting for 50 - 60% of the expenses over the past two years. By comparison, network upgrades consumed 16 - 30% of the effort, while PCs only represented 8 - 15% of the pie.

In our sampling, PC upgrades played a fairly minor role. Keep in mind that it is standard practice to replace PCs every three years anyway. For instance, TI regularly refreshes its machines on a rolling basis every three years, while North Pacific Insurance replaced half of its 500 desktops during its two-year Y2K project.

In scattered cases, older PCs were retired; for instance, MedStar Health replaced 300 machines out of its 4000 population. San Francisco Newspapers used the occasion to replace 300 of its 800 desktops with new, standardized configurations. In many cases, older boxes could be upgraded with simple BIOS patches, a strategy which MedStar Health also used. And of course, in our earlier research on Windows thin clients (see Computer Finance, December 1998, January and February 1999), we found that this was another way to extend the operating lives of older desktops.

Testing. As we've noted all along, testing accounts for the vast majority of Y2K work. Gartner estimates that testing could account for up to 75% of all Y2K project costs. A typical response was provided by Mercantile Bank, which estimated that testing accounted for 40% of the Y2K effort. Although that proportion sounded a bit conservative to us, for Mercantile, that was still the biggest chunk of their project.

Another typical strategy was to add mainframe capacity, and a logical partition (LPA), to accommodate the test environment. For instance, MedStar Health spent a significant sum adding disk, memory, and processing power to accommodate the testbed.

Embedded Devices. 'Fear of the Unknown' would have been the best way to characterize this part of the Y2K problem, as we noted in our earlier report (see Computer Finance, May 1998).

Unlike IT systems, embedded devices are always made by someone else, and are not always easily under the organization's control. For organizations with capital-intensive facilities, they can represent a large chunk of the Y2K pie. For instance, at Dow Chemical, manufacturing process control equipment assessment and remediation cost approximately $15 million, compared to $42 million spent on IT (business systems and computing infrastructure) sprojects.

Embedded devices tend to live on for decades. As for date sensitivity, we learned that:

* Devices made before 1980 lacked microprocessors, and therefore had no Y2K issue.
* Devices made between 1980 - 1995 often were equipped with microprocessors, and might have Y2K compliance problems.
* Devices made after 1995 were probably Y2K compliant.

This brings up several issues. First, how important are the devices? At Penn, devices were categorized as biomedical, facility, and miscellaneous (e.g., phones, faxes, and other office equipment), and then prioritized. At MedStar Health, manufacturers were contacted for documentation. Given that biomedical devices are regulated by the FDA, MedStar Health's information requests had some clout. 'Although some manufacturers were slow to respond, we ultimately received the information in plenty of time,' said Caroline Campbell, who was responsible for biomedical device Y2K compliance at MedStar Health.

In non-regulated industries, prying loose Y2K compliance information was often like pulling teeth, especially for devices made by companies which have since been acquired by new owners. That was the case at San Francisco Newspapers, where much of the press room equipment had been modernized in the early 1980s with early-generation microprocessor-based controllers. Up to half the devices had Y2K issues, according to David Byers, Y2K project manager, who indicated that tracking down the specs of some of the equipment required detective work.

The operable question on compliance boils down to those devices manufactured during the 1980s and early 1990s. In most cases, compliance issues turned out to be minor. At Penn, less than 10% of the 28,000 devices were not compliant, and in many cases, it was a matter of simply adjusting displays; the equipment continued to work. At PHS, a relatively new healthcare network, much of the equipment was new to begin with, and therefore carried few compliance problems.

At the Univ. of Maryland telecom office, over 90% of the devices either were compliant or didn't count years, while the remaining 10% was 'almost compliant,' according to Mark Katsouros, Y2K coordinator for the University of Maryland's telecommunications office.

In many cases, the oldest switches were in the process of being replaced anyway. However, in a few isolated instances, obsolete equipment could not be replaced until after the date change, dictating interim solution. In at least one case, the workaround was to rest the device's date back to 1972 (a year where the days fall the same as this year). 'Whenever we dealt with that, we made sure to wear bell bottoms and flannel shirts,' said Katsouros.

Y2K Impact on IT projects. Some of the results weren't surprising. According to Rubin's data, ERP and application outsourcing initiatives were the hardest hit. But, if a project was important enough, it was still funded; not surprisingly, CRM (customer relationship management) and e commerce projects were less impacted.

Our sample was more of a mixed bag, with some indicating severe impact while others claiming that IT projects were not slowed at all. Thanks to having relatively newer IT infrastructures, Home Depot, US Cellular, and PHS Health System experienced little impact. 'We consciously decided that we weren't going to let Y2K delay IT projects,' said Jim West, CIO at US Cellular. Y2K was the rationale for retiring a DOS-based property management system earlier than planned at Choice Hotels.

For some organizations, Y2K was the clarion call to accelerate replacement of older systems. For instance, San Francisco Newspapers replaced its advertising application, migrated from Dun & Bradstreet (now Geac) Millennium, with Oracle financials, flushed out all pre-Pentium desktops, and implemented a new help desk and asset tracking system. In the case of the advertising system, noted Byers, the project was moved up about 3 - 4 years as a result of Y2K.

At other organizations, results were mixed. For instance, Y2K helped spur Mercantile Bank to move up the replacement of its retail banking system, but held off on web banking. Premera deferred numerous projects, but, as mentioned earlier, accelerated some such as consolidating systems from a recently acquired business unit. At the other end of the spectrum, organizations such as Lockheed Martin Tactical Aircraft and MedStar Health reported that many IT projects were slowed down.

Even in smaller companies, with shorter-fuse Y2K requirements, the impacts were significant. At the Ezralow Companies, a southern California realty system, Y2K remediation only involved getting BIOS patches to 20 486 machines and an upgrade to an old, DOS-based general ledger system. Nonetheless, roughly $60,000 worth of IT upgrades (including migration from Citrix WinFrame to MetaFrame, and an upgrade to its property management system) were put on hold, but not because of staffing or time crunches. 'We wanted to make sure that we got through the date change first,' said Steve Tamkin, IT manager.

Respondents polled by the Arizona Millennium Group indicated most IT projects were slowed down, excluding those judged strategic or legally mandated. According to Jim West at Choice Hotels, many projects, such as a guest feedback system and integration of email with call centers was pushed back from 1998 to 1999. At Prudential Fox & Roach, about half of all projects suffered delays during 3Q 1999.

Business Partner Compliance. In most cases, a two-tier strategy was used: direct interaction with strategic suppliers and pro forma compliance letters with everybody else. The letters had to be written, but did they provide any useful guidance? In the long run, response improved in relation to the amount of follow-up invested by the sender.

Lockheed Martin Tactical Aircraft Systems has approximately 4000 active suppliers, of which about 1700 are critical to aircraft production, according to Guthrie. In some cases, they performed on site reviews, a strategy similar to Oce Printing, which performed such reviews as part of its ISO 9000 process.

Home Depot conducted EDI tests, and found only 20% of its trading partners not in compliance. Letters went out to other partners. According to Ron Kerr, senior IS manager, getting response to the letters took a degree of 'arm twisting.' The responses, when they finally came, were often vague, indicating that the lawyers were leery about guaranteeing the unknown.

Home Depot's experiences proved typical for all sectors except where government regulation provided added clout. For instance, all healthcare institutions that we surveyed indicated few problems getting responses from suppliers of biomedical devices because of FDA requirements.

Supplier compliance involved a delicate balancing act, according to Karla Barber, director of the Y2K program office at TI. When the tables were turned - when TI itself was asked by its customers about Y2K compliance - the requirements for response became a moving target.

'Our customers kept evolving their demands,' she said, noting, 'We initially got simple letters of awareness, but eventually, some of them became multi-page questionnaires that you practically had to sign with blood.' At that point, TI bit the bullet and posted its own Y2K compliance website. 'Without our website, [business partner] compliance would have been a monumental task for us,' she said.

Contingency Planning. Also known as the answer to Murphy's Law, a major benefit of Y2K projects was that many organizations beefed up their contingency plans. That was the case with many B.C. provincial government ministries. 'Contingency planning had not been considered as seriously before this,' said Paul Sihota, who headed the provincial governments Y2K task force. He added, 'That was critical given the earthquake threat in our area.'

Contingency planning was no stranger at Oce Printing, whose south Florida headquarters is extremely vulnerable to hurricanes. 'Our existing contingency plans addressed 70% of our Y2K needs,' explained John Wolf, who managed compliance efforts. The major distinction was that, instead of preparing to move central office functions, Y2K would involve more forward movements of inventory out in the field.

Respondents such as Mercantile Bank and TI also upgraded their existing contingency plans for Y2K. 'We tried to base contingency plans off existing plans, but at many sites, the existing plans were only about 80% complete,' said TI's Barber. Not surprisingly, healthcare providers were the most vigilant, because, unlike most businesses, failure could be a matter of life and death. For instance, MedStar Health planned for 3-day outages, involving measures such as buying extra auxiliary power sources and preparing to track patient information on laptops. At Penn, all operational areas had to update their contingency plans, but, noted Opland, they didn't attempt to plan for every contingency. 'There was a moderate level of detail, but we didn't go hog wild.' PHS was more ambitious. 'We had numerous fallback positions,' noted Ralph Wallace, who oversaw Y2K compliance, adding, 'Certainly we could have spent less, but who is to say that all this contingency planning won't serve us well in years to come.'

New Years Weekend. Organizations with 24 x 7 operations tended to beef up their staffs over New Years weekend, but in most cases, the all-night vigils over new years proved anticlimactic. Typical experiences included:

* Premera deploying about 300 additional IT and business staff over New Years weekend, encountering less than a dozen minor program errors.
* Mercantile Bank stationing 50 IT and business staff on standby during New Years Eve, and 400 people the following morning for conducting tests. At least they brought in some catering. The only departure from normal was slightly higher ATM traffic, which was easily handled.
* Texas Instruments deployed 200 IT staff over the weekend (double the normal weekend staffing), encountering only minor problems, such as incorrect report screens.
* British Columbia provincial government catching only a half dozen reported glitches, such as the rebooting of a tape management system.
* Choice Hotels encountering a handful of errors in shell scripts, which transfer data between systems.
* Oce Printing deploying two additional customer service representatives over the weekend, who never had to page IT staff for backup.

WAS IT WORTH IT?

'It was a great experience. I'm glad it won't happen again,' said Mercantile Bank's Bloom, summarizing the general sentiment. Given the huge build-up on the potentially disastrous effects of Y2K bugs, the anticlimax of nothing going wrong makes it tempting to wonder if the hundreds of billions spent on remediation were all for naught. 'If we hadn't renovated our code, it would have been death by a thousand cuts,' responded Premera's Smit.

Nonetheless, cynics cited the Italian telco paradox, where supposedly, nothing much was spent, and nothing bad happened. Rubin dismisses the Italian paradox as myth, countering that, while the government spent little, private industry didn't ignore Y2K. He also stated that the tools and expertise improved significantly to allow even late starters to catch up and fix the bug, an observation that was backed up by Sperling of the Arizona Millennium Group. 'The latecomers clearly didn't have to spend as much because they learned valuable lessons from the trailblazers,' Sperling said.

Finally, Rubin adds, 'Critical infrastructures were never that vulnerable to the bug in the first place,' noting that only the most advanced systems had the types of digital controls that would have been susceptible. 'The people who own those systems knew about the bug and fixed it,' he said, adding other critical systems, such as air traffic control, had manual fallback procedures.

THE Y2K LEGACY

There were some obvious benefits, including the clearing out of the cobwebs of ancient, disused code, and taking inventory of the usable code that was left. Some organizations, such as Dow, already had robust IT assets tracked already, and therefore, Y2K didn't really change anything. For others, it was a different story.

The same was true for contingency planning, although some organizations such as Oce Printing concluded that they already had comprehensive plans in place.

As a huge enterprise wide initiative, Y2K projects required project management discipline. And, because Y2K often impacted core transaction systems, careful change management procedures had to be put in place in order to keep programs in use, while carefully tracking what could be taken out or migrated back into production.

The experience with Y2K taught the benefits of carefully controlling code releases, according to Home Depot's Kerr. 'We learned that it pays to bundle code releases together, issuing them less frequently, rather than constantly dribbling out program updates,' he said.

According to TI's Barber, the exercise helped identify the real 'owners' of applications - the people who had the domain expertise to take responsibility for legacy programs, many of which may have become orphaned as the original sponsors or IT project managers for the affected programs left the organization.

At North Pacific, an IT staff retention bonus program paid results in limiting turnover, and ensuring that Y2K renovations would be completed on time (the program expires this march, and probably won't be renewed by the new management).

In essence, the benefits and lessons that were learned revolved around one concept: discipline. It's human nature to mobilize for an emergency, placing stronger management practices into effect. The results showed up in better project management discipline, change management practices, more accurate code inventories, and improved software quality. It's another challenge to maintain the discipline after the urgency has worn off.

Lacking management follow-through, the hundreds of billions spent will simply amount to another quick fix if IT organizations are unable to entrench their newly-improved practices.

© 2000 ComputerWire Inc