Outside Threats can Wreck Your Business. Protect Yourself!

Previous Page: Previous: Remote Alarm Block

Next Page: Next: Success Story

The blackout of August 14-15 cut off power to eight American states, one Canadian province, and at least 50 million people. Among the blackout victims were telecom providers in the affected area, who had to scramble to provide constant customer service during the outage. And while 99 percent of customers had service during the blackout, that still means that hundreds of thousands of customers had dead phones.

An event of this magnitude clearly demonstrates that network outages can happen for reasons totally outside the network. Many network outages are caused by factors that network operators cannot control. In recent years there have been a number of network failures linked to outside causes:

• Cable cut by outside personnel: Utility workers, public employees, and construction workers have halted telephone service for thousands of customers for hours by inadvertently cutting underground fiber while digging.
• Power Failure: Failure of commercial power creates many potential sources of network outage - and not just the obvious ones. Besides straining backup power sources, blackouts and power surges can blow fuses and trigger automatic shutdowns, cutting off service to customers.
• Weather: Storms regularly damage phone lines and commercial power transmission, resulting in dead phones for thousands of customers.

These factors don't just affect your quality of service report. They hit your bottom line - and the bottom line of your customers.

Network Failures are Expensive for You and Your Customers

For a quick lesson in network crash and revenue burn, one need only look back to the well-publicized August 1999 case of MCI Worldcom. It began about 10 p.m. on August 5, when technicians noticed a high level of congestion on the frame relay network. It evolved into a nightmarish house of cards for both MCI and its customers.

MCI had recently upgraded to a more scalable infrastructure, a move that reportedly caused the initial congestion and led to under-performance and complete network instability for over a week. As efforts to fix the problem repeatedly failed, MCI was forced to shut down the whole system for 24 hours.

The Chicago Board of Trade was one of MCI's 3,000 customers rendered helpless by the outage. The failure disabled the electronic system that governs the board's exchange leading to an estimated loss of some 180,000 trades. At anywhere between $10,000 and $100,000 per trade, the loss of business was significant and tough to calculate.

The same could be said for national truck stop operator TravelCenters of America, another customer for whom the wheels of commerce ground to a halt. In an InternetWeek story published at the time of the meltdown, Bill Bartkus, vice president of information systems for TCA, said he would seek compensation for lost business. "We're not satisfied at all with this," he said. "There has been a serious impact on our business."

Service Level Agreements Don't Protect Against a Bad reputation

Under real-world conditions, 100% reliability is a practical impossibility; outages are bound to occur sometimes, no matter how robust the network. Your customers need to know this, and it should be spelled out in your service level agreement (SLA).

But SLAs have their limits. If an outage happens, your SLA may protect you from legal liability, but it can't protect your from a loss of reputation with your customers.

According to Bill Harris, a senior advisor at the telecom consulting and management company QCI, sticking to the letter of your SLA can do more harm than good. How you handle a service outage can make or break your business reputation.

"It really depends on lot of things," Harris said. "Was the outage covered in the press? Did the service provider immediately come forward and address the issue or let it linger? Did they make good even if they didn't need to?"

Network Monitoring is Crucial for Preventing and Mitigating Outages

Once an outage occurs, for whatever reason, the question often becomes one of duration. A study of headlines coming out of the telecommunications industry reveals one reason why failures take a long time to notice and an even longer time to fix: a scaling back of the workforce. Companies, said Harris, "are reducing headcount across the board."

Some of these reductions are coming in the area of network support with service providers using fewer people to monitor an increasing number of network miles and switches. The logical result of less dependence upon human capital is greater need, according to Harris, for "more robust, better installed and better maintained" alarm reporting and control systems.

For example, a major carrier experienced a major service failure after an 18-hour power outage drained emergency batteries. The real culprit in the service failure was a malfunctioning power alarm.

"Monitoring services were not activated in a database and the carrier did not know what was happening at the switch," Harris said. Had the monitoring system been functioning appropriately, network downtime would have been significantly reduced.

Bob Berry, president and chief executive officer of Fresno, CA-based DPS Telecom, agreed. "Our busiest days are typically Monday mornings when administrators find out the hard way that their network alarms were not adequately monitored. However, the overall expense of network outages is oftentimes underestimated by people." Berry adds, "In addition to the loss of revenue associated with outages, companies are also faced with FCC fines, SLA penalties, customer churn, and even a damaged reputation."

Network Visibility is Essential to Avoid Outages and Penalties

While some outages cannot be avoided, they can certainly be corrected quickly with the proper monitoring system in place. Today's advanced remote site monitoring equipment can notify administrators across the country of an event that is happening or about to happen to their network. Some systems can even notify multiple people and NOCs so that a technician can be deployed to the troubled site while an administrator in another region can reroute traffic through a different path.

Berry adds, "Today's monitoring systems allow you to build rules for derived alarms and notification escalation. For example, if two noncritical events occur at the same time, it could be considered very serious. And alarm escalation lets upper management know when SLA sensitive clients may be affected." In the telecom industry, an ounce of prevention truly is worth a pound of cure.