When backup power fails at water and wastewater treatment plants, the results can be catastrophic. But the good news is that such events are often avoidable by a proper assessment of backup equipment and disaster planning strategy. For example, a thorough assessment of equipment could have avoided a disaster in Connecticut in late 2011, when tropical storm Irene and October snow resulted in 47 spills with severity ranging from 42.7 million gallons of partially treated sewage dumped into Long Island Sound in Stamford, to 250 gallons of raw sewage poured into the Farmington River in Simsbury.
In almost all cases, the spills were triggered by flooding and plant power failures that should have been prevented by backup generators, but those generators failed. Officials cited old and underpowered generators as a large part of the problem, but also noted that the utilities were unprepared for power outages and weather events that would exceed a week.
Connecticut’s utilities should have looked to the experiences of West Virginia American Water. Earlier in June 2011, severe weather and outages exceeding 10 days tested the utility’s staff and backup equipment when a lethally destructive storm known as a “derecho” drove winds of 70 mph and higher through Indiana, Ohio, West Virginia, Virginia, Maryland, plus parts of New Jersey. Fallen limbs and trees took down power lines and left 680,000 West Virginia residences in the dark. And of course, utility power dropped out for 350 booster pumps, pressure reducing valves, and other equipment sites, all dependent upon electricity, to deliver 50 million gallons of water produced each day through 3,500 miles of water main across the state.
The crisis lasted 10 days, and while West Virginia American Water owned 28 portable backup generators, the widespread power outages overwhelmed their inventory, forcing them to borrow seven more from other American Water subsidiaries and purchase another 15. Crews shuttled generators around the system to keep the pumps and other equipment running, but in something of a domino effect, leaks developed because of the fluctuating water pressure, causing repairs and requiring water sampling and testing. But even though there were some scattered water outages, most customers saw little to no disruptions in service. Within eight days of the initial storm, crews had restored service fully to all but about 200 of its 171,000 customers.
It was another close call at the West Virginia American Water wastewater treatment plant in Fayetteville, where crews rotated portable generators around five pump stations at least twice a day for up to 10 days. An outside septic pumping company joined the effort to pump another site twice a day for a week. Some of the borrowed generators came from Pennsylvania American Water, and according to Dan Hufton, senior director of production for Pennsylvania American Water, disasters are just one of many factors that impact a backup strategy.
“When you consider a typical water system’s treatment plant, which is where everything starts on the water side, you’re looking at a fixed location, and you have a sizable load to back up, typically provided by a fixed generator,” says Hufton. “But out in the distribution system with hills and mountains, we have different pressure zones, and water has to be lifted from lower to higher elevations. So you may have a large pump station, and in those cases you often have a fixed generator because of the critical nature of the unit. But then you might have a series of small generators for smaller groups of customers where it’s not cost-effective to keep a fixed generator. We equip them with a switch because we like everything rigged up, so we have a quick connection without a lot of wiring.”
As for the power output, at booster stations the range is from 50 to 100 kW, with larger stations and plant
locations ranging from 200 to 500 kW. From a historical perspective, Hufton notes that grid reliability and backup power expectations have changed. “Several years ago we designed a generator to produce half the average day’s delivery from a plant because it was just there for backup and not for long-term power, and not too long ago if you had two independent feeds from an electric utility in your treatment plant, that was acceptable. But right now the standard in the water industry is really for two independent feeds from the utility, and you also want to have a generator that can produce an average day’s demand. Then, if you’re in the middle of the summer and it’s a high demand, the generators aren’t typically sized for that large of an event, so you would probably have to request use restrictions from customers to make up the difference.”
Because the plant’s load determines the generator’s capacity, it makes sense to wring as much efficiency as possible from the plant’s operations as part of the backup strategy, says Pete Chase of Honeywell Building Solutions, Minneapolis, MN. Generally, Honeywell’s projects are related to energy services contracts (ESCOs) so energy and operational efficiency improvements are a means of funding projects.
“You don’t want to put in backup generation on something that’s wasteful,” says Chase. “So, we want to get the plant running at a better efficiency and get energy use lowered, and then look at what’s needed for backup generation or if there are peak shaving opportunities. I had a situation with a plant where the biggest energy users were 500-horsepower pumps for moving water out of the high zones. We found that there were some inefficiencies in the pumps with the system getting some artificial head, and we were able to redesign the system to use 350-horsepower pumps. Then we logged all the runtime and other demands on the system.”
The head, or water pressure the pump must overcome to move water, is a critical factor in specifying a pump’s capacity, and moreover, whether a variable frequency drive (VFD) design is possible. “It’s very tough in municipal pumping because if you slow the drive down and you still have the head, the drive can actually make the system much more inefficient unless you have a very tight range of operation for that particular pump,” says Chase. “I’ve seen this happen with operators of plants where they are turning the pump down, and all of a sudden it stabilizes out and you find after checking the data that it wasn’t pushing any water at all.”
Rather than default to VFD pumps, another way to handle the energy load and accomplish peak shaving is through a process of load distribution. Says Chase, “We had a 350-horsepower unit running two hours a day so I could cut it back to 275 horsepower and run it seven hours a day, and now we’re doing the load shedding by chopping up the loads rather than having giant pumps running. A lot of these plants get oversized and you see 500 to 1,000-horsepower pumps, but why not have a 500 and a combination of 700 or some other smaller size that can run in flexible schedules?”
In one instance, the Honeywell team found an opportunity for peak shaving at a utility, where the load from a 500-hp pump could be offset with a generator that could also function in a backup capacity. When the pump’s operations were evaluated in the context of an overall efficiency program, the pump’s capacity was lowered to 350 hp, so savings were seen in the cost of a smaller pump, and a smaller generator.
“There are other ways to do peak shaving, and with that generator there is an option for a second generator to cover the plant’s load,” adds Chase. “So the peak shaver handles the critical load of the pump, and another generator can cover the balance of the plant load. Now you have a generator running every day, and you don’t have an asset that just sits there. So the utility is spending money on an asset that brings in a benefit all the time.”
Chase also notes that equipment upgrades need to be incorporated into disaster recovery strategies and plans. “The staff has to know their responsibilities, the sequence of starting the plant in an emergency, and also what do if a backup generator does fail.”
Such failures can be costly if they involve an environmentally sensitive area, according to Bill Cunningham, an engineer with Malcolm Pirnie, the water division of ARCADIS, Cunningham is working on a project in Henrico County to implement a regional water supply program in central Virginia. The $43.4 million contract is part of a $280 million endeavor that will span eight years, when reservoir filling commences. The project involves upgrades to the York River Water Treatment Plant, and Cunningham notes that backup power is critical to the plant and surrounding areas.
“They feed into the York River, which is part of the Chesapeake Bay watershed, so it’s all under a number of regulations and requirements that are fairly stringent. If they lose power and bypass their system, there’s all sorts of issues they have to go through with permitting and the State of Virginia.”
For York River, ARCADIS created a complete building to house an existing 1,350-kW generator plus a new 2,250-kW unit, because the plant was close to an urban environment. Engineers also had to create a solution for fuel storage.
“Fuel storage is an issue because requirements for running duration have increased,” says Cunningham. “Different agencies have requirements from 72 to 96 hours of fuel availability for the onsite generation. We had to build a fuel area capable of handling about 80,000 gallons of diesel fuel. You can imagine that these are very large storage facilities and needed a hundred-by-hundred foot lot.”
Just the logistics of building such products can be complex, but Cunningham notes that the impact to a plant’s staff should also be considered. “You want to make sure that the plant operator knows that there’s going to be issues of testing, and they have to understand that some equipment is going to be lost,” he explains. “Typically, there is redundancy in the plant and the operators can shift the process to supporting sources or use storage, and in a wastewater treatment plant, they can make sure the flow can handle the equalization. But they have to have a good plan and understand that prior to the commissioning there will be calamities if they don’t prepare themselves. If the plant operators stop the test to recover the plant, your contract could be put on hold while the problem gets resolved, and it can be very stressful.”
Coordinating with the local electrical utility is another consideration, as well as consulting with a plant’s operators regarding sensitive equipment. “Every time you stop and start a plant you put some stress on the system,” says Cunningham. “If you have older equipment that you’re worried about, sometimes it’s better to take it offline during the commissioning process. We had a plant with some 800-horsepower blowers, and you don’t want to be tripping those on and offline every couple of minutes. You want to plan with the plant operator some sort of strategy to remove the blowers from the system.”
Regarding the electricity supplier, Cunningham advises plants to protect their backup systems after an outage by reestablishing their grid connection slowly. “Every time you have an outage, there will always be a transfer where you lose one side and the other side picks up the voltage. But when you return to the grid, you can parallel the generators with the utility company and have a soft transfer so the plant doesn’t experience the higher level of stress.”
The Helix Water District in El Cajon, CA, recently had the need for a soft transfer after a full night without utility power. The experience tested the District’s backup equipment and, moreover, its emergency plans. Their plan provides a good example of a coordinated effort, and includes, ” policies, procedures, and an organizational structure for response to major emergencies.” It complies with the National Incident Management System ( www.fema.gov/national-incident-management ). The District holds regularly scheduled tests through exercises and drills that are designed to simulate actual events.
In an unscheduled event during a regional power outage on September 8, 2011, Helix employees had a chance to put their plans into action, with an all-night, but successful, effort that included trucking generators and fuel to pump stations, communicating with the San Diego County Office of Emergency Services, coordinating activities with the San Diego County Water Authority, and keeping the media appraised.
According to Kyle Swanson, Helix Water District’s system operations supervisor, the District saw several areas where it could dramatically improve reliability during power outages of longer duration. “Part of the assessment is looking at what you would expect in your average in peak demand,” explains Swanson. “Then, we determined the threshold for the district, so we have the ability to provide water to the customers with emergency backup power if there’s a district-wide power outage. When we looked at it globally, we wanted to keep service for customers at least 24 to 48 hours. So we asked ourselves how many generators do we need, and where can we use them strategically to combine and boost the system? Then we could have one generator at the right pumping facility rather than deploying two generators for two pumping facilities.”
The District’s improvements include an emergency generator standby program with a fleet of seven mobile generators, from 45 kW to 350 kW, to provide emergency power to the administration office, operations center, and any of the distribution pump stations. Critical facilities, such as the R. M. Levy Water Treatment Plant, have permanent standby generators. Additionally, new electrical connections were upgraded at various pump stations, and the District purchased 2,000-W, suitcase-style generators to power emergency electrical, electronic, and radio telemetry equipment at storage tank sites.
“One of the things that came out of this was our goal of getting the best bang for our buck,” adds Swanson. “If we have an area supplying 20 customers versus another system that has 2,000 customers, we want to give them both service. But it may take staff time and resources, so part of this program incorporates stationery generators at some of the smaller service locations, and then we could maintain service there rather than deploying staff members and a mobile generator to those resources that could go to the location with 2,000 accounts. So, you’re taking the services and distributing them to everybody, but getting more bang for your buck by allocating the mobile generators to a larger service area.”
In March 2011, Helix announced the launch of a new source of power that could keep its operations center running if the electrical utility suffers another power outage. Unfortunately, the new 290-kW solar photovoltaic system at its Nat L. Eggert Operations Center (OC) in El Cajon won’t be of much help at night. Nonetheless, the zero-emission system boosts the District’s sustainability factor and saves money. It’s deployed through a partnership with Borrego Solar Systems Inc., San Diego, a designer and installer of grid-tied solar electric power systems, and the California Center for Sustainable Energy.
Protecting operating centers is critical, according to Anusha Yalamanchili, senior engineer, Westin Engineering, Rancho Cordova, CA. Yalamanchili has specialized in project management, PLC programming, SCADA design, demand management, and more. She notes that the risk to control systems is a critical factor in keeping a plant operating.
“It’s more than power,” says Yalamanchili. “There’s also the need to restore communications and IT and programmable logic controllers. Plants should have copies of all programmable logic controller codes in one location. With the SCADA system, you have to make sure everything follows proper security procedures. And people are a component in a debt disaster recovery plan. We do workshops and help water utilities understand the organization and who the go-to people are. Things such as who are the first responders and what is the call sequence to get things done.”
If a plant is undertaking a backup upgrade, it’s the ideal time to consider an integrated approach with control systems, adds James Glegg, P.E., manager, Control System Engineering, Westin Engineering. “Some folks at plants have been pretty slow to modernize in terms of getting computerized controls, and backup power wasn’t always given the same degree of attention as power generation,” says Glegg. “So when they modernize, they have to look at backup power and UPS systems and infrastructure that can support more reliable robust and failure-free systems.”
Creating a failure-free system actually starts with finding the weakest points in a system. Glegg adds, “Look at the whole plant and find your points of failure and your failure modes. You have to do a failure mode analysis of the entire plant to figure out where the key points are.”
Finding key failure points could have been a deciding factor in avoiding the disasters of 2011 in Connecticut. But obviously, the subject of backup power involves an in-depth look at every facet of a plant’s operations-from hardware such as pumps; to water transport systems; to backup generation within a plant and in the transmission and service area; and, of course, the control systems, the IT systems, and the people that operate them. Ultimately, it’s a complex endeavor, but well worth the undertaking when compared to consequences of spills and disrupted water services.