THE MYSTERY
There was a power outage at a municipal waste water collection system, which left a lift station dead in the water. The station wouldn’t run on generator power, and once regular power resumed the station wouldn’t come to life in auto or bypass mode either, so the customer called ACES to the scene.
THE CLUES
Upon inspecting the PLC (Programmable Logic Controller), the ACES’ Control Systems Investigator (CSI) found the PLC faulted out. He sleuthed around the SCADA backup batteries and ascertained they were at 21V when they should be at 24V minimum. Even more puzzling, the battery charger was on and should be putting out up to 28V.
Then he discovered that, although the battery charger was on, it wasn’t generating power.
The CSI cycled the power switch, with no results. He went a step further and unplugged the charger for a few seconds and plugged it back in, and it fired up and started generating 26 volts.
THE SOLUTION
This sparked the VFDs (Variable Frequency Drive) to start working — in bypass mode only — but not the transducers or floats. Further scrutiny revealed that one of the fuses was blown, and after it was replaced, the transducer and floats started working again.
The CSI surmised that when the power went out and the standby generator failed to kick in, the backup batteries ran until they drained low enough to fault the PLC. At some point the fuse blew, probably due to a power surge. When VFDs are in bypass mode, but without DC power to pull in a control relay, they won’t come online; without control voltage the floats and ultrasonic sensor are equally inoperable. (The pumps would run in bypass mode, but at only at full bore, without modulation from the VFDs.)
A FEW WEEKS LATER — A SECOND MYSTERY
One day the customer changed the backup batteries, and when he tried to power up the lift station it refused to run.
THE CLUES
The CSI determined that there were no blown fuses, and any faults on the PLC were resolved by cycling power to the unit. When he turned the power back on the “OK” light glowed green, indicating everything was normal.
However, even after a computer reboot the HMI still didn’t register data. The CSI checked communications and recorded a ping from the computer to the PLC, confirming that the computer could send data to the PLC.
At this point the CSI called the ACES’ HMI programmer and worked with the CSI to troubleshoot the Wonderware software. They ran through several checks, revealing that Wonderware diagnostics could not communicate with the PLC.
THE PERP
The CSI discovered that the PLC didn’t have a program. The “OK” light on the PLC wasn’t flashing, but this is normal operation for this particular model of PLC. The low battery indicator for the PLC backup battery is another matter: This battery holds the program in the PLC when the main power is off, and the low battery indicator should switch on when there is less than 5% battery life remaining.
In this case the low battery indicator stayed dark until the battery was totally removed, and then it popped on.
Tech support said that the battery backup circuit in the PLC was probably not working, so the battery circuit may not have been working at all.
THE SOLUTION
The CSI reloaded the firmware and lift station software to the PLC and the HMI started working normally. He also installed a new PLC backup battery, and tested it by powering down the unit for a few moments to ensure the battery was holding the program.
Now water is flowing freely through the lift station, and systems are go for both SCADA and PLC backup power and PLC program retention in the event of another power failure.