Taming the Wild Alarm System, Part 5

October 31, 2022

Bill Hollifield

Young, enthusiastic engineer: “I’ve been doing some research and examining our control system. I think we need to do something called alarm rationalization.”

Production department head: “Really? Tell me more about that.”

Engineer: <Provides detailed explanation.>

Department head: “Hmmm, sounds interesting. Maybe we can do that in a year or two. That reminds me, I need to schedule my colonoscopy.”

OK, alarm rationalization is not (quite) as bad as that. There are some similarities, in that it is highly recommended, proven in practice and can find and fix minor issues before they become big, expensive, and dangerous problems (and some good technology makes it pretty straightforward and easy to do).

If you are not rationalized...

An alarm system that has never been rationalized is also likely to be one that was created without the benefit of a consistent alarm philosophy. It is often a collection of many different people’s ideas about what should be an alarm, implemented over many years, and changed at the whim of other people to reflect their personal preferences. There will be major inconsistencies in such a system.

Unrationalized alarm systems will have hundreds of meaningless and distracting alarms that make little sense to the operator. These are often (rightfully) ignored and the operators will have little trust in the overall system. This can result in other actually important alarms being ignored as well, because – how can you tell the difference? Alarm suppression methods may be in use in an uncontrolled way, with some alarms being suppressed for days or months or longer. These are upsets and accidents waiting to happen.

The alarm system is supposed to reliably inform operators of process conditions that need their attention. An unrationalized system will not do this.

How to do it

Alarm rationalization is a proven methodology for getting your alarm system in alignment with your alarm philosophy and making it work like it is supposed to. Hexagon has performed thousands of successful alarm rationalizations in plants of all types in dozens of countries. We know exactly how to do it correctly and cost-effectively. In The Alarm Management Handbook – Second Edition , we wrote a highly detailed chapter on exactly how to go about rationalization.

It includes lots of tips, as well as pitfalls to avoid. If you have never done a rationalization, we advise that you get some experienced assistance. Doing so will save you far more time and money than what you will spend by going it alone. We know how to minimize the impact on your plant resources – but you must participate! Don’t believe anyone that says they can rationalize your alarm system without your involvement.

For this blog we thought it more interesting to write about some of the strange and disturbing things we have found from doing so many successful rationalization projects!

Before and after

Before beginning rationalization, you will usually find that

Operator frustration is high
Alarms are occurring constantly
The alarm summary display is always many pages long
Most alarms are nuisances, with some even occurring based on timers
Operators are constantly hitting the acknowledge button (which does not mean that the alarm was read or understood!)

When you are done with rationalization, what will be the result? Here are some quotes and reactions.

From operators:

“Finally the alarm system makes sense.”
“The alarm system is useful now. It sure wasn’t before.”
“You can understand the alarms now – they have real meaning.”
“I’m not constantly dealing with a bunch of incomprehensible alarms anymore.”
“This is the best thing we have ever done.”

Quote from an Operations Supervisor: “… this rationalization project gave us a lot more benefit than just improving the alarm system. In fact, it helped us discover mistakes in our piping and instrument drawings (P&IDs), incorrect procedures, wrong point descriptors, and in general, misunderstandings we had about how our plant works. It also found a number of ‘gotchas,’ which should contribute to improved process safety and operability.”

Note that rationalization is not “just getting rid of alarms.” You will always be surprised to find that there are alarms you badly need that are not configured at all!

What we've found

Here are some of the things we’ve learned in doing so many projects.

From a presentation at a power industry conference: “The contribution of alarm handling to upsets, emergencies and incidents is not always recognized. The Operations and Maintenance team often accept alarm floods and nuisance alarms as the status quo.”

Those figures are not just “a little bit off of” the recommended guidelines, they are horrific!

Poor MOC and documentation are rampant

Poor management-of-change practices and out-of-date or “just plain wrong” documentation are common! In many instances, alarm rationalization is the first time anything that even vaguely looks like a process review has been conducted since the plant’s initial start-up. It is rare to find any effort that has involved comparing what is actually configured in the control system with what is detailed on a P&ID or in loop diagrams!

Here are some typical findings. They result in badly needed changes to the process documentation, operating displays, programming logic, procedures and sometimes the equipment itself. How safe is a plant with the following issues?

Wrong or missing tag descriptors
Wrong alarms configured on a graphic
Important alarms missing from graphics
Instrument loops exist that are not included on the P&ID
Instrument loops are shown on the P&ID that do not exist in the DCS
Demolished equipment still shown on P&IDs
Equipment exists that is not shown on P&IDs
Incorrectly configured instrument ranges (where’d all those bad-PV alarms come from?)
Redundant instruments with different configured ranges and other characteristics
Switches (quite unreliable) are used for providing critical alarms where a transmitter is available. (This will almost always be found in older plants that have gone through an upgrade.)

Here are some items found in doing a major “big name” refinery rationalization. Being big does not make you accurate!

A lot of the documentation was missing and needed updating
The management of change “procedure” and work process were very complex, and errors in the documentation showed it was unreliable
P&IDs, equipment manuals, and operations training manuals were not updated after project changes
There were a large number of incorrect or contradictory findings in the operating limits and the operating envelopes in use
There were many tags related to safety and interlock systems that did not contain alarms, but should have. This was very surprising to the operators
Maintenance ignores malfunctioning instruments and so the diagnostic alarms are just disabled and ignored

Now, a rationalization does not go into these things without reason. When prepping for a rationalization, Hexagon uses software that actually imports the settings for every alarm, straight from the control system. Then, in the alarm discussions, sometimes things like this example occur:

Hexagon: “OK, for this reactor pre-shutdown alarm, is 350 degrees the correct setpoint?”
Customer: “Yes. The DCS alarm warns the operator that the safety system is about to shutdown the reactor.”
Hexagon, later: “So, the safety system alarms when it shuts down the reactor. Hmmm – this shutdown is set at 350 degrees. Wait – how can the DCS alarm at 350 be a ‘pre-alarm’ for the operator to react and avoid the shutdown?”
Customer: “Wow, looks like it can’t.”
Hexagon: “And look – the other reactor is set at 360 degrees? Shouldn’t that be the same?”
Customer: “Hmmm, let’s get some more documents. I think there was a project that might have done that…”

Here are some more from a major “big name” chemical plant rationalization:

The design basis for the plant itself had undergone some revisions in years past, but the documentation was never fully updated before the project team was reassigned to another project. The resulting P&IDs were a patchwork of long since abandoned processes and improvements.
Operator comments during the rationalization:
- "Ok, so I know that’s how it's drawn, but that's not how it actually is."
- "That hasn't worked in years."
- "That system got removed about a decade ago."
- "That was for the old process, it needs to be taken out."
- "That's been bypassed for ages, we need to get someone to make a call as to whether we're going to keep it."
Projects never altered existing tags in the DCS, they just made new ones and left the old ones idle in the system: “We had to weed through a number of long-obsolete logic blocks and calculations.”

Many 40+ year-old power plants do not have P&IDs at all, or the P&IDs have not been updated for years. A surprising but not-uncommon statement: “We know our P&IDs are not reliable at all, we prefer you do not use them.”

Horrifying stories from the field

“On the **** Gas project, there are a huge number of DCS tags mapped through MODBUS from various packaged systems. These tags have absolutely no traceability with the P&ID drawings or DCS graphics for the respective systems. These include inputs, outputs, logic blocks (sequences), diagnostics and so on. Almost all of them have alarms configured. The descriptions on many of these tags were identical and sometimes did not even make sense. Most of these alarms turned out to be representing normal events and were removed.”

“I was doing rationalization in **** for **** Refinery. They have a Foxboro system there. There were about 25,000 points in the system. Alarms were configured as if it was a testing database for the people who did control logic engineering work during database configuration. You just name it and you will find alarms configured on it. When we were done, not only did they get a rationalized alarm system, but they got a list of great process and documentation improvements that would help them dig out of constant fire-fighting mode and focus on long-term, lasting improvements.”

Alarms and personnel safety

Here are some interesting discussions:
Customer: “That alarm is before the steam safety valve blows. It has to be the highest priority.”
Hexagon: “Why?”
Customer: “Because the safety valve is pointed at the walkway.”
(Conversation ensues about how alarm priority does not divert live steam, but a piece of vent piping does.)

There was an alarm on the coal conveyor to indicate a person’s hand or body could be caught in it.
Customer: “Set that to low priority.”
Hexagon (aghast): “WHY??!!”
Customer: “Most of the times it is a false alarm due to equipment malfunction.”
(Conversation ensues about alarms that relate to personnel safety, lawyers, and the need to maintain important sensors!)

Summary

Rationalization isn’t just an extremely good idea; it is also a requirement in the ISA 18.2 and IEC 62682 Alarm Management Standards. What you want is a productive and efficient rationalization with the best possible outcome. To get this, you should read the Handbook for how to go about it, follow the proven practices, get some help if you are new to it, and use the right tools. It does take a bit of effort, but the benefits are huge. Not once have we had a customer say: “That was a big waste of time!” Quite the opposite – we often have operators (a highly sceptical bunch!) start lobbying to help out other units that have not yet been rationalized. So don’t keep putting off alarm rationalization (or that colonoscopy …)!

Please contact us for more information or if you have questions

Review other Taming the Wild Alarm System topics in this blog series:

Articles & Blogs