Arc Flash and Power Protection Lessons Learned from Data Centers

An energy-intensive data center requires multiple high capacity sources to function. When there is a failure, the arc flash energies are also notably greater than a small industry network.

• By Zarheer Jooma, John Mason, Hugh Hoagland
• Sep 24, 2018

1. Introduction
Data has become the cornerstone of daily life, making data centers a crucial infrastructure. Modern data centers are energy-hungry, and competition for efficiency and reliability is fierce. Microprocessors run billions of commands, generating substantial heat that is detrimental to both efficiency and reliability.  Electrical supply problems such as voltage dips, power outages, and harmonics can be catastrophic to any data center and can hamper anything from a simple transaction like online shopping to a major event that could even influence security.  Something this important will naturally inhibit a complex electrical system. Complex systems are training camps for protection engineers and knowledge is best when shared. This article discusses some learning points from a recent system study that included the network modeling, electrical protection coordination, arc flash incident energy analysis, and labeling based on a brownfield data center infrastructure expansion project. It aims to present some challenges and best practices that may help the wider industry in safety and arc flash program improvement opportunities.

2. The Data Center
Each data center is unique but contains a generic layout[1] consisting of three main systems, namely: a) the electrical system; b) the mechanical system; and c) the information technology (IT) system. Due to the massive processing power required by complex software used by millions at any given time, IT systems are large energy consumers. The primary energy source is electricity and it is used to produce useful information (data processing) and heat as a waste product. Electricity is again required to remove the waste heat. The mechanical cooling and venting systems may use refrigeration systems or evaporative cooling systems that could also produce waste water (or loss of water through evaporation) and all auxiliary processes are usually electrical. 

3. Learning Opportunities
In order to specify the electrical protection settings for a data center, in-depth knowledge of the process train is crucial. It doesn't take much to conclude that everything revolves around redundancy (and an abundance) of electricity. Ensuring that power is ALWAYS available is clearly the cornerstone of the electrical design. Understanding this, however, doesn't make the study any easier. In order to assist both data centers and design engineers as well as the general industry, knowledge gained from performing electrical design work and arc flash studies at data centers is presented here. 

3.1 Every backup or alternate source has to be considered

The situation: Data centers may have more than one utility supply. When two utility transformers are present, the system may be "split" in two (Section A and Section B) with contingency for either transformer to feed either section. Confusing? It gets even worse when each utility transformer has a generator backup and each data center section has a UPS. Trying to keep track of each can get very confusing indeed.

The lesson: Similar to data centers, the general industry these days has generators and UPS systems to ensure business continuation. In this case, it is imperative that each possible system configuration is simulated using electrical network analysis software in order to consider or determine the following:

i) Equipment rating: Does paralleling any two or more systems exceed the peak asymmetrical current rating of any electrical equipment in the system? Equipment ratings could be exceeded even when a single source is feeding an alternate load. Finally, each utility stipulates a maximum and minimum short circuit capacity; the maximum capacity situation could over-duty certain equipment. The design engineer should ensure that all scenarios are simulated and the operational limitations clearly defined and communicated. If possible, software interlocks (e.g., breaker close inhibit/blocking signals) or hardware interlocks (e.g., interlocking key systems) can be recommended by the engineer.

ii) Protection settings: Does switching in alternate supplies guarantee that protection devices will operate as designed? As the source is altered, the capacity of the system is altered. As an example, perhaps the plant could start multiple cooling systems simultaneously from the normal utility transformer. The backup transformer, however, may limit operations and warrant a staggered startup sequence. The design engineer can simulate a motor start and determine the voltage drop at a remote location. Most software packages offer this functionality and are able to auto-generate problematic areas in the network.

If the source capacity is vastly different, short circuit or fault current values will also vary. This implies that the default protection settings may not work for all scenarios. In order to address this issue, the design engineer may provide two settings groups (equipment dependent) or may notify the client of equipment that may not operate correctly. If known at an early stage of the project, equipment changes (that allow multiple settings groups) can be processed without penalties.

iii) Arc flash energy: Arc flash energy in the event of a failure is driven by multiple factors; however, at low voltage the short circuit current (or more accurately, the resultant arcing current) and associated fault clearing time are the key contributors. An energy-intensive data center requires multiple high capacity sources to function. When there is a failure, the arc flash energies are also notably greater than a small industry network. In certain cases, a person may be far away (distance-wise and system-wise) from a low voltage main distribution panel but may require full arc rated daily work wear with a faceshield and balaclava (among others). If an alternate source is switched, the previously used daily work wear may now be insufficient. The design engineers can run the various scenarios but should avoid making the decision on the PPE by themselves. Instead, the results should be discussed with the client -- PPE costs, human factors (ergonomics) considerations, and safety risk may influence the final decision.

3.2 Is the scope limited to the infrastructure expansion parts only?

The situation: As demand grows for faster IT, data centers are continuously upgrading to increase capacity. As contractors vie for the contract, pricing the scope of work is tightly controlled. Many contractors (if not all) tend to limit the scope exclusively to equipment that they will have "hands on" and exclude everything else.

The lesson: If cooling towers are upgraded with larger motors, larger UPS systems are installed, or an additional generator is introduced, the entire power system is influenced. The NFPA 70E-2018[2] Article 130.5(G) requires that the incident energy analysis be updated if changes influence the electrical distribution system. It does not limit the evaluation of the incident arc flash energy to equipment that the contractor had "hands on." The client also has a responsibility to specify these requirements and not merely leave it to the discretion of the contractors submitting a bid. Reviewing the entire electrical subsystem or network may identify issues brought about by the upgrade. This may also serve as the very important incident energy analysis required by the NFPA 70E-2018 every five years.

3.3 Does it add value?

The situation: Networks containing several tiers of incomer and feeder breakers are generally unable to activate an instantaneous trip. This means that arcs close to breakers without "instantaneous" protection will have a time delay, leading to a higher incident arc flash energy. In order to address this situation, equipment manufacturers provide a "maintenance mode" switch that allows an operator to implement a protection system change (removing the time delay) by simply activating a switch. The system can then be returned to "normal mode" after the task is completed. If implemented correctly, this function can reduce the need for PPE from a flash suit down to not requiring arc rated protection at all.

The lesson: The system engineer should carefully evaluate the protection device trip curves (current versus time) to ensure that this function is used appropriately. If the system coordination allows for activating instantaneous protection (minimum current pickup without a time delay), then activating the maintenance mode switch does not serve any purpose. Depending on the cost of the equipment and the number of protective devices being installed, this can be a wasteful expenditure on equipment costs, design costs, commissions costs, and testing costs. With that being said, if cost isn't a factor, the "maintenance mode" is strongly recommended even if it is redundant.

The engineer should take care to ensure that the "maintenance mode" is modeled correctly. Comparing several packages, it appears that the library files differ and the software architecture on executing the function differs between the various software packages. Even if there is confidence in the model, having an independent engineer review the various arc flash energies is a good due diligence exercise.

3.4 Love reinventing the wheel?

The situation: Most engineering consultants retain the software model on the basis of intellectual property. When an infrastructure expansion occurs, the system or network has to be remodeled or the client remains bound to use the same consultant. Paying for something that has already been paid for doesn't make much sense.

The lesson: Although there are merits to modeling from scratch and a genuine case can be made for intellectual property claims, there are several benefits to sharing the model. It saves the design engineer time and it saves the client on consulting fees to collect data, establish a new model, report, and label equipment that was already compliant. More importantly, it allows for several eyes on a single model, rather than a new model (every few years) that only one or two engineers have reviewed. The use of a singular software model across different engineering firms or consultancies can be challenging and may produce errors (different library files, various format differences, etc.) unless a software simulation related "scope of work" is specified. This leads to the lesson that industry should start specifying the technical requirements for arc flash studies instead of being handed the flavor-of-the-day from a consultant. The specification may include software version, library files, one-line layouts, and reporting requirements, among others.

3.5 It must be the worst case…right?

The situation: In a panel or switchboard, the main incoming breaker will most likely be assigned a greater incident arc flash energy since arcing faults will be cleared by an upstream device that may have a time delay. Devices downstream of the main breaker will most likely be assigned a lower incident arc flash energy since arc faults are cleared by the main breaker. Is it always best to go with the highest value?

The lesson: Unfortunately, there is no generic "one size fits all" in this case. Switchboard design considerations, human factors (ergonomics), and the specifics of the network (consider large downstream motor contributions) will all factor into the decision. Several authors[3] have written on this topic, and it is recommended that a novice engineer review such literature before starting the model. Also, when modeling, be aware that running a line side versus load side together with activating a "maintenance mode" simulation may provide undesired or incorrect results. Sharing a generic example with the software company's technical support department may be beneficial.

3.6 This is construction, not a training class

The situation: Infrastructure expansion field workers do not have the luxury of working around de-energized equipment. Teams are likely to be exposed to a greater level of risk during construction as compared to regular day-to-day operations.

The lesson: The contractor needs to ensure that all workers receive awareness training or the equivalent as per the Regulations for Construction OSHA 1926.21(b)(2), "The employer shall instruct each employee in the recognition and avoidance of unsafe conditions and the regulations applicable to his work environment to control or eliminate any hazards or other exposure to illness or injury." Since the data center operations continue as usual when construction is going on, staff exposed to construction-related electrical hazards or activities may also require awareness training. 

Finally, when the new infrastructure is complete, it may result in previously low arc flash incident energy panels now requiring arc rated PPE or non-melting clothing as per OSHA 1910 Subpart I and Subpart S, respectively. Further reading on the topic is presented by Hugh Hoagland[4].

Conclusion
Undertaking a network protection analysis combined with an arc flash engineering study provides several learning opportunities. Although the specifics cannot be adopted by the general industry, the concepts and principles can be applied generically.

Energy-intensive plants will produce greater arcing currents; however, newer and faster protective devices can be set to drastically reduce the arcing time, thereby reducing the incident arc flash energy. This can only be achieved if the system is modeled optimally and the protective devices set correctly.  Also, a system with several redundancies creates a melting pot of scenarios that require modeling. Although this exercise can be tedious, its importance should never be underestimated. Before jumping into a data center model, ensure that the consultant has had previous experience working on data centers or at least works under the direct (and close) supervision of an experienced consultant.   

In conclusion, it is important to understand that every "lesson" cannot be conveyed in a single article. The topics covered in this article were presented in a simplified way geared toward non-specialist readers, and not all considerations were discussed.  These concepts are complex, and it is recommended that professional guidance be sought when specifying technical documents for arc flash studies and network protection design.

References