OPAL-RT’s HYPERSIM & SCALABLE’s EXata CPS: Real-Time Cyber-Physical Simulation of the Electric Power Grid for Cybersecurity Studies

/in /by

The Status Quo
21st century power grids face a number of make-or-break challenges on the Infrastructure/Security/Wide-Area Monitoring, Protection and Control fronts, notably mixed and hybrid old/new analog/digital equipment; newer energy sources like wind and solar integrated into older networks; as well as faster-switching converters and newer digital equipment that make the grid ‘smarter’, more sustainable, future-proofed and flexible—but that also leave it open to all the vulnerabilities associated with networked resources and communications.











Smart Grids & Their Inherent Complexities…
Modern power grids have become Cyber-Physical Systems (CPS) composed of electrical and communication infrastructure. As opposed to the analog networks of, say, 100 years ago that were made of cables and switches and hardware, today’s grids are studded with communications, administration and protection equipment that has been being ushered in for its precision and superior oversight functions since the dawn of the digital age.

Today’s grid is becoming more ‘intelligent’ through the:

  • Wide deployment of new technologies
  • Substation, transmission and distribution automation
  • Increased Distributed Energy Resources (DER) integration
  • Advanced two-way communication networks, and the
  • Development of synchro phasor systems














However, as an unavoidable consequence of the above, as newer technologies are adopted, the grid is becoming more vulnerable to cybersecurity threats of all kinds as well as communication equipment failures. Mixed technologies are harder to test; hybrid networks offer unique challenges as diagnostics for one simply aren’t adequate solutions for the other.











…& the Real-World Outcomes
We have seen that connecting Supervisory Controls and Data Acquisition (SCADA) systems and Operational Technology (OT) devices via the internet has significantly improved accessibility, automation, and efficiency of vast networks, but it also introduces vulnerabilities.

Without hyperbole, we can say that this makes every communication line a potential attack surface. Because of this, cyber threats against public utilities and other critical infrastructure are just as ubiquitous as attacks on government and corporate computing infrastructures.

These attacks may cause loss, and/or denial of access or manipulation of system views and control. Cyber-attacks against SCADA systems, such as power generation and distribution systems, water treatment plants, and transportation facilities, can cause widespread disruption of commerce and daily life.



Besides cyber-attacks, a larger amount of communications equipment also means more potential for human error, operator carelessness or negligence, and equipment failures that can also lead to serious consequences.



“There is a pressing need for operators of SCADA systems, microgrids, substations and other infrastructures to determine how resilient their operational systems are to cyberattacks and to develop plans to mitigate the associated risks.”











The Partnership











This is why 2018’s partnership between OPAL-RT and SCALABLE was so exciting and groundbreaking for both parties:
OPAL-RT TECHNOLOGIES are experts in real-time simulation of power systems and power electronics:





  • We provide real-time simulation technology and engineering R&D, both of which are used extensively in the development and testing of operational technology within the electric power grid.


            • We’re focused on improving the security and reliability of systems used to control, protect and monitor the grid.










    SCALABLE Network Technologies are experts in real-time simulations of communication network infrastructures:





        • Their EXata network emulation platform, with its cyber library of simulated attacks and vulnerabilities, is used to analyze and test the resilience of critical communication networks effectively.


            • Tools like EXata CPS allow customers to visualize their specific environments in a manageable laboratory setting and quickly evaluate a range of ‘what if’ scenarios to determine the impact on their systems if subjected to cyber-attack.


    The Much-Awaited HYPERSIM 2019.2, Featuring EXata CPS



























    The companies’ collaboration has borne fruit in HYPERSIM 2019.2:

    “EXata CPS is integrated in HYPERSIM 2019.2 on the same hardware to offer a complete real-time cyberphysical solution for the development, testing, and assessment of electrical grids with communication networks,” said Etienne Leduc, Product Owner of HYPERSIM. “HYPERSIM, which simulates the physical system, is the only real-time digital simulator with the power to simulate electromagnetic transients of large-scale power systems, tackling operational and reliability issues threatening a power system’s cybersecurity. This integration of EXata CPS and HYPERSIM provides a means to test the resilience of power systems to cyber-attacks and improve their cyber defenses, thereby helping to ensure cybersecurity, reliability, and efficiency of such systems.”






    Typical Configuration/Application










    The figure below shows the integration between the OPAL-RT simulator, at left, communicating with both EXata CPS and the devices under test (controllers or ECUs or other networked IT/Security devices) with monitoring, storing and interaction abilities, represented to the right.







    Types of Attack Supported
    The hybrid best-in-class duo of HYPERSIM and EXata CPS can model any number of types of attacks.

    The most significant attacks which can impact power systems are:
















                              • Denial of Service (DOS): These attacks can bring down or make unavailable a critical piece of equipment
                              • Packet Modification Attacks: These attacks make changes to the payload of packets and can result in:







                                        • Bogus input, such as modified sensor data, which can lead to erroneous decisions by the controllers
                                        • Bogus output, such as manipulated or misleading data sent, which can lead to unintended or incorrect actions 





























                                                    Communications Protocols Supported


                                                    Both companies’ support of Communication Protocols is extensive, as is evidently required in a context entirely dependent on I/O, digital communications and both IT and security infrastructure:





                                                    EXata CPS communicates with HYPERSIM through the following protocols:





















                                                                              • Generic Object-Oriented Substation Events (GOOSE), a subset of IEC 61850
                                                                              • 118 (over TCP/IP), used by synchrophasors
                                                                              • DNP3 (over TCP/IP)
                                                                              • Modbus (over TCP/IP)
                                                                              • IEC 60870-5-104 (over TCP/IP)

















                                                                                  Two Sample Attacks: Scenarios










                                                                                  The following graphic depicts the SCADA dashboard where two possible scenarios are modeled and simulated using the hybrid HYPERSIM/EXata CPS toolbox:






                                                                                  Scenario 1









                                                                                  In Scenario 1, we simulate a message delay attack once the grid is islanded:
















                                                                                                            • As there’s not enough generational power for all the loads in this microgrid, the residential load L1 needs to be shed upon islanding the grid






















                                                                                                                                          • By delaying the GOOSE message aimed at the breaker by 3 seconds, the frequency and voltage become unstable, which can lead to equipment damage or backup protections kicking in

















                                                                                                                                              Scenario 2









                                                                                                                                              In Scenario 2, while islanded, we simulate a packet value multiplication attack:
















                                                                                                                                                                        • By intercepting the power measurement of the L2 industrial load going to the microgrid controller and multiplying its value by 2, the controller thinks that it needs to react as there’s not enough generational power for all the loads






















                                                                                                                                                                                                      • In consequence, the residential load L3 needs to be shed by the microgrid controller, cutting power for families and small businesses






























                                                                                                                                                                                                          Highlights of this Article








                                                                                                                                                                                                        • OPAL-RT and SCALABLE partner to develop joint cybersecurity solutions based on OPAL-RT’s proven real-time simulators and solvers and SCALABLE’s accumulated expertise in cyber-physical security. Learn more >
                                                                                                                                                                                                        • EXata CPS, SCALABLE’s flagship CPS solution is supported in HYPERSIM 2019.2, OPAL-RT’s premiere real-time simulation platform for power systems and power electronics. Learn more >
                                                                                                                                                                                                        • OPAL-RT and SCALABLE co-host what promises to be a valuable and illuminating webinar on ensuring the safety and protection of electric power grids with OFFIS as special guest. Learn more >
                                                                                                                                                                                                        • OPAL-RT and SCALABLE published a white paper on this collaboration in August Learn more >


                                                                                                                                                                                                          •



                                                                                                                                                                                                          Traveling Wave Relay Testing

                                                                                                                                                                                                          /in , /by

                                                                                                                                                                                                          When we last spoke with Shijia Li, in November, she told us about Protection Relay Testing. She has since been made team leader for Protection and Smart Grid team within OPAL-RT’s AXES (Application, eXpertise and Electrical Simulation) division. This time, she is speaking to OPAL-RT Product News about OPAL-RT’s HIL Traveling Wave Test System.
                                                                                                                                                                                                          —

                                                                                                                                                                                                          Interviewer (IV): “Hello Shijia. First, can you tell us when we introduced the Traveling Wave test system?”

                                                                                                                                                                                                          Shijia Li (SL): “We developed it about 1.5 years ago.”

                                                                                                                                                                                                          IV: “Our software has been simulating faults on FPGAs for a while; why hadn’t we used this method previously?”

                                                                                                                                                                                                          SL: “Previously, the FPGA had not been used for protection system testing. It was used for simulating power electronics devices, or motors or drives, but not to simulate a power system with transmission lines, etc. This was the first time we’d tested the power system components on the FPGA; it was a new way to use the FPGA. It has the fast time step required to precisely locate (within a few meters) and diagnose faults on power system lines.”

                                                                                                                                                                                                          IV: “So prior to that, all protection was run on CPU? How did we make this breakthrough?”

                                                                                                                                                                                                          SL: “This actually came about because of a request from a client. They built a device containing an algorithm and needed a way to test it. The conventional tests [Editor’s note: CPU-based] wouldn’t work with their device, so we had to use an FPGA model to achieve a much smaller time step. We had an engineer developing a model–more of a mathematical model–to make it run much faster on an FPGA. That innovation also prompted us to improve our solver. The client’s engineers were so impressed with the results from our constant parameter (CP) line model that they’re eager to see our frequency dependent (FD) line model.”

                                                                                                                                                                                                          IV: “So we currently have two different line models?”

                                                                                                                                                                                                          SL: “As of now, we only have the CP line model, but our R&D department is finalizing the FD line model.”

                                                                                                                                                                                                          IV: “What’s the difference between the two models?”

                                                                                                                                                                                                          SL: “The FD line model more accurately represents overhead lines than the CP model. It has a richer harmonic content, which represents with higher fidelity a line during a fault; with the FD line model, we’ll be able to test single-ended TW fault locating algorithms, which is more challenging.”

                                                                                                                                                                                                          IV: “Impressive. This is a fairly new innovation, then, FPGAs being used to do work this precise, in this context?”

                                                                                                                                                                                                          SL: “Yes. The travelling wave is a very high-frequency phenomenon, so it requires faster simulation as well as faster hardware. Our usual I/O boards take one sample every microsecond, which is sufficient for simulations in the range of 10 to 50 µs, but when simulating the travelling wave phenomenon at 500 ns on the FPGA, we need I/O boards that can follow at this speed, to get better accuracy. Fortunately, we already have a board with a sampling rate of 2 MS/s.”

                                                                                                                                                                                                          IV: “What did utilities do before this? Did they simply say, ‘there’s a fault somewhere between kilometre 364 and 365’, for example?”

                                                                                                                                                                                                          SL: “We could say that. There are other ways of detecting the fault location that are not as accurate as this one; it really depends on the manufacturer. It is, for example, often expressed in a percentage of the setting, which relates to the length of the line, so the longer the line, the lower the accuracy.”

                                                                                                                                                                                                          IV: “This was a breakthrough in terms of narrowing the range?”

                                                                                                                                                                                                          SL: “Yes, absolutely. And this idea was floated a long time ago, but, at the time, the relay itself didn’t have enough computational power. The processor wasn’t fast enough to run the algorithm. But since technology has evolved, they can implement it on the hardware. To understand how much of a breakthrough that is, you have to look at it in the operational context. When there’s a fault on a line, there are some strategies that can be used to avoid sending out a team to investigate. These strategies vary from one utility to the next and are based on the environment around the fault location. Since there aren’t cameras everywhere, some assumptions must be made.”

                                                                                                                                                                                                          “For example, in rural areas, some might successively reclose and reopen breakers to try and clear a fault (in case a tree fell on a line, for example) to liberate the line. If every automated strategy fails, or, in dense urban areas, it is most often necessary to send out a team to investigate, which can be very costly. If the team has to search over a few kilometres for the location of the fault, it can take a lot of time. It’s even more difficult, for obvious reasons, with cables that are buried underground. Travelling wave relays might mean a high-cost reduction in many cases. This is the breakthrough.”

                                                                                                                                                                                                          IV: “This is an HIL process, right?”

                                                                                                                                                                                                          SL: “Yes, this is a hardware-in-the-loop (HIL) process, but obviously not on the lines themselves. There are testers that can be used in the field to perform simple signal injection tests. But what we’re doing is more in the lab: we’re using the same devices, the same settings. The device we are testing is monitoring the line. What we’re doing is we’re replacing that actual power line with our simulator: we send signals to the device, but the device is monitoring the lines on the simulator.”

                                                                                                                                                                                                          IV: “And the larger context for this is control and protection, one of your specialties. Was travelling wave testing something people had wanted to do for a while?”

                                                                                                                                                                                                          SL: “Well, it is not a new idea, but it’s not that long since it has actually been put into use. It is a new feature, and we do have customers expressing interest in it. Generally speaking, in the context of the protection industry, this would be considered an innovation: it’s not been widely used or adopted by most utilities. And we’re seeing some of our clients out there in the early stages, trying to convince people to adopt this technology.”

                                                                                                                                                                                                          IV: “How does this technology fit in, in terms of the industry in general?”

                                                                                                                                                                                                          SL: “The protection and control sectors are very well-established, mature sectors or fields, within the power system industry. The current devices and schemes or implementations we have are good enough to protect most power systems. For now, there are some new perspectives—the broader introduction of renewable energy—that may introduce some new challenges. And travelling wave technology, which brings a challenge in terms of testing: this is ultimately why we’re developing this solution.”

                                                                                                                                                                                                          “The microgrid also, and its protection, is a very hot topic in this field. Other than that, from the communication-aided protections: we use more and more fibre optics, so that’s something we could test as well. And that brings us to the IEC61850 digital substation concepts [Editor’s note: Product News blog post to come]: so let’s say, with one relay now, we can do a lot of complex functions and so, of course, the testing becomes exponentially more complex as well.”

                                                                                                                                                                                                          IV: “Thanks for speaking with us again, Shijia.”

                                                                                                                                                                                                          —

                                                                                                                                                                                                          About the Interviewee


                                                                                                                                                                                                          Shijia Li received her Bachelor’s degree from Zhejiang University, China in 2012 and Master’s degree from McGill University, Canada in 2015, both in the field of power engineering. She joined OPAL-RT in March 2015, where her work focuses on power system modelling and real-time simulation applications with protective relays and PMUs. Shijia is actively involved in developing technical solutions and providing advanced training to help users better utilize real-time simulation techniques for exploring the latest P&C/smart grid technologies. Currently, Shijia leads the Protection and Smart Grid team in OPAL-RT’s AXES (Application, eXpertise and Electrical Simulation division).


                                                                                                                                                                                                          Protection Relay Testing with HYPERSIM

                                                                                                                                                                                                          /in /by

                                                                                                                                                                                                          HYPERSIM is a high-end modelling and simulation platform intended for those managing and supporting large-scale power networks requiring constant and exacting monitoring, tuning and maintenance.  

                                                                                                                                                                                                          We spoke with Shijia Li, Team Lead—Protection and Smart Grid at OPAL-RT, about the efforts her department has made in developing HYPERSIM’s Protection Relay Library. Shijia took a few moments from her day to speak to us about this enhancement to HYPERSIM’s functionality. 
                                                                                                                                                                                                          —

                                                                                                                                                                                                          INTERVIEWER [IV]: “So, thanks for speaking with us, Shijia! You’ve told me these Protection Relay Library items are a sort of workflow improvement for those who work in Protection?” 

                                                                                                                                                                                                          Shijia Li [SL]: “Yes, so much so that they are pretty universal. They’re a library of pre-made components, or blocks, for a modelling/simulation platform like HYPERSIM. They’re usually signal processing modules, plus math-controlled functions and logical operations. They would be normally installed in sub-stations or the like.” 

                                                                                                                                                                                                          IV: “What are some examples of some of the Protection Relay Library items in HYPERSIM?” 

                                                                                                                                                                                                          SL: “Well we’re just getting started, but so far: overcurrent protection; under- or over- frequency relay; under- or over-voltage relay; distance protection (measured in impedance); transformer differential—which measures current on both sides—and the loss of excitation voltage for the generator. These models represent the generic functions of the protective relays installed in substations. And we have plans for more to come.” 

                                                                                                                                                                                                          IV: “Would someone generally… know which of these to use in a situation, or is it automated?” 

                                                                                                                                                                                                          “This is a vastly complex topic—like you could write a book about it. But in a Distribution system, let’s say, of around 10 kV, with distribution feeders headed to the users, we would tend to use over-current protection and reclosers. But say, for example, at a Transmission level of 500 kV, we would use distance protection and line differential protection. So they’re different functions we use in different situations.” 

                                                                                                                                                                                                          “But even beyond this, there are considerations about the type of equipment requiring protection and how that is prioritized, the criticality of that equipment—and how much the equipment itself costs is a big factor in how, when and what we protect. As I said, it’s a pretty complex topic, and I’m over-simplifying here to give you a brief answer.” 

                                                                                                                                                                                                          IV: “What would be the advantage of using a real-time simulator for performing protection studies?“ 

                                                                                                                                                                                                          SL: “Well, first, there’s no practical way to test in real life situations because it could cause a service interruption. When using a real-time simulator to test the relays with realistic behaviour and scenarios, all the relay equipment ‘thinks’ it’s cabled to the real thing; this is why we call it a real-time simulator, because for all intents and purposes the relay equipment is behaving and reacting as it would in real life situations. So it’s all advantages and no setbacks—close to realistic with little risk. I think the real costs of these events may be lost to people, but it may easily cost millions of dollars per hour if a major city like Montreal, say, is without power for some while.” 

                                                                                                                                                                                                          “At this point [2018], real-time simulators are a mature technology, and we use them extensively for calculations in offline testing. But we’re now getting one step beyond that. The conventional way of testing is just to run some calculations in software or to connect a signal generator to the device under test—which just does unit tests to see if the relay will operate at certain thresholds. But this doesn’t test anything in a more in-context way. What we do more of now is to generate a gamut of more realistic signals in real time, and so this approach is more true-to-life—to what is actually happening in real life.” 

                                                                                                                                                                                                          “What we’re now doing, is called model-based testing or system-based testing—it’s like the next wave. With the combination of the real-time simulator and both the virtual and real hardware components, it’s an enhanced level of testing.”


                                                                                                                                                                                                          Editor’s Note:
                                                                                                                                                                                                          For a video that examines Distance Relay Type Testing, as well as demonstrates strategies for automating iterative testing with Excel-based spreadsheets, please see this video clip.


                                                                                                                                                                                                          IV: “What would be the adva                                                                                                                                                                                                          ntage of having a virtual library of relays to the users?” 

                                                                                                                                                                                                          SL: “In this way, we can simulate various parts of the protection system and integrate real-world hardware devices (Hardware-in-the-Loop or HIL). So it’s easier to test complex scenarios involving many relays and the virtual. Several types of power system studies don’t require real relays. The accuracy of the virtual is good enough for the first step of proof of concept. Users don’t need to buy the relays, don’t have to set them up (which helps avoid configuration or connectivity issues)….” 

                                                                                                                                                                                                          “There’s also different layers of modelling and simulation that we can look at. When we’re simulating a power system, it’s mostly the primary equipment. But we can also simulate the protection control layer, which is the secondary equipment, and it’s not at a very high voltage or current. And there are also various communications between layers. So using a real-time simulator and virtual devices means you can test a more complete system more thoroughly and accurately—and sub out the real for the virtual where circumstances permit.” 

                                                                                                                                                                                                          IV: “Can you tell me about some of the other great features available for protection studies in HYPERSIM?” 

                                                                                                                                                                                                          SL: “Well, we can automate tests for testing complex protection schemes using TestView, and by combining virtual relays and real devices, as we’ve already covered. We can automate sequences for dedicated types of protection equipment, such as distance protection. We can also use advanced testing functionalities for communication protocols. For example, we developed what we call Data Integrity Manipulation for IEC 61850-9-2 that allows the user to test the robustness of the protection system would there be issues with the Ethernet network or even would it be cyberattacked. Another feature is the compatibility with MATLAB/Simulink or other external code so a user who’d already have protection algorithms programmed in another tool could import it to HYPERSIM. There’s any number of time- and labour-saving ways these features can be used.” 

                                                                                                                                                                                                          IV: “Could a user build their own protection block based on our library, if they wanted to?” 

                                                                                                                                                                                                          SL: “Yes, absolutely. We could provide source code if they want. So a user can save a lot of time building their own library: starting from an existing block, they could add functionality to the extent they wished to.” 

                                                                                                                                                                                                          IV: “I’d like to thank you very much for taking the time to speak with us, Shijia!” 

                                                                                                                                                                                                          SL: “It’s been my pleasure.” 
                                                                                                                                                                                                          —

                                                                                                                                                                                                          Please see OPAL-RT’s web page on Protection Systems for more on our solutions: opal-rt.com/protection-system-overview

                                                                                                                                                                                                          About the Interviewee


                                                                                                                                                                                                          Shijia Li received her Bachelor’s degree from Zhejiang University, China in 2012 and Master’s degree from McGill University, Canada in 2015, both in the field of power engineering. She joined OPAL-RT in March 2015, where her work focuses on power system modelling and real-time simulation applications with protective relays and PMUs. Shijia is actively involved in developing technical solutions and providing advanced training to help users better utilize real-time simulation techniques for exploring the latest P&C/smart grid technologies. Currently, Shijia leads the Protection and Smart Grid team in OPAL-RT’s Application, eXpertise and Electrical Simulation division.