> For the complete documentation index, see [llms.txt](https://www.opal-rt.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.opal-rt.com/docs/rt-lab/model-and-simulation/co-simulation/run-geographically-distributed-real-time-simulations.md).

# Run Geographically Distributed Real-Time Simulations

## Introduction

Geographically Distributed Real-Time Simulation (GD-RTS) connects real-time simulations running at different physical locations so they can participate in a common experiment.

Each simulator executes its own model independently in real time. The simulators exchange measurements, commands, states, or other application data through an IP network, while a common time reference can be used to timestamp and correlate information produced at the different sites.

This architecture makes it possible to combine simulation resources, models, controllers, and laboratory equipment that do not need to be physically located in the same facility.

{% hint style="danger" %}
**Important:** Geographically distributed real-time simulation is sensitive to network latency, jitter, packet loss, and the dynamics of the coupled models.
{% endhint %}

## Architecture

A geographically distributed simulation consists of two or more independent real-time simulation sites.

Each site typically includes:

* an OPAL-RT real-time simulator;
* a simulation model;
* a communication interface;
* access to the inter-site IP network; and
* when required, a source of precise absolute time.

For example, one site could simulate an electrical network while another executes a protection, control, aggregation, or supervisory model.

Each simulation:

* is built independently;
* runs on its own real-time simulator;
* has its own real-time execution loop;
* can use its own simulation time step; and
* communicates with remote simulations through standard communication interfaces.

The models are therefore **independent simulations**, not different partitions of the same <code class="expression">space.vars.Software</code> model.

## Configure data exchange

### Select the data exchange protocol

Use a communication protocol appropriate for the application and the type of data being transferred.

Examples include:

* UDP/IP
* TCP/IP
* IEEE C37.118
* IEC 61850
* DNP3
* Modbus
* a custom Ethernet protocol

The communication interface transfers measurements, states, commands, or other application data between sites independently of the mechanism used to synchronize their clocks.

## Configure simulation sites synchronization

For applications involving timestamped measurements, event correlation, synchrophasors, or other time-sensitive information, synchronize the participating simulators to a common absolute time reference.

OPAL-RT simulators can use the **Oregano Syn1588 PCIe** synchronization card, with support for:

* IEEE 1588 PTP;
* IRIG-B; and
* 1 PPS.

To configure synchronization on each simulator, refer to [Time Synchronization Driver](https://www.opal-rt.com/docs/rt-lab/configure-ios/time-synchronization-driver.md).

A key distinction is that **time synchronization and data communication serve different purposes**:

* **Time synchronization** provides the participating simulators with a common absolute time reference.
* **Data communication** transfers simulation measurements, states, commands, or other information between the sites.

A simulator can therefore receive its time from GPS/PTP while exchanging application data over a separate Ethernet connection.

{% hint style="danger" %}
**Important:** Synchronizing the simulator clocks does not eliminate network delay.
{% endhint %}

Verify synchronization before configuring the distributed application.

### Exchange timestamped data

When the application uses an absolute-time protocol, associate transferred measurements or events with timestamps derived from the synchronized time source.

This allows the receiving simulation to determine **when data was generated**, independently of when it arrives.

This is different from requiring the remote simulators to execute the corresponding calculation at exactly the same instant.

### Account for network latency

A geographically distributed WAN introduces communication delay that may be both **large and variable** compared with the simulation time step.

Consider:

* one-way latency;
* round-trip time;
* jitter;
* packet loss;
* available bandwidth;
* routing changes; and
* traffic congestion.

The simulators should therefore not generally be expected to execute every real-time step in strict lockstep across the WAN.

Instead, each simulator continues executing locally in real time. Information is exchanged at an application-appropriate rate and processed when available.

The simulators are aligned to **physical time**, but are not necessarily locked to each other's numerical solver steps.

### Assess loop delay

When exchanged data is part of a closed loop, the communication network becomes part of the simulated system dynamics.

The key question is therefore not simply:

> Can the simulated system remain meaningful and stable with the actual communication delay between the sites?

In a distributed control loop:

**Plant → Measurement → Network delay → Remote controller → Network delay → Command → Plant**

Round-trip network latency becomes part of the closed-loop dynamics.

Geographically distributed simulation is therefore more suitable for phenomena whose characteristic times are compatible with the communication delays between sites.

## Handle late or missing data

The model must define what happens when remote data:

* arrives late;
* arrives out of order;
* is duplicated;
* is corrupted; or
* never arrives.

Depending on the application, possible strategies include:

* holding the previous value;
* interpolating between valid values;
* extrapolating;
* rejecting data older than a defined threshold;
* using a fallback state or command;
* temporarily opening the distributed control loop; or
* stopping the simulation.

The appropriate strategy depends on the simulated system and must be part of the model design.

Do not assume that the network will always deliver each update within the expected interval.

## Limitations and considerations

* **Fast EMT coupling is particularly challenging:** Models running with microsecond-scale steps should not generally depend on receiving remote WAN data every simulation step. Keep fast local dynamics local whenever possible, and exchange higher-level measurements, states, references, or control information at rates compatible with the communication network.
* **Cybersecurity and network policies matter:** Communication between geographically separated laboratories may cross institutional firewalls, VPNs, research networks, public networks, network-address translation, and cybersecurity appliances. Identify the required addresses, ports, multicast behavior, routing, and security policies before the experiment.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://www.opal-rt.com/docs/rt-lab/model-and-simulation/co-simulation/run-geographically-distributed-real-time-simulations.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
