5.7 Simulation and XiL

Build intuition

As we saw in 5.6, the number of scenarios needed runs into the millions. They cannot all be tried with real vehicles. This is where simulation comes in: roads, vehicles, pedestrians, weather and sensors are built inside a computer, and tests can be run millions of times, repeating the same conditions exactly.

A simulation, however, is always “made up”. Raising realism reduces the number of tests possible, and raising the number lowers realism. So tests that change, step by step, how much is real and how much is virtual are used side by side. This is called XiL (X-in-the-Loop).

A simulation combines models of vehicle motion, sensor models (camera images, LiDAR point clouds, radar), models of how surrounding vehicles and pedestrians behave, reproduction of weather and lighting, and machinery to run scenarios. Every part differs from reality to some extent (the gap between simulation and reality), and the size of that gap determines how far results can be trusted. Common standards such as ASAM OpenSCENARIO and OpenDRIVE describe scenarios and roads, so scenarios can be passed between tools.

Check with math

① Simulation piles up the count

With m computers in parallel, each running r scenarios per unit of time, the time needed to run N scenarios is

The time per scenario, however, depends heavily on how much detail is reproduced. A coarse simulation that moves surrounding vehicles and pedestrians as “boxes” (object level) can compute a 30-second situation in about 10 seconds, but a sensor-level simulation that also generates camera images and LiDAR point clouds takes about 2 minutes per scenario even on a GPU (illustrative figures). With 50 computers running the coarse simulation, that is 50 × 8,640 = 432,000 a day, so the 3 million scenarios of 5.6 are done in about 7 days. Running the same 3 million at sensor level on 20 GPU machines gives 20 × 720 = 14,400 a day — about 210 days.

In practice, therefore, the whole set is first run in the coarse simulation, and only the situations with little stopping margin (say the 1%, or 30,000) are checked at sensor level. 30,000 take about 2 days, so in about 9 days in total the dangerous regions are examined in detail (the idea of searching for dangerous situations is in 5.8). Note that perception errors do not occur in the coarse simulation, so hazards involving perception must be covered either by feeding miss and error probabilities measured on real data into the model (5.9) or by searching separately at sensor level. On a test track at 50 a day, the 3 million would take 60,000 days, about 160 years. Meeting validation targets without simulation is practically impossible.

② Measure the gap with reality and add it as a margin

Whether simulation results can be trusted is checked by running the same situations both in reality and in simulation and comparing them. Suppose stopping distances for 20 situations were measured on a test track and compared with simulation, and the differences \(e_i = x_{\text{real}} - x_{\text{sim}}\) had a mean of \(\bar{e} = 0.4\) m and a standard deviation of \(s_e = 0.3\) m. The simulation underestimates stopping distance by 0.4 m on average — it is optimistic. So the pass judgement adds to the simulation result a

(using the standard deviation of 1.4; ±2σ covers about 95%). In other words, “only situations where the simulated vehicle stopped with at least 1.0 m to spare are taken to stop in reality too”. But the gap is known only within the conditions compared (a dry road, say). On a wet road it must be compared again.

③ Check in reality the dangerous situations found in simulation

Checking can also run the other way. Suppose 10 of the situations where simulation said “collision” were reproduced on a test track, and 9 of them came similarly close to a collision. The agreement is 90%, but the confidence interval of 1.6,

is wide: all we can say is “roughly 70% or more agree”. With 90 of 100, it narrows to ±6%. Even more important are situations where simulation says “no collision” but reality collides. If there are many of these, showing safety in simulation means nothing. Situations near the boundary that simulation rated “safe” are selected and tried in reality to check for such misses.

Summary

• Simulation piles up counts but is “made up”. Tests with different shares of real components — MIL, SIL, HIL, VIL, test track, public road — are used side by side (XiL)
• Large numbers of scenarios can be run in N/(mr), but the gap with reality is measured and added to the pass margin (ē + 2sₑ). The gap is known only within the range compared
• Simulation results are checked by reproducing them in reality, looking especially for “safe in simulation, dangerous in reality” misses. What is not modelled cannot be tested, so assessment uses multiple pillars

Quick quiz

Further reading

• Next lesson: 5.8 Searching for unknown hazards
• Related: 5.6 Scenario-based testing, 5.5 Acceptance criteria and validation targets, 4.3 Sensors and fusion, 4.6 Modular vs. end-to-end (closed-loop evaluation), 2.8 Data splits and distribution shift, 1.6 Estimation and confidence intervals
• Glossary: XiL (MIL, SIL, HIL, VIL), Gap between simulation and reality, Multi-pillar assessment, Open-loop / closed-loop evaluation, Scenario-based testing
• Video lecture: coming soon
• References (this lesson summarizes the standards and documents in its own words; check the originals for exact definitions and requirements)
  • UNECE WP.29, “New Assessment/Test Method for Automated Driving (NATM) — Master Document,” 2021.
  • ASAM, “OpenSCENARIO” and “OpenDRIVE” standards.
  • A. Dosovitskiy et al., “CARLA: An Open Urban Driving Simulator,” Conference on Robot Learning, 2017.