Long-Range Obstacle Detection for Advanced Driver Assistance Systems (LROD-ADAS)

Life-threatening risks to humans, damage to the vehicles, and extensive disruptions to the operation of the railway network can be prevented through the early enough detection of possible obstacles on the train tracks. Today, this is solely the task of the train driver, who is, however, limited by his range of vision, his personal fitness, as well as current environmental conditions, such as lighting and weather. Infrastructure-side video surveillance is being utilized near train stations or railway crossings, but equipping an entire railway network with such sensors results in high installation and operating costs. Advanced Driver Assistance Systems (ADAS) with object detection capabilities could support the train driver through modern on-board sensor technologies and data-driven object detection. The required range for obstacle detection should as an optimum exceed the individual braking distance of a train, which due to its weight, high travelling speed and low braking force, from metall wheels on metal tracks, often surpasses the 500 m mark. Various sensor technologies are already in use for obstacle detection in the field of Autonomous Mobile Robots (AMRs), such as Light Detection And Ranging (LiDAR), radar systems or stereo vision. However, for use in rail vehicles, these do not have a sufficient detection range or are limited by the particular reflective properties required of a target, consequently rendering them inadequate for a direct broad adaptation.

Within a recently concluded Mobility Initiative project, in collaboration with SBB, Cornelius von Einem, and his colleagues from the Autonomous Systems Lab led by Professor Roland Siegward, explored the feasibility of an on-board long-range obstacle detection system, capable of detecting and localizing any known and unknown obstacles on train tracks at long-range. The work comprised the

• design of a novel active long-range sensor suite,
• the development of a data-driven obstacle detection algorithm,
• and a flexible sensor fusion method for vehicle state estimation.

The main challenges involved designing a sufficiently precise sensor solution, as well as algorithmic challenges in detecting possible hazards, without available training data thereof, and estimating the train’s position for accurate map to camera associations.

High-precision long-range sensor suite for railway applicaitons

The team developed a high-precision long-range sensor suite, consisting of high focal-length cameras, a long-distance 1D LiDAR and most notably a novel, high-precision, low-cost actuation platform for actively pointing the sensor suite at possible regions of interest on the horizon. Our custom gimballing platform demonstrates pointing accuracy of less than 7 mdeg, sufficient to detect human-sized obstacles at a range of 1500 m. Furthermore, we demonstrate a railway-specific calibration procedure utilizing railway tracks as a georeferenced long-range calibration target.

Figure 2: Figure (a) shows the detection process for potential obstacles. A map overlay obtained from the vehicle position and map information (Figure (b)) provide an RoI. Obstacles are detected using a neural network and their relative position is determined using a long range LiDAR. Finally, the absolute obstacle position is shown on the map in Figure (b).

A novel railway anomaly detection method

The project addresses then the visual detection of possible hazards on the railway tracks ahead. Common obstacle detection systems are trained on a pre-defined set of possible objects but are thus, by design, also limited to these categories and fail to generalize to unknown obstacle types, such as fallen trees after a storm. Furthermore, the amount of training data available in the railway domain is severely limited, often leading to the use of artificially augmented datasets, resulting in unexpected detector biases. We developed a novel railway anomaly detection method by reformulating it as a railway track segmentation task, specifically excluding obstacles from the training process. By analyzing the difference of the actual train track detection, to where we expect to detect tracks, we are able to reliably detect obstacles, without taking any opinion on the nature of the obstacle. We further propose two new methods and evaluate them against multiple existing approaches from the state-of-the-art in an in-depth ablation study.

Improved state estimation of rail vehicles

In order to associate global map data and the knowledge of the position of train tracks ahead, with what the sensors observe, it is crucial to know the vehicle’s current position. Individual sensor modalities are not capable of providing an accurate, robust, and high-frequency state estimate, necessitating a sensor fusion approach. As trains move in virtually one dimension on accurately mapped train tracks, we present a path-constrained multi-hypothesis Extended Kalman Filter (EKF), capable of fusing absolute and relative sensor measurement for an accurate pose estimate. We recorded multiple datasets on a tram in the city of Zurich to demonstrate the accuracy and reliability of our method.

Collaboration

Cornelius conducted his dissertation on the project, he found this collaborative research project an interesting and challenging opportunity.  

"Particularly constraining, and this is something this project evidenced for all stakeholders, was the lack of publicly existing railway datasets. Particularly for the vision components. This is also where the collaboration with Siemens and SBB was critical, they shared datasets and also enabled us to record the needed datasets on our own, which was key for the progress of this project. I would like to particularly thank Lisa Sidler from SBB, who helped us organize data collections, not only at the beginning of the project. Other important collaborators were Kristian Weiss from Siemens and Simon Steiner as the second team member from SBB."