MeBeSafe
MeBeSafe is a project in which Cygnify works together with multiple world-class European automotive OEMs and research institutes, to develop in-vehicle (ADAS) and infrastructure systems to "nudge" drivers to safer behavior. The consortium, coordinated by one of Germany's leading automotive research institutes, RWTH Aachen's Institute for Automotive Engineering (IKA), was awarded EU Horizon2020 Research & Innovation funding aimed at bringing Europe's best innovations to the market.
For the project Cygnify has, among other things, further developed its deep learning-powered road user forecasting method. We focus in particular on predicting the behavior of vulnerable road users (pedestrians and especially bicyclists), for a time horizon of 5 to 10 seconds ahead. The main sensor inputs come from forward-looking cameras, optionally combined with lidar or radar sensors, allowing state-of-the-art detection, tracking, and forecasting of road users' trajectories. Based in part on those forecasts and the predicted likelihood of the various options, hazardous situations associated with predicted intersecting trajectories are identified.
For each road user tracked using the forward-looking camera, a multi-level forecast is produced in real time, taking into account its recent trajectory, road infrastructure and lane choice, and other road users. Both the high-level and the low-level forecast is produced using a deep learning neural network with LSTMs at its core, to exploit the sequential nature of the input data.
The high-level prediction produces multi-modal probabilistic forecasts which accurately reflect the probability of each likely direction choice (e.g. going straight, making a left turn, making a right turn). Each of those high-level direction choices (or "destinations") are sent to the low-level trajectory prediction network. That network produces a precise, second-to-second trajectory prediction of (x,y) coordinates for each likely direction choice. Each such low-level trajectory prediction takes into account the specific infrastructure layout, other road users, and the target road user's observed trajectory and speed behavior. Importantly, the method does not require detailed HD-map type infrastructure information (even though that helps). It only requires "drivable area" information which can be determined based on forward-looking camera information combined with simple maps.
In addition, our algorithms can combine this information with driver gaze direction information, from inward-looking cameras. Thus, estimates are made of whether the driver may have missed relevant road users and associated hazards; which can trigger a visual or auditory "nudge" to direct the driver's attention appropriately.
Visualization of the multi-level road user forecasting process. This example is using Argo.ai's open-sourced vehicle trajectories and maps datasets. The ego-vehicle is depicted as a blue dot. The high-level network predicts the high-level direction choice of the target vehicle (red dot) based on a grid cell, occupancy map-type discretization of the intersection layout. These high-level predictions are probabilistic, giving each likely option a matching probability value. For each likely option, a plausible low-level trajectory prediction is produced, which takes the target's recent specific trajectory and speed, specific infrastructure layout, and other road users into account. Note that as time progresses, for this target vehicle the "going straight" option becomes increasingly unlikely, whereas the "turning right" option increasingly becomes the only likely option.
Demonstration of driver gaze direction estimation combined with outside driving context object detection and tracking. Based on this, our algorithms can estimate whether the driver has seen a certain likely hazard, and if necessary "nudge" the driver's attention in the right direction.
See more on the MeBeSafe website.
Contact us for more information about our work in MeBeSafe.