Dynamic loss balancing and sequential enhancement for road-safety assessment and traffic scene classification

TL;DR

This work proposes to reduce dependency on tedious human labor by automating attribute collection through a two-stage deep learning approach that alleviates extreme class imbalance by incorporating a multi-task variant of recall-based dynamic loss weighting.

Abstract

Road-safety inspection is an indispensable instrument for reducing road-accident fatalities contributed to road infrastructure. Recent work formalizes road-safety assessment in terms of carefully selected risk factors that are also known as road-safety attributes. In current practice, these attributes are manually annotated in geo-referenced monocular video for each road segment. We propose to reduce dependency on tedious human labor by automating recognition with a two-stage neural architecture. The first stage predicts more than forty road-safety attributes by observing a local spatio-temporal context. Our design leverages an efficient convolutional pipeline, which benefits from pre-training on semantic segmentation of street scenes. The second stage enhances predictions through sequential integration across a larger temporal window. Our design leverages per-attribute instances of a lightweight bidirectional LSTM architecture. Both stages alleviate extreme class imbalance by incorporating a multi-task variant of recall-based dynamic loss weighting. We perform experiments on the iRAP-BH dataset, which involves fully labeled geo-referenced video along 2,300 km of public roads in Bosnia and Herzegovina. We also validate our approach by comparing it with the related work on two road-scene classification datasets from the literature: Honda Scenes and FM3m. Experimental evaluation confirms the value of our contributions on all three datasets.

Figures (3)

• Figure 1: Our multi-frame local recognition pipeline recognizes road-safety attributes in multi-frame input $\mathbf{X}$. Data tensors are represented as cuboids, while processing modules are shown as rounded rectangles. The shared front-end (red) maps each input frame into convolutional features ($\mathbf{F}$) that are subsequently pooled by the $\mathbf{SPP}$ module. Attribute-specific back-ends (green) produce attention pools (yellow) and concatenate them with the shared spatial pools (blue). Fully-connected layers ($\textrm{FC}_i$) map the concatenated descriptors into attribute-specific logits.
• Figure 2: Sequential enhancement corrects local predictions with per-attribute Bi-LSTM models. For each attribute $a$, the model $\mathrm{Bi\text{-}LSTM}_{a}$ outputs corrected logits $\mathbf{s'^a_t}$ in segment t by observing T = 21 vectors that correspond to segments from (t-10) to (t+10). Each of these vectors is a concatenation of the logits $\mathbf{s_i^a}$ and the jointly learned embedding $\mathbf{e_i^a}$ of the the most probable class according to the local model.
• Figure 3: Four iRAP-BH examples where sequential enhancement (LSTM) succeeds to correct our local visual predictions (CNN). For each of the five consecutive segments (columns), we display categorical predictions by both models (top) and the ground truth label (bottom right). Row 1 involves a single-peak attribute - Intersection type. We observe that the local model incorrectly assigns a positive class (3-way intersection) in column 4. Rows 2 and 3 involve smooth attributes - Bicycle facility and Number of lanes. We observe that the local model mistakes a motorcyclist near a tram rail for a dedicated bicycle lane in column 3 and fails to predict the correct number of lanes again in column 3. Row 4 involves the Street lighting attribute. We observe that the upcoming lighting poles are obscured by the road curvature and overgrown roadside bushes. Consequently, the local model fails in columns 2-4. In all cases, the sequential model succeeds to correct the mistakes by leveraging a larger temporal context.