[paper reading] icml2020: can autonomous vehicles identify, recover from, and adapt to distribution shifts?

Column: January 6, 2022 7:18 PM
Last edited time: January 30, 2022 12:14 AM
Sensor/ organization : Oxford
Status: Finished
Summary: RIP out-of-distribution, How to consider the phenomenon of uncertainty identification Deal with
Type: ICML
Year: 2020
The amount of citation : 46

In progress → Reference Youtube Better than the paper

• References and foreword

  Project home page ：

  Robust Imitative Planning

  PMLR pdf + supplementary pdf：

  Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?

  github：https://github.com/OATML/oatomobile

  arxiv Address ：

  Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?

  youtube ( Advice to see Although it's a little long The comparison Easy to understand )：

  https://youtu.be/p73GZZFGkq4

1. Motivation

• link ：out-of-distribution

  Deep neural networks usually use Closed world hypothesis Training assumes that the distribution of test data is similar to that of training data . However , When used in real-world tasks , This assumption does not hold , Resulting in a significant decline in their performance .

  Why does the model have OOD brittleness ？

  1. Neural network models may Heavily dependent on training data False clues and annotation artifacts in (spurious cues and annotation artifacts), and OOD The example is unlikely to contain the same false patterns as the examples in the distribution .
  2. Training data cannot cover all aspects of distribution , Therefore, the generalization ability of the model is limited .

Familiarize yourself with abbreviations ：
OOD = out of distribution / Data that does not appear in the training set
RIP = robust imitative planning
Ada RIP = adaptive robust imitative planning

That is, if you encounter a scenario that you did not encounter during training at runtime / data , Although the variance of the model is relatively large , But this information is not used for processing , This paper mainly does ：

1. When testing the model , distinguish Scenes not encountered in the training set , namely OOD scene
2. stay OOD Scene On the model Judge Whether to believe and which one to believe or choice AdaRIP sampling

Problem scenario

[Sugiyama & Kawanabe, 2012; Amodei et al., 2016; Snoek et al., 2019] It has been proved many times , When ML When the model is exposed to a new environment （ That is to say Deviate from the training set In the case of the observed distribution ） when , Because they cannot be generalized , Its reliability will drop sharply , Which leads to disastrous results

give an example ： In this picture , Different Model given ${\mathbf{y}}^1,{\mathbf{y}}^3$ Is pretty good , But this is because of this scene ( Large disc ) It did not appear in the training set, and the existing model evaluation did not consider completely , Therefore, this paper proposes RIP In this scenario, it will be given for follow path For better ${\mathbf{y}}^3$, In the figure ${\text{min}}_k$ It refers to the same track $y_i$ Corresponding q1, q2, q3 The smallest

RIP Consider the differences between the models , To avoid the OOD Overconfidence in tasks leads to catastrophic Path result expansion

Although there are other comparisons trick Methods , such as Directly restrict vehicles within the lane line , Based on perception 、e2e Method , But this is also vulnerable spurious correlations. You will also get non causal characteristics And lead to OOD The chaos of actions in the scene

• Dolls about non-causal features that lead to confusion in OOD scenes (de Haan et al., 2019).

In the second half of the introduction, some people's work is cited The existing Baseline But they can't solve this out of distribution problem such as lbc, R2P2

Contribution

stay conclusion Conclusion There is a concise version in Mainly ：formulate out of distribution dataset The uncertainty of , Put forward RIP To solve this uncertain problem makes model robust, One last thing benchmark It is used to make your own model for everyone robust and OOD Performance under the event .

1. Epistemic uncertainty-aware planning：RIP In fact, it can be regarded as a Simple quantification of epistemic uncertainty with deep ensembles enables detection of distribution shifts.

   By using Bayesian decision theory and robust control objectives , Shows how to take conservative actions in unfamiliar situations , This usually enables us to recover from changes in distribution （ Pictured 1）

   link ：Monte-Carlo Dropout（ Monte Carlo dropout）,Aleatoric Uncertainty,Epistemic Uncertainty

2. Uncertainty-driven online adaptation： Adaptive robust imitation programming (AdaRIP), Use RIP Cognitive uncertainty estimation to effectively query expert feedback , For immediate adaptation , Without affecting security . therefore ,AdaRIP Can be deployed in the real world ： It can reason what it doesn't know , And in these cases Require manual guidance To ensure current security and improve future performance .

3. Autonomous car novel-scene benchmark： One benchmark be used for Evaluate the robustness of autonomous driving to a set of distributed tasks . Evaluation indicators ：

   1. testing OOD event , Measure by the correlation between violations and model uncertainty
   2. recover from distribution shift, Quantify by the percentage of successful maneuvers in the new scene
   3. Adapt effectively OOD scene , Provide online supervision

2. Method

The first is a few assumptions and formula explanation ：

1. Expert data ：$\mathcal{D}={\left\{\left({\mathbf{x}}^i,{\mathbf{y}}^i\right)\right\}}_{i=1}^N$; among ,$\mathbf{x}$ It is a high-dimensional observation input , $\mathbf{y}$ yes time-profiled Expert track , So expert strategy expert policy It can be expressed in this way ：$\mathbf{y}\sim {\pi }_{\text{expert }}(\cdot \mid \mathbf{x})$

   Imitation learning will be used in the method approximate the unkonwn expert policy

2. hypothesis Inverse Dynamics： Use PID Conduct Low-level control, This only needs to be aimed at the trajectory $\mathbf{y}=(s_1,\cdots ,s_T)$ The action is made by low-level controller To output $a_t=\mathbb{I}\left(s_t,s_{t+1}\right),\forall t=1,\dots ,T-1$

3. Suppose the global planning has , Assume truth location information get

concise ： The imitation learning results are expressed by Gaussian probability , Just learn the parameters of the distribution ; And then pass by aggregate and plan Make the final choice

Figure 1 ： From link youtube

among aggregate step That's the top $\oplus$ operator , There are two types shown in the yellow box calculation in Figure 1 ：

• Take the worst of the strategies Choose the higher of the poor （RIP-WCM） worst case model
  Inspired by (Wald,1939)

• The other is to add all Divide by quantity （RIP-MA） model averaging

  Inspired by Bayesian decision theory (Barber, 2012)

• There is another kind in the paper RIP-BCM It's found by the author's experience → ${\max }_k\log q_k$

The formula says

2.1 Expert data

give experts plan Distribution of → Because usually it is a direct action however softmax The previous one should also count distribution Well ？

Bayesian Imitative Model

• training via MLE

  In the data set $\mathcal{D}$ Next distribution density models $q(\mathbf{y}|\mathbf{x};\theta )$ A posteriori of $p(\mathbf{\theta }|\mathcal{D})$ , That is, the model parameters learned first through the data set , Then take it as a priori , Get into probabilisitc imitative mode

  $${\mathbf{\theta }}_{\mathrm{M}\mathrm{L}\mathrm{E}}=\underset{\mathbf{\theta }}{\mathrm{argmax}}{\mathbb{E}}_{(\mathbf{x},\mathbf{y})\sim \mathcal{D}}[\log q(\mathbf{y}\mid \mathbf{x};\mathbf{\theta })]\text{\hspace{1em}(1)}$$

• using probabilisitc imitative model $q(\mathbf{y}|\mathbf{x};\theta )$; Different from before (Rhinehart et al., 2020; Chen et al., 2019) Here is a prior distribution of model parameters $p(\mathbf{\theta })$ Used to substitute

  Observing $\mathbf{x}$ Next , Experts make $\mathbf{y}$ The probability of is → The author himself said Empirically I found it very effective

  $$\begin{array}{rl}q(\mathbf{y}\mid \mathbf{x};\mathbf{\theta })& =\prod _{t=1}^{T}p\left(s_t\mid {\mathbf{y}}_{<t},\mathbf{x};\mathbf{\theta }\right)\\ & =\prod _{t=1}^T\mathcal{N}\left(s_t;\mu \left({\mathbf{y}}_{<t},\mathbf{x};\mathbf{\theta }\right),\Sigma \left({\mathbf{y}}_{<t},\mathbf{x};\mathbf{\theta }\right)\right)\end{array}\text{\hspace{1em}(2)}$$

  among $\mu (\cdot ;\mathbf{\theta })$ , $\Sigma (\cdot ;\mathbf{\theta })$ Are the two RNN, Although the normal distribution is unimodal unimodality , But autoregression （ namely , Future samples depend on the sequential sampling of the past normal distribution ） Allow for multimodal distribution multi-model distribution Modeling

The whole process

1. Use deep imitative models As a posterior $p(\theta |D)$ A simple approximation of
2. Consider one K A collection of components , Use ${\theta }_k$ To refer to our first k A model $q_k$ Parameters of
3. Through maximum likelihood training （ See formula 1 and Frame diagram b part ）

2.2 distinguish OOD

Main comparison posterior $p(\mathbf{\theta }|\mathcal{D})$ Under each plan disagreement Use $\log q(\mathbf{y}\mid \mathbf{x};\mathbf{\theta })$ The variance of To point out the same policy How different are the different trajectories

$$u(\mathbf{y})\triangleq {\mathrm{Var}}_{p(\mathbf{\theta }\mid \mathcal{D})}[\log q(\mathbf{y}\mid \mathbf{x};\mathbf{\theta })]\text{\hspace{1em}(3)}$$

Low variance proof in-distribution, The high square difference is OOD

• How to determine this high and low threshold ？

2.3 Post Planning

Alternative planning strategies under cognitive uncertainty （ The red part of the picture below ）

Excerpt from youtube in ppt

First formulate Target location under cognitive uncertainty G Planning problems , That is, model parameters $p(\mathbf{\theta }|\mathcal{D})$ A posteriori of , Optimization as a general goal (Barber, 2012), We call it robust imitation programming (RIP)

$$\begin{array}{rl}{\mathbf{y}}_{\text{RIP }}^{\mathcal{G}}& \triangleq \underset{\mathbf{y}}{\mathrm{argmax}}\stackrel{\text{aggregation operator }}{\overbrace{\underset{\mathbf{\theta }\in \text{ supp }(p(\mathbf{\theta }\mid \mathcal{D}))}{\oplus }}}\log \underset{\text{imitation posterior }}{\underbrace{p(\mathbf{y}\mid \mathcal{G},\mathbf{x};\mathbf{\theta })}}\\ & =\underset{\mathbf{y}}{\mathrm{argmax}}\underset{\mathbf{\theta }\in \text{ supp }}{\oplus }\log \underset{\text{imitation prior }}{\underbrace{q(\mathbf{y}\mid \mathbf{x};\mathbf{\theta })}}+\log \underset{\text{goal likelihood }}{\underbrace{p(\mathcal{G}\mid \mathbf{y})}}\end{array}\text{\hspace{1em}(4)}$$

among $\oplus$ Is applied to a posteriori $p(\mathbf{\theta }|\mathcal{D})$ The operator of （ See the definition above ）, And the target likelihood is determined by, for example, the final target position $s_T^{\mathcal{G}}$ Gaussian for center and pre specified tolerance $p$ give $p(\mathcal{G}\mid \mathbf{y})=\mathcal{N}\left({\mathbf{y}}_T;{\mathbf{y}}_T^{\mathcal{G}},{ϵ}^{2}I\right)$

• It's a bit like Gaussian process Because in the whole T Gaussian process distribution value in time

Just like the formula 4 in plan ${\mathbf{y}}_{RIP}^G$ , What we maximize is mainly two parts ： From expert data imitation prior, and Close to the final target point G

• What is pointed out in the original text about posterior $p(\mathbf{\theta }|\mathcal{D})$ Yes our belief about the true expert model

  emmm But isn't this the model parameter trained through the data set ？ Why is a right expert model Of true Probability ？ You need to look at the code

  It means how much the corresponding model parameters are like this expert plan Do you ？ → It's like this

Although the description Deep imitation model DML (Rhinehart et al., 2020) It's a $\oplus$ selects a single ${\theta }_k$ from posterior The experiment partially proved So for OOD direct gg

2.4 AdaRIP

can we do better? → Experts step in The blue part

3. experimental result

• stay nuScenes： About out-of-distribution How to deal with it ？ Do you divide data manually ？

  Because in carla in It can be seen that these scenarios are manually selected for testing

stay nuScenes It's done on the dataset

I put forward a benchmark CARNOVEL

4. Conclusion

1. Put forward RIP Yes distribution shift Scene recognition and recovery
2. AdaRIP Act after understanding the uncertainty , According to online experts feedback Carry out parameter adaptation
3. Put forward a benchmark baseline Do this out of distribution problem

Excerpt from the original

twitter

This article The author's youtube It's really easier to understand than the paper hhhh ppt It's really good , Each step is also very clear , About open-question I also put forward my own shortcomings in the video （ There are still many deficiencies Mainly involving real vehicles real-time）

1. Real time cognitive uncertainty evaluator
2. Live online planning
3. Resistance to catastrophic forgetting in online adaptation → How to do incremental learning ？
   Resistance to catastrophic forgetting in online adaptation

• But I feel that the first real-time method from the paper It can be done , And there is no comparison of time in the full text ？ why open question Put forward ？ Namely ： The experimental part does not explain the real-time effect

Post the code later when you look at the code

• During the group meeting , The little friend pointed out $q_1,q_2,q_3$ If they are all from the same data set , The same training parameters , The model that should be trained is similar , There will be no diversity of choices Pictured 1
• There is also the fact that the theme of this article should be adaptive … But I didn't do it … adapt Artificially , In the words of brothers ： It turned out to be adding a layer embedding.