[paper reading] trajectory guided control prediction for end to end autonomous driving: a simple yet strong Ba

References and foreword

Summary: leaderboard The existing first place TCP, very simple Set up Good results were achieved

Thesis link ：Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline

Code link ：https://github.com/OpenPerceptionX/TCP The author's reply is expected to 9 Monthly opening

jg When the brush arrives I took a look at it Why are these frames so familiar … Look carefully at the back to find out Learn from each family's strengths High scores for simple sensors That's amazing

1. Motivation

For end-to-end driving tasks , Generally, the trajectory is predicted , Then follow with the controller ; The other is direct prediction controller Output ; Inspired by two methods , Author time ： We can learn from each other's advantages , Then the method of this paper

Contribution

1. The research tried two methods , Then the paper puts forward the combined pipeline
2. multi-step control Prediction makes temporal reasoning
3. CARLA Top of the list （ Ranking list viewing time June 25, 2022 ）

2. Method

Experts use Roach 【 use RL An expert strategy trained 】, here Whisper I collected some existing open source experts 【Roach, SEED, AUTO, MMFN】, Then I gave a repo carla-expert To collect for you … If you are interested, you can go and order star; If you want to do this later It must be useful , After all, the first time I did a task I've been working on data for a month

original e2e You can also write formulas Gain insight , It is probably the learned strategy and expert data loss In the entire dataset $\mathrm{D}=\{(x,a^{\ast })\}$ Next Take the smallest

$$\arg \underset{\theta }{\min }{\mathbb{E}}_{\left(\mathrm{x},{\mathbf{a}}^{\ast }\right)\sim \mathrm{D}}\left[\mathcal{L}\left({\mathbf{a}}^{\ast },{\pi }_{\theta }(\mathbf{x})\right)\right]\text{\hspace{1em}(1)}$$

2.1 frame

• Overall front encoder frame It can be compared with CILRS
• Loss Design can see Roach One article

Input ： A picture ; The speed of the current frame ,high command, Target point, etc

Network details , There are concise descriptions in the appendix and the text ：

1. The camera size is 900x256,FOV Directly fill it with 100
2. K = 4 That is, choose the next four steps steps The action of / Track into
3. Image Encoder It uses pre trained ResNet-34
4. measurement encoder It is a MLP [ The full text does not elaborate on the specific layers ; For the time being, consider and CILRS Consistent settings That is to say According to the input num Corresponding to the corresponding number of linear To encoder in ] → 128
5. Their respective encoder Output concat together form ${\mathbf{j}}^{\mathbf{t}\mathbf{r}\mathbf{a}\mathbf{j}}$ → 256

Next Two branch Are directly taken from the common output feature Enter the respective branches

• that … loss Yes? backpropagate?

  see loss part , Their respective loss Add up , direct .backward()

2.2 Trajectory Branch

and [16] The same will ${\mathbf{j}}^{\mathbf{t}\mathbf{r}\mathbf{a}\mathbf{j}}$ Deliver to GRU in , It should be and transfuser The operation is the same , It's probably the feeling in the figure below ：

Excerpt from transfuser One article

among Because there is experts The true value of (x,y) Track points as a reference , So it can be counted as loss

Pay attention to the overall frame diagram , After getting these trajectory points , It uses two horizontal and vertical PID Transfer the trajectory to the control quantity , Different from the previous method , there PID Control is horizontal and vertical error, Most of the previous methods are basically a speed control （ Given a speed ）, One steering control is primary

2.3 Multi-Step Control Branch

Here is an important point ： Before this Based on the current input Predict the next output , The overall output is assumed to be independent and identically distributed . But actually Current input and historical input , And future output are not independent . Contrast and use MDP（ Markov decision making process ） and RL Go back / forward , Proposed a simpler way

First, just look at the corresponding light yellow modules in the main frame diagram ,

temporal module Input is ${\mathbf{j}}_{\mathrm{t}}^{\text{ctl }}$ and ${\mathbf{a}}_t^{ctl}$ concat After , among ：

$${\mathbf{j}}_{\mathrm{t}}^{\text{ctl }}=\mathrm{MLP}\left(\text{ Concat }\left[\mathrm{Sum}\left(\mathrm{Softmax}\left({\mathbf{w}}_{\mathrm{t}}\right)\odot \mathbf{F}\right),{\mathbf{h}}_{\mathrm{t}}^{\text{ctl }}\right]\right)$$

among ${\mathbf{w}}_{\mathbf{t}}$ By t The moment hidden state concat Together , See the formula below ：

and ${\mathbf{h}}_t^{traj},{\mathbf{h}}_t^{ctl}$ It's from their own Trajectory Branch and Control Branch And come

---

Problem area ：

• But there is a little doubt here branch Which part of the interior hidden state？

  Both modules in Figure 3 are GRU Realized ,hidden state From each GRU Get in

• Trajectory yes GRU May be t Next GRU, therefore K Yes, it means how many GRU modular ？

  Yes

• but Control branch What kind of network MLP？ Which floor is that hidden state Well ？

  yes MLP hidden state It's all their own GRU From inside , The two before the output in Figure 3 are MLP

2.4 Loss Design

There are mainly three kinds , The trajectory 、 Control and attachment loss

For trajectories ：

with transfuser The difference is to join feature Of loss Come in ,j yes 128 Dimensional

For control ：

For additional , from image feature Added speed prediction head L1 loss And a value predict forecast expected return Same as [55] equally ,L2 Loss

Then three loss Add up ：

$$\mathcal{L}={\lambda }_{traj}\cdot {\mathcal{L}}_{traj}+{\lambda }_{ctl}\cdot {\mathcal{L}}_{ctl}+{\lambda }_{aux}\cdot {\mathcal{L}}_{aux}$$

---

Problem area ：

• Why feature… How do you do it? loss,expert Of feature It is also output from the same network You can play like this ？

  roach practice , from BEV Under the gt Make different inputs , But I don't know if it is collected directly expert Of feature Or use BEV Input to this network loss

  • You may have to go and have a look roach Of loss How to play

2.5 Fusion Approach

The integration method is also quite Engineering , Direct judgment In the past 1s Internal steer Absolute value addition Is it more than 0.1, If you exceed, you are considered to be making a turn , Just switch to Mainly track branches ; Otherwise, the multi-step control branch is mainly used

3. Experiments and results

Experiments and results Personally, I think the author is a little easy … Because I smoked directly leaderboard, however ！CARLA The difference between leaderboards and others is ： Data sets are created by you , In general , Everyone will use their own data sets to train other methods more or less agent In the same expert training The contrast between the two … 【 The general meaning is ： The experimental conditions are not consistent , Although you are good online But maybe someone else just used you 1/2 The data of …】

however This does not mean that the method is not effective , After all This is a fraction of what a camera can do , I deeply admire ！ The following is just a screenshot of each experiment table and visualization , Because I am too familiar with this task No more , Interested in If you want to try the list, you can have a look relevant CARLA leaderbaord Configure teaching articles It contains a description of the indicators

Some in their own way Control variable Ablation Experiment

twitter

This part is read in pieces Some finished reading in the previous experiment hhhh come from jg I'm sorry ： Yes CARLA leaderboard Complete overfitting of hhhh After all, this method will not be tried in the car … LAV At least one condition of this method is that it can measure a wave on a real vehicle , It will be much more interpretable such as Find out why , however LAV The real wife is too big , This method The first time my grades were anonymous I don't think it's possible … Return to yourself quick setting A wave of a camera experiment But the effect is not good ( at that time TCP The article hasn't been posted yet )… So let's see if there's a chance Look at the article and repeat it briefly

PS Now, your data set One by one , When I was doing it 100K The frame data is less than … LAV 390K,TCP 420K, Now that I think about it I seem to know why my score is so different It's also a task that I can't understand at the back , however planning This is the only task … The list can be brushed

---

Give someone a compliment Fragrance in hand ; Positive feedback To better open the record hhh