[point cloud processing paper crazy reading frontier version 11] - unsupervised point cloud pre training via occlusion completion

Abstract

• Method ： A pre training method for point cloud is proposed Occlusion Completion (OcCo)
• Technical details ：
  1. mask The blocked point in the camera's perspective
  2. Learn one encoder-decoder Model , Used to reconstruct occluded points
  3. Use encoder As the initialization of the downstream point cloud task
• application ： object classification & part-based and semantic segmentation
• Code ：https://github.com/hansen7/OcCo ( Support PyTorch and TensorFlow)

introduction

OcCo It has the following properties ：

1. Study in small samples (few-shot learning) The sampling efficiency can be improved in the experiment
2. It can improve generalization in classification and segmentation tasks
3. After fine adjustment, it is easier to find the local minimum
4. adopt network dissection Able to describe a more semantic representation
5. stay jittering, translation and rotation Better classification quality can still be maintained under transformation

Method

remember $\mathcal{P}$ by 3D A group of point clouds in European space ,$\mathcal{P}=\left\{p_1,p_2,\dots ,p_n\right\}$, Each of these points $p_i$ It contains coordinates $\left(x_i,y_i,z_i\right)$ And other features ( Color and normal vector ) Vector . First from occlusion mapping $o(\cdot )$ Begin to describe , And then introduce ompletion model$c(\cdot )$, Pseudocode and structural details are shown in the appendix .

Generating Occlusions

Define a randomised occlusion mapping $o:\mathbb{P}\to \mathbb{P}$ , among $\mathbb{P}$ It's point cloud space , The description is from all point clouds $\mathcal{P}$ To cover the point cloud $\tilde{\mathcal{P}}$ Mapping between . This mapping is done by removing $\mathbb{P}$ Those points that cannot be seen from a specific point of view $\tilde{\mathcal{P}}$, Steps are as follows ：

1. The complete point cloud in the world coordinate system is projected onto the coordinates in the camera coordinate system according to the viewpoint of the camera
2. Determine the occluded point under this viewpoint
3. Then back project the points in the camera coordinate system to the world coordinate system

Viewing the point cloud from a camera

Define from... Through pinhole camera 3D Mapping between the world coordinate system and a specific camera coordinate system ：

among $(x,y,z)$ Is the coordinate of the original point cloud in the world coordinate system , The camera viewpoint is rotated matrix $\mathbf{R}$ Vector of peaceful shift $\mathbf{t}$ decision . Inside the camera $\mathbf{K}$ By focal length $f$,skewness $\gamma$, The width of the image $w$, high $h$ decision . After giving the above parameters , You can calculate the coordinates of the point in the camera coordinate system $\left(x_{\mathrm{c}\mathrm{a}\mathrm{m}},y_{\mathrm{c}\mathrm{a}\mathrm{m}},z_{\mathrm{c}\mathrm{a}\mathrm{m}}\right)$.

Determining occluded points

Handle points in two ways $\left(x_{\mathrm{c}\mathrm{a}\mathrm{m}},y_{\mathrm{c}\mathrm{a}\mathrm{m}},z_{\mathrm{c}\mathrm{a}\mathrm{m}}\right)$ ：

1. In the camera coordinate system 3D spot $\left(x_{\mathrm{c}\mathrm{a}\mathrm{m}},y_{\mathrm{c}\mathrm{a}\mathrm{m}},z_{\mathrm{c}\mathrm{a}\mathrm{m}}\right)$
2. Depth is $z_{\mathrm{c}\mathrm{a}\mathrm{m}}$ Of 2D Pixel coordinates $\left(f{x}_{\mathrm{c}\mathrm{a}\mathrm{m}}/z_{\mathrm{c}\mathrm{a}\mathrm{m}},f{y}_{\mathrm{c}\mathrm{a}\mathrm{m}}/z_{\mathrm{c}\mathrm{a}\mathrm{m}}\right)$

In this way , If some points obtained by projection have the same pixel coordinates , But the depth value is different , Then there may be an occlusion relationship between these points . To determine which points are obscured , We use... First Delaunay triangulation To rebuild a polygon mesh, Then remove those belonging to hidden surface The point of , This hidden surface adopt z-buffering decision .

Mapping back from camera frame to world frame

Once the occluded point is removed , We can re project the point into the original world coordinate system , The principle used is formula 1 The inverse transformation of . therefore randomised occlusion mapping $o(\cdot )$ The construction steps of are as follows ：

1. Fix a set of initial point clouds $\mathcal{P}$
2. Given the camera's internal parameter matrix $\mathbf{K}$, External parameters under multiple viewpoints $\left[\left[{\mathbf{R}}_1\mid {\mathbf{t}}_1\right],\dots ,\left[{\mathbf{R}}_V\mid {\mathbf{t}}_V\right]\right]$, among $V$ Indicates the number of viewpoints
3. For each viewpoint $v\in [V]$, Use the formula 1 take $\mathcal{P}$ Are projected into the corresponding camera coordinate system
4. Find the occlusion points and remove them
5. Back project the remaining points into the world coordinate system , For each viewpoint $v\in [V]$, Get the final occlusion point cloud ${\tilde{\mathcal{P}}}_v$

The Completion Task

Given by occlusion mapping $o(\cdot )$ Obtained point cloud $\tilde{\mathcal{P}}$, The goal of the completion task is to start from $\tilde{\mathcal{P}}$ Learn one completion mapping $c:\mathbb{P}\to \mathbb{P}$, Used to complete the point cloud $\hat{\mathcal{P}}$. If meet ${\mathbb{E}}_{\tilde{\mathcal{P}}\sim o(\mathcal{P})}\ell (c(\tilde{\mathcal{P}}),\mathcal{P})\to 0$, It means that completion mapping To be accurate , among $\ell (\cdot ,\cdot )$ Is the loss function . The structure of the complement model is a encoder-decoder Network of ,encoder Map the occluded network into a vector ,decoder Complete the point cloud . After pre training ,encoder The weight of can be used as the initial value of downstream tasks .

experiment

OcCo Pre-Training Setup

Used in all experiments ModelNet40 As a pre training data set . The internal parameter of the camera is set to $\left\{f=1000,\gamma =0,w=1600,h=1200\right\}$ . For each group of point clouds , Random selection 10 Group viewpoint , The viewpoint rotation is different , Pan is set to 0.

completion model in ,encoder It can be set to PointNet, PCN and DGCNN.decoder choice folding operation , The reconstruction step is divided into two steps , The first step will be 1024 The occlusion vector of dimension is converted to include 1024 Point coarse Point cloud ${\hat{\mathcal{P}}}_{\text{coarse }}$, Then on ${\hat{\mathcal{P}}}_{\text{coarse }}$ Every point in the uses $4\times 4$ Of 2D Mesh reconstruction with 16384 Point fine shape ${\hat{\mathcal{P}}}_{\text{fine }}$, Use Chamfer Distance (CD) As a prediction $\hat{\mathcal{P}}$ and ground truth $\mathcal{P}$ Loss function between ：
$$\begin{array}{rl}\mathrm{C}\mathrm{D}(\hat{\mathcal{P}},\mathcal{P})& =\frac{1}{|\hat{\mathcal{P}}|}\sum _{\hat{x}\in \hat{\mathcal{P}}}\underset{x\in \mathcal{P}}{\min }\Vert \hat{x}-x{\Vert }_2+\frac{1}{|\mathcal{P}|}\sum _{x\in \mathcal{P}}\underset{\hat{x}\in \hat{\mathcal{P}}}{\min }\Vert x-\hat{x}{\Vert }_2\end{array}$$
The ultimate model loss is coarse and fine The shape of the Chamfer distances Weighted sums ：
$$\ell :=\mathrm{CD}\left({\hat{\mathcal{P}}}_{\text{coarse }},{\mathcal{P}}_{\text{coarse }}\right)+\alpha \mathrm{C}\mathrm{D}\left({\hat{\mathcal{P}}}_{\text{fine }},{\mathcal{P}}_{\text{fine }}\right)$$

Fine-Tuning Setup

Few-shot learning

The goal of small sample learning is to use very limited data to train accurate models , During the training , Random selection $K$ individual classes, Every category Both contain $N$ Samples .

Object classification

Part segmentation

Semantic segmentation

analysis

Visualisation of optimisation landscape

Visualisation of learned features

Unsupervised mutual information probe

Detection of semantic concepts

Discuss

some time , It would be more interesting to design a complete model that shows the occlusion steps considered . The model may converge faster , You need fewer parameters , It can also be used as a strong bias in training .

New words

• a flurry of A lot of