ADOP: Approximate Differentiable One-Pixel Point Rendering

Related tags

Deep LearningADOP
Overview

ADOP: Approximate Differentiable One-Pixel Point Rendering

Abstract: We present a novel point-based, differentiable neural rendering pipeline for scene refinement and novel view synthesis. The input are an initial estimate of the point cloud and the camera parameters. The output are synthesized images from arbitrary camera poses. The point cloud rendering is performed by a differentiable renderer using multi-resolution one-pixel point rasterization. Spatial gradients of the discrete rasterization are approximated by the novel concept of ghost geometry. After rendering, the neural image pyramid is passed through a deep neural network for shading calculations and hole-filling. A differentiable, physically-based tonemapper then converts the intermediate output to the target image. Since all stages of the pipeline are differentiable, we optimize all of the scene's parameters i.e. camera model, camera pose, point position, point color, environment map, rendering network weights, vignetting, camera response function, per image exposure, and per image white balance. We show that our system is able to synthesize sharper and more consistent novel views than existing approaches because the initial reconstruction is refined during training. The efficient one-pixel point rasterization allows us to use arbitrary camera models and display scenes with well over 100M points in real time.

  • The source code will be published after the paper has been accepted to a conference.

[Full Paper]

Video

Watch the video

Comments
  • Guide on exporting scene from COLMAP

    Guide on exporting scene from COLMAP

    Hi! I have some troubles exporting point cloud from COLMAP to compatible with ADOP format. Here is what I did:

    1. Autoreconstruction in COLMAP (both sparse and dense)
    2. File > Export As > as .ply
    3. preprocessed it with colmap2adop tool
    4. started training
    5. got an error that my point cloud file does not contain normal data
    6. I checked that your .ply files have normal data in headers, but I struggle to make COLMAP include it in mine
    opened by DenShlk 22
  • RAM usage when training dataset

    RAM usage when training dataset

    Through a lot of trial and error I managed to get both the ADOP viewer and trainer set up on Ubuntu 20.04.

    SETUP I'm trying to run my own dataset through the trainer to see how it is. I only have an Nvidia GTX 1070 card with 8GB of VRAM, 16GB of physical RAM but plenty of hard disk space. I've gotten to the point that the trainer runs without crashing. My dataset is 175 images, downscaled to 1000x562, converted from a ~7M point-cloud project in COLMAP.

    PROBLEM However, running the trainer seems to take up over 1GB per epoch on top of the base amount. This quickly sends my RAM usage to near full. Luckily I have a large swap partition but that too fills up over hundreds of epochs. Is this expected behaviour or a memory leak? Or could it be because my install or conda environment might be faulty? Or is it my incapable hardware? In any case, once all my swap space gets filled the program is unfortunately killed on its own.

    If tweaking any of the training config or dataset parameters could potentially fix or mitigate this issue, please do let me know. Thank you for your work on this, I find this to be a very interesting project.

    opened by ShinyLuxray 19
  • How to optimize for rendering

    How to optimize for rendering

    Hi I have a question regarding the adop_viewer. l tried the adop_viewer with pretrained models on tanks and temples dataset, it works perfectly on a laptop with a 3080 GPU (8GB VRAM) on all scenes.

    I guess you are using the 40GB A100 ini files for training these models, so I also tried training my data on a GPU with similar VRAM using exact settings from train_tank_and_temples_multi.ini. However, when I attempt to visualize my checkpoints on the 3080 GPU it shows CUDA out of memory for all my scenes.

    My question is: for rendering, did you make some optimizations to make the trained models fit in a GPU with less VRAM? Or is there certain configurations/arguments that allows the adop_viewer to take a model trained with high-end GPUs and render on low-end GPUs? Thanks in advance!

    opened by qiaosongwang 12
  • Issue with multi-camera COLMAP dataset

    Issue with multi-camera COLMAP dataset

    I have successfully trained with ADOP on scenes that use a single camera. I am having difficulty, however, with a larger dataset comprised of 882 cameras, 2/3 of which are from drone footage, the other 1/3 being a hand-held DSLR.

    COLMAP successfully generates the camera registrations and point cloud, and running colmap2adop correctly creates all the necessary files for training. I am noticing, however, that COLMAP resizes every single image, effectively generating 882 individual camera profiles. I thought at first this wouldn't necessarily be an issue, as adop_train can parse multiple cameras. I am running into this error, though:

    Assertion 'cam.w == image_size_input(0)' failed!
  File: /home/visionarymind/Downloads/ADOP/src/lib/data/Dataset.cpp:49
  Function: SceneDataTrainSampler::SceneDataTrainSampler(std::shared_ptr<SceneData>, std::vector<int>, bool, Saiga::ivec2, int, bool)
Aborted (core dumped)

    I have looked at each camera.ini file, and the dimensions match each image. Would you have any idea why this error would be thrown and any way to work around it? Technically, there were only two cameras, but COLMAP resizes the images during registration.

    opened by VisionaryMind 10
  • Something worry in PointRenderer.cu & PointRenderer.h

    Something worry in PointRenderer.cu & PointRenderer.h

    2021-11-10 15-59-34屏幕截图 Can not Build CUDA object src/lib/CMakeFiles/NeuralPoints.dir/rendering/PointRenderer.cu.o

    /home/xgenie/projectfold/ADOP/src/lib/./config.h(25): error: inline specifier allowed on function declarations only

    /home/xgenie/projectfold/ADOP/src/lib/./config.h(25): error: inline specifier allowed on function declarations only

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(37): error: qualified name is not allowed

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(39): error: Function is not a template

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(39): error: not a class or struct name

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "AutogradContext" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "IValue" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "AutogradContext" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: identifier "K" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: identifier "aff" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: identifier "K" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: identifier "g_point" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: identifier "aff" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: identifier "g_point" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(527): warning: variable "iw" was declared but never referenced

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(615): warning: variable "cam2" was set but never used

    opened by XGenietony 10
  • Unable to get output in the neural view

    Unable to get output in the neural view

    Hello, thanks for releasing your code!

    I tried running the program (the compilation was successful) but I am unable to obtain any output in the neural view.

    image

    This is the output I get from the terminal. I am using Ubuntu 20.04.3 LTS.

    (base) [email protected]:~/Documents/ADOP$ ./build/bin/adop_viewer --scene_dir scenes/tt_lighthouse/
register neural render info
Ref. 
=============== Saiga ===============
| Saiga Version     1.3.2           |
| Eigen Version     3.3.91          |
| Compiler          GNU             |
|   -> Version      9.3.0           |
| Build Type        RelWithDebInfo  |
| Debug             0               |
| Eigen Debug       0               |
| ASAN              0               |
| Asserts           1               |
| Optimizations     0               |
=====================================
Initializing GLFW.
Initializing GLFW sucessfull!
Creating GLFW Window. 1920x1080 Mode=3 Fullscreen=1 Borderless=0
=========================== OpenGL ===========================
| OpenGL Version    3.2.0 NVIDIA 495.29.05                   | 
| GLSL Version      1.50 NVIDIA via Cg compiler              | 
| Renderer          NVIDIA GeForce GTX 1660 SUPER/PCIe/SSE2  | 
| Vendor            NVIDIA Corporation                       | 
==============================================================
[Renderer] Target resized to 1848x1016
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/imgui_gl.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map_linear.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_extract_bright.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_downsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_upsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_combine_simple.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/copy_image.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_directional.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_directional.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/stenciltest.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/post_processing/post_processing.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/blitDepth.glsl
Deferred Renderer initialized. Render resolution: 1848x1016
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/post_processing/imagedisplay.glsl
Program Initialized!
Loading Scene scenes/tt_lighthouse/
====================================
Scene Loaded
  Name       tt_lighthouse
  Path       /home/jeremy/Documents/ADOP/scenes/tt_lighthouse
  Image Size 2048x1080
  Aspect     1.8963
  K          1666.68 1661.18 1024    540     0      
  ocam       2048x1080 affine(1, 0, 0, 0, 0) cam2world() world2cam()
  ocam cut   1
  normalized center 0 0
  dist       -0.126731   0.0217503   0           0           0           0           0.000406465 0.000123456
  Points     12313620
  Colors     1
  Normals    1
  Num Images 309
  Num Cameras 1
====================================
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map_linear.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_extract_bright.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_downsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_upsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_combine_simple.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/copy_image.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
Found experiment experiments//2021-10-30_09:17:38_multi_scene_pose/ with 1 epochs
Found experiment experiments//2021-10-30_09:14:22_boat/ with 1 epochs
loading checkpoint 2021-10-30_09:17:38_multi_scene_pose -> ep0400
Load Checkpoint render
Total Model Params: 574651
> Reset cuda cache
GPU memory - Point Cloud: 295.527MB
Pinhole Intrinsics:
Tensor [1, 13] float cuda:0 Min/Max -0.126731 1666.68 Mean 376.289 Sum 4891.75 req-grad 1
GPU memory - Texture: 197.018MB
Load Checkpoint points 12313620 max uv: 12313619
Load Checkpoint texture. Texels: 12313620 Channels: 4
Load Checkpoint pose
First pose before SE3(Quatwxyz(0.990142,-0.0887136,0.107975,0.00948205),Vec3(0.115149,0.198961,4.53537))
First pose after SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
Load Checkpoint intrinsics
Load Checkpoint environment_map
Load Checkpoint vignette
Load Checkpoint response
Load Checkpoint exposures_values
Using Adam texture optimzier
optimizing texture with lr 0.08/0.004
optimizing environment_map with lr 0.02
optimizing response with lr 0.001
optimizing exposure with lr 0.0005
optimizing vignette with lr 5e-06
optimizing poses with lr 0.005
GPU memory - Texture: 197.018MB
Current Best (img,cam) = (0,0) EV: -0.239136
[Renderer] Target resized to 733x354
[DeferredRenderer] Resize 733x354
loading shader shader//point_render.glsl
[Renderer] Target resized to 545x249
[DeferredRenderer] Resize 545x249
loading checkpoint 2021-10-30_09:17:38_multi_scene_pose -> ep0400
Load Checkpoint render
Total Model Params: 574651
> Reset cuda cache
GPU memory - Point Cloud: 295.527MB
Pinhole Intrinsics:
Tensor [1, 13] float cuda:0 Min/Max -0.126731 1666.68 Mean 376.289 Sum 4891.75 req-grad 1
GPU memory - Texture: 197.018MB
Load Checkpoint points 12313620 max uv: 12313619
Load Checkpoint texture. Texels: 12313620 Channels: 4
Load Checkpoint pose
First pose before SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
First pose after SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
Load Checkpoint intrinsics
Load Checkpoint environment_map
Load Checkpoint vignette
Load Checkpoint response
Load Checkpoint exposures_values
Using Adam texture optimzier
optimizing texture with lr 0.08/0.004
optimizing environment_map with lr 0.02
optimizing response with lr 0.001
optimizing exposure with lr 0.0005
optimizing vignette with lr 5e-06
optimizing poses with lr 0.005
GPU memory - Texture: 197.018MB
Current Best (img,cam) = (0,0) EV: -0.239136

    Any help would be greatly appreciated! Thank you!

    opened by mickey1356 9
  • A question about CmakeList.txt

    A question about CmakeList.txt

    Hello, I am a beginner of neural network. I want to run your program on my computer, but when I Compiling ADOP,in the command "cmake -DCMAKE_PREFIX_PATH="${CONDA}/lib/python3.8/site-packages/torch/;${CONDA}" ..",there was an error,it was "The source directory XXXXdoes not appear to contain CMakeLists.txt."How can i solve this problem?Should i write the CmakeList on myself ?Is there sth wrong in other parts ?this may easy for most people,but i sincerely hope you can answer me,thank you!

    opened by Lu-pin-an 9
  • Getting

    Getting "Loss not finite :(".

    Hello, I'm preparing a dataset of the interior of an appartment. I checked and COLMAP seems to work well, despite that there is a fair amount of white walls. After a random number of epochs, I'm getting the following (pasting only the tail of the output):

    === Epoch 81 ===

Train 81 |   0% |                              |    0/1016 [00:00:0000] [0.00 e/s] 
Train 81 |   0% |                              |    0/1016 [00:00:0000] [0.00 e/s] 
Train 81 |   4% |#                             |   40/1016 [00:05:0000] [8.00 e/s]  Cur=212.233490 Avg=302.391327
Train 81 |   9% |##                            |   88/1016 [00:10:0000] [8.80 e/s]  Cur=333.448730 Avg=287.600220
Train 81 |  13% |####                          |  136/1016 [00:15:0000] [9.07 e/s]  Cur=441.840668 Avg=295.512512
Train 81 |  18% |#####                         |  184/1016 [00:20:0000] [9.20 e/s]  Cur=379.657135 Avg=317.960571
Train 81 |  22% |######                        |  224/1016 [00:25:0000] [8.96 e/s]  Cur=277.145172 Avg=296.512878
Train 81 |  27% |########                      |  272/1016 [00:30:0000] [9.07 e/s]  Cur=310.707367 Avg=302.186951
Train 81 |  31% |#########                     |  320/1016 [00:35:0000] [9.14 e/s]  Cur=34.233173 Avg=287.412201
Train 81 |  36% |##########                    |  368/1016 [00:40:0001] [9.20 e/s]  Cur=33.885960 Avg=282.051544
Train 81 |  41% |############                  |  416/1016 [00:45:0001] [9.24 e/s]  Cur=267.568176 Avg=272.566742
Train 81 |  46% |#############                 |  464/1016 [00:50:0001] [9.28 e/s]  Cur=376.509857 Avg=253.497742
Train 81 |  50% |###############               |  512/1016 [00:55:0001] [9.31 e/s]  Cur=347.692017 Avg=255.870377
Train 81 |  55% |################              |  560/1016 [01:00:0001] [9.33 e/s]  Cur=222.076523 Avg=251.570724
Train 81 |  60% |#################             |  608/1016 [01:05:0001] [9.35 e/s]  Cur=184.181763 Avg=259.685944
Train 81 |  65% |###################           |  656/1016 [01:10:0002] [9.37 e/s]  Cur=30.161480 Avg=255.755127
Train 81 |  69% |####################          |  704/1016 [01:15:0002] [9.39 e/s]  Cur=34.554646 Avg=257.362579
Train 81 |  74% |######################        |  752/1016 [01:20:0002] [9.40 e/s]  Cur=239.453049 Avg=248.403091
Train 81 |  78% |#######################       |  792/1016 [01:25:0002] [9.32 e/s]  Cur=33.262161 Avg=248.473404
Train 81 |  83% |########################      |  840/1016 [01:30:0002] [9.33 e/s]  Cur=348.131836 Avg=255.551346
Train 81 |  87% |##########################    |  888/1016 [01:35:0003] [9.35 e/s]  Cur=547.986633 Avg=261.241058
Train 81 |  92% |###########################   |  936/1016 [01:40:0003] [9.36 e/s]  Cur=268.251221 Avg=261.505127
Tensor [8, 4, 512, 512] float cuda:0 Min/Max 1.3125e-06 20599.7 Mean 0.520595 Sum 4.36707e+06 sdev 7.16297 req-grad 1
Tensor [8, 4, 256, 256] float cuda:0 Min/Max 3.05836e-05 34.5411 Mean 0.535766 Sum 1.12358e+06 sdev 0.771836 req-grad 1
Tensor [8, 4, 128, 128] float cuda:0 Min/Max 1.22896e-05 33.5968 Mean 0.552693 Sum 289770 sdev 0.658228 req-grad 1
Tensor [8, 4, 64, 64] float cuda:0 Min/Max 0.000194362 36.972 Mean 0.557636 Sum 73090.4 sdev 0.61885 req-grad 1

Tensor [8, 2, 512, 512] float cuda:0 Min/Max -1 0.999414 Mean 0.083806 Sum 351508 sdev 0.473772 req-grad 0
Tensor [1] float cuda:0 Min/Max nan nan Mean nan Sum nan sdev nan req-grad 1
Tensor [8, 3, 512, 512] float cuda:0 Min/Max nan nan Mean nan Sum nan sdev nan req-grad 1
Tensor [1, 1, 512, 512] float cuda:0 Min/Max 0 1 Mean 0.878906 Sum 230400 sdev 0.326237 req-grad 0
Tensor [8, 3, 512, 512] float cuda:0 Min/Max 0 0.956863 Mean 0.653581 Sum 4.11197e+06 sdev 0.274293 req-grad 0

Scene:
Scene Log - Texture: Tensor [4, 11655018] float cuda:0 Min/Max -22.6205 15.4725 Mean -0.8064 Sum -3.75944e+07 sdev 1.10892 req-grad 1
Background Desc:  -0.970885 -0.863956 -0.900406 -0.882389 
Environment map: Tensor [1, 4, 1024, 512] float cuda:0 Min/Max -6.2707 3.55126 Mean -0.982767 Sum -2.06101e+06 sdev 0.274843 req-grad 1
Poses: Tensor [254, 8] double cuda:0 Min/Max -6.83462 8.02766 Mean 0.196939 Sum 400.181 sdev 1.62054 req-grad 0
Vignette params:  -0.0149647 -0.00850714 -0.00492687 |   0.00124693 -0.000540384
terminate called after throwing an instance of 'std::runtime_error'
  what():  Loss not finite :(

    Could it be due to the lack of enough points on the white walls? If so, how would you go about it? I'm willing to provide any data if required to better assess this.

    opened by cduguet 8
  • Something wrong with the neural view using my own data

    Something wrong with the neural view using my own data

    After using the ADOP viewer with 2 kinds of my trained data, I got the following results.

    2022-06-29 10-50-44 的屏幕截图

    2022-06-29 11-08-36 的屏幕截图

    All neural views of them are definitely wrong. Is there the same problem with me?

    By the way, when I used the ADOP viewer with the scene of "boat", there is a correct colored display in the neural view.

    opened by HandsomeFYM 5
  • Strange random color when training

    Strange random color when training

    Hi，I have some doubt of my train result on other dataset. It looks like below:( left picture is the train result, right is ground trurh) image

    I can not find out why it happens, is it caused by the rasterization method?

    Below show some of the rasterization result:( left: train render result, medium: rgb ground truth, right: rasterization mask) image

    opened by Sylvia6 4
  • How to generate a video from a given trajectory?

    How to generate a video from a given trajectory?

    Hi, I saw you provided some videos for the trained models on the different scenes. I was wondering how you generated them? Is it possible to generate them with the viewer?

    opened by vahidEttehadi 4
  • How to use the mask function？

    How to use the mask function？

    Hi, I'm wondering to use the mask to filter out the dynamic objects in my scene. But how to use the mask function？I have set the mask_dir in the dataset.ini? Should I also specify the classifaction_dir in the dataset.ini？

    opened by Gatsby23 0
  • can't move in vr_viewer using oculus quest 2

    can't move in vr_viewer using oculus quest 2

    Thank you for the excellent work. VR is being tested.

    The build is successful, but moving the VR headset does not change the rendering screen.

    If there is a solution, please share it with me.

    opened by minsoo5403 1
  • fail to run ./build/bin/adop_viewer --scene_dir scenes/boat

    fail to run ./build/bin/adop_viewer --scene_dir scenes/boat

    optimizing texture with lr 0.01/0.0002 optimizing environment_map with lr 0.002 optimizing response with lr 0.001 optimizing vignette with lr 5e-06 optimizing poses with lr 0.005 signal 11 (Segmentation fault), address is 0x31f from 0x7fbbb8c33eb5 [bt]: (0) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga17printCurrentStackEv+0x38) [0x7fbbb7c3b468] [bt]: (1) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(+0xe5871) [0x7fbbb7c3b871] [bt]: (2) /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fbba6312420] [bt]: (3) /lib64/ld-linux-x86-64.so.2(+0x17eb5) [0x7fbbb8c33eb5] [bt]: (4) /usr/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x88) [0x7fbb96bf0928] [bt]: (5) /usr/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_error+0x33) [0x7fbb96bf09f3] [bt]: (6) /usr/lib/x86_64-linux-gnu/libdl.so.2(+0x1b59) [0x7fbb96947b59] [bt]: (7) /usr/lib/x86_64-linux-gnu/libdl.so.2(dlclose+0x28) [0x7fbb96947468] [bt]: (8) /lib/x86_64-linux-gnu/libcuda.so(+0x217ec1) [0x7fbb31383ec1] [bt]: (9) /lib/x86_64-linux-gnu/libcuda.so(+0x217f33) [0x7fbb31383f33] [bt]: (10) /lib/x86_64-linux-gnu/libcuda.so(+0x2f42a2) [0x7fbb314602a2] [bt]: (11) /opt/conda/envs/adop/lib/libcudart.so.11.0(+0x1b408) [0x7fbbb6553408] [bt]: (12) /opt/conda/envs/adop/lib/libcudart.so.11.0(cudaGraphicsGLRegisterImage+0x1d1) [0x7fbbb65a34c1] [bt]: (13) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN5Saiga4CUDA7Interop9initImageEjN2gl6GLenumE+0x32) [0x7fbbb8b126b2] [bt]: (14) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN16RealTimeRenderer6RenderE9ImageInfo+0x78b) [0x7fbbb8affd2b] [bt]: (15) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN16RealTimeRenderer7ForwardEPN5Saiga6CameraE9ImageInfo+0x32b) [0x7fbbb8b03f5b] [bt]: (16) ./build/bin/adop_viewer(+0x21e07) [0x55e456e21e07] [bt]: (17) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_opengl.so(_ZN5Saiga14OpenGLRenderer12PrepareImguiEb+0x160) [0x7fbbb87e7510] [bt]: (18) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_opengl.so(_ZN5Saiga14OpenGLRenderer6renderERKNS_10RenderInfoE+0x5b) [0x7fbbb87e834b] [bt]: (19) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga10WindowBase6renderEv+0x4c) [0x7fbbb7c1745c] [bt]: (20) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga8MainLoop6renderEff+0x7a) [0x7fbbb7c1561a] [bt]: (21) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga8MainLoop13startMainLoopENS_18MainLoopParametersE+0x25c) [0x7fbbb7c159fc] [bt]: (22) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga10WindowBase13startMainLoopENS_18MainLoopParametersE+0x2d) [0x7fbbb7c1788d] [bt]: (23) ./build/bin/adop_viewer(+0x1adfa) [0x55e456e1adfa] [bt]: (24) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fbb96ab4083] [bt]: (25) ./build/bin/adop_viewer(+0x1b1de) [0x55e456e1b1de] Assertion '0' failed! File: /host/home//G/1workspace/NeRF-space/ADOP/External/saiga/src/saiga/core/util/crash.cpp:109 Function: void Saiga::crit_err_hdlr(int, siginfo_t*, void*) Aborted (core dumped)

    opened by hongdayu 1
  • python setup.py install gives us an error

    python setup.py install gives us an error

    Building wheel torch-1.10.0a0+git302ee7b Traceback (most recent call last): File "setup.py", line 309, in cmake = CMake() File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 102, in init self._cmake_command = CMake._get_cmake_command() File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 126, in _get_cmake_command elif cmake is not None and CMake._get_version(cmake) >= distutils.version.LooseVersion("3.10.0"): File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 137, in _get_version return distutils.version.LooseVersion(line.strip().split(' ')[2]) AttributeError: module 'distutils' has no attribute 'version'

    opened by CHenYYff 0
  • make -j10 Failed

    make -j10 Failed

    image Thanks for your excellent work! But when I compile the adop project using "make -j10", it seems like meet errors. I'm following the steps of "compile instructions". Please help me out! Thanks~

    opened by Yukun66 0
Releases(v1.0)
Owner
Darius Rückert
Darius Rückert
GitHub Repository
Tensorflow2.0 🍎🍊 is delicious, just eat it! 😋😋

How to eat TensorFlow2 in 30 days ? 🔥 🔥 Click here for Chinese Version（中文版） 《10天吃掉那只pyspark》 🚀 github项目地址: https://github.com/lyhue1991/eat_pyspark

lyhue1991 9.7k Jan 01, 2023
VisionKG: Vision Knowledge Graph

VisionKG: Vision Knowledge Graph Official Repository of VisionKG by Anh Le-Tuan, Trung-Kien Tran, Manh Nguyen-Duc, Jicheng Yuan, Manfred Hauswirth and

Continuous Query Evaluation over Linked Stream (CQELS) 9 Jun 23, 2022
[CoRL 2021] A robotics benchmark for cross-embodiment imitation.

x-magical x-magical is a benchmark extension of MAGICAL specifically geared towards cross-embodiment imitation. The tasks still provide the Demo/Test

Kevin Zakka 36 Nov 26, 2022
Implementations of orthogonal and semi-orthogonal convolutions in the Fourier domain with applications to adversarial robustness

Orthogonalizing Convolutional Layers with the Cayley Transform This repository contains implementations and source code to reproduce experiments for t

CMU Locus Lab 36 Dec 30, 2022
Compact Bilinear Pooling for PyTorch

Compact Bilinear Pooling for PyTorch. This repository has a pure Python implementation of Compact Bilinear Pooling and Count Sketch for PyTorch. This

Grégoire Payen de La Garanderie 234 Dec 07, 2022
[CVPR 2022 Oral] EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

EPro-PnP EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation In CVPR 2022 (Oral). [paper] Hanshen

同济大学智能汽车研究所综合感知研究组 ( Comprehensive Perception Research Group under Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University) 842 Jan 04, 2023
InvTorch: memory-efficient models with invertible functions

InvTorch: Memory-Efficient Invertible Functions This module extends the functionality of torch.utils.checkpoint.checkpoint to work with invertible fun

Modar M. Alfadly 12 May 12, 2022
Time Series Forecasting with Temporal Fusion Transformer in Pytorch

Forecasting with the Temporal Fusion Transformer Multi-horizon forecasting often contains a complex mix of inputs – including static (i.e. time-invari

Nicolás Fornasari 6 Jan 24, 2022
Pure python implementation reverse-mode automatic differentiation

MiniGrad A minimal implementation of reverse-mode automatic differentiation (a.k.a. autograd / backpropagation) in pure Python. Inspired by Andrej Kar

Kenny Song 76 Sep 12, 2022
Code for paper Adaptively Aligned Image Captioning via Adaptive Attention Time

Adaptively Aligned Image Captioning via Adaptive Attention Time This repository includes the implementation for Adaptively Aligned Image Captioning vi

Lun Huang 45 Aug 27, 2022
Template repository to build PyTorch projects from source on any version of PyTorch/CUDA/cuDNN.

The Ultimate PyTorch Source-Build Template Translations: 한국어 TL;DR PyTorch built from source can be x4 faster than a naïve PyTorch install. This repos

Joonhyung Lee/이준형 651 Dec 12, 2022
Constructing interpretable quadratic accuracy predictors to serve as an objective function for an IQCQP problem that represents NAS under latency constraints and solve it with efficient algorithms.

IQNAS: Interpretable Integer Quadratic programming Neural Architecture Search Realistic use of neural networks often requires adhering to multiple con

0 Oct 24, 2021
DPC: Unsupervised Deep Point Correspondence via Cross and Self Construction (3DV 2021)

DPC: Unsupervised Deep Point Correspondence via Cross and Self Construction (3DV 2021) This repo is the implementation of DPC. Tested environment Pyth

Dvir Ginzburg 30 Nov 30, 2022
Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning Source Code

Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning Trevor Ablett*, Bryan Chan*,

STARS Laboratory 8 Sep 14, 2022
Code for the paper "A Study of Face Obfuscation in ImageNet"

A Study of Face Obfuscation in ImageNet Code for the paper: A Study of Face Obfuscation in ImageNet Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng,

35 Oct 04, 2022
A full-fledged version of Pix2Seq

Stable-Pix2Seq A full-fledged version of Pix2Seq What it is. This is a full-fledged version of Pix2Seq. Compared with unofficial-pix2seq, stable-pix2s

peng gao 205 Dec 27, 2022
[arXiv'22] Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Panoptic NeRF Project Page | Paper | Dataset Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation Xiao Fu*, Shangzhan zhang*,

Xiao Fu 111 Dec 16, 2022
PyTorch code for Vision Transformers training with the Self-Supervised learning method DINO

Self-Supervised Vision Transformers with DINO PyTorch implementation and pretrained models for DINO. For details, see Emerging Properties in Self-Supe

Facebook Research 4.2k Jan 03, 2023
Bayesian regularization for functional graphical models.

BayesFGM Paper: Jiajing Niu, Andrew Brown. Bayesian regularization for functional graphical models. Requirements R version 3.6.3 and up Python 3.6 and

0 Oct 07, 2021
Implementation detail for paper "Multi-level colonoscopy malignant tissue detection with adversarial CAC-UNet"

Multi-level-colonoscopy-malignant-tissue-detection-with-adversarial-CAC-UNet Implementation detail for our paper "Multi-level colonoscopy malignant ti

CVSM Group - email: <a href=[email protected]"> 84 Nov 22, 2022