ADOP: Approximate Differentiable One-Pixel Point Rendering

Related tags

Deep LearningADOP
Overview

ADOP: Approximate Differentiable One-Pixel Point Rendering

Abstract: We present a novel point-based, differentiable neural rendering pipeline for scene refinement and novel view synthesis. The input are an initial estimate of the point cloud and the camera parameters. The output are synthesized images from arbitrary camera poses. The point cloud rendering is performed by a differentiable renderer using multi-resolution one-pixel point rasterization. Spatial gradients of the discrete rasterization are approximated by the novel concept of ghost geometry. After rendering, the neural image pyramid is passed through a deep neural network for shading calculations and hole-filling. A differentiable, physically-based tonemapper then converts the intermediate output to the target image. Since all stages of the pipeline are differentiable, we optimize all of the scene's parameters i.e. camera model, camera pose, point position, point color, environment map, rendering network weights, vignetting, camera response function, per image exposure, and per image white balance. We show that our system is able to synthesize sharper and more consistent novel views than existing approaches because the initial reconstruction is refined during training. The efficient one-pixel point rasterization allows us to use arbitrary camera models and display scenes with well over 100M points in real time.

  • The source code will be published after the paper has been accepted to a conference.

[Full Paper]

Video

Watch the video

Comments
  • Guide on exporting scene from COLMAP

    Guide on exporting scene from COLMAP

    Hi! I have some troubles exporting point cloud from COLMAP to compatible with ADOP format. Here is what I did:

    1. Autoreconstruction in COLMAP (both sparse and dense)
    2. File > Export As > as .ply
    3. preprocessed it with colmap2adop tool
    4. started training
    5. got an error that my point cloud file does not contain normal data
    6. I checked that your .ply files have normal data in headers, but I struggle to make COLMAP include it in mine
    opened by DenShlk 22
  • RAM usage when training dataset

    RAM usage when training dataset

    Through a lot of trial and error I managed to get both the ADOP viewer and trainer set up on Ubuntu 20.04.

    SETUP I'm trying to run my own dataset through the trainer to see how it is. I only have an Nvidia GTX 1070 card with 8GB of VRAM, 16GB of physical RAM but plenty of hard disk space. I've gotten to the point that the trainer runs without crashing. My dataset is 175 images, downscaled to 1000x562, converted from a ~7M point-cloud project in COLMAP.

    PROBLEM However, running the trainer seems to take up over 1GB per epoch on top of the base amount. This quickly sends my RAM usage to near full. Luckily I have a large swap partition but that too fills up over hundreds of epochs. Is this expected behaviour or a memory leak? Or could it be because my install or conda environment might be faulty? Or is it my incapable hardware? In any case, once all my swap space gets filled the program is unfortunately killed on its own.

    If tweaking any of the training config or dataset parameters could potentially fix or mitigate this issue, please do let me know. Thank you for your work on this, I find this to be a very interesting project.

    opened by ShinyLuxray 19
  • How to optimize for rendering

    How to optimize for rendering

    Hi I have a question regarding the adop_viewer. l tried the adop_viewer with pretrained models on tanks and temples dataset, it works perfectly on a laptop with a 3080 GPU (8GB VRAM) on all scenes.

    I guess you are using the 40GB A100 ini files for training these models, so I also tried training my data on a GPU with similar VRAM using exact settings from train_tank_and_temples_multi.ini. However, when I attempt to visualize my checkpoints on the 3080 GPU it shows CUDA out of memory for all my scenes.

    My question is: for rendering, did you make some optimizations to make the trained models fit in a GPU with less VRAM? Or is there certain configurations/arguments that allows the adop_viewer to take a model trained with high-end GPUs and render on low-end GPUs? Thanks in advance!

    opened by qiaosongwang 12
  • Issue with multi-camera COLMAP dataset

    Issue with multi-camera COLMAP dataset

    I have successfully trained with ADOP on scenes that use a single camera. I am having difficulty, however, with a larger dataset comprised of 882 cameras, 2/3 of which are from drone footage, the other 1/3 being a hand-held DSLR.

    COLMAP successfully generates the camera registrations and point cloud, and running colmap2adop correctly creates all the necessary files for training. I am noticing, however, that COLMAP resizes every single image, effectively generating 882 individual camera profiles. I thought at first this wouldn't necessarily be an issue, as adop_train can parse multiple cameras. I am running into this error, though:

    Assertion 'cam.w == image_size_input(0)' failed!
  File: /home/visionarymind/Downloads/ADOP/src/lib/data/Dataset.cpp:49
  Function: SceneDataTrainSampler::SceneDataTrainSampler(std::shared_ptr<SceneData>, std::vector<int>, bool, Saiga::ivec2, int, bool)
Aborted (core dumped)

    I have looked at each camera.ini file, and the dimensions match each image. Would you have any idea why this error would be thrown and any way to work around it? Technically, there were only two cameras, but COLMAP resizes the images during registration.

    opened by VisionaryMind 10
  • Something worry in PointRenderer.cu & PointRenderer.h

    Something worry in PointRenderer.cu & PointRenderer.h

    2021-11-10 15-59-34屏幕截图 Can not Build CUDA object src/lib/CMakeFiles/NeuralPoints.dir/rendering/PointRenderer.cu.o

    /home/xgenie/projectfold/ADOP/src/lib/./config.h(25): error: inline specifier allowed on function declarations only

    /home/xgenie/projectfold/ADOP/src/lib/./config.h(25): error: inline specifier allowed on function declarations only

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(37): error: qualified name is not allowed

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(39): error: Function is not a template

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(39): error: not a class or struct name

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "AutogradContext" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "IValue" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "AutogradContext" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: identifier "K" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: identifier "aff" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: identifier "K" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: identifier "g_point" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: identifier "aff" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: identifier "g_point" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(527): warning: variable "iw" was declared but never referenced

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(615): warning: variable "cam2" was set but never used

    opened by XGenietony 10
  • Unable to get output in the neural view

    Unable to get output in the neural view

    Hello, thanks for releasing your code!

    I tried running the program (the compilation was successful) but I am unable to obtain any output in the neural view.

    image

    This is the output I get from the terminal. I am using Ubuntu 20.04.3 LTS.

    (base) [email protected]:~/Documents/ADOP$ ./build/bin/adop_viewer --scene_dir scenes/tt_lighthouse/
register neural render info
Ref. 
=============== Saiga ===============
| Saiga Version     1.3.2           |
| Eigen Version     3.3.91          |
| Compiler          GNU             |
|   -> Version      9.3.0           |
| Build Type        RelWithDebInfo  |
| Debug             0               |
| Eigen Debug       0               |
| ASAN              0               |
| Asserts           1               |
| Optimizations     0               |
=====================================
Initializing GLFW.
Initializing GLFW sucessfull!
Creating GLFW Window. 1920x1080 Mode=3 Fullscreen=1 Borderless=0
=========================== OpenGL ===========================
| OpenGL Version    3.2.0 NVIDIA 495.29.05                   | 
| GLSL Version      1.50 NVIDIA via Cg compiler              | 
| Renderer          NVIDIA GeForce GTX 1660 SUPER/PCIe/SSE2  | 
| Vendor            NVIDIA Corporation                       | 
==============================================================
[Renderer] Target resized to 1848x1016
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/imgui_gl.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map_linear.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_extract_bright.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_downsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_upsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_combine_simple.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/copy_image.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_directional.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_directional.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/stenciltest.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/post_processing/post_processing.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/blitDepth.glsl
Deferred Renderer initialized. Render resolution: 1848x1016
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/post_processing/imagedisplay.glsl
Program Initialized!
Loading Scene scenes/tt_lighthouse/
====================================
Scene Loaded
  Name       tt_lighthouse
  Path       /home/jeremy/Documents/ADOP/scenes/tt_lighthouse
  Image Size 2048x1080
  Aspect     1.8963
  K          1666.68 1661.18 1024    540     0      
  ocam       2048x1080 affine(1, 0, 0, 0, 0) cam2world() world2cam()
  ocam cut   1
  normalized center 0 0
  dist       -0.126731   0.0217503   0           0           0           0           0.000406465 0.000123456
  Points     12313620
  Colors     1
  Normals    1
  Num Images 309
  Num Cameras 1
====================================
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map_linear.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_extract_bright.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_downsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_upsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_combine_simple.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/copy_image.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
Found experiment experiments//2021-10-30_09:17:38_multi_scene_pose/ with 1 epochs
Found experiment experiments//2021-10-30_09:14:22_boat/ with 1 epochs
loading checkpoint 2021-10-30_09:17:38_multi_scene_pose -> ep0400
Load Checkpoint render
Total Model Params: 574651
> Reset cuda cache
GPU memory - Point Cloud: 295.527MB
Pinhole Intrinsics:
Tensor [1, 13] float cuda:0 Min/Max -0.126731 1666.68 Mean 376.289 Sum 4891.75 req-grad 1
GPU memory - Texture: 197.018MB
Load Checkpoint points 12313620 max uv: 12313619
Load Checkpoint texture. Texels: 12313620 Channels: 4
Load Checkpoint pose
First pose before SE3(Quatwxyz(0.990142,-0.0887136,0.107975,0.00948205),Vec3(0.115149,0.198961,4.53537))
First pose after SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
Load Checkpoint intrinsics
Load Checkpoint environment_map
Load Checkpoint vignette
Load Checkpoint response
Load Checkpoint exposures_values
Using Adam texture optimzier
optimizing texture with lr 0.08/0.004
optimizing environment_map with lr 0.02
optimizing response with lr 0.001
optimizing exposure with lr 0.0005
optimizing vignette with lr 5e-06
optimizing poses with lr 0.005
GPU memory - Texture: 197.018MB
Current Best (img,cam) = (0,0) EV: -0.239136
[Renderer] Target resized to 733x354
[DeferredRenderer] Resize 733x354
loading shader shader//point_render.glsl
[Renderer] Target resized to 545x249
[DeferredRenderer] Resize 545x249
loading checkpoint 2021-10-30_09:17:38_multi_scene_pose -> ep0400
Load Checkpoint render
Total Model Params: 574651
> Reset cuda cache
GPU memory - Point Cloud: 295.527MB
Pinhole Intrinsics:
Tensor [1, 13] float cuda:0 Min/Max -0.126731 1666.68 Mean 376.289 Sum 4891.75 req-grad 1
GPU memory - Texture: 197.018MB
Load Checkpoint points 12313620 max uv: 12313619
Load Checkpoint texture. Texels: 12313620 Channels: 4
Load Checkpoint pose
First pose before SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
First pose after SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
Load Checkpoint intrinsics
Load Checkpoint environment_map
Load Checkpoint vignette
Load Checkpoint response
Load Checkpoint exposures_values
Using Adam texture optimzier
optimizing texture with lr 0.08/0.004
optimizing environment_map with lr 0.02
optimizing response with lr 0.001
optimizing exposure with lr 0.0005
optimizing vignette with lr 5e-06
optimizing poses with lr 0.005
GPU memory - Texture: 197.018MB
Current Best (img,cam) = (0,0) EV: -0.239136

    Any help would be greatly appreciated! Thank you!

    opened by mickey1356 9
  • A question about CmakeList.txt

    A question about CmakeList.txt

    Hello, I am a beginner of neural network. I want to run your program on my computer, but when I Compiling ADOP,in the command "cmake -DCMAKE_PREFIX_PATH="${CONDA}/lib/python3.8/site-packages/torch/;${CONDA}" ..",there was an error,it was "The source directory XXXXdoes not appear to contain CMakeLists.txt."How can i solve this problem?Should i write the CmakeList on myself ?Is there sth wrong in other parts ?this may easy for most people,but i sincerely hope you can answer me,thank you!

    opened by Lu-pin-an 9
  • Getting

    Getting "Loss not finite :(".

    Hello, I'm preparing a dataset of the interior of an appartment. I checked and COLMAP seems to work well, despite that there is a fair amount of white walls. After a random number of epochs, I'm getting the following (pasting only the tail of the output):

    === Epoch 81 ===

Train 81 |   0% |                              |    0/1016 [00:00:0000] [0.00 e/s] 
Train 81 |   0% |                              |    0/1016 [00:00:0000] [0.00 e/s] 
Train 81 |   4% |#                             |   40/1016 [00:05:0000] [8.00 e/s]  Cur=212.233490 Avg=302.391327
Train 81 |   9% |##                            |   88/1016 [00:10:0000] [8.80 e/s]  Cur=333.448730 Avg=287.600220
Train 81 |  13% |####                          |  136/1016 [00:15:0000] [9.07 e/s]  Cur=441.840668 Avg=295.512512
Train 81 |  18% |#####                         |  184/1016 [00:20:0000] [9.20 e/s]  Cur=379.657135 Avg=317.960571
Train 81 |  22% |######                        |  224/1016 [00:25:0000] [8.96 e/s]  Cur=277.145172 Avg=296.512878
Train 81 |  27% |########                      |  272/1016 [00:30:0000] [9.07 e/s]  Cur=310.707367 Avg=302.186951
Train 81 |  31% |#########                     |  320/1016 [00:35:0000] [9.14 e/s]  Cur=34.233173 Avg=287.412201
Train 81 |  36% |##########                    |  368/1016 [00:40:0001] [9.20 e/s]  Cur=33.885960 Avg=282.051544
Train 81 |  41% |############                  |  416/1016 [00:45:0001] [9.24 e/s]  Cur=267.568176 Avg=272.566742
Train 81 |  46% |#############                 |  464/1016 [00:50:0001] [9.28 e/s]  Cur=376.509857 Avg=253.497742
Train 81 |  50% |###############               |  512/1016 [00:55:0001] [9.31 e/s]  Cur=347.692017 Avg=255.870377
Train 81 |  55% |################              |  560/1016 [01:00:0001] [9.33 e/s]  Cur=222.076523 Avg=251.570724
Train 81 |  60% |#################             |  608/1016 [01:05:0001] [9.35 e/s]  Cur=184.181763 Avg=259.685944
Train 81 |  65% |###################           |  656/1016 [01:10:0002] [9.37 e/s]  Cur=30.161480 Avg=255.755127
Train 81 |  69% |####################          |  704/1016 [01:15:0002] [9.39 e/s]  Cur=34.554646 Avg=257.362579
Train 81 |  74% |######################        |  752/1016 [01:20:0002] [9.40 e/s]  Cur=239.453049 Avg=248.403091
Train 81 |  78% |#######################       |  792/1016 [01:25:0002] [9.32 e/s]  Cur=33.262161 Avg=248.473404
Train 81 |  83% |########################      |  840/1016 [01:30:0002] [9.33 e/s]  Cur=348.131836 Avg=255.551346
Train 81 |  87% |##########################    |  888/1016 [01:35:0003] [9.35 e/s]  Cur=547.986633 Avg=261.241058
Train 81 |  92% |###########################   |  936/1016 [01:40:0003] [9.36 e/s]  Cur=268.251221 Avg=261.505127
Tensor [8, 4, 512, 512] float cuda:0 Min/Max 1.3125e-06 20599.7 Mean 0.520595 Sum 4.36707e+06 sdev 7.16297 req-grad 1
Tensor [8, 4, 256, 256] float cuda:0 Min/Max 3.05836e-05 34.5411 Mean 0.535766 Sum 1.12358e+06 sdev 0.771836 req-grad 1
Tensor [8, 4, 128, 128] float cuda:0 Min/Max 1.22896e-05 33.5968 Mean 0.552693 Sum 289770 sdev 0.658228 req-grad 1
Tensor [8, 4, 64, 64] float cuda:0 Min/Max 0.000194362 36.972 Mean 0.557636 Sum 73090.4 sdev 0.61885 req-grad 1

Tensor [8, 2, 512, 512] float cuda:0 Min/Max -1 0.999414 Mean 0.083806 Sum 351508 sdev 0.473772 req-grad 0
Tensor [1] float cuda:0 Min/Max nan nan Mean nan Sum nan sdev nan req-grad 1
Tensor [8, 3, 512, 512] float cuda:0 Min/Max nan nan Mean nan Sum nan sdev nan req-grad 1
Tensor [1, 1, 512, 512] float cuda:0 Min/Max 0 1 Mean 0.878906 Sum 230400 sdev 0.326237 req-grad 0
Tensor [8, 3, 512, 512] float cuda:0 Min/Max 0 0.956863 Mean 0.653581 Sum 4.11197e+06 sdev 0.274293 req-grad 0

Scene:
Scene Log - Texture: Tensor [4, 11655018] float cuda:0 Min/Max -22.6205 15.4725 Mean -0.8064 Sum -3.75944e+07 sdev 1.10892 req-grad 1
Background Desc:  -0.970885 -0.863956 -0.900406 -0.882389 
Environment map: Tensor [1, 4, 1024, 512] float cuda:0 Min/Max -6.2707 3.55126 Mean -0.982767 Sum -2.06101e+06 sdev 0.274843 req-grad 1
Poses: Tensor [254, 8] double cuda:0 Min/Max -6.83462 8.02766 Mean 0.196939 Sum 400.181 sdev 1.62054 req-grad 0
Vignette params:  -0.0149647 -0.00850714 -0.00492687 |   0.00124693 -0.000540384
terminate called after throwing an instance of 'std::runtime_error'
  what():  Loss not finite :(

    Could it be due to the lack of enough points on the white walls? If so, how would you go about it? I'm willing to provide any data if required to better assess this.

    opened by cduguet 8
  • Something wrong with the neural view using my own data

    Something wrong with the neural view using my own data

    After using the ADOP viewer with 2 kinds of my trained data, I got the following results.

    2022-06-29 10-50-44 的屏幕截图

    2022-06-29 11-08-36 的屏幕截图

    All neural views of them are definitely wrong. Is there the same problem with me?

    By the way, when I used the ADOP viewer with the scene of "boat", there is a correct colored display in the neural view.

    opened by HandsomeFYM 5
  • Strange random color when training

    Strange random color when training

    Hi，I have some doubt of my train result on other dataset. It looks like below:( left picture is the train result, right is ground trurh) image

    I can not find out why it happens, is it caused by the rasterization method?

    Below show some of the rasterization result:( left: train render result, medium: rgb ground truth, right: rasterization mask) image

    opened by Sylvia6 4
  • How to generate a video from a given trajectory?

    How to generate a video from a given trajectory?

    Hi, I saw you provided some videos for the trained models on the different scenes. I was wondering how you generated them? Is it possible to generate them with the viewer?

    opened by vahidEttehadi 4
  • How to use the mask function？

    How to use the mask function？

    Hi, I'm wondering to use the mask to filter out the dynamic objects in my scene. But how to use the mask function？I have set the mask_dir in the dataset.ini? Should I also specify the classifaction_dir in the dataset.ini？

    opened by Gatsby23 0
  • can't move in vr_viewer using oculus quest 2

    can't move in vr_viewer using oculus quest 2

    Thank you for the excellent work. VR is being tested.

    The build is successful, but moving the VR headset does not change the rendering screen.

    If there is a solution, please share it with me.

    opened by minsoo5403 1
  • fail to run ./build/bin/adop_viewer --scene_dir scenes/boat

    fail to run ./build/bin/adop_viewer --scene_dir scenes/boat

    optimizing texture with lr 0.01/0.0002 optimizing environment_map with lr 0.002 optimizing response with lr 0.001 optimizing vignette with lr 5e-06 optimizing poses with lr 0.005 signal 11 (Segmentation fault), address is 0x31f from 0x7fbbb8c33eb5 [bt]: (0) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga17printCurrentStackEv+0x38) [0x7fbbb7c3b468] [bt]: (1) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(+0xe5871) [0x7fbbb7c3b871] [bt]: (2) /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fbba6312420] [bt]: (3) /lib64/ld-linux-x86-64.so.2(+0x17eb5) [0x7fbbb8c33eb5] [bt]: (4) /usr/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x88) [0x7fbb96bf0928] [bt]: (5) /usr/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_error+0x33) [0x7fbb96bf09f3] [bt]: (6) /usr/lib/x86_64-linux-gnu/libdl.so.2(+0x1b59) [0x7fbb96947b59] [bt]: (7) /usr/lib/x86_64-linux-gnu/libdl.so.2(dlclose+0x28) [0x7fbb96947468] [bt]: (8) /lib/x86_64-linux-gnu/libcuda.so(+0x217ec1) [0x7fbb31383ec1] [bt]: (9) /lib/x86_64-linux-gnu/libcuda.so(+0x217f33) [0x7fbb31383f33] [bt]: (10) /lib/x86_64-linux-gnu/libcuda.so(+0x2f42a2) [0x7fbb314602a2] [bt]: (11) /opt/conda/envs/adop/lib/libcudart.so.11.0(+0x1b408) [0x7fbbb6553408] [bt]: (12) /opt/conda/envs/adop/lib/libcudart.so.11.0(cudaGraphicsGLRegisterImage+0x1d1) [0x7fbbb65a34c1] [bt]: (13) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN5Saiga4CUDA7Interop9initImageEjN2gl6GLenumE+0x32) [0x7fbbb8b126b2] [bt]: (14) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN16RealTimeRenderer6RenderE9ImageInfo+0x78b) [0x7fbbb8affd2b] [bt]: (15) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN16RealTimeRenderer7ForwardEPN5Saiga6CameraE9ImageInfo+0x32b) [0x7fbbb8b03f5b] [bt]: (16) ./build/bin/adop_viewer(+0x21e07) [0x55e456e21e07] [bt]: (17) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_opengl.so(_ZN5Saiga14OpenGLRenderer12PrepareImguiEb+0x160) [0x7fbbb87e7510] [bt]: (18) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_opengl.so(_ZN5Saiga14OpenGLRenderer6renderERKNS_10RenderInfoE+0x5b) [0x7fbbb87e834b] [bt]: (19) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga10WindowBase6renderEv+0x4c) [0x7fbbb7c1745c] [bt]: (20) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga8MainLoop6renderEff+0x7a) [0x7fbbb7c1561a] [bt]: (21) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga8MainLoop13startMainLoopENS_18MainLoopParametersE+0x25c) [0x7fbbb7c159fc] [bt]: (22) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga10WindowBase13startMainLoopENS_18MainLoopParametersE+0x2d) [0x7fbbb7c1788d] [bt]: (23) ./build/bin/adop_viewer(+0x1adfa) [0x55e456e1adfa] [bt]: (24) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fbb96ab4083] [bt]: (25) ./build/bin/adop_viewer(+0x1b1de) [0x55e456e1b1de] Assertion '0' failed! File: /host/home//G/1workspace/NeRF-space/ADOP/External/saiga/src/saiga/core/util/crash.cpp:109 Function: void Saiga::crit_err_hdlr(int, siginfo_t*, void*) Aborted (core dumped)

    opened by hongdayu 1
  • python setup.py install gives us an error

    python setup.py install gives us an error

    Building wheel torch-1.10.0a0+git302ee7b Traceback (most recent call last): File "setup.py", line 309, in cmake = CMake() File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 102, in init self._cmake_command = CMake._get_cmake_command() File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 126, in _get_cmake_command elif cmake is not None and CMake._get_version(cmake) >= distutils.version.LooseVersion("3.10.0"): File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 137, in _get_version return distutils.version.LooseVersion(line.strip().split(' ')[2]) AttributeError: module 'distutils' has no attribute 'version'

    opened by CHenYYff 0
  • make -j10 Failed

    make -j10 Failed

    image Thanks for your excellent work! But when I compile the adop project using "make -j10", it seems like meet errors. I'm following the steps of "compile instructions". Please help me out! Thanks~

    opened by Yukun66 0
Releases(v1.0)
Owner
Darius Rückert
Darius Rückert
GitHub Repository
Official release of MSHT: Multi-stage Hybrid Transformer for the ROSE Image Analysis of Pancreatic Cancer axriv: http://arxiv.org/abs/2112.13513

MSHT: Multi-stage Hybrid Transformer for the ROSE Image Analysis This is the official page of the MSHT with its experimental script and records. We de

Tianyi Zhang 53 Dec 27, 2022
Leveraging Two Types of Global Graph for Sequential Fashion Recommendation, ICMR 2021

This is the repo for the paper: Leveraging Two Types of Global Graph for Sequential Fashion Recommendation Requirements OS: Ubuntu 16.04 or higher ver

Yujuan Ding 10 Oct 10, 2022
Easy to use Python camera interface for NVIDIA Jetson

JetCam JetCam is an easy to use Python camera interface for NVIDIA Jetson. Works with various USB and CSI cameras using Jetson's Accelerated GStreamer

NVIDIA AI IOT 358 Jan 02, 2023
Efficient Training of Visual Transformers with Small Datasets

Official codes for "Efficient Training of Visual Transformers with Small Datasets", NerIPS 2021.

Yahui Liu 112 Dec 25, 2022
Readings for "A Unified View of Relational Deep Learning for Polypharmacy Side Effect, Combination Therapy, and Drug-Drug Interaction Prediction."

Polypharmacy - DDI - Synergy Survey The Survey Paper This repository accompanies our survey paper A Unified View of Relational Deep Learning for Polyp

AstraZeneca 79 Jan 05, 2023
A colab notebook for training Stylegan2-ada on colab, transfer learning onto your own dataset.

Stylegan2-Ada-Google-Colab-Starter-Notebook A no thrills colab notebook for training Stylegan2-ada on colab. transfer learning onto your own dataset h

Harnick Khera 66 Dec 16, 2022
Re-implement CycleGAN in Tensorlayer

CycleGAN_Tensorlayer Re-implement CycleGAN in TensorLayer Original CycleGAN Improved CycleGAN with resize-convolution Prerequisites: TensorLayer Tenso

89 Aug 15, 2022
Birthday-problem - The birthday problem asks for the probability that, in a set of n randomly chosen people, at least two will share a birthday

Birthday-problem In probability theory, the birthday problem asks for the probab

5 Jan 05, 2023
Predicts an answer in yes or no.

Oui-ou-non-prediction Predicts an answer in 'yes' or 'no'. It is based on the game 'effeuiller la marguerite' in which the person plucks flower petals

Ananya Gupta 1 Jan 15, 2022
Gin provides a lightweight configuration framework for Python

Gin Config Authors: Dan Holtmann-Rice, Sergio Guadarrama, Nathan Silberman Contributors: Oscar Ramirez, Marek Fiser Gin provides a lightweight configu

Google 1.7k Jan 03, 2023
DP-CL(Continual Learning with Differential Privacy)

DP-CL(Continual Learning with Differential Privacy) This is the official implementation of the Continual Learning with Differential Privacy. If you us

Phung Lai 3 Nov 04, 2022
[NeurIPS 2021] PyTorch Code for Accelerating Robotic Reinforcement Learning with Parameterized Action Primitives

Robot Action Primitives (RAPS) This repository is the official implementation of Accelerating Robotic Reinforcement Learning via Parameterized Action

Murtaza Dalal 55 Dec 27, 2022
A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution.

Awesome Pretrained StyleGAN2 A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution. Note the readme is a

Justin 1.1k Dec 24, 2022
[NeurIPS-2021] Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

MosaicKD Code for NeurIPS-21 paper "Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data" 1. Motivation Natural images share common l

ZJU-VIPA 37 Nov 10, 2022
Tensorflow implementation of "BEGAN: Boundary Equilibrium Generative Adversarial Networks"

BEGAN in Tensorflow Tensorflow implementation of BEGAN: Boundary Equilibrium Generative Adversarial Networks. Requirements Python 2.7 or 3.x Pillow tq

Taehoon Kim 922 Dec 21, 2022
Running Google MoveNet Multipose Tracking models on OpenVINO.

MoveNet MultiPose Tracking on OpenVINO

60 Nov 17, 2022
A TensorFlow implementation of SOFA, the Simulator for OFfline LeArning and evaluation.

SOFA This repository is the implementation of SOFA, the Simulator for OFfline leArning and evaluation. Keeping Dataset Biases out of the Simulation: A

22 Nov 23, 2022
People log into different sites every day to get information and browse through these sites one by one

HyperLink People log into different sites every day to get information and browse through these sites one by one. And they are exposed to advertisemen

0 Feb 17, 2022
Lane assist for ETS2, built with the ultra-fast-lane-detection model.

Euro-Truck-Simulator-2-Lane-Assist Lane assist for ETS2, built with the ultra-fast-lane-detection model. This project was made possible by the amazing

36 Jan 05, 2023
NaijaSenti is an open-source sentiment and emotion corpora for four major Nigerian languages

NaijaSenti is an open-source sentiment and emotion corpora for four major Nigerian languages. This project was supported by lacuna-fund initiatives. Jump straight to one of the sections below, or jus

Hausa Natural Language Processing 14 Dec 20, 2022