Differences

This shows you the differences between two versions of the page.

--- tamiwiki:projects:scanning-tami [2025/02/05 13:28] – wissotsky
+++ tamiwiki:projects:scanning-tami [2025/02/24 13:24] (current) – wissotsky
@@ Line 1: / Line 1: @@
 <WRAP center round todo 60%>
-Very WIP page(as of 5th February 2025), you can help by asking questions in the telegram group or in-person on Mondays
+Very WIP page(as of 24th February 2025), you can help by asking questions in the telegram group or in-person on Mondays
 </WRAP>
 d scan of tami
+{{tamiwiki:projects:he-is-speaking-guy-explaining-with-a-whiteboard.gif}}
 the scan is based on a phone video recording of me walking around tami.
@@ Line 21: / Line 23: @@
 After that I switched to superpoint for the feature detection and lightglue for feature matching which seems to be a fairly popular combo currently.
+{{tamiwiki:projects:animation_tamiscan_superpoint_lightglue.gif}}
 First I matched every frame with its 30 nearest frames(by time).
@@ Line 36: / Line 40: @@
 I tried projecting points from a monocular depth estimate(apple depth pro) but it was too globally unstable between frames.
-#TODO
+Fortunately OpenMVS's patchmatch implementation was much faster and had more flexible parameters compared to colmap
+I was able to run a quick low resolution depth estimate that I then used as ground truth for realigning a neural monocular depth estimate.
+OpenMVS .dmap files include a depth map and a confidence map, I was able to use this with RANSAC to fit a polynomial which offset the neural depth map
+The confidence map was used to change the contribution of samples and allowed me to fit even very hard cases.
+Now I was able to reproject all the realigned depths into a very dense pointcloud for the scene. Much denser that would even be possible with patchmatch
+Albeit this point cloud had visible layers because of small discrepancies between the depths and views
+Fortunately 3dgs is specifically good at optimizing these kinds of cases into a multiview consistent scene.
+After running 3dgs the results were quite good, and I decided to move onto creating a mesh that could be used in software like blender.
+Trying a bunch of papers, the best results I got were from ones using TSDF integration for their meshing workflow, but none of them had all the features that I wanted so I decided to do it myself.
+After rendering depth maps from 3dgs and integrating them into a tsdf volume I saw that there were alot of "spikes" and outliers in the depth(due to just splat ordering)
+I was able smooth the depth over by rasterizing the median depth(according to RaDe-GS)
+{{tamiwiki:projects:combined_depths_animation.gif}}
+But I still had some outliers in the depth data.
+I tried again fitting neural depth estimates to my 3dgs depth data, but at this point their level of detail was lower than what I was getting from 3dgs.
+I then tried PromptDA which is meant to upscale depths from smarthpones and lidars using a low-rez depth + rgb image pairs -> hi-rez depth.
+But the problem I got there is that the outliers were visibly clearly still in the depth data but brought into the distribution and blended into it.
-coarse patchmatch mvs depth estimation
+After plotting the rasterized median depth from gaussian splats into a frequency histogram I was able to see that in problematic images there are two distinct spikes and a long trail of depths.
-alignment of neural monocular estimate to patchmatch(ransac polynomial fitting)
+I was able to fit a kernel density estimate to the depth data and then I manually found cutoff value where if the density after the global peak becomes lower it means that were past the primary peak any depth beyond that is an outlier.
-dense point cloud integration
+{{tamiwiki:projects:threshold_animation.gif}}
-dgs scene refinement
+After removing the depth outliers I was able to get much cleaner results
-dgs depth rasterization
+To get a mesh from the depth images I used TSDF integration, I used the VoxelBlockGrid implementation from open3d.
-depth kernel density outlier detection
+But the gpu vram wasnt enough for me to extract mesh detail down to 1mm. And running the integration purely on cpu was too slow.
-tsdf integration
+So I ended up computing the tsdf volume in batches on the gpu and them merging them onto a uniform voxel grid on the cpu, where there was overlap between the grids I used trilinear interpolation.
-mesh compresson
+#TODO mesh compresson
 voxelization

TAMI - wiki

User Tools

Site Tools

Differences

Page Tools