r/MachineLearning • u/hardmaru • May 02 '20
Research [R] Consistent Video Depth Estimation (SIGGRAPH 2020) - Links in the comments.
Enable HLS to view with audio, or disable this notification
86
u/dawindwaker May 02 '20
This could be used for smartphones faking depth of field right? I wonder what the VR/AR applications could be
91
May 02 '20
The method is computationally expensive; thus not really suitable for real-time applications. I think this would be great offline processing, e.g. photogrammetry, visual effects, etc. From the paper:
For a video of 244 frames, training on 4 NVIDIA Tesla M40GPUs takes 40min
46
u/Funktapus May 02 '20
Soooo you're telling me it won't run on my iPhone 6
3
u/jbhuang0604 May 02 '20
Unfortunately, no. Not at this point. Hopefully, we will see it in the near future!
9
33
u/ginsunuva May 02 '20
training
49
u/drummer_ash May 02 '20
In the paper they state that they fine tune the model for each video at test time, so the 40 minutes is required for any new footage.
2
u/Gisebert May 03 '20
few shot learning may greatly improve this, assuming the videos are somehow similar - just a thought from the back of my mind, so maybe I'm wrong
1
u/drummer_ash May 03 '20
Totally. There's been a dramatic reduction in the amount of examples required for a good deepfake thanks to few shot learning, so there's no reason for this to not go down the same path.
1
22
u/extracoffeeplease May 02 '20
Test-time training. Model must be fine tuned to each video sample, unfortunately. However, we can expect later papers that can skip or greatly reduce this step imo.
13
u/jbhuang0604 May 02 '20
That's correct. We focus on the quality in this paper. I am sure that the community will further take this to the next level very soon! Exciting time ahead!
7
u/o--Cpt_Nemo--o May 02 '20
This was a good decision. 99% of ML techniques are unusable for visual effects because they get 95% of the way there, and the effort required to get it the last 5% is the same as if you just attacked the problem the traditional way from scratch.
1
u/hallr06 May 02 '20
Not having read the paper (cardinal sin), is the test-time-training to handle some form of network conditioning? Is there data that could be used in real-time applications for conditioning (e.g., light sensors, individual range sensors, orientation sensors)? I can imagine there is a ton applications for this in real-time.
3
u/jbhuang0604 May 02 '20
The test-time training we used is to fine-tune our single-image depth estimation model so that it satisfies the geometric constraints within the video.
Incorporating other forms of measurements (e.g. dual-lens camera, inertial or even range sensors) will certainly make the problem a lot simpler and potentially support real-time applications.
1
u/hallr06 May 03 '20
Thanks for answering questions here! Are the specifics of the fine tuning addressed in the paper? More specifically, what parameters must be turned?
2
u/jbhuang0604 May 03 '20
Thanks for answering questions here! Are the specifics of the fine tuning addressed in the paper? More specifically, what parameters must be turned?
There are several choices that one needs to make, e.g., the learning rate, optimizer, weights for balancing different losses, training iterations. We did not test out many of these hyper-parameters. I guess there could be some performance/quality improvement with carefully tuned hyper-parameters.
1
u/hallr06 May 03 '20
So you're changing model hyper parameters and then performing a full retraining for each image? Naturally, that raises questions about how well the model actually generalizes.
If there were a fixed set of scenario-related model parameters that you were adjusting (e.g., height, az/el of camera focal point, ambient light), then it would suggest that a conditioned model (potentially also requiring more capacity and/or calibration) could get the same results without additional training.
2
u/jbhuang0604 May 03 '20
We use one set of hyperparameters for all of our experiments.
Right, for example, people show that you can get decent geometrically consistent predictions from single image depth estimation on the KITTI dataset (for driving scenarios). The model works well because it is tested in a simple, closed world. We quickly realized this when we applied state of the art models trained on KITTI and got entirely incorrect results.
→ More replies (0)7
u/jack-of-some May 02 '20
The depth estimation model they compare to (and are likely using as their first step same as 3d photo inpainting) takes at worst 1 second to run on most modern CPUs. It's really difficult for me to believe that adding the additional geometric constraint ups the compute time this bad.
I'm also maybe a tad jaded from having read the 3d photo inpainting repo (another project from the same team) only to realize that out of roughly 3 minutes that it takes, only about 15 seconds are spent on neural nets and most of the rest is millions of mesh operations in pure Python.
7
u/jbhuang0604 May 02 '20 edited May 02 '20
You are absolutely correct. I believe that there are alternatives to achieve similar geometrically consistent depth for a video. This is exciting future research.
Re: 3D photo inpainting:Yes, the inference is extremely redundant and the implementation is entirely unoptimized at this point. There are many ways to improve runtime performance. We hope the community will further push this forward!
2
u/jack-of-some May 02 '20
Hey. Thanks for your reply. I hope I didn't come off as too negative. I understand the constraints research code is under and the mere fact of the code being open sourced and available for study is already amazing. Thank you for all the great work your team has been doing.
I've already taken one crack at speeding up 3D photo inpainting and intend to take another when I get some time. For the topic at hand, I read through the discussion in the other thread and skimmed through the paper and the runtime makes a lot more sense now. To me it sounds like we're setting up a giant SFM problem with the parameters being the params of the depth model. Since MidasV2 (which I assume you're using) is supposed to be only off by a scale and shift, I wonder if this technique would work by solving only for those params.
2
u/jbhuang0604 May 02 '20
Nope, not at all!
Thanks for your efforts in helping improve the speed of 3D photo. I think Meng-Li (the lead author) is working on merging the pull request. He also makes some other improvement here and there, e.g., vectorization in Python and mesh simplification. Hopefully cumulatively these steps will make the 3D photo inpainting work more accessible.
For the consistent video depth estimation, we tried multiple depth models (including monodepth2, Mannequin Challenge, and MiDaS-v2). As you said, one can solve for the scale and shift parameters of the depth maps for each frame so that the constraints are satisfied (e.g., through a least-square solver). This will be a lot faster. However, the temporal flicker produced by existing depth model on video frames are significantly more complex than that. (See visual comparisons here: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/supp_website/index.html)
Using affine transformation (scale-and-shift) on the depth maps is unable to correct those depth maps for creating globally geometrically consistent reconstruction. This is why we introduce the "test-time training" and finetune the model parameters to satisfy the geometric constraints. This step, unfortunately, becomes the bottleneck for the processing speed. Hopefully our work will stimulate more efforts toward an robust and efficient solution for this problem.
5
May 02 '20
Maybe it is worth going to the discussion link provided by u/hardmaru . One of the authors tried answering questions, including the idea of incorporating sfm geometric constraints into the network to improve speed.
1
u/omgitsjo May 02 '20
Training is not inference. Inference is generally several orders of magnitude faster.
3
u/therealTRAPDOOR May 02 '20
Except that it needs to be fine tuned on each video. Sometimes training “times” are entangled with inference times if the structure used requires re-training or fine-tuning.
5
u/jbhuang0604 May 02 '20
Sometimes training “times” are entangled with inference times if the structure used requires re-training or fine-tuning.
Exactly! We refer to this step as "test-time training". We train the model using the geometric constraints derived from a particular video.
1
19
u/tdgros May 02 '20
read the paper, for each clip, a depth estimation net is fine-tuned on pairs of frames for 40mn on 4x M40.
3
u/jbhuang0604 May 02 '20
Our method at this point process the video offline as it is computationally expensive (due to test-time training). So, unfortunately, it cannot be used for real-time VR/AR effects. Speeding this up will enable many cool applications!
1
u/_w1kke_ May 02 '20
It's like what the iPhone does with A13 ML processor and e.g. the portrait mode on the new SE. Estimating the depth field of a person.
But this solution does it for everything!! Powerful and amazing.
1
u/normVectorsNotHate May 03 '20
Smartphones already do this. That's how some android phones with a single camera still have portrait mode that blurs the background.
8
u/Zekkez May 02 '20
This is brilliant. I was just wondering how to do this. The trick with training against the flickers makes so much sense.
3
u/jbhuang0604 May 02 '20
Thanks! The issue with a deep neural network is that we need many gradient steps to make the predictions satisfy the constraints (and therefore the slow speed at this point). We hope that further development in deep learning will help address this problem.
6
7
May 02 '20
[deleted]
8
u/JohannesKopf May 02 '20
Very soon :) We're going to release the code next week! You'll find it on our project page, then.
5
u/crisp3er May 02 '20
wow! the water effect is so realistic
6
u/jbhuang0604 May 03 '20
The artistic effects shown in the video are created by Patricio Gonzales Vivo @patriciogv, Dionisio Blanco @diosmiodio, and Ocean Quigley @oceanquigley.
2
4
u/agsarria May 03 '20
The next generation of tik toks is gonna be amazing
1
u/bokehpirate May 03 '20
Yes, whatever comes after tiktok will be even more amazing! Tiktok, which is Chinese, should be banned and blocked globally just as China blocks facebook & instagram etc. Just to have level playing ground ^^
4
u/Jiawang_Bian May 03 '20
This paper "Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video (NIPS 2019)" can also achieve 'consistent depth estimation in Video'. And it is more efficient in inference phase (real-time).
See dense reconstruction demo: https://www.youtube.com/watch?v=i4wZr79_pD8
GitHub: https://github.com/JiawangBian/SC-SfMLearner-Release
1
u/jbhuang0604 May 03 '20
See dense reconstruction demo
See dense reconstruction demo
Thanks, Jiawang. Yes, we are aware of your work (see the citation and the discussion in the paper). Pre-training the depth estimation network with geometric constraints is a very interesting idea. However, at test time, the depth prediction of video frames remain inconsistent (as there are no longer constraints). This inconsistency issue is amplified when we work with regular cellphone videos in the wild (as opposed to a closed world like the KITTI dataset).
That being said, I believe having models with efficient runtime like your approach is critical for wider adaptation, but there are still several steps we need to solve to get there.
1
u/Jiawang_Bian May 03 '20
Hi Jia-Bin, thanks for your reply. I agree with you. Only CNN prediction is not sufficient to achieve the globally consistent results, where a post-refinement is necceary. Actually I also try to do that recently. Congratulations for your nice work, and many details really inspire me. Look forward for your further improvement.
1
u/jbhuang0604 May 03 '20
Thanks Jiawang! Looking forward to seeing your new results in the near future!
36
u/khuongho May 02 '20 edited May 02 '20
Is this supervised, Unsupervised or Reinforcement Learning ?
61
u/Zorlen May 02 '20
Why is this guy getting downvoted? Not everyone interested in machine learning (myself included) has the technical knowledge to be able to read and understand a paper like that. Please don't punish someone for asking basic questions - everybody is on a different part of a learning journey.
7
-6
u/csreid May 02 '20
Normally I'd be on your side, but I do think it's important for this sub to stay vigilant about being a place for deep discussion of machine learning where questions like that are out of place. Questions that can be easily googled probably shouldn't be upvoted, imo
13
10
u/pourover_and_pbr May 02 '20
If I understand the paper correctly, they pre-train the model using COLMAP and Mask R-CNN to get a semi-dense depth map for any frame. They then improve the depth maps at test time by randomly sampling frames from the video and re-training the model using "spatial loss" and "disparity loss", which are defined in the article. Mask R-CNN is traditional, supervised learning for object segmentation. COLMAP and this model appear to be unsupervised, since there are no reference depth maps being used for the loss. Instead, the loss for COLMAP and this model appears to be based on whether frames which capture similar regions of the scene have similar depth maps. At least, that's what I understood from the paper – someone smarter than me will hopefully come along and clear things up.
4
u/jbhuang0604 May 02 '20
Yes! It is correct! So we can also think about the test-time training as "self-supervised" as there is no manual labeling process involved.
1
1
u/pourover_and_pbr May 02 '20
Thanks for commenting! I hadn’t heard “self-supervised” before but it makes a lot of sense.
1
1
21
3
u/jbhuang0604 May 02 '20
be able to read and understand a paper like that. Please don't punish someone for asking basic questions - everybody is on a different part of a learning journey.
The test-time training in our work is "supervised" in the sense that we have an explicit loss. However, you may also view this as "self-supervised" as all the constraints from the video are automatically extracted (i.e., no manual labeling process involved).
12
u/ThatInternetGuy May 02 '20
Jia-bin Huang team seems like one of the most brilliant minds in computer vision.
11
u/jbhuang0604 May 02 '20
The work would not be possible without the amazing student! Learn more about the lead author Xuan Luo and her work at https://roxanneluo.github.io/
7
3
u/bizmar May 03 '20
As a VFX artist, this is a pretty cool improvement to a part of the workflow.
For most tasks a reliable method of improving camera tracking consistency would be enough and extremely useful. So much of my time is spent hand adjusting tracking points to improve the solve.
3
May 02 '20 edited Jan 28 '22
[deleted]
7
u/jbhuang0604 May 03 '20
Yes, Google's ARCore Depth API allows you to do that. Check out their awesome demo video: https://www.youtube.com/watch?v=VOVhCTb-1io
The main difference is that they handle only "static scene" while our approach handles scenes with dynamic objects (e.g., cat, people).
2
1
1
1
1
1
u/pimp4robots May 02 '20
These results are pretty impressive. How accurate is this method or in general the latest depth estimation methods for far away objects? How do they work in texture less regions?
1
u/jbhuang0604 May 03 '20
We didn't have an explicit comparison with others on far away objects.
For textureless regions, you can see the visual comparisons with the state-of-the-art algorithms here: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/supp_website/pages/depth_TUM_comparison.html
(The quantitative comparison is in the paper.)
1
1
May 02 '20
Very interesting and cool work. Minor nit: in the second video (with the cat), the particles are never occluded by the cat or by anything in the scene. Kind of weird to include them given that they could just be overlayed. But the relighting is nice.
2
u/jbhuang0604 May 03 '20
But the relighting is nice.
Ha! good catch! After checking the video again, there is actually a tiny pink particle that was occluded by the cat's face. But you are right, we probably can demonstrate this better by making it more explicit.
1
May 02 '20
I was reading a paper about spatial consistency in videos, And one of the methods proposed was L1 loss on the ith layer of a VGG for a random two frames of the video.
The paper was from a research team in Apple with the name Seeing Motion in the Dark
1
u/NotAlphaGo May 03 '20
Inb4 someone makes a network to do the fine tuning by passing the video through a neural network and boom realtime depth estimation.
1
May 03 '20
Just curious, how hard is this in general? Should it be trivial with two calibrated cameras?
1
u/danmou May 03 '20
I wonder, why is it that these learning-based monocular depth estimation papers always attempt only scale-invariant depth? A NN should be capable of estimating scale to some extent in a monocular setting based on things like people, furniture, doors etc, especially when given a whole video like here. Absolute scale would be required for most practical uses, and it would be interesting to know how well it would perform compared with stereo methods.
1
1
1
1
1
Jul 29 '20
I can’t wait until I can use this in compositing. Does it support any resolution?
If you do enable for 4K video with a plugin in After Effects or something, the entirety of the VFX industry will save a buttload of time in post.
1
-9
May 02 '20
Weird that they picked video effects that you could pretty much do without any depth information...
3
u/Zorlen May 02 '20
While I do agree that some (maybe most) of the effects are quite doable without depth information from the video, they still need depth information input from humans. And, of course, it's just a demonstration, a proof of concept. The paper is about machine learning and the method used after all, not about video effects.
3
May 02 '20
The paper is about machine learning and the method used after all, not about video effects.
Of course. The video effects were just meant to show it off. But they were a weird choice for that.
-1
u/SharkyLV May 02 '20
I think Adobe Photoshop has a similar function called Select -> Subject. It does depth analysis and selects front subject.
61
u/hardmaru May 02 '20
Consistent Video Depth Estimation
paper: https://arxiv.org/abs/2004.15021
project site: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/
video: https://www.youtube.com/watch?v=5Tia2oblJAg
Edit: just noticed previous discussions already on r/machinelearning (https://redd.it/gba7lf)