Universal Correspondence Network | Projects

MEDIA ANALYTICS

PROJECTS

PEOPLE

PUBLICATIONS

PATENTS

Universal Correspondence Network

We present deep metric learning to obtain a feature space that preserves geometric or semantic similarity. Our visual correspondences span across rigid motions to intra-class shape or appearance variations. Our fully convolutional architecture, along with a novel correspondence contrastive loss, allows faster training by effective reuse of computations, accurate gradient computation and linear time testing instead of quadratic time for typical patch similarity methods. We propose a convolutional spatial transformer to mimic patch normalization in traditional features like SIFT.

Collaborators: Christopher B. Choy, Junyoung Gwak, Silvio Savarese, Manmohan Chandraker

Universal Correspondence Network Paper

Christopher B. Choy, Stanford University; JunYoung Gwak, Stanford University, Silvio Savarese, Stanford University; Manmohan Chandraker, NEC Laboratories America, Inc.

[Download PDF] [Supplemental] [Project Site]

Abstract

We have proposed a novel deep metric learning approach to visual correspondence estimation, that is shown to be advantageous over approaches that optimize a surrogate patch similarity objective. We propose several innovations, such as a correspondence contrastive loss in a fully convolutional architecture, on-the-fly active hard negative mining and a convolutional spatial transformer. These lend capabilities such as more efficient training, accurate gradient computations, faster testing and local patch normalization, which lead to improved speed or accuracy. We demonstrate in experiments that our features perform better than prior state-of-the-art on both geometric and semantic correspondence tasks, even without using any spatial priors or global optimization. In future work, we will explore applications of our correspondences for rigid and non-rigid motion or shape estimation as well as applying global optimization.

Project Data

Citing this work

If you find this work useful in your research, please consider citing:

@incollection{choy_nips16,
  title = {Universal Correspondence Network},
  author = {Choy, Christopher B and Gwak, JunYoung and Savarese, Silvio and Chandraker, Manmohan},
  booktitle = {Advances in Neural Information Processing Systems 30},
  year = {2016},
}

Please follow the instructions below to download the source code
Paper
Slides
Blog post about the UCN and the FAQ
Trained network weights
- googlenet-conv-spatial-transformer trained on CUB2011, and prototxts
- googlenet-conv-spatial-transformer trained on Flowweb, and prototxts
- googlenet-conv-spatial-transformer trained on VOC2011, and prototxts
- googlenet-conv-spatial-transformer trained on SINTEL, and prototxts
- googlenet-conv-spatial-transformer trained on KITTI Flow, and prototxts
Supplementary Materials
CUB2011 Dataset with Filtered Annotations
VOC2011 Correspondence Annotations

Patent

US Patent 15045 (449-460) END-TO-END FULLY CONVOLUTIONAL FEATURE LEARNING FOR PATCH SIMILARITY

Code

This source code corresponds to the paper, “Universal Correspondence Network”, as it appears in NIPS 2016. To obtain the code, please download the linked zip file and the license agreement.

Important: The following instructions must be followed exactly:

The license agreement must be signed by the appropriate manager responsible for enforcing copyright. For a student, it may be the faculty advisor. For a professor, it may be the department chair. For a researcher, it may be the lab or company manager.
All three pages of the agreement must be scanned and emailed.
Please use your official university or company email address.

You will receive a password to unzip the source repository in your official email address.

Please cite the following paper if you find the code useful in your work:

@inproceedings{Choy_etal_2016,
  author = {Christopher B. Choy and JunYoung Gwak and Silvio Savarese and Manmohan Chandraker},
  title = {Universal Correspondence Network},
  booktitle = {Advances in Neural Information Processing Systems 29},
  editor = {D. D. Lee and M. Sugiyama and U. V. Luxburg and I. Guyon and R. Garnett},
  pages = {2414--2422},
  year = {2016},
  publisher = {Curran Associates, Inc.},
  url = {http://papers.nips.cc/paper/6487-universal-correspondence-network.pdf}
}