Machine Learning

Aiming to achieve near-human video understanding, in this project we analyze several gigabytes of spatiotemporal data to perform action recognition, multi-person tracking, object permanence and video reasoning. eigen has built a scalable video-understanding platform for long-form video reasoning that scales to new environments and camera angles without any re-training. eigen also provides a system platform. Running both on the cloud (AWS) and on-prem, it can scale up to thousands of streams into it for cloud-based AI processing. Our AI video algorithms provide efficient streaming and inference. eigen has a web frontend and support for iOS/Android playback. eigen has been tested in several retail POCs serving 200+ streams; its behavioral analytics have also been evaluated through various NEC customers. Using mixed precision and TensorRT, eigen is extremely efficient and incurs very low cloud costs. PublicationsHopper: Multi-hop Transformer for Spatiotemporal Reasoning. In ICLR 2021. PDFPoster 15 Keypoints is All You Need. Michael Snower, Asim Kadav, Farley Lai, Hans Peter Graf. In CVPR 2020. PDFRanked #1 in PoseTrack Tripping Through Time: Efficient Localization of Activities in Videos. (Spotlight) Meera Hahn, Asim Kadav, James M. Rehg, Hans Peter Graf. In CVPR Workship on Language and Vision, 2019. Also, appears in BMVC'20. PDF Visual Entailment: A Novel Task for Fine-Grained Image Understanding. Ning Xie, Farley Lai, Derek Doran, Asim Kadav. In NeurIPS Workshop on Visually-Grounded Interaction and Language, 2018 (ViGIL’18). PDFDatasetleaderboard Teaching Syntax using Adverserial Distraction. Juho Kim, Christopher Malon, and Asim Kadav. In FEVER-EMNLP, 2018. PDF Attend and Interact: Higher-Order Object Interactions for Video Understanding. Chih-Yao Ma, Asim Kadav, Iain Melvin, Zsolt Kira, Ghassan AlRegib, and Hans Peter Graf. In CVPR, 2018. PDFVideoLive Demo

We develop several types of algorithms for high-level semantic analysis, which are used for tasks such as scene interpretation, document retrieval and question answering. For text interpretation, a syntactic analysis extracts relevant elements followed by concept interpretation. To combine different data types, a data-specific module first generates metadata representations that are integrated into a deep learning network. Supervised sequence embedding (SSE) is a simple and efficient technique to interpret shorter segments of text, such as product reviews or e-mail messages. Short phrases (n-grams) are modeled in a latent space. The phrases are then combined to form document-level latent representations, where position of an n-gram in the document is used to compute corresponding combining weight. The resulting two-stage supervised embedding is then coupled with a classifier to form an end-to-end system that we apply to the large-scale sentiment classification task. SSE does not require feature selection, and its parameter space grows only linearly with the size of the n-grams.

Accurate and fast diagnosis based on histological samples is crucial for prevention, early detection and treatment of cancer. We have developed a digital pathology system that uses machine learning algorithms to analyze images of tissue to assist in cancer diagnosis. This system is used in major diagnostics laboratories in Japan for quality control.