INTEGRATED SYSTEMS
PROJECTS
PEOPLE
PUBLICATIONS
PATENTS
Somewhere inside every large organization is the answer to a question someone is asking right now. But often, there is no practical way to find it. An analyst needs a figure buried in a chart on page sixty of a filing. A safety engineer needs the two minutes of facility surveillance video when an intruder trespassed. A clinician needs the prior study that most closely resembles the one on the screen. In each case, the evidence already exists.
But finding it requires a person with enough time and expertise to search manually. In practice, that often means the answer might as well not exist.
The problem is the form that enterprise knowledge takes.
Keyword Tag: grounded multimodal
Much of an organization’s most valuable information is not plain text. It lives in charts and tables, years of video, satellite imagery, radiology studies, maintenance records, and continuously generated data streams. Generative AI has made it possible to ask natural-language questions of this information, and retrieval-augmented generation (RAG) has become the standard architecture for connecting models to proprietary data. But today’s RAG systems were largely built for text documents. They struggle when the evidence is visual, temporal, spatial, or spread across multiple modalities.
This limitation matters most in industries where the stakes are highest. Finance, law, healthcare, defense, and industrial operations hold enormous amounts of multimodal data, and they cannot rely on answers without evidence. In these settings, the product is not just the answer; it is the provenance behind it. A system that produces a convincing response but cannot show exactly where that response came from has limited commercial value.
We are building toward a retrieval and reasoning layer that works across documents, video, imagery, and data streams. Users should be able to ask open-ended questions in their own language, rather than conforming to a predefined taxonomy, and receive answers grounded directly in the underlying evidence. The goal is to make an organization’s entire body of knowledge searchable enough to use, efficient enough to query continuously, and traceable enough to trust and defend.
Three barriers stand between today’s multimodal RAG systems and reliable enterprise deployment: grounding, cost, and scale. First, correctness is difficult to measure. In multimodal retrieval, there is no established way to determine whether a retrieved image, video segment, or other piece of evidence actually supports a generated claim. As a result, hallucinations can be difficult to detect and grounding is often assumed rather than measured. Second, the economics work against enterprise-scale deployment. Enterprise context can be enormous, while LLM APIs are typically priced by the amount of context they process. A naive system therefore spends much of its cost sending information the model does not actually need. Third, video and live streams break the traditional indexing model. Converting an entire video archive into text before anyone asks a question is prohibitively expensive, while a live stream cannot be fully indexed in advance at all. Retrieval must instead happen incrementally, as questions are asked and new information arrives.
Our approach makes grounding, efficiency, and incremental retrieval first-class design constraints. RAG-check provides a methodology for evaluating multimodal RAG, turning grounding into something that can be measured rather than assumed. Our work on re-ranking retrieved context improves the quality and relevance of the evidence that reaches the model in the first place. We attack cost by reducing context rather than simply paying to process it. LeanContext and query-aware data reduction identify the information relevant to a particular question before it is sent to the model, while EcoDoc applies the same principle to multimodal document pipelines.
For video and streaming data, iRAG replaces exhaustive up-front indexing with an incremental approach that processes content as queries demand it. ViTA makes video-to-text conversion efficient enough for practical deployment, while StreamingRAG extends retrieval and reasoning to live multimodal streams. Finally, we address open-vocabulary grounding in specialist domains where predefined labels and taxonomies are especially limiting. Open-SAT uses LLM guidance to improve free-form retrieval from satellite imagery. Our work on visual alignment grounds generated radiology text in what is actually visible in medical images, while knowledge-enhanced prompt learning helps models bridge the commonsense gaps that emerge when reasoning about the open world.
Our AI systems move multimodal RAG from simply generating plausible answers toward something enterprises can actually depend on: retrieval that is measurable in its grounding, economical in its use of context, scalable to massive and continuous data, and flexible enough to answer questions in the language users naturally use.
Satellite imagery is vast, high-resolution, and rich with information, but finding specific objects within it using natural language has remained a stubborn challenge. Open-SAT, developed by researchers at NEC Laboratories America and North South University, tackles this problem without retraining any
In satellite applications, user queries often take the form of open-ended natural language, extending beyond a fixed set of predefined categories. This open-vocabulary nature poses significant challenges for retrieving relevant image tiles, as the retrieval system must generalize to a wide range of unseen
Radiology Report Generation (RRG) is a critical step toward automating healthcare workflows, facilitating accurate patient assessments, and reducing the workload of medical professionals. Despite recent progress in Large Medical Vision-Language Models (Med-VLMs), generating radiology reports that are
Enterprises are increasingly adopting Generative AI applications to extract insights from large volumes of multimodal documents in domains such as finance, law, healthcare, and industry. These documents contain structured and unstructured data (images, charts, handwritten texts, etc.) requiring robust
Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating external knowledge to generate a response within a context with improved accuracy and reduced hallucinations. However, multi-modal RAG systems face unique challenges: (i) the retrieval process may select irrelevant
Retrieval-augmented generation (RAG) improves large language models (LLMs) by using external knowledge to guide response generation, reducing hallucinations. However, RAG, particularly multi-modal RAG, can introduce new hallucination sources: (i) the retrieval process may select irrelevant pieces (e.g.,
Retrieval-augmented generation (RAG) systems combine the strengths of language generation and information retrieval to power many real-world applications like chatbots. Use of RAG for understanding of videos is appealing but there are two critical limitations. One-time, upfront conversion of all content
Neural language models for commonsense reasoning often formulate the problem as a QA task and make predictions based on learned representations of language after fine-tuning. However, without providing any fine-tuning data and pre-defined answer candidates, can neural language models still answer commonsense
Costs of LLM API usage rise rapidly when proprietary enterprise data is used as context for user queries to generate more accurate responses from LLMs. To reduce costs, we propose LeanContext, which generates query-aware, compact and AI model-friendly summaries of relevant enterprise data context. This
Retrieval-augmented generation (RAG) is used in natural language processing (NLP) to provide query-relevant information in enterprise documents to large language models (LLMs). Such enterprise context enables the LLMs to generate more informed and accurate responses. When enterprise data is primarily
Extracting real-time insights from multi-modal data streams from various domains such as healthcare, intelligent transportation, and satellite remote sensing remains a challenge. High computational demands and limited knowledge scope restrict the applicability of Multi-Modal Large Language Models (MM-LLMs)
Question-answering (QA) is a significant application of Large Language Models (LLMs), shaping chatbot capabilities across healthcare, education, and customer service. However, widespread LLM integration presents a challenge for small businesses due to the high expenses of LLM API usage. Costs rise rapidly
Retrieval augmented generation (RAG) systems combine the strengths of language generation and information retrieval to power many real-world applications like chatbots. Use of RAG for combined understanding of multimodal data such as text, images and videos is appealing but two critical limitations exist: