DATA SCIENCE & SYSTEM SECURITY
PROJECTS
PEOPLE
PUBLICATIONS
PATENTS
Multimodal data are prevalent in industrial monitoring, finance and healthcare. In particular, time series are often tagged with text comments from experts that provide layman users with the domain knowledge to understand the charts. Texts give the patterns qualitative meaning while time series makes the words quantitative. Analyzing the relationship between different data types is the key to unraveling the hidden structure of such data.
This project aims to develop machine learning and data mining algorithms that provide insight about multimodal data through joint modeling of time series, natural language texts and data of other types. Through tasks such as automatic time series explanation, cross-modal retrieval, time series QA and knowledge discovery, we create virtual domain experts that can comprehend domain-specific terms and use them to explain time series data. Automated financial analyst, plant operator, health advisor and fitness coach are just a few examples of the next generation AI-human interaction paradigm enabled by multimodal learning.
Team Members: Zhengzhang Chen, Wenchao Yu, Haifeng Chen
Publication Tags: multimodal, multi modal, multimodal data
Time series, typically represented as numerical sequences, can also be transformed into images and texts, offering multi-modal views (MMVs) of the same underlying signal. These MMVs can reveal complementary patterns and enable the use of powerful pre-trained large models, such as large vision models
Time series analysis provides essential insights for real-world system dynamics and informs downstream decision-making, yet most existing methods often overlook the rich contextual signals present in auxiliary modalities. To bridge this gap, we introduce TimeXL, a multi-modal prediction framework that
Time series data is essential in various applications, including climate modeling, healthcare monitoring, and financial analytics. Understanding the contextual information associated with real-world time series data is often essential for accurate and reliable event predictions. In this paper, we introduce
Retrieval-augmented generation (RAG) systems combine the strengths of language generation and information retrieval to power many real-world applications like chatbots. Use of RAG for understanding of videos is appealing but there are two critical limitations. One-time, upfront conversion of all content
Our new project, Time Series Language Model for Explainable AI, represents a significant leap forward in the field of forecasting and explainable AI. By combining advanced forecasting techniques with explainable AI, we are paving the way for a future where data-driven insights are not only accurate but
Weakly Supervised Temporal Action Localization (WSTAL) aims to jointly localize and classify action segments in untrimmed videos with only video level annotations. To leverage video level annotations most existing methods adopt the multiple instance learning paradigm where frame/snippet level action
Retrieval-augmented generation (RAG) is used in natural language processing (NLP) to provide query-relevant information in enterprise documents to large language models (LLMs). Such enterprise context enables the LLMs to generate more informed and accurate responses. When enterprise data is primarily
Retrieval augmented generation (RAG) systems combine the strengths of language generation and information retrieval to power many real-world applications like chatbots. Use of RAG for combined understanding of multimodal data such as text, images and videos is appealing but two critical limitations exist:
Existing visual reasoning datasets, such as Visual Question Answering (VQA), often suffer from biases conditioned on the question, image or answer distributions. The recently proposed CLEVR dataset addresses these limitations and requires fine-grained reasoning, but the dataset is synthetic and consists
We introduce a new inference task – Visual Entailment (VE) – which differs from traditional Textual Entailment (TE) tasks whereby a premise is defined by an image, rather than a natural language sentence as in TE tasks. A novel dataset SNLI-VE is proposed for VE tasks based on the Stanford Natural Language
