Multi-modal Time Series Analysis: A Tutorial and Survey

/in /by

Publication Date: 8/7/2025

Event: 31st ACM SIGKDD Conference on Knowledge Discover and Data Mining (ACM KDD 2025)

Reference: pp. 6043-6053, 2025

Authors: Yushan Jiang, University of Connecticut; Kanghui Ning, University of Connecticut; Zijie Pan, University of Connecticut; Xuyang Shen, University of Connecticut; Jingchao Ni, University of Houston; Wenchao Yu, NEC Laboratories America, Inc.; Anderson Schneider, Morgan Stanley; Haifeng Chen, NEC Laboratories America, Inc.; Yuriy Nevmyvaka, Morgan Stanley; Dongjin Song, University of Connecticut

Abstract: Multi-modal time series analysis has recently emerged as a prominent research area, driven by the increasing availability of diverse data modalities, such as text, images, and structured tabular data from real-world sources. However, effective analysis of multi-modal time series is hindered by data heterogeneity, modality gap, misalignment, and inherent noise. Recent advancements in multi-modal time series methods have exploited the multi-modal context via cross-modal interactions based on deep learning methods, significantly enhancing various downstream tasks. In this tutorial and survey, we present a systematic and up-to-date overview of multi-modal time series datasets and methods. We first state the existing challenges of multi-modal time series analysis and our motivations, with a brief introduction of preliminaries. Then, we summarize the general pipeline and categorize existing methods through a unified cross-modal interaction framework encompassing fusion, alignment, and transference at different levels (i.e., input, intermediate, output), where key concepts and ideas are highlighted. We also discuss the real-world applications of multi-modal analysis for both standard and spatial time series, tailored to general and specific domains. Finally, we discuss future research directions to help practitioners explore and exploit multi-modal time series. The up-to-date resources are provided in the GitHub repository. https://github.com/UConn-DSIS/Multi-modal-Time-Series-Analysis.

Publication Link: https://dl.acm.org/doi/10.1145/3711896.3736567