DataX: A system for Data eXchange and transformation of streams

/in /by

Publication Date: 9/26/2021

Event: The 14th International Symposium on Intelligent Distributed Computing (IDC 2021)

Reference: pp. 1-10, 2021

Authors: Giuseppe Coviello, NEC Laboratories America, Inc.; Kunal Rao, NEC Laboratories America, Inc.; Murugan Sankaradas, NEC Laboratories America, Inc.; Srimat T. Chakradhar, NEC Laboratories America, Inc.

Abstract: The exponential growth in smart sensors and rapid progress in 5G networks is creating a world awash with data streams. However, a key barrier to building performant multi-sensor, distributed stream processing applications is high programming complexity. We propose DataX, a novel platform that improves programmer productivity by enabling easy exchange, transformations, and fusion of data streams. DataX abstraction simplifies the application’s specification and exposes parallelism and dependencies among the application functions (microservices). DataX runtime automatically sets up appropriate data communication mechanisms, enables effortless reuse of microservices and data streams across applications, and leverages serverless computing to transform, fuse, and auto-scale microservices. DataX makes it easy to write, deploy and reliably operate distributed applications at scale. Synthesizing these capabilities into a single platform is substantially more transformative than any available stream processing system.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-030-96627-0_29

Additional Publication Link: https://arxiv.org/pdf/2111.04959.pdf