LogLens: A Real-time Log Analysis System

/in /by

Publication Date: 7/2/2018

Event: 38th IEEE International Conference on Distributed Computing Systems (ICDCS 2018)

Reference: pp. 1052-062, 2018

Authors: Biplob Debnath, NEC Laboratories America, Inc.; Mohiuddin Solaimani, University of Texas at Dallas; Muhammad Ali Gulzar, University of California, Los Angeles ; Nipun Arora, NEC Laboratories America, Inc.; Cristian Lumezanu, NEC Laboratories America, Inc.; Jianwu Xu, NEC Laboratories America, Inc.; Bo Zong, NEC Laboratories America, Inc.; Hui Zhang , NEC Laboratories America, Inc.; Guofei Jiang, NEC Laboratories America, Inc.; Latifur Khan, University of Texas at Dallas

Abstract: Administrators of most user-facing systems depend on periodic log data to get an idea of the health and status of production applications. Logs report information, which is crucial to diagnose the root cause of complex problems. In this paper, we present a real-time log analysis system called LogLens that automates the process of anomaly detection from logs with no (or minimal) target system knowledge and user specification. In LogLens, we employ unsupervised machine learning based techniques to discover patterns in application logs, and then leverage these patterns along with the real-time log parsing for designing advanced log analytics applications. Compared to the existing systems which are primarily limited to log indexing and search capabilities, LogLens presents an extensible system for supporting both stateless and stateful log analysis applications. Currently, LogLens is running at the core of a commercial log analysis solution handling millions of logs generated from the large-scale industrial environments and reported up to 12096x man-hours reduction in troubleshooting operational problems compared to the manual approach.

Publication Link: https://ieeexplore.ieee.org/document/8416368