GLAD: Content-Aware Dynamic Graphs for Log Anomaly Detection

/in /by

Logs play a crucial role in system monitoring and debugging by recording valuable system information, including events and status. Although various methods have been proposed to detect anomalies in log sequences, they often overlook the significance of considering relationships among system components, such as services and users, which can be identified from log contents. Understanding these relationships is vital for identifying anomalies and their underlying causes. To address this issue, we introduce GLAD, a Graph-based Log Anomaly Detection framework designed to detect relational anomalies in system logs. GLAD incorporates log semantics, relationship patterns, and sequential patterns into a unified framework for anomaly detection. Specifically, GLAD first introduces a field extraction module that utilizes prompt-based few-shot learning to extract essential field information, such as services and users, from log contents. Then GLAD constructs dynamic log graphs for sliding windows by leveraging the log events and extracted fields. These graphs represent events and fields as nodes and their relationships as edges. Subsequently, we propose atemporal-attentive graph edge anomaly detection model for identifying anomalous relationships in the dynamic log graphs. This model employs a Graph Neural Network (GNN)-based encoder enhanced with transformers to capture structural, content, and temporal features. We evaluate our proposed method on three datasets, and the results demonstrate the effectiveness of GLAD in detecting anomalies indicated by varying relation patterns.

Read more

Long Term Monitoring and Analysis of Brood X Cicada Activity by Distributed Fiber Optic Sensing Technology

/in /by

Brood X is the largest of the 15 broods of periodical cicadas, and individuals from this brood emerged across the Eastern United States in spring 2021. Using distributed acoustic sensing (DAS) technology, the activity of Brood X cicadas was monitored in their natural environment in Princeton, NJ. Critical information regarding their acoustic signatures and activity level is collected and analyzed using standard outdoor-grade telecommunication fiber cables. We believe these results have the potential to be a quantitative baseline for regional Brood X activity and pave the way for more detailed monitoring of insect populations to combat global insect decline. We also show that it is possible to transform readily available fiber optic networks into environmental sensors with no additional installation costs. To our knowledge, this is the first reported use case of a distributed fiber optic sensing system for entomological sciences and environmental studies.

Read more

Real-Time Photonic Blind Interference Cancellation

/in /by

mmWave devices can broadcast multiple spatially-separated data streams simultaneously in order to increase data transfer rates. Data transfer can, however, be compromised by interference. Photonic blind interference cancellation systems offer a power-efficient means of mitigating interference, but previous demonstrations of such systems have been limited by high latencies and the need for regular calibration. Here, we demonstrate real-time photonic blind interference cancellation using an FPGA-photonic system executing a zero-calibration control algorithm. Our system offers a greater than 200-fold reduction in latency compared to previous work, enabling sub-second cancellation weight identification. We further investigate key trade-offs between system latency, power consumption, and success rate, and we validate sub-Nyquist sampling for blind interference cancellation. We estimate that photonic interference cancellation can reduce the power required for digitization and signal recovery by greater than 74 times compared to the digital electronic alternative.

Read more

Beyond Communication: Telecom Fiber Networks for Rain Detection and Classification

/in /by

We present the field trial of an innovative neural network and DAS-based technique, employing a pre-trained CNN fine-tuning strategy for effective rain detection and classification within two practical scenarios.

Read more

Distributed Fiber-Optic Sensor as an Acoustic Communication Receiver Array

/in /by

A novel acoustic transmission technique using distributed acoustic sensors is introduced. By choosing better incident angles for smaller fading and employing an 8- channel beamformer, over 10KB data is transmitted at a 6.4kbps data rate.

Read more

OFDM Signal Transmission Using Distributed Fiber-Optic Acoustic Sensing

/in /by

Acoustic data transmission with the Orthogonal Frequency Division Multiplexing (OFDM) signal has been demonstrated using a Distributed Acoustic Sensor (DAS) based on Phase-sensitive Optical Time-Domain Reflectometry (?-OTDR).

Read more

Adaptation Speed Analysis for Fairness-Aware Causal Models

/in /by

For example, in machine translation tasks, to achieve bidirectional translation between two languages, the source corpus is often used as the target corpus, which involves the training of two models with opposite directions. The question of which one can adapt most quickly to a domain shift is of significant importance in many fields. Specifically, consider an original distribution p that changes due to an unknown intervention, resulting in a modified distribution p*. In aligning p with p*, several factors can affect the adaptation rate, including the causal dependencies between variables in p. In real-life scenarios, however, we have to consider the fairness of the training process, and it is particularly crucial to involve a sensitive variable (bias) present between a cause and an effect variable. To explore this scenario, we examine a simple structural causal model (SCM) with a cause-bias-effect structure, where variable A acts as a sensitive variable between cause (X) and effect (Y). The two models respectively exhibit consistent and contrary cause-effect directions in the cause-bias-effect SCM. After conducting unknown interventions on variables within the SCM, we can simulate some kinds of domain shifts for analysis. We then compare the adaptation speeds of two models across four shift scenarios. Additionally, we prove the connection between the adaptation speeds of the two models across all interventions.

Read more

Calibrate Graph Neural Networks under Out-of-Distribution Nodes via Deep Q-learning

/in /by

Graph neural networks (GNNs) have achieved great success in dealing with graph-structured data that are prevalent in the real world. The core of graph neural networks is the message passing mechanism that aims to generate the embeddings of nodes by aggregating the neighboring node information. However, recent work suggests that GNNs also suffer the trustworthiness issues. Our empirical study shows that the calibration error of the in-distribution (ID) nodes would be exacerbated if a graph is mixed with out-of-distribution (OOD) nodes, and we assume that the noisy information from OOD nodes is the root for the worsened calibration error. Both previous study and our empirical study suggest that adjusting the weights of edges could be a promising way to reduce the adverse impact from the OOD nodes. However, how to precisely select the desired edges and modify the corresponding weights is not trivial, since the distribution of OOD nodes is unknown to us. To tackle this problem, we propose a Graph Edge Re-weighting via Deep Q-learning (GERDQ) framework to calibrate the graph neural networks. Our framework aims to explore the potential influence of the change of the edge weights on target ID nodes by sampling and traversing the edges in the graph, and we formulate this process as a Markov Decision Process (MDP). Many existing GNNs could be seamlessly incorporated into our framework. Experimental results show that when wrapped with our method, the existing GNN models can yield lower calibration error under OOD nodes as well as comparable accuracy compared to the original ones and other strong baselines. The source code is available at:https://github.com/DamoSWL/Calibration-GNN-OOD.

Read more

Citizen Science for the Sea with Information Technologies: An Open Platform for Gathering Marine Data and Marine Litter Detection from Leisure Boat Instruments

/in /by

Data crowdsourcing is an increasingly pervasive and lifestyle-changing technology due to the flywheel effect that results from the interaction between the Internet of Things and Cloud Computing. This paper presents the Citizen Science for the Sea with Information Technologies (C4Sea-IT) framework. It is an open platform for gathering marine data from leisure boat instruments. C4Sea-IT aims to provide a coastal marine data gathering, moving, processing, exchange, and sharing platform using the existing navigation instruments and sensors for today’s leisure and professional vessels. In this work, a use case for the detection and tracking of marine litter is shown. The final goal is weather/ocean forecasts argumentation with Artificial Intelligence prediction models trained with crowdsourced data.

Read more

Source-Free Domain Adaptive Fundus Image Segmentation with Class-Balanced Mean Teacher

/in /by

This paper studies source-free domain adaptive fundus image segmentation which aims to adapt a pretrained fundus segmentation model to a target domain using unlabeled images. This is a challenging task because it is highly risky to adapt a model only using unlabeled data. Most existing methods tackle this task mainly by designing techniques to carefully generate pseudo labels from the model’s predictions and use the pseudo labels to train the model. While often obtaining positive adaption effects, these methods suffer from two major issues. First, they tend to be fairly unstable – incorrect pseudo labels abruptly emerged may cause a catastrophic impact on the model. Second, they fail to consider the severe class imbalance of fundus images where the foreground (e.g., cup) region is usually very small. This paper aims to address these two issues by proposing the Class-Balanced Mean Teacher (CBMT) model. CBMT addresses the unstable issue by proposing a weak-strong augmented mean teacher learning scheme where only the teacher model generates pseudo labels from weakly augmented images to train a student model that takes strongly augmented images as input. The teacher is updated as the moving average of the instantly trained student, which could be noisy. This prevents the teacher model from being abruptly impacted by incorrect pseudo-labels. For the class imbalance issue, CBMT proposes a novel loss calibration approach to highlight foreground classes according to global statistics. Experiments show that CBMT well addresses these two issues and outperforms existing methods on multiple benchmarks.

Read more