Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion

/in /by

Symmetry in the parameter space of deep neural networks (DNNs) has proven beneficial for various deep learning applications. A well-known example is the permutation symmetry in Multi-Layer Perceptrons (MLPs), where permuting the rows of weight matrices in one layer and applying the inverse permutation to adjacent layers yields a functionally equivalent model. While permutation symmetry fully characterizes the equivalence set for MLPs, its discrete nature limits its utility for transformers. In this paper, we introduce rotation symmetry, a novel form of parameter space symmetry for transformers that generalizes permutation symmetry by rotating parameter matrices in self-attention layers. Unlike permutation symmetry, rotation symmetry operates in a continuous domain, thereby significantly expanding the equivalence set for transformers. Based on this property, we propose a theoretically optimal parameter matching algorithm as a plug-and-play module to enhance model fusion. We evaluate our approach using pre-trained transformers across diverse natural language and vision tasks. Experimental results demonstrate that our rotation symmetry based matching algorithm substantially improves model fusion, highlighting the potential of parameter space symmetry to facilitate model fusion. Our code is available on https://github.com/zhengzaiyi/RotationSymmetry.

Read more

Pathologist-Read vs AI-Driven Assessment of Tumor-Infiltrating Lymphocytes in Melanoma

/in /by

Tumor-infiltrating lymphocytes (TILs) are a provocative biomarker in melanoma, influencing diagnosis, prognosis, and immunotherapy outcomes; however, traditional pathologistreadTIL assessment on hematoxylin and eosin–stained slides is prone to interobserver variability, leading to inconsistent clinical decisions. Therefore, development of newer TIL scoring approachesthat produce more reliable and consistent readouts is important.

Read more

Toward Intelligent and Efficient Optical Networks: Performance Modeling, Co-existence, and Field Trials

/in /by

Optical transmission networks require intelligent traffic adaptation and efficient spectrum usage. We present scalable machine learning (ML) methods for network performance modeling, andfield trials of distributed fiber sensing and classic optical network traffic coexistence.

Read more

Span-based Polarization Sensing in Cables Without Reflectors

/in /by

Polarization-based, multi-span sensing over a link without reflection-back circuits is demonstrated experimentally. It is shown that distributed reflection from Rayleigh scattering can serveas an alternative to reflectors after spatial averaging of received state-of-polarization

Read more

Robust Phase Noise Power Spectral Density Estimation Using Multi-Laser Interferometry

/in /by

We jointly estimate the phase noise power spectral densities of multiple lasers using interferometry between different combinations of laser pairs. We demonstrate a beat-frequency trackingmethod that allows under-sampling of interferometric products without phase jumps.

Read more

QoT-Driven Control and Optimization in Fiber-Optic WDM Network Systems

/in /by

This paper outlines QoT-driven optimization strategies in coherent fiber-optic WDM networks, addressing distinct transmission scenarios, QoT metrics, control-plane methodologies, and emerging trends to enhance network reliability, flexibility and capacity.

Read more

High Definition-Distributed Fiber Optic Sensing and Smart Intersection application

/in /by

Distributed fiber optics sensing is applied for traffic management in the intersection. The high-definition fiber sensing data streaming is applied as source and YOLO computer vision model isemployed for event detection classification and localization.

Read more

First City-Scale Deployment of DASs with Satellite Imagery and AI for Live Telecom Infrastructure Management

/in /by

We demonstrate real-time fiber risk assessment and dynamic network routing in live metro networks using deployed DASs, satellite imagery, and large-scale AI, achieving the first significantreduction in fiber failures in four years

Read more

Accelerating Distributed Machine Learning with AllReduce Reconfiguration Based on Optical Circuit Switching

/in /by

We propose to apply optical circuit switching to enable dynamic AllReduce reconfiguration for accelerating distributed machine learning. With simulated annealing-based optimization, theproposed AllReduce reconfiguration approach achieves 31% less average training time than existing solutions.

Read more

Where’s the Liability in the Generative Era? Recovery-based Black-Box Detection of AI-Generated Content

/in /by

The recent proliferation of photorealistic images created by generative models has sparked both excitement and concern, as these images are increasingly indistinguishable from real ones to the human eye. While offering new creative and commercial possibilities, the potential for misuse, such as in misinformation and fraud, highlights the need for effective detection methods. Current detection approaches often rely on access to model weights or require extensive collections of real image datasets, limiting their scalability and practical application in real-world scenarios. In this work, we introduce a novel black-box detection framework that requires only API access, sidestepping the need for model weights or large auxiliary datasets. Our approach leverages a corrupt-and-recover strategy: by masking part of an image and assessing the model’s ability to reconstruct it, we measure the likelihood that the image was generated by the model itself. For black-box models that do not support masked-image inputs, we incorporate a cost-efficient surrogate model trained to align with the target model’s distribution, enhancing detection capability. Our framework demonstrates strong performance, outperforming baseline methods by 4.31% in mean average precision across eight diffusion model variant datasets.

Read more