Re-ranking the Context for Multimodal Retrieval Augmented Generation

/in /by

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating external knowledge to generate a response within a context with improved accuracy and reduced hallucinations. However, multi-modal RAG systems face unique challenges: (i) the retrieval process may select irrelevant entries to user query (e.g., images, documents), and (ii) vision-language models or multi-modal language models like GPT-4o may hallucinate when processing these entries to generate RAG output. In this paper, we aim to address the first challenge, i.e, improving the selection of relevant context from the knowledge-base in retrieval phase of the multi-modal RAG. Specifically, we leverage the relevancy score (RS) measure designed in our previous work for evaluating the RAG performance to select more relevant entries in retrieval process. The retrieval based on embeddings, say CLIP-based embedding, and cosine similarity usually perform poorly particularly for multi-modal data. We show that by using a more advanced relevancy measure, one can enhance the retrieval process by selecting more relevant pieces from the knowledge-base and eliminate the irrelevant pieces from the context by adaptively selecting up-to-?? entries instead of fixed number of entries. Our evaluation using COCO dataset demonstrates significant enhancement in selecting relevant context and accuracy of the generated response.

Read more

PPDiff: Diffusing in Hybrid Sequence-Structure Space for Protein-Protein Complex Design

/in /by

Designing protein-binding proteins with high affinity is critical in biomedical research and biotechnology. Despite recent advancements targeting specific proteins, the ability to create high-affinity binders for arbitrary protein targets on demand, without extensive rounds of wet-lab testing,remains a significant challenge. Here, we introduce PPDiff, a diffusion model to jointly design the sequence and structure of binders for arbitrary protein targets in a non-autoregressive manner. PPDiff builds upon our developed Sequence Structure Interleaving Network with Causal attention layers (SSINC), which integrates interleaved self-attention layers to capture global amino acid correlations, k-nearest neighbor (kNN) equivariant graph layers to model local interactions in three-dimensional (3D) space, and causal attention layers to simplify the intricate interdependencies within the protein sequence. To assess PPDiff, we curate PPBench, a general protein complex dataset comprising 706,360 complexes from the Protein Data Bank (PDB). The model is pretrained on PPBench and finetuned on two real-world applications: target-protein mini-binder complex design and antigen-antibody complex design. PPDiff consistently surpasses baseline methods, achieving success rates of 50.00%, 23.16%, and 16.89% for the pretraining task and the two downstream applications, respectively.

Read more

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion

/in /by

Symmetry in the parameter space of deep neural networks (DNNs) has proven beneficial for various deep learning applications. A well-known example is the permutation symmetry in Multi-Layer Perceptrons (MLPs), where permuting the rows of weight matrices in one layer and applying the inverse permutation to adjacent layers yields a functionally equivalent model. While permutation symmetry fully characterizes the equivalence set for MLPs, its discrete nature limits its utility for transformers. In this paper, we introduce rotation symmetry, a novel form of parameter space symmetry for transformers that generalizes permutation symmetry by rotating parameter matrices in self-attention layers. Unlike permutation symmetry, rotation symmetry operates in a continuous domain, thereby significantly expanding the equivalence set for transformers. Based on this property, we propose a theoretically optimal parameter matching algorithm as a plug-and-play module to enhance model fusion. We evaluate our approach using pre-trained transformers across diverse natural language and vision tasks. Experimental results demonstrate that our rotation symmetry based matching algorithm substantially improves model fusion, highlighting the potential of parameter space symmetry to facilitate model fusion. Our code is available on https://github.com/zhengzaiyi/RotationSymmetry.

Read more

Pathologist-Read vs AI-Driven Assessment of Tumor-Infiltrating Lymphocytes in Melanoma

/in /by

Tumor-infiltrating lymphocytes (TILs) are a provocative biomarker in melanoma, influencing diagnosis, prognosis, and immunotherapy outcomes; however, traditional pathologistreadTIL assessment on hematoxylin and eosin–stained slides is prone to interobserver variability, leading to inconsistent clinical decisions. Therefore, development of newer TIL scoring approachesthat produce more reliable and consistent readouts is important.

Read more

Toward Intelligent and Efficient Optical Networks: Performance Modeling, Co-existence, and Field Trials

/in /by

Optical transmission networks require intelligent traffic adaptation and efficient spectrum usage. We present scalable machine learning (ML) methods for network performance modeling, andfield trials of distributed fiber sensing and classic optical network traffic coexistence.

Read more

Span-based Polarization Sensing in Cables Without Reflectors

/in /by

Polarization-based, multi-span sensing over a link without reflection-back circuits is demonstrated experimentally. It is shown that distributed reflection from Rayleigh scattering can serveas an alternative to reflectors after spatial averaging of received state-of-polarization

Read more

Robust Phase Noise Power Spectral Density Estimation Using Multi-Laser Interferometry

/in /by

We jointly estimate the phase noise power spectral densities of multiple lasers using interferometry between different combinations of laser pairs. We demonstrate a beat-frequency trackingmethod that allows under-sampling of interferometric products without phase jumps.

Read more

QoT-Driven Control and Optimization in Fiber-Optic WDM Network Systems

/in /by

This paper outlines QoT-driven optimization strategies in coherent fiber-optic WDM networks, addressing distinct transmission scenarios, QoT metrics, control-plane methodologies, and emerging trends to enhance network reliability, flexibility and capacity.

Read more

High Definition-Distributed Fiber Optic Sensing and Smart Intersection application

/in /by

Distributed fiber optics sensing is applied for traffic management in the intersection. The high-definition fiber sensing data streaming is applied as source and YOLO computer vision model isemployed for event detection classification and localization.

Read more

First City-Scale Deployment of DASs with Satellite Imagery and AI for Live Telecom Infrastructure Management

/in /by

We demonstrate real-time fiber risk assessment and dynamic network routing in live metro networks using deployed DASs, satellite imagery, and large-scale AI, achieving the first significantreduction in fiber failures in four years

Read more