CATR-Net: A Cross-Attention-Driven Transformer–CNN Framework with Bidirectional Feature Fusion for High-Accuracy Image Recognition

Abstract

Introduction: Convolutional neural networks (CNNs) encode strong local inductive biases but capture global context poorly, whereas vision transformers (ViTs) model long-range dependencies yet are data-hungry and weak at fine spatial detail. Hybrid CNN–transformer models attempt to reconcile these properties, but the majority couple the two streams through sequential stacking or one-directional attention, so the interaction between local and global evidence remains shallow and the fusion weights are fixed rather than input-adaptive.

Objectives: This study designs a dual-branch architecture in which convolutional and transformer representations are exchanged symmetrically and combined through a data-dependent gate, with the aim of raising recognition accuracy without a disproportionate increase in cost.

Methodology and proposed model: We propose CATR-Net, a framework whose parallel ConvNeXt-style local branch and Swin-style global branch are linked by a Bidirectional Cross-Attention Fusion (BCAF) block. In BCAF, convolutional queries attend to transformer keys and values and, simultaneously, transformer queries attend to convolutional keys and values; the mutually enhanced descriptors are then merged by an Adaptive Gated Fusion module that predicts per-channel mixing coefficients. The network is trained end-to-end with label-smoothed cross-entropy.

Results: On CIFAR-100 and CUB-200-2011, CATR-Net attains 86.8% and 91.2% top-1 accuracy, improving over the strongest recent baseline (MaxViT) by 1.7 and 1.1 percentage points, respectively, and over CoAtNet by 1.9 and 1.6 points, while retaining a comparable parameter budget (28.6 M) and 4.7 GFLOPs. Macro-averaged F1 reaches 0.865 and 0.909, and an ablation confirms that bidirectionality and gated fusion each contribute measurable gains.

Conclusion: Symmetric cross-branch attention with adaptive gating is an effective mechanism for high-accuracy recognition.

Real-time applications—The framework is applicable to fine-grained visual inspection, biomedical image screening, intelligent surveillance, and on-device retrieval where both discriminative detail and global scene understanding are required.