We design a Fusion-Mamba block (FMB) to map cross-modal features into a hidden state space for interaction, thereby reducing disparities between cross-modal features and enhancing the representation consistency of fused features. FMB contains two modules: the State Space Channel Swapping (SSCS) ...
3.2 Dynamic vision state space module Inspired by EVMamba [23], we propose the DVSS mod- ule as a modification of the SSM block for image fusion process. In Fig. 2(b), starting with the input deep feature, we initially apply LayerNorm (LN) followed by the efficient state space module...
Secondly, to effectively combine spatial and spectral information, we extend the Mamba block to accommodate dual inputs. This expansion leads to the creation of a new module called the FusionMamba block, which outperforms existing fusion techniques such as concatenation and cross-attention. We ...
class VSSBlock(nn.Module): def __init__( self, hidden_dim: int = 0, drop_path: float = 0, norm_layer: Callable[..., torch.nn.Module] = partial(nn.LayerNorm, eps=1e-6), attn_drop_rate: float = 0, d_state: int = 16, **kwargs, ): super().__init__() self.ln_1 ...
The network integrates a Multi-Head Mamba Block (MH-Mamba Block) for enhanced multi-scale feature representation, an Adaptive Boundary Enhancement Fusion Module (ABEFM) for improved boundary-aware feature fusion, and an edge-detail auxiliary training branch to capture fine-grained details. The ...
最后,通过扫描合并S6 Block的输出以创建最终的 2D 特征图。 SS2D 目的:扫描扩展用于生成不同的排列方式或视角来捕获图像块的多样信息,以提高后续融合过程的效果。 步骤:每个输入块根据不同的扫描顺序(如从左到右、从上到下、对角线等)被重新排列成新的序列。这些排列有助于模型从不同的视角提取特征,提高数据的...
The implementation of the Multi-modal Mamba Block (M3 Block) is located in this file. 3.Checkpoint Checkpoints arelocated in the folder ./Model/Infrared_Visible_Fusion/Infrared_Visible_Fusion/models 4.Data Preparation MSRS, RoadScene, M3FD, Harvard medical dataset Download the Infrared-Visible ...
To capitalize on the distinct attributes of the multi-modal remote sensing data from both branches, a feature fusion block (FFB) is designed to synergistically enhance and integrate the features extracted from the dual-branch structure at each stage. Extensive experiments on the Vaihingen and the ...
In the following sections, we will present results from ablation experiments to justify the selection of the c2f module as the foundational block. These experiments will demonstrate that the network exhibits superior performance when utilizing the c2f module compared to other blocks, such as c3, bo...
This method designs a cross-scanning visual state space block (CVSSBlock) that uses cross 2D scanning (CS2D) to fully capture global information from multiple directions, while by incorporating convolutional neural network branches to overcome the constraints of Vision Mamba (VMamba) in acquiring ...