In visible-SAR cross-modal remote sensing target detection tasks, due to the significant differences between the two modalities in imaging mechanisms and feature representations, as well as the susceptibility of SAR images to speckle noise interference and the inadequate utilization of structural information, existing methods often struggle to balance detection accuracy with model lightweighting and practical deployment requirements. This is particularly true in resource-constrained scenarios, such as space-borne or airborne platforms, where models need to have low parameter counts, reduced computational overhead, and robust adaptability to complex environments. To address these issues, this paper proposes a lightweight dual-branch fusion network, DBF-YOLO, for visible-SAR cross-modal remote sensing target detection. Based on the YOLOv10 framework, the method constructs a visible-SAR dual-branch feature extraction structure and designs an intermediate cross-modal fusion path at three levels (P3, P4, P5) to achieve progressive interaction of dual-modal features at different scales. To overcome the strong speckle noise and edge degradation in SAR images, a SAR Gradient Enhancement Module (SGM) is introduced to enhance the structural representation capability of SAR inputs. Additionally, an Adaptive Gated Dual-Modal Fusion Module (AGD) is proposed to enable dynamic selection and effective complementarity of dual-modal information based on different scales and spatial positions. Experimental results on the OGSOD 1.0 dataset show that DBF-YOLO achieves 94.4% mAP50% and 70.1% mAP50-95, with only 5.0 M parameters and 20.5 GFLOPs, striking a good balance between detection accuracy and computational complexity. Furthermore, experimental results on the OSPRC dataset demonstrate the robustness of DBF-YOLO under different resolutions, polarization modes, and cloud cover conditions. Particularly in low-contrast dense target scenes and cloud cover conditions, the model demonstrates stronger environmental adaptability. This method can provide a reference for designing lightweight multi-modal remote sensing target detection models and deploying them on resource-limited platforms.