Exploring The Current State of Multimodal Alignment and Fusion

School of Science, The Hong Kong University of Science and Technology, Hong Kong, China

This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

Multimodal alignment and fusion technology is the core driving force for the transformation of artificial intelligence from single-modal perception to multimodal cognition. This technology has shown great application potential in the fields of medicine, transportation, etc. This paper aims to deeply explore the technical evolution, application innovation and key challenges of multimodal alignment and fusion. This paper concludes that dynamic time warping (DTW), contrastive learning (CLIP-like) and causal reasoning constitute the three major milestones of technological development. Among them, the accuracy of medical multimodal diagnosis (96.2%±1.5%) and the fusion accuracy of autonomous driving (mAP=0.912, nuScenes benchmark) significantly surpass the single-modal method. However, modal heterogeneity, computational efficiency bottlenecks and fragmentation of evaluation systems are still the main bottlenecks. This paper proposes future directions such as photonic chip fusion and standardization system construction, which provide important references for theoretical innovation and industrial implementation of multimodal technology, and have far-reaching significance for promoting the development of general artificial intelligence.