China AI Hub
Technology / Multimodal AI
Multimodal AI
Multimodal AI refers to models that process more than one input type — text, images, audio, video — in a unified model, enabling tasks like image understanding, OCR, and video analysis alongside language.
Technical background
Multimodality came in two waves: early systems pipelined separate encoders (vision tower → LLM, as in LLaVA), while newer designs train unified models over interleaved text and media. Chinese labs were early movers on open vision-language models; Qwen2-VL (2024) set the reference for open multimodal capability that many downstream projects adopted.
How it works
A vision encoder converts images or video frames into token-like embeddings aligned with the language space; the LLM attends over both. Video adds temporal structure (frame sampling or dedicated video encoders); audio follows the same pattern with audio encoders. The result is one model that can caption, OCR, reason over charts, and follow instructions about media.
Why it matters
Most enterprise documents are not text: scans, screenshots, diagrams, video. Multimodal models unlock those corpora, and are the perception layer for computer use and agentic browsing.
Chinese adoption
Multimodal support is broad across the China AI Hub database (last verified 2026-09-22). Models listing vision: DeepSeek-V4.1-Flash, Qwen3.8-Max and Qwen3.8-Flash, GLM-5.3-Flash, Kimi K3, MiniMax M3 and the Doubao Seed 2.1 series. Models listing video: Qwen3.8-Max/Flash, GLM-5.3-Flash, Kimi K3 and MiniMax M3. Company projects include Zhipu’s GLM-V/CogVideo, MiniMax’s video models and ByteDance’s Seed multimodal line.
Major Chinese companies and models
- Alibaba Cloud — Qwen3.8-Max lists vision and video; the Qwen-VL line is open.
- Moonshot AI — Kimi K3 lists vision and video; Kimi-VL is in the open project list.
- Zhipu AI — GLM-5.3-Flash lists vision and video; GLM-V, GLM-Image, GLM-ASR and CogVideo projects.
- MiniMax — MiniMax M3 lists vision and video; MiniMax H3 is a video-generation model.
- ByteDance — Doubao Seed 2.1 series lists vision (and multimodal generation via Seedream/Seedance).
Practical applications
Document understanding and OCR, screenshot-based automation, video summarization and moderation, product and defect inspection, and accessibility tooling.
Limitations
Perceptual errors persist (fine print, charts, counting); video understanding is expensive; unified models trade single-modality depth for breadth; and “multimodal” labels hide large capability variance across modalities.
Deployment considerations
Choose model per modality mix (text-only work rarely justifies multimodal cost); benchmark on your own document types; pair with OCR pre-processing where precision is critical.
Future development
Convergence of understanding and generation (omni-models), native video reasoning, and multimodal agents that see and act in the same loop.