Zhensong Zhang

Multimodal Agents

Visual understanding · Spatial intelligence · Wearable AI · GUI agents

Biography

I am a principal researcher at Huawei, working on multimodal agents. For years I studied how intelligent systems perceive and understand the visual world — across 3D and spatial perception, long-video and egocentric understanding, and human-centric generation. My focus has since shifted to how they remember, reason, and act in it. Today that means multimodal agents for personal devices — egocentric perception, always-on sensing and long-horizon memory — and I lead the interaction-data effort behind them, with GUI agents (phone and computer use) as one current focus. I value shipping as much as publishing: I led the core technology behind Huawei's sign-language digital human, which interpreted the keynote live at the Huawei Developer Conference in two consecutive years and shipped to third-party developers. Research from my team has also shipped on Huawei phones — face recognition, gaze estimation, on-device visual understanding — and into Huawei Cloud’s MetaStudio.

I obtained my Ph.D. from The Chinese University of Hong Kong in 2018, and earlier received my B.Eng. and M.S. degrees from Xidian University and the University of Chinese Academy of Sciences in 2011 and 2014, respectively.

28Patents filed

50+Publications

We are hiring — interns and full-time researchers in Multimodal Agents, GUI Agents and VLMs. The work: GUI agent data, evaluation and root-cause analysis; mid-training, SFT and post-training for multimodal agents. See what past interns published, and where they are now. Email me a CV and a line on what you want to work on.

Selected Projects & Impact

Research

Multimodal Agents for Personal Devices

Role: Initiated and now lead the research direction.

Multimodal agents that perceive and act through real devices — starting with AI glasses and wearable hardware, spanning egocentric perception and intent understanding, always-on streaming video sensing, episodic memory and spatial reasoning, and budget-aware long-video understanding. The direction now centres on interaction data for GUI agents across phone and desktop. Our team placed second in the HD-EPIC VQA Challenge 2025.

CFD Egocentric Co-Pilot PnP Clarifier ColorTrigger Map2Thought

Product

Sign-Language Digital Human

Role: Designed the core algorithm and data specification; drove both into production.

Huawei's real-time sign-language avatar, launched on stage by Richard Yu at the Huawei Developer Conference in two consecutive years. It interpreted the conference keynotes live: 3+ hours in 2022, for an audience of 10M+ online and on-site. Shipped to third-party developers as HMS Core's SignPal Kit, covering 20,000+ Chinese signs and 26 facial expressions.

HDC 2021 HDC 2022 Tech Deep-Dive

Research

3D/4D Reconstruction & Novel-View Synthesis

Role: Led the research direction.

Work across feed-forward 3D Gaussian Splatting, video-diffusion 4D reconstruction, monocular depth and geometry, and multi-view harmonization — from benchmarks to fast, in-the-wild scene capture. The on-device reconstruction prototype we built seeded REMY, a HarmonyOS 3D spatial-memory app announced at HDC 2025; our monocular-depth model won the ECCV 2022 RVC challenge.

WildAnySplat Off The Grid Charge GRVS ViDAR CHROMA SCRREAM

Research

Conversational & Interactive Digital Humans

Role: Led the research direction.

The digital-human direction I led at Huawei — spanning speech- and audio-driven co-speech gesture generation, human-motion video generation, 2D/3D avatars and talking humans, and conversational 3D virtual humans — the research behind the sign-language digital human above. Our entry won the Reproducibility Award at the GENEA Challenge 2023. Algorithms from this line were delivered by my team into Huawei Cloud's MetaStudio, whose digital-human generation and driving services were announced at HDC.Cloud 2023.

ICo3D Motion-Video Survey Co-Speech Gesture Video QPGesture DiffuseStyleGesture