Shuaiyi Huang

I'm a PhD candidate in the department of Computer Science at University of Maryland, College Park, advised by Prof. Abhinav Shrivastava.

I obtained my M.S. degree in Computer Science from ShanghaiTech University in 2020, advised by Prof. Xuming He. I received my B.Eng. degree in Software Engineering from Tongji University in 2017.

My research lies in the intersection of Computer Vision and Autonomous Agents, with a focus on video understanding, object recognition, multimodal learning, and generative world modeling. I aim to build foundation models that seamlessly integrate visual, linguistic, and action understanding for real-world applications.

I am looking for full-time jobs in industry. Please feel free to contact me!

Email  /  CV  /  Scholar  /  LinkedIn  /  Github

Experiences

• Research Intern, Generative World Modeling at NVIDIA May - Aug, 2023
  Work on video diffusion for robot manipulation.
  Mentor: De-An Huang, Linxi "Jim" Fan, and Yuke Zhu


• Research Intern, Video Understanding and Object Recognition at NVIDIA May - Nov, 2022
  Work on video instance segmentation with point supervision.
  Mentor: Zhiding Yu, De-An Huang, Shiyi Lan

Publications

Preference feedback collected by human or VLM annotators is often noisy, presenting a significant challenge for preference-based reinforcement learning. To address this challenge, we propose TREND, a novel framework that integrates few-shot expert demonstrations with a tri-teaching strategy for effective noise mitigation.

In this work, we introduce visual trace prompting, a simple yet effective approach to facilitate Vision Language Action Models' spatial- temporal awareness for action prediction. visually.

A benchmark framework that automatically generates hallucination cases in Vision-Language models to evaluate their robustness and accuracy.

We propose a novel method for universal policy learning via active region video diffusion models, focusing on task-critical regions in videos.

This work explores the impact of point-level supervision in the context of video instance segmentation, offering insights into its effectiveness.

We propose an unsupervised approach for video instance segmentation, leveraging self-supervised learning methods to improve object instance tracking and segmentation across video frames.

We propose D-NeRV, a novel implicit neural representation based framework designed to encode large-scale and diverse videos. It achieves state-of-the-art performances on video compression.

We address the challenge of label sparsity in semantic correspondence by enriching supervision signals from sparse keypoint annotations. We first propose a teacher-student learning paradigm for generating dense pseudo-labels and then develop two novel strategies for denoising pseudo-labels.

This paper explores a confidence-aware adversarial learning framework to enhance self-supervised semantic matching with improved robustness and accuracy.

This work presents real-world benchmark datasets, evaluation criteria, and baseline approaches for assessing dehazing methods in image processing.

We introduce a Dynamic Context Correspondence Network (DCCN) to improve semantic alignment by leveraging dynamic feature contexts across images.

This paper provides a real-world benchmark dataset for defogging, a new evaluation criterion, and baseline approaches to assess defogging techniques.

This work proposes structured attention mechanisms for visual question answering, enabling more precise reasoning over complex visual scenes.

Services

• Reviewers: CVPR, ECCV, ICCV, IJCV, TIP, WACV, ACCV, ICRA

Thank Dr. Jon Barron for sharing the source code of his personal page.