Research
1. 3D segmentation from 2D multi-view images
3D segmentation from 2D multi-view images, refers to segmenting the 3D structure of arbitrary objects from multi-view 2D images, enabling the rapid and convenient creation of 3D assets. Its capability facilitates applications such as robotic navigation and embodied intelligence simulations. However, this task is much more difficult than 2D image segmentation, requiring solutions to challenges such as establishing cross-view consistent constraints between pixels. We achieve this task by lifting the Segment Anything Model (SAM) from 2D to 3D with the assistance of radiance fields, such as NeRFs and 3DGS, which are known as efficient representations to connect 2D multi-view images with 3D structures.
Publications
- Jiazhong Cen, Jiemin Fang, Zanwei Zhou, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian. Segment Anything in 3D with Radiance Fields. International Journal of Computer Vision, 133(8): 5138-5160, 2025. (PDF)
- Chen Yang, Sikuang Li, jiemin Fang, Ruofan Liang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian. GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting. ACM Trans. Graphics (Siggraph Asia), 43(6)， 2024. (PDF)
- Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian. Segment Anything in 3D with NeRFs. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, USA, 2023. (PDF) (CODE)
- Jiazhong Cen, Xudong Zhou, Jiemin Fang, Changsong Wen, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian. Tackling View-Dependent Semantics in 3D Language Gaussian Splatting. International Conference on Machine Learning (ICML), Vancouver, Canada, 2025. (PDF)
- Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian. Segment Any 3D Gaussians. AAAI Conference on Artificial Intelligence (AAAI), Philadelphia, USA, 2025. (PDF)
2. Tissue reconstruction from endoscopic surgery videos
Reconstructing deformable tissues from endoscopic videos in robotic surgery is crucial for various clinical applications, such as intraoperative assistance, surgery simulation and training. We propose highly-efficient tissue reconstruction methods based on NeRFs and 3DGS, which not only significantly accelerate the reconstruction process as well as the rendering process (up to real-time), but also maintain or even improve the reconstruction quality across a variety of non-rigid deformations. Our work received Young Scientist Award of MICCAI 2023.
Publications
- Chen Yang, Kailing Wang, Yuehao Wang, Qi Dou, Xiaokang Yang, Wei Shen. Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane. IEEE Trans. Medical Imaging, 43(9): 3211-3223, 2024. (PDF)
- Chen Yang, Kailing Wang, Yuehao Wang, Xiaokang Yang, Wei Shen. Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Vancouver, Canada, 2023. [Young Scientist Award][STAR Award] (PDF)
- Kailing Wang, Chen Yang, Yuehao Wang, Sikuang Li, Yan Wang, Qi Dou, Xiaokang Yang, Wei Shen. EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries using Gaussian Splatting. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Marrakesh, Morocco, 2024. (PDF)