Hsiao-Yu (Fish) Tung 童筱妤

I am looking for AI-related positions in the industry! I was a postdoc at MIT BCS working with Josh Tenenbaum and Dan Yamins. I work on self-supervised 3D perception and common sense learning for embodied agents. I study how 3D perception can improve the way machines see, act, reason, and understand language, and how machines can acquire and improve their 3D perception and common sense knowledge through their interactions with the physical world. I received my PhD from CMU Machine Learning Department under the supervision of Katerina Fragkiadaki. I received my M.S. in ML from CMU and my B.S. in Electrical Engineering from National Taiwan University. I worked with Professor Alex Smola on spectral method for Bayesian models during my masters.

contact: sfish0101 AT gmail DOT com
CV/ Google scholar / Twitter



2018 Summer

2017 Summer

2016 Summer

2015 Summer



Recent News



3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes
Haotian Xue, Antonio Torralba, Joshua Tenenbaum, Daniel Yamins, Yunzhu Li, Hsiao-Yu Fish Tung
CVPR 2023 Precognition Workshop / Paper


Conference Papers

Physion++: Evaluating Physical Scene Understanding with Objects Consisting of Different Physical Attributes in Humans and Machines
Hsiao-Yu Fish Tung*, Mingyu Ding*, Zhenfang Chen*, Sirui Tao, Vedang Lal, Daniel Bear, Gan Chuang, Joshua B Tenenbaum, Daniel L. K. Yamins, Judith E. Fan, Kevin Smith
CogSci 2023 / VSS 2023 / Paper (Coming Soon)

3D View Prediction Models of the Dorsal Visual Stream
Gabriel Sarch, Hsiao-Yu Fish Tung, Aria Wang, Jacob Prince, Michael Tarr
CCN 2023 / Paper (Coming Soon)

Tactile-Filter: Interactive Tactile Perception for Part Mating
Kei Ota, Devesh K Jha, Hsiao-Yu Fish Tung, Joshua B Tenenbaum
RSS 2023 / Paper / Video

Fluidlab: A differentiable environment for benchmarking complex fluid manipulation
Zhou Xian, Bo Zhu, Zhenjia Xu, Hsiao-Yu Fish Tung, Antonio Torralba, Katerina Fragkiadaki, Chuang Gan
ICLR 2023 / Paper / Website / Code

H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions
Kei Ota, Hsiao-Yu Fish Tung, Kevin A. Smith, Anoop Cherian, Tim K. Marks, Alan Sullivan, Asako Kanezaki, Joshua B. Tenenbaum
ICRA 2023 / Paper /

Visually-Grounded Library of Behaviors for Manipulating Diverse Objects across Diverse Configurations and Views
Jingyun Yang*, Hsiao-Yu Fish Tung*, Yunchu Zhang*, Gaurav Pathak, Ashwini Pokle, Christopher G Atkeson, Katerina Fragkiadaki
CoRL 2021 / Paper / Website

Physion: Evaluating Physical Prediction from Vision in Humans and Machines
Daniel M. Bear*, Elias Wang*, Damian Mrowca*, Felix J. Binder*, Hsiao-Yu Fish Tung, R.T. Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher Joshua B. Tenenbaum Daniel L.K. Yamins Judith E. Fan
NeurIPS 2021, Datasets and Benchmarks Track / Arxiv / Code / Code for the particle-based dynamics models

Disentangling 3D Prototypical Networks for Few-Shot Concept Learning
Mihir Prabhudesai*, Shamit Lal*, Darshan Patil*, Hsiao-Yu Fish Tung, Adam W Harley, Katerina Fragkiadaki
ICLR 2021 / Arxiv / project page

HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworks
Zhou Xian, Shamit Lal, Hsiao-Yu Fish Tung, Emmanouil Antonios Platanios, Katerina Fragkiadaki
ICLR 2021 / paper

3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators
Hsiao-Yu Fish Tung*, Zhou Xian*, Mihir Prabhudesai, Shamit Lal, Katerina Fragkiadaki
CoRL 2020 / paper / project page

Embodied Language Grounding with Implicit 3D Visual Feature Representations
Mihir Prabhudesai*, Hsiao-Yu Fish Tung*, Syed Ashar Javed*, Maximilian Sieb, Adam W. Harley, Katerina Fragkiadaki
CVPR 2020 / paper / supp / project page / ArXiv Extension

Learning from Unlabelled Videos Using Contrastive Predictive Neural 3D Mapping
Adam W. Harley, Fangyu Li, Shrinidhi K. Lakshmikanth, Xian Zhou, Hsiao-Yu Fish Tung, Katerina Fragkiadaki,
ICLR 2020 / paper / project page

Learning Spatial Common Sense with Geometry-Aware Recurrent Networks
Hsiao-Yu Fish Tung*, Ricson Cheng*, Katerina Fragkiadaki
CVPR 2019 / paper / project page / code
Oral presentation

Reward Learning from Narrated Demonstrations
Hsiao-Yu Fish Tung, Adam W. Harley, Liang-Kang Huang, Katerina Fragkiadaki
CVPR 2018 / paper

Self-supervised learning of motion capture
Hsiao-Yu Fish Tung, Hsiao-Wei Tung, Ersin Yumer, Katerina Fragkiadaki
NIPS 2017 / paper / project page / code
Spotlight presentation

Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision
Hsiao-Yu Fish Tung*, Adam W. Harley*, William Seto*, Katerina Fragkiadaki
ICCV 2017 / paper / code

Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
Dougal J. Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alexander J. Smola, Arthur Gretton
ICLR 2017 / paper

Fast and Guaranteed Tensor Decomposition via Sketching
Yining Wang, Hsiao-Yu Tung, Alexander J. Smola, Animashree Anandkumar
NIPS 2015 / paper
Spotlight presentation

Spectral Methods for Indian Buffet Process Inference
Hsiao-Yu Fish Tung, Alexander J. Smola
NIPS 2014 / paper

Novel traffic signal timing adjustment strategy based on Genetic Algorithm
Hsiao-Yu Tung*, Wei-Chiu Ma*, Tian-Li Yu
CEC 2014 / paper
Oral presentation


Journal Papers

Spectral Methods for Nonparametric Models
Hsiao-Yu Fish Tung, Chao-Yuan Wu, Manzil Zaheer, Alexander J. Smola
Master Thesis / Arxiv

Combination of Feature Engineering and Ranking Models for Paper-Author Identification in KDD Cup 2013
C.-L. Li, Y.-C. Su, T.-W. Lin, C.-H. Tsai, W.-C. Chang, K.-H. Huang, T.-M. Kuo, S.-W. Lin, Y.-S. Lin, Y.-C. Lu, C.-P. Yang, C.-X. Chang, W.-S. Chin, Y.-C. Juan, H.-Y. Tung, J.-P. Wang, C.-K. Wei, Felix Wu, T.-C. Yin, T. Yu, Y. Zhuang, S.-d. Lin, H.-T. Lin, C.-J. Lin
JMLR 2015 / paper

Effective String Processing and Matching for Author Disambiguation
W.-S. Chin, Y.-C. Juan, Y.-Zhuang, Felix Wu, H.-Y. Tung, T. Yu, J.-P. Wang, C.-X. Chang, C.-P. Yang, W.-C. Chang, K.-H. Huang, T.-M. Kuo, S.-W. Lin, Y.-S. Lin, Y.-\ C. Lu, Y.-C. Su, C.-K. Wei, T.- C. Yin, C.-L. Li, T.-W. Lin, C.-H. Tsai, S.-d. Lin, H.-T. Lin, C.-J. Lin
JMLR 2014 / paper



  • I am from Taiwan. My Chinese name is My Chinese Name (TUNG, HSIAO-YU). Hsiao-Yu shares the same pronouciation as little fish in Chinese, so I picked Fish as my English name.