Me Papers CV Research Contact Useful Link

Hsiao-Yu Fish Tung

Machine Learning Department

Carnegie Mellon University

mail: htung at

About me


I am a final-year PhD student in the Machine Learning Department at CMU, advised by Professor Katerina Fragkiadaki. My life-long goal is to build machines that can acquire common sense without human labeling and can be taught and instructed like humans: by observing how other humans complete tasks, and sometimes with the help of language.

I believe common sense comes as a byproduct of using the right biases. For example, for visual understanding, humans have a strong bias on object permanence and most people understand scenes in terms of 3D. That is why we get suprised when a magician pulls a rabbit out of an empty hat. I have developed learning-based methods that are equipped with such biases and these models show strong generality on unseen situations. I have also designed algorithms for machines to learn simple concepts by watching human teachers doing things. Using the acquired knowledge, they are able to imagine their goal, self-supervise their own progress and efficiently pick up skills. I aim at designing learning algorithms that require only natural supervisions, e.g., self-supervision from predicting their raw sensory inputs (e.g., images and videos), supervision from human demonstrations, and priors learned from raw data.

I received my M.S. in CMU MLD and my B.S. in Electrical Engineering from National Taiwan University in 2013. During my master, I worked with Professor Alex Smola on spectral method for Bayesian models and had succefully designed efficient and provable algorithms for topic modeling. In my undergraduate, I was one of the member in team "Algorithm @ National Taiwan University" led by Professor Chih-Jen Lin 's and we won the KDD CUP 2013 Championship on both Track1 and Track2.

Conference Papers

Embodied Language Grounding with Implicit 3D Visual Feature Representations [Arxiv Extension]

Mihir Prabhudesai, Hsiao-Yu Fish Tung, Syed Ashar Javed, Maximilian Sieb, Adam W. Harley, Katerina Fragkiadaki
Conference on Computer Vision and Pattern Recognition, CVPR 2020.

Learning from Unlabelled Videos Using Contrastive Predictive Neural 3D Mapping [Paper]

Adam W. Harley, Fangyu Li, Shrinidhi K. Lakshmikanth, Xian Zhou, Hsiao-Yu Fish Tung, Katerina Fragkiadaki
International Conference on Learning Representations, ICLR 2020.

Learning Spatial Common Sense with Geometry-Aware Recurrent Networks [Arxiv Extension]

Hsiao-Yu Fish Tung, Ricson Cheng, Katerina Fragkiadaki
Conference on Computer Vision and Pattern Recognition, CVPR 2019, oral.

Reward Learning from Narrated Demonstrations [Paper]

Hsiao-Yu Fish Tung, Adam W. Harley, Liang-Kang Huang, Katerina Fragkiadaki
Conference on Computer Vision and Pattern Recognition, CVPR 2018.

Self-supervised learning of motion capture [Paper] [Code] [Website]

Hsiao-Yu Fish Tung, Hsiao-Wei Tung, Ersin Yumer, Katerina Fragkiadaki
Neural Information Processing Systems, NIPS 2017, spotlight.

Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision [Paper]

Hsiao-Yu Fish Tung, Adam W. Harley, William Seto, Katerina Fragkiadaki
International Conference on Computer Vision, ICCV 2017.

Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy [Paper]

Dougal J. Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alexander J. Smola, Arthur Gretton
International Conference on Learning Representations, ICLR 2017.

Fast and Guaranteed Tensor Decomposition via Sketching [Paper]

Yining Wang, Hsiao-Yu Tung, Alexander J. Smola, Animashree Anandkumar
Neural Information Processing Systems, NIPS 2015, spotlight.

Spectral Methods for Indian Buffet Process Inference. [Paper]

Hsiao-Yu Fish Tung, Alexander J. Smola
Neural Information Processing Systems, NIPS 2014.

Novel Traffic Light Timing Adjustment Strategy Based On Genetic Algorithm. [Paper]

Hsiao-Yu Tung, Wei-Chiu Ma, and Tian-Li Yu
IEEE Congress on Evolutionary Computation, IEEE CEC 2013, Oral.

Journal Paper

Spectral Methods for Nonparametric Models [Paper]

Hsiao-Yu Fish Tung, Chao-Yuan Wu, Manzil Zaheer, Alexander J. Smola
Master Thesis, CMU MLD

Combination of Feature Engineering and Ranking Models for Paper-Author Identification in KDD Cup 2013 [Paper]

C.-L. Li, Y.-C. Su, T.-W. Lin, C.-H. Tsai, W.-C. Chang, K.-H. Huang, T.-M. Kuo, S.-W. Lin, Y.-S. Lin, Y.-C. Lu, C.-P. Yang, C.-X. Chang, W.-S. Chin, Y.-C. Juan, H.-Y. Tung, J.-P. Wang, C.-K. Wei, Felix Wu, T.-C. Yin, T. Yu, Y. Zhuang, S.-d. Lin, H.-T. Lin, C.-J. Lin
Journal of Machine Learning Research (JMLR), 2015

Effective String Processing and Matching for Author Disambiguation. [Paper]

W.-S. Chin, Y.-C. Juan, Y.-Zhuang, Felix Wu, H.-Y. Tung, T. Yu, J.-P. Wang, C.-X. Chang, C.-P. Yang, W.-C. Chang, K.-H. Huang, T.-M. Kuo, S.-W. Lin, Y.-S. Lin, Y.-\ C. Lu, Y.-C. Su, C.-K. Wei, T.- C. Yin, C.-L. Li, T.-W. Lin, C.-H. Tsai, S.-d. Lin, H.-T. Lin, C.-J. Lin
Journal of Machine Learning Research (JMLR), 2014

Resume(Short version)[ Last Update: 2017/10 ]


Carnegie Mellon University


PhD Student
Machine Learning Department, School of Computer Science

Carnegie Mellon University


Master Student
Machine Learning Department, School of Computer Science

National Taiwan University


B.S. in Electrical Engineering

Work Experience

OpenAI Inc.

2018 Summer (Expected)

Research Intern

Adobe Research

2017 Summer

Research Intern with Ersin Yumer

Google Brain, Google Inc.

2016 Summer

Software Enginnering Intern with Andrew Dai

Intel Parallel Computing Lab

2015 Summer

Research Intern with Sheng Li


2012 Summer

Summer Intern, Home Entertainment group

Honor and Awards

2018 Qualcomm QinF fellowship Finalist (Attending Final Round in mid-April !)

2018 Open Philanthropy Project AI Fellows Finalists (~10% selected for interview)

2018 Facebook Fellowship Finalists

2014, 2017 NIPS Travel Award

Member of Eta Kappa Nu

An academic honor society

2013 KDD Cup Award, Track1&2 Champion

The leading Data Mining competition in the world, organized by ACM.
Competition topic on Author-Paper recognition for data from Microsoft Academic Search

2 times NTU President Award

Awarded to top 5% of students in each department of National Taiwan University.
Rank 1/245 in 2012 Spring

NTU 2013 Student Outstanding Performance Scholarship

2012 Altera Innovate Asia FPGA Design Competition, Outstanding Achievement

Most prestigious FPGA design competition in Asia.
Project on FPGA-based gesture recognition 3-D control panel

2nd Place in NTU Green Grass Award (without a first prize winner)

NTU Art Festival Installation Art Contest

* I am from Taiwan. My Chinese name is My Chinese Name (TUNG, HSIAO-YU). Hsiao-Yu shares the same pronouciation as little fish in Chinese, so I picked Fish as my English name.

* I enjoy all kinds of music in English, Chinese, Taiwanese, Korean, Japanese. I also listen to classic music, light music, Hip-Hop and Jazz.

* I enjoy art works, especially art in the 20th century.

Contact Information


GHC-8215, Gates Hillman Center,
Machine Learning Department,
Carnegie Mellon University,
5000 Forbes Ave,
Pittsburgh, PA 15213


htung at
sfish0101 at