Collaborators

      

Leo Guibas, Stanford University Marcus Rohbrach  Research Scientist at Facebook AI Research (FAIR),  co-author  Devi Parikh, Georgia Tech/FAIR MPK,   co-author Dhruv Batra, Georgia Tech/FAIR MPK,   co-author Camille Couprie, FAIR Paris,   co-author Othman Sbai, FAIR Paris,   co-author Antoine Bordes, FAIR Paris,   co-author Yann LeCun,  Professor of Computer Scienc at NYU and Director of AI Research at Facebook. Sayna Ebrahimi, PhD Student at UC Berkeley,  co-author Othman Sbai, University of Paris;  co-author Tinne Tuytelaars, Professor of Computer Science at KULauven;  co-author Ahmed Elgammal,  Professor of Computer Science at Rutgers University:  MS & PhD advisor.  Dimitris Metaxas, Professor of Computer Science at Rutgers University: co-author Harpreet Sawhney(CTO SRI International): Mentor at SRI International,   co-author Hui Cheng (Program Director, SRI International): Mentor at SRI International,   co-author Ji Zhang, Rutgers, co-author Ethan Zhu, Rutgers, co-author' Mennatullah Siam, University of Alberta;  co-author Tao Xu, Lehigh University: co-author;  co-author

Trevor Darrel, Professor at UC Berkeley,  co-author Philip Torr, University of Oxford Arslan Chaudry,University of Oxford  Scott Cohen, Adobe Research Brian Price, Adobe Research Walter Chang, Adobe Research Manohar Paluri, Facebook Research Yannis Kalantidis, Facebook Research   Jingen Liu (Senior Research Scientist, SRI International): Mentor at SRI International,   co-author Xiaolei Huang (Associate Professor of Computer Science and Engineering at Lehigh University):  co-author Shaoting Zhang (Associate Professor of Computer Science at  UNC at Charlotte):  co-author Scott Cohen (Principal Scientist at Adobe Research): Mentor at Adobe Research,   co-author Brian Price (Senior Research Scientist at Adobe Research):  Mentor at Adobe Research, co-author Walter Chang (Principal Scientist at  Adobe Research): Mentor at Adobe Research, co-author Sheng Huang, Chongqing University: co-author Han Zhang, Rutgers University: co-author Hossam Faheem (Ain Shams University, Egypt); co-author

Research Statement: Imagination Inspired Computer Vision

Imagination is one of the key properties of human intelligence that enables us not only to generate creative products like art and music but also to understand the visual world. My research focuses mostly on developing imagination-inspired techniques that empower AI machines to see (computer vision) or to create (e.g., fashion and art); “Imagine to See” and “Imagine to Create”. 

Imagine to See. There are over 10,000 living bird species, yet most computer vision datasets of birds have only 200-500 categories. Typically, there are few images available for training classifiers for most of these categories (long-tail classes) and hence the number of training images per category shows a Zipf distribution [25]. How could imagination help understand visual classes with zero/few examples? Many people might not know what “Parakeet Auklet” is but can imagine it when described in language by saying that “Parakeet Auklet is a bird that has an orange bill, dark above and white below.”. If we give this description to an average person, he will be able to select the relevant bird among other different birds, due to our capability to imagine the “Parakeet Auklet” class from the language description. I have worked on setting up tasks that are inspired by imagination to study visual recognition of unseen classes and long-tail classes guided by language. Since ICCV2013, I have proposed and pioneered the “Write a Classifier” task [8,13,16,17,44] which got recently recognized at the United Nations conference on biodiversity[45]. Recently expanding from the “Sherlock” model and dataset I proposed at AAAI17 [20,21,46], I have been working on developing methods with a deeper understanding and a capacity to learn an ever-growing set of concepts [23,24] (covered by the media); see the figure (left). Learning millions of these concepts impacts not only our relevant content experiences on Google and Facebook but also gets us closer to helpful robots that are continually learning like us about the visual world. 

Imagine to Create. Imagination has been the source of novel ideas that enable humanity to progress at an ever-faster rate. Creative AI is a relatively understudied direction in machine learning where the goal is for machines to generate original items with realistic, aesthetic and/or thoughtful attributes, usually in artistic contexts. In the short term, Creative AI has a high potential to speed up our rate of generating creative products like paintings, music, animations, etc. as a source of inspiration. As I detail later, I have worked on modeling Creative AI to produce art [18] and fashion [19]; see the figure (right). Our pioneering creativity work grabbed attention from the scientific community, media, and industry. One of the exciting results we achieved in [19] is that our model was able to create new pants with additional arm sleeves (non-existing in the dataset). The surprising aspect of this design is that professional fashion designers found it inspirational for designing new pants, showing how creativity may impact the fashion industry. I am also excited about future exploration of Creative AI in producing 3D models, videos, and animation. It also may help imagine unseen likely dangerous situations to make self-driving cars more reliable.

Key Projects

Sherlock (2016-present)    Goal Scalable Fact Learning images (objects, attributes, actions, and interactions), publications at AAAI2017  and ACLW2016 Write a Classifier (2013-present) Goal  Language Guided Visual Perception of object categories by pure text articles.  publications  at ICCV 2013, CVPRW 2015, EMNLPW 2015, TPAMI 2016, ongoing work.

• Write a Classifier Project: Zero Shot Learning from Pure Text description for Fine-Grained Categories, ICCV13, ICCVW13, TPAMI15(to be submitted).
• Hypergraph based approach for Zero-Shot Learning, N-Shot Learning, and Attribute Prediction, CVPR 2015
• Image Classification and Matching by local descriptors (e.g. SIFT), ICIP13

• Joint Category and Pose Recognition with Deep Neural Networks, submitted
• Weather Classification using Convolutional Neural Networks, ICIP, 2015

•  Structured Regression: Structured Regression Methods applied to 3D Pose Estimation, Image Reconstruction, Toy Examples, Machine Learning Journal (accepted).

• English2MindMap:   Pure Text In- Hierarchical Pictorial Visualization Out, SWWS09, ISM12, MTAP15. 
• Undergraduate-Research Mentored Projects (2007-2011),

•  Activity Recognition and Motion Characterization, CVPRW13
•  Undergraduate-Research Mentored Projects (2007-2011), IPCV09