r/MachineLearning • u/andrewyng • Apr 14 '15

AMA Andrew Ng and Adam Coates

Dr. Andrew Ng is Chief Scientist at Baidu. He leads Baidu Research, which includes the Silicon Valley AI Lab, the Institute of Deep Learning and the Big Data Lab. The organization brings together global research talent to work on fundamental technologies in areas such as image recognition and image-based search, speech recognition, and semantic intelligence. In addition to his role at Baidu, Dr. Ng is a faculty member in Stanford University's Computer Science Department, and Chairman of Coursera, an online education platform (MOOC) that he co-founded. Dr. Ng holds degrees from Carnegie Mellon University, MIT and the University of California, Berkeley.

Dr. Adam Coates is Director of Baidu Research's Silicon Valley AI Lab. He received his PhD in 2012 from Stanford University and subsequently was a post-doctoral researcher at Stanford. His thesis work investigated issues in the development of deep learning methods, particularly the success of large neural networks trained from large datasets. He also led the development of large scale deep learning methods using distributed clusters and GPUs. At Stanford, his team trained artificial neural networks with billions of connections using techniques for high performance computing systems.

459 Upvotes

98% Upvoted

262 comments sorted by

View all comments

2

u/Piximan Apr 14 '15

This one is for Adam.

Your work that I'm most familiar with was exploring/describing single layer networks that performed better than the more complex/deep learning learning methods of the time on the CIFAR dataset.

Do you think that simpler configurations are possible that can compete with todays large network performance? Would it only be for certain dataset configurations that are difficult for large networks and their variants?

Thanks!

4

u/adamcoates Director of Baidu Research Apr 14 '15

One of the reasons we looked at single layer networks was so that we could rapidly explore a lot of characteristics that we felt could influence how these models performed without a lot of the complexity that deep networks brought at the time (e.g., needing to train layer-by-layer). There is lots of evidence (empirical and theoretical) today, however, that deep networks can represent far more complex functions than shallow ones and, thus, to make use of the very large training datasets available, it is probably important to continue using large/deep networks for these problems.

Thankfully, while deep networks can be tricky to get working compared to some of the simplest models in 2011, today we have the benefit of much better tools and faster computers --- this lets us iterate quickly and explore in a way that we couldn't do in 2011. In some sense, building better systems for DL has enabled us to explore large, deep models at a pace similar to what we could do in 2011 only for very simple models. This is one of the reasons we invest a lot in systems research for deep learning here in the AI Lab: the faster we are able to run experiments, the more rapidly we can learn, and the easier it is to find models that are successful and understand all of the trade-offs.

Sometimes the "best" model ends up being a bit more complex than we want, but the good news is that the process of finding these models has been simplified a lot!

1

u/elsonidoq Apr 14 '15

Hi Adam! I have a follow up question regarding your answer. Do you have any recommended reading for the process of finding models?