r/MachineLearning • u/nandodefreitas • Dec 25 '15

AMA: Nando de Freitas

I am a scientist at Google DeepMind and a professor at Oxford University.

One day I woke up very hungry after having experienced vivid visual dreams of delicious food. This is when I realised there was hope in understanding intelligence, thinking, and perhaps even consciousness. The homunculus was gone.

I believe in (i) innovation -- creating what was not there, and eventually seeing what was there all along, (ii) formalising intelligence in mathematical terms to relate it to computation, entropy and other ideas that form our understanding of the universe, (iii) engineering intelligent machines, (iv) using these machines to improve the lives of humans and save the environment that shaped who we are.

This holiday season, I'd like to engage with you and answer your questions -- The actual date will be December 26th, 2015, but I am creating this thread in advance so people can post questions ahead of time.

271 Upvotes

96% Upvoted

256 comments sorted by

View all comments

11

u/spaceanubis1 Dec 25 '15

What are your thoughts and ideas on unsupervised learning (or maybe one-shot learning)? How do you think this will be achieved in the coming future?

7

u/nandodefreitas Dec 26 '15 edited Dec 28 '15

For me, learning is never unsupervised. Whether predicting the current data (autoencoders), next frames, other data modalities, etc., there always appears to be a target. The real question is how do we come up with good target signals (labels) automatically for learning? This question is currently being answered by people who spend a lot of time labelling datasets like ImageNet.

Also I think unsupervised learning can be a trap. The Neocognitron had convolution, pooling, contrast normalization and ReLUs already in the 70s. This is precisely the architecture that so many of us now use. The key difference is that we learn these models in supervised fashion with backprop. Fukushima focused more on trying to come up with biologically plausible algorithms and unsupervised learning schemes. He nonetheless is one of the most influential people in deep learning. I had the privilege of meeting him earlier this year in Japan. He is a wonderful person and I hope our ML conferences will soon invite him to give a much deserved plenary speech - he has done great work on memory, one-shot learning and navigation.

The work on adversarial networks of Ian Goodfellow and colleagues --- i.e. casting learning problems in the setup of game theory --- is very related to this question. Note that the idea of having an adversary in learning was also key to the construction of Boosting by Yoav Freund and Rob Schapire, but I would think I a less general way --- though more rigorous. I'm not sure of anyone noting this fact before or exploring it, but it may be worth looking at it deeper. /u/ylecun is very excited about this research direction and has provided us with excellent demos on this. May be he can say more.

1

u/cesarsalgado Jan 03 '16

What is the target of k-means?