Steve Jurvetson: Deep Learning

black_boxbeyond_engineeringDeep Learning: Intelligence from Big Data
Sep 16, 2014
Stanford Graduate School of Business
vlabvideos
https://www.youtube.com/watch?v=czLI3oLDe8M

A machine learning approach inspired by the human brain, Deep Learning is taking many industries by storm. Empowered by the latest generation of commodity computing, Deep Learning begins to derive significant value from Big Data. It has already radically improved the computer’s ability to recognize speech and identify objects in images, two fundamental hallmarks of human intelligence.

Industry giants such as Google, Facebook, and Baidu have acquired most of the dominant players in this space to improve their product offerings. At the same time, startup entrepreneurs are creating a new paradigm, Intelligence as a Service, by providing APIs that democratize access to Deep Learning algorithms.

Moderator
Steve Jurvetson, Partner, DFJ Ventures

Panelists
Adam Berenzweig, Co-founder and CTO, Clarifai
Naveen Rao, Co-founder and CEO, Nervana Systems
Elliot Turner, Founder and CEO, AlchemyAPI
Ilya Sutskever, Research Scientist, Google Brain

19:24 Subsystem inscrutability. Black box.

20:04 Beyond Engineering
Danny Hillis, The Pattern on the Stone: “The greatest achievement of our technology may well be the creation of tools that allow us to go beyond engineering–that allow us to create more than we can understand”

33:40 Convolutional Neural Networds

34:10 current deep learning model: recognizes 10,000 categories (classes).
average adult: 90,000 words (in English, not counting personal names)

1:09:00 we can propagate a signal with almost 0 loss. Biology can’t.
Will AI look like human intelligence? The underlying structure is different (substrate: silicon vs. neurons). With constraints, AI’s behavior will look however we want.

==========

http://www.npr.org/sections/alltechconsidered/2014/02/20/280232074/deep-learning-teaching-computers-to-tell-things-apart

state of the art in reverse image search (01/01/2015):
the original (and only) source of the following image is placed low in results:
https://1-ps.googleusercontent.com/sxk/rMkKc5ekI1lndxMkjJEGNPjM35/www.edsurge.com/d3e7x39d4i7wbe.cloudfront.net/uploads/photo/image/1878/x1-1419622484.jpg.pagespeed.ic.z4DSB56envp7OAryUx7p.png

Deep learning

Deep Learning
Deterministic Neural Networks
Zhirong Wu
http://vision.princeton.edu/courses/COS598/2014sp/slides/lecture05_cnn/lecture05_cnn.pdf

from:
COS598C Spring 2014: Scene Understanding

MIT Technology Review: 10 Breakthrough Technologies 2013

see also:
In the beginning was the code: Juergen Schmidhuber
TEDxUHasselt, 2013
https://www.youtube.com/watch?v=T1Ogwa76yQo

http://en.wikipedia.org/wiki/Deterministic_algorithm

Stochastic neural networds
http://neuron.eng.wayne.edu/tarek/MITbook/chap8/8_3.html

An approach to predicting non-deterministic neural network behavior
2005
http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=1556389&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D1556389

Non-Deterministic Learning Dynamics in Large Neural Networks due to Structural Data Bias (2000)
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.31.9712

Nondeterministic Discretization of Weights Improves Accuracy of Neural Networks
Marcin Wojnarski

https://medium.com/backchannel/the-deep-mind-of-demis-hassabis-156112890d8a

When You’re Talking or Typing, AI is There

Structure Data 2014: When You’re Talking or Typing, AI is There
Apr 11, 2014
Despite the hard work that goes into building systems for deep learning and other methods of understanding human language, users might never know they’re powering their favorite apps. And that’s kind of the point.
https://www.youtube.com/watch?v=5qcAOkNOX5c

ambience search
context about the user
11:35 written in Python, …
12:40 latency … real-time learning
13:50 terabite on the device … Google’s entire knowledge graph