30 papers
This website is based on a rumoured list of papers that Ilya Sutskever gave to John Carmack. We currently only have a list of 27. If you or anyone you know has the full, canonical list please feel free to reach out.
Contributors
Andrej Karpathy
Fei-Fei Li- +many other researchers
CS231n: Convolutional Neural Networks for Visual Recognition
The course notes that teach convolutional networks from first principles, from linear classifiers up to deep architectures for images.
CS231n: Convolutional Neural Networks for Visual Recognition
The course notes that teach convolutional networks from first principles, from linear classifiers up to deep architectures for images.
Andrej KarpathyFei-Fei Li Plus many more
Contributors
- Andrej Karpathy
The Unreasonable Effectiveness of Recurrent Neural Networks
A hands on blog post that trains character level RNNs to generate text and shows, with vivid examples, how much structure they capture.
The Unreasonable Effectiveness of Recurrent Neural Networks
A hands on blog post that trains character level RNNs to generate text and shows, with vivid examples, how much structure they capture.
Andrej Karpathy
Contributors
Christopher Olah
Understanding LSTM Networks
The clearest visual explanation of how LSTM gates carry information across long sequences, widely used as a first introduction.
Understanding LSTM Networks
The clearest visual explanation of how LSTM gates carry information across long sequences, widely used as a first introduction.
Christopher Olah
Contributors
Alex Krizhevsky
Ilya Sutskever
Geoffrey Hinton
ImageNet Classification with Deep Convolutional Neural Networks
AlexNet. The convolutional network that won ImageNet by a wide margin and set off the modern deep learning era.
ImageNet Classification with Deep Convolutional Neural Networks
AlexNet. The convolutional network that won ImageNet by a wide margin and set off the modern deep learning era.
Alex KrizhevskyIlya Sutskever
Contributors
Kaiming He
Xiangyu Zhang
Shaoqing Ren- JS Jian Sun
Deep Residual Learning for Image Recognition
ResNet. Introduces residual connections that let networks grow to hundreds of layers by learning changes to the input rather than full transformations.
Deep Residual Learning for Image Recognition
ResNet. Introduces residual connections that let networks grow to hundreds of layers by learning changes to the input rather than full transformations.
Kaiming HeXiangyu Zhang
Contributors
- FY Fisher Yu
Vladlen Koltun
Multi-Scale Context Aggregation by Dilated Convolutions
Shows how dilated convolutions expand the receptive field without losing resolution, which sharpened dense prediction tasks like segmentation.
Multi-Scale Context Aggregation by Dilated Convolutions
Shows how dilated convolutions expand the receptive field without losing resolution, which sharpened dense prediction tasks like segmentation.
FY Fisher YuVladlen Koltun
Contributors
- Kaiming He
- Xiangyu Zhang
- Shaoqing Ren
- JS Jian Sun
Identity Mappings in Deep Residual Networks
A follow up to ResNet that studies why identity shortcuts work so well and proposes a cleaner pre-activation residual block.
Identity Mappings in Deep Residual Networks
A follow up to ResNet that studies why identity shortcuts work so well and proposes a cleaner pre-activation residual block.
Kaiming HeXiangyu Zhang
Contributors
Wojciech Zaremba- Ilya Sutskever
Oriol Vinyals
Recurrent Neural Network Regularization
Shows how to apply dropout to LSTMs correctly, on the non-recurrent connections, so large recurrent models stop overfitting.
Recurrent Neural Network Regularization
Shows how to apply dropout to LSTMs correctly, on the non-recurrent connections, so large recurrent models stop overfitting.
Wojciech ZarembaIlya Sutskever
Contributors
Dario Amodei- RA Rishita Anubhai
- +many other researchers
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
An end to end speech recognition system trained with connectionist temporal classification that worked across two very different languages.
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
An end to end speech recognition system trained with connectionist temporal classification that worked across two very different languages.
Dario Amodei RA Rishita Anubhai Plus many more
Contributors
- Oriol Vinyals
Samy Bengio- MK Manjunath Kudlur
Order Matters: Sequence to Sequence for Sets
Examines how the order of inputs and outputs affects sequence to sequence models, and how to handle data that is really a set.
Order Matters: Sequence to Sequence for Sets
Examines how the order of inputs and outputs affects sequence to sequence models, and how to handle data that is really a set.
Oriol VinyalsSamy Bengio
Contributors
Dzmitry Bahdanau
Kyunghyun Cho
Yoshua Bengio
Neural Machine Translation by Jointly Learning to Align and Translate
Introduces the attention mechanism, letting a translation model look back at the relevant source words instead of a single fixed summary.
Neural Machine Translation by Jointly Learning to Align and Translate
Introduces the attention mechanism, letting a translation model look back at the relevant source words instead of a single fixed summary.
Dzmitry BahdanauKyunghyun Cho
Contributors
- Oriol Vinyals
Meire Fortunato
Navdeep Jaitly
Pointer Networks
A sequence model whose outputs point back at positions in the input, which suits problems whose answer is a selection or ordering of the inputs.
Pointer Networks
A sequence model whose outputs point back at positions in the input, which suits problems whose answer is a selection or ordering of the inputs.
Oriol VinyalsMeire Fortunato
Contributors
Ashish Vaswani
Noam Shazeer- +many other researchers
Attention Is All You Need
The Transformer. Replaces recurrence entirely with self attention, the architecture that underpins almost every modern large language model.
Attention Is All You Need
The Transformer. Replaces recurrence entirely with self attention, the architecture that underpins almost every modern large language model.
Ashish VaswaniNoam Shazeer Plus many more
Contributors
Sasha Rush- +many other researchers
The Annotated Transformer
A line by line, runnable reimplementation of the Transformer that turns the original paper into working, readable code.
The Annotated Transformer
A line by line, runnable reimplementation of the Transformer that turns the original paper into working, readable code.
Sasha Rush Plus many more
Contributors
Alex Graves- GW Greg Wayne
- ID Ivo Danihelka
Neural Turing Machines
Couples a neural network to an external memory it can read and write with differentiable attention, learning simple algorithms from examples.
Neural Turing Machines
Couples a neural network to an external memory it can read and write with differentiable attention, learning simple algorithms from examples.
Alex Graves GW Greg Wayne
Contributors
Adam Santoro- DR David Raposo
- +many other researchers
A Simple Neural Network Module for Relational Reasoning
Introduces the relation network, a small plug in module that lets a network reason about how pairs of objects relate to each other.
A Simple Neural Network Module for Relational Reasoning
Introduces the relation network, a small plug in module that lets a network reason about how pairs of objects relate to each other.
Adam Santoro DR David Raposo Plus many more
Contributors
- Adam Santoro
- RF Ryan Faulkner
- +many other researchers
Relational Recurrent Neural Networks
Adds a self attention based memory to recurrent networks so that stored memories can interact, improving tasks that need relational reasoning over time.
Relational Recurrent Neural Networks
Adds a self attention based memory to recurrent networks so that stored memories can interact, improving tasks that need relational reasoning over time.
Adam Santoro RF Ryan Faulkner Plus many more
Contributors
Justin Gilmer- SS Samuel S. Schoenholz
- PF Patrick F. Riley
- Oriol Vinyals
- D Dahl
Neural Message Passing for Quantum Chemistry
Unifies many graph neural networks under a message passing framework and applies it to predicting molecular properties.
Neural Message Passing for Quantum Chemistry
Unifies many graph neural networks under a message passing framework and applies it to predicting molecular properties.
Justin Gilmer SS Samuel S. Schoenholz
Contributors
Jared Kaplan- SM Sam McCandlish
- +many other researchers
Scaling Laws for Neural Language Models
Measures how language model loss falls as a smooth power law in model size, data, and compute, the empirical basis for building ever larger models.
Scaling Laws for Neural Language Models
Measures how language model loss falls as a smooth power law in model size, data, and compute, the empirical basis for building ever larger models.
Jared Kaplan SM Sam McCandlish Plus many more
Contributors
- YH Yanping Huang
- YC Youlong Cheng
- +many other researchers
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
A pipeline parallelism library that splits a giant model across devices and keeps them busy, making it practical to train very large networks.
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
A pipeline parallelism library that splits a giant model across devices and keeps them busy, making it practical to train very large networks.
YH Yanping Huang YC Youlong Cheng Plus many more
Contributors
- Geoffrey Hinton
- DV Drew van Camp
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
An early information-theoretic argument that good networks are ones whose weights can be described with few bits, linking generalization to compression.
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
An early information-theoretic argument that good networks are ones whose weights can be described with few bits, linking generalization to compression.
Geoffrey Hinton DV Drew van Camp
Contributors
Peter Grunwald
A Tutorial Introduction to the Minimum Description Length Principle
A readable introduction to choosing models by how well they compress the data, treating learning as finding the shortest description.
A Tutorial Introduction to the Minimum Description Length Principle
A readable introduction to choosing models by how well they compress the data, treating learning as finding the shortest description.
Peter Grunwald
Contributors
Scott Aaronson
The First Law of Complexodynamics
A blog essay asking for a formal law that explains why the complexity of a closed system rises, peaks, and falls, rather than simply tracking entropy.
The First Law of Complexodynamics
A blog essay asking for a formal law that explains why the complexity of a closed system rises, peaks, and falls, rather than simply tracking entropy.
Scott Aaronson
Contributors
- Scott Aaronson
Sean M. Carroll- LO Lauren Ouellette
Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton
Uses a simple cellular automaton model of coffee mixing with cream to ask why complexity rises and then falls as a system moves toward equilibrium.
Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton
Uses a simple cellular automaton model of coffee mixing with cream to ask why complexity rises and then falls as a system moves toward equilibrium.
Scott AaronsonSean M. Carroll
Contributors
Thomas M. Cover- JA Joy A. Thomas
Kolmogorov Complexity
The textbook treatment of the shortest program that produces a string, the formal backbone behind description length and algorithmic randomness.
Kolmogorov Complexity
The textbook treatment of the shortest program that produces a string, the formal backbone behind description length and algorithmic randomness.
Thomas M. Cover JA Joy A. Thomas
Contributors
- XC Xi Chen
Diederik P. Kingma- +6 6 more researchers
Variational Lossy Autoencoder
Combines variational autoencoders with autoregressive decoders, and shows how to control which information the latent code is forced to keep.
Variational Lossy Autoencoder
Combines variational autoencoders with autoregressive decoders, and shows how to control which information the latent code is forced to keep.
XC Xi ChenDiederik P. Kingma Plus 6 more
Contributors
Shane Legg
Machine Super Intelligence
A doctoral thesis that proposes a formal, universal measure of machine intelligence and explores its consequences for very capable agents.
Machine Super Intelligence
A doctoral thesis that proposes a formal, universal measure of machine intelligence and explores its consequences for very capable agents.
Shane Legg