# Language Model 单词级 n-gram 前向神经网络语言模型 [A Neural Probabilistic Language Model](http://jmlr.org/papers/volume3/bengio03a/bengio03a.pdf) (Bengio et al., 2001; 2003) 字符级 RNN 语言模型介绍 [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) 字符级 n-gram 语言模型跟 RNN 对比 [The unreasonable effectiveness of Character-level Language Models](https://nbviewer.jupyter.org/gist/yoavg/d76121dfde2618422139) [好玩的文本生成](https://www.msra.cn/zh-cn/news/features/ruihua-song-20161226) # Word Embedding review: - https://ruder.io/word-embeddings-1/ - https://ruder.io/word-embeddings-softmax/index.html word2vec tutorial: - [skip-gram](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/) by Chris McCormick - [negative sample](http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/) by Chris McCormick word2vec paper: - [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/abs/1301.3781) - [Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546) embedding for downstream tasks: - [A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning](https://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf)(Collobert and Weston 2008) - [Natural Language Processing (almost) from Scratch](https://arxiv.org/abs/1103.0398) - http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/ distributional semantic model: - [From frequency to meaning: Vector space models of semantics](https://www.aaai.org/Papers/JAIR/Vol37/JAIR-3705.pdf) - [A Brief History of Word Embeddings](https://www.gavagai.io/text-analytics/a-brief-history-of-word-embeddings/) - https://rare-technologies.com/making-sense-of-word2vec/ - https://ruder.io/secret-word2vec/index.html - [Glove: Global Vectors for Word Representation](https://nlp.stanford.edu/pubs/glove.pdf) - [Improving Distributional Similarity with Lessons Learned from Word Embeddings](https://transacl.org/ojs/index.php/tacl/article/viewFile/570/124) sentence embedding: [Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882) [A Convolutional Neural Network for Modelling Sentences](https://www.aclweb.org/anthology/P14-1062) [paragraph vector](https://cs.stanford.edu/~quocle/paragraph_vector.pdf) [Skip-Thought Vectors](https://arxiv.org/abs/1506.06726) [Semi-supervised Sequence Learning](https://arxiv.org/abs/1511.01432) [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://arxiv.org/abs/1705.02364) https://github.com/Separius/awesome-sentence-embedding https://www.dataiku.com/product/plugins/sentence-embedding/ [Are distributional representations ready for the real world?](https://arxiv.org/abs/1705.11168) more aboue embedding: [Enriching Word Vectors with Subword Information](https://arxiv.org/pdf/1607.04606v1.pdf) https://ruder.io/word-embeddings-2017/ [A Survey of Cross-lingual Word Embedding Models](https://arxiv.org/abs/1706.04902) visual embedding: - https://distill.pub/2016/misread-tsne/ # Machine Learning [Linear Regression](http://ufldl.stanford.edu/tutorial/supervised/LinearRegression/) [Logistic Regression](http://ufldl.stanford.edu/tutorial/supervised/LogisticRegression/) [Softmax Regression](http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/) https://cs231n.github.io/linear-classify/ [Linear model by Andrew Ng](http://cs229.stanford.edu/notes/cs229-notes1.pdf) [An overview of gradient descent optimization algorithms](http://ruder.io/optimizing-gradient-descent/) # Neural Network 神经网络入门介绍 [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/index.html) by Nielsen RNN: - LSTM 介绍 [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Colah - https://distill.pub/2016/augmented-rnns/ CNN: - http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ - http://colah.github.io/posts/2014-07-Understanding-Convolutions/ - http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/ - [Dilated CNN](https://arxiv.org/abs/1610.10099) RecNN: - [RecNN](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf) # Seq2Seq From SMT to NMT - [Statistical phrase-based translation](https://www.aclweb.org/anthology/N03-1017/) - [Recurrent continuous translation models](https://www.aclweb.org/anthology/D13-1176.pdf) - [Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation](https://arxiv.org/abs/1406.1078) - [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215) - [On the properties of neural machine translation: Encoder–Decoder approaches](https://www.aclweb.org/anthology/W14-4012.pdf) Attention - [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://arxiv.org/abs/1502.03044) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473v7) - [Effective Approaches to Attention-based Neural Machine Translation](https://nlp.stanford.edu/pubs/emnlp15_attn.pdf) - https://distill.pub/2016/augmented-rnns/ Transformer: - Transformer 介绍 [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) by Jay Alammar - Transformer 论文 [Attention Is All You Need]() - Transformer [Pytorch 实现](http://nlp.seas.harvard.edu/2018/04/03/attention.html) # Natural Language Processing 基于 NLTK 库的自然语言处理实践教程 [The NLTK Book](http://www.nltk.org/book/) 基于神经网络的自然语言处理方法的历史演进 [A Review of the Neural History of Natural Language Processing](http://ruder.io/a-review-of-the-recent-history-of-nlp/) by Sebastian Ruder 基于神经网络的自然语言处理常用方法简介 [A Primer on Neural Network Models for Natural Language Processing](http://u.cs.biu.ac.il/~yogo/nnlp.pdf) by Yoav Goldberg 基于深度学习的自然语言处理技术最佳实践 [Deep Learning for NLP Best Practices](http://ruder.io/deep-learning-nlp-best-practices/index.html#attentionhttp://ruder.io/deep-learning-nlp-best-practices/index.html) [基于 Pytorch 深度学习和自然语言处理入门](https://nlp-pt.apachecn.org/) # Pretrained language models [NLP's ImageNet moment has arrived](http://ruder.io/nlp-imagenet/) [The Illustrated BERT, ELMo, and others](http://jalammar.github.io/illustrated-bert/) [Semi-supervised Sequence Learning](https://arxiv.org/abs/1511.01432) [ELMo](https://arxiv.org/abs/1802.05365) [ULMFiT](https://arxiv.org/abs/1801.06146) [OpenAI-Transformer](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) [BERT](https://arxiv.org/abs/1810.04805) [BERT Word Embeddings](http://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/) http://ruder.io/multi-task/ [auxiliary task](http://ruder.io/multi-task-learning-nlp/) https://github.com/huggingface/transformers https://github.com/hanxiao/bert-as-service # Text Classification [Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882) [A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1510.03820) http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/ https://github.com/yoonkim/CNN_sentence http://albertxiebnu.github.io/fasttext/ # Text embedding search https://github.com/facebookresearch/faiss https://engineering.fb.com/data-infrastructure/faiss-a-library-for-efficient-similarity-search/ https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch https://hanxiao.io/2019/11/22/Video-Semantic-Search-in-Large-Scale-using-GNES-and-TF-2-0/ # Tools fasttext: https://github.com/facebookresearch/fastText textcnn: https://github.com/dennybritz/cnn-text-classification-tf # Resources [深度学习 500 问](https://github.com/scutan90/DeepLearning-500-questions) [中文自然语言处理相关资料](https://github.com/crownpku/awesome-chinese-nlp) [中文自然语言处理各任务最新进展](https://chinesenlp.xyz/#/zh/) by 滴滴人工智能实验室 http://ruder.io/ http://www.hankcs.com/ [liuhuanyong 常见自然语言处理任务项目](https://liuhuanyong.github.io/)