update

A Brief History of Word Embeddings _ Gavagai.md

@@ -0,0 +1,43 @@
+One of the strongest trends in Natural Language Processing (NLP) at the moment is the use of **word embeddings**, which are vectors whose relative similarities correlate with semantic similarity. Such vectors are used both as an end in itself (for computing similarities between terms), and as a representational basis for downstream NLP tasks like text classification, document clustering, part of speech tagging, named entity recognition, sentiment analysis, and so on. That this is a trend is obvious when looking at the proceedings from the recent large conferences in NLP, e.g. [ACL](https://aclanthology.info/) or [EMNLP](https://aclweb.org/anthology/D/D15/). For the first time (ever), semantics was the dominating subject at EMNLP (“Empirical Methods in NLP”) this year. In fact, some people even suggested the conference be renamed “Embedding Methods in NLP”, due to the large amount of papers covering various types of methods, applications and evaluations for word embeddings.
+
+This is a positive trend (if you like semantics and embeddings), and there is a lot of progress being made in NLP currently. However, many recent publications (and talks) on word embeddings are surprisingly oblivious of the large body of previous work in fields like computer science, cognitive science, and computational linguistics. Apart from this being bad academic manners, it also risks delaying progress by reinventing the wheel rather than building on existing knowledge.
+
+The point of this post is to provide a brief history of word embeddings, and to summarize the current state of the art. The usual caveat applies: this is by no means meant as a complete list of all relevant research on the subject; on the contrary, it is intended as a reference and starting point for those interested in exploring the field further.
+
+First, a note on terminology. **Word embedding** seems to be the dominating term at the moment, no doubt because of the current popularity of methods coming from the [deep learning](https://en.wikipedia.org/wiki/Deep_learning) community. In computational linguistics, we often prefer the term **distributional semantic model** (since the underlying semantic theory is called [distributional semantics](https://en.wikipedia.org/wiki/Distributional_semantics)). There are also many other alternative terms in use, from the very general **distributed representation** to the more specific **semantic vector space** or simply **word space**. For consistency, I will adhere to current practice and use the term word embedding in this post.
+
+Word embeddings are based on the idea that contextual information alone constitutes a viable representation of linguistic items, in stark contrast to formal linguistics and the Chomsky tradition. This idea has its theoretical roots in structuralist linguistics and ordinary language philosophy, and in particular in the works of [Zellig Harris](https://en.wikipedia.org/wiki/Zellig_Harris), [John Firth](https://en.wikipedia.org/wiki/John_Rupert_Firth), and [Ludwig Wittgenstein](https://en.wikipedia.org/wiki/Ludwig_Wittgenstein), all publishing important works in the 1950s (in the case of Wittgenstein, posthumously). The earliest attempts at using feature representations to quantify (semantic) similarity used hand-crafted features. Charles Osgood’s [semantic differentials](https://en.wikipedia.org/wiki/Semantic_differential) in the 1960s is a good example, and similar representations were also used in early works on connectionism and artificial intelligence in the 1980s.
+
+Methods for using automatically generated contextual features were developed more or less simultaneously around 1990 in several different research areas. One of the most influential early models was [Latent Semantic Analysis/Indexing](https://en.wikipedia.org/wiki/Latent_semantic_analysis) (LSA/LSI), developed in the context of information retrieval, and the precursor of today’s **topic models**. At roughly the same time, there were several different models developed in research on artificial neural networks that used contextual representations. The most well-known of these are probably [Self Organizing Maps](https://en.wikipedia.org/wiki/Self-organizing_map) (SOM) and [Simple Recurrent Networks](https://en.wikipedia.org/wiki/Recurrent_neural_network) (SRN), of which the latter is the precursor to today’s **neural language models**. In computational linguistics, Hinrich Schütze developed models that were based on word co-occurrences, which was also used in [Hyperspace Analogue to Language](https://en.wikipedia.org/wiki/Hyperspace_Analogue_to_Language) (HAL) that was developed as a model of semantic memory in cognitive science.
+
+Later developments are basically only refinements of these early models. **Topic models** are refinements of LSA, and include methods like [probabilistic LSA](https://en.wikipedia.org/wiki/Probabilistic_latent_semantic_analysis) (PLSA) and [Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) (LDA). **Neural language models** are based on the same application of neural networks as SRN, and include architectures like [Convolutional Neural Networks](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN) and [Autoencoders](https://en.wikipedia.org/wiki/Autoencoder). **Distributional semantic models** are often based on the same type of representation as HAL, and includes models like [Random Indexing](https://en.wikipedia.org/wiki/Random_indexing) and [BEAGLE](http://www.indiana.edu/~clcl/BEAGLE/).
+
+The main difference between these various models is the type of contextual information they use. LSA and topic models use *documents* as contexts, which is a legacy from their roots in information retrieval. Neural language models and distributional semantic models instead use *words* as contexts, which is arguably more natural from a linguistic and cognitive perspective. These different contextual representations capture different types of semantic similarity; the document-based models capture **semantic relatedness** (e.g. “boat” – “water”) while the word-based models capture **semantic similarity** (e.g. “boat” – “ship”). This very basic difference is too often misunderstood.
+
+Speaking of common misunderstandings, there are two other myths that need debunking:
+
+– There is no need for *deep* neural networks in order to build good word embeddings. In fact, two of the most successful and acknowledged recent models – the Skipgram and CBoW models included in the [word2vec](https://code.google.com/p/word2vec/) library – are shallow neural networks of the same flavor as the original SRN.
+
+– There is no qualitative difference between (current) *predictive* neural network models and *count-based* distributional semantics models. Rather, they are different computational means to arrive at the same type of semantic model; several recent papers have demonstrated both theoretically and empirically the correspondence between these different types of models [[Levy and Goldberg (2014)](https://levyomer.files.wordpress.com/2014/09/neural-word-embeddings-as-implicit-matrix-factorization.pdf), [Pennington et al. (2014)](http://www.nlp.stanford.edu/pubs/glove.pdf), [Österlund et al. (2015](http://aclweb.org/anthology/D/D15/D15-1024.pdf))].
+
+So that’s a very brief history and a couple of clarifications. What about the current state of the art? The boring answer is that it depends on what task you want to solve, and how much effort you want to spend on optimizing the model. The somewhat more enjoyable answer is that you will probably do fine whichever current method you choose, since they are more or less equivalent. A good bet is to use a factorized model – either using explicit factorization of a distributional semantic model (available in e.g. the [PyDSM](https://github.com/jimmycallin/pydsm) python library or the [GloVe](http://nlp.stanford.edu/projects/glove/) implementation), or using a neural network model like those implemented in [word2vec](https://code.google.com/p/word2vec/) – since they produce state of the art results ([Österlund et al., 2015](https://www.aclweb.org/anthology/D15-1024)) and are robust across a wide range of semantic tasks ([Schnabel et al., 2015](https://www.aclweb.org/anthology/D15-1036)).
+
+For those interested in playing around with word embeddings, I recommend the following libraries:
+
+### **Standalone implementations**
+
+### – [word2vec](https://code.google.com/p/word2vec/) (also available in e.g. [Spark MLlib](http://spark.apache.org/mllib/) and [DL4J](http://deeplearning4j.org/))
+
+### – [GloVe](http://nlp.stanford.edu/projects/glove/)
+
+### **Frameworks**
+
+### – [S-Space](https://github.com/fozziethebeat/S-Space) (Java)
+
+### – [SemanticVectors](https://github.com/semanticvectors/semanticvectors) (Java)
+
+### – [Gensim](https://radimrehurek.com/gensim/) (Python)
+
+### – [PyDSM](https://github.com/jimmycallin/pydsm) (Python)
+
+### – [DISSECT](http://clic.cimec.unitn.it/composes/toolkit/) (Python)

README.md

@@ -1,3 +1,78 @@
+# Language Model
+
+单词级 n-gram 前向神经网络语言模型 [A Neural Probabilistic Language Model](http://jmlr.org/papers/volume3/bengio03a/bengio03a.pdf) (Bengio et al., 2001; 2003)
+
+字符级 RNN 语言模型介绍 [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
+
+字符级 n-gram 语言模型跟 RNN 对比 [The unreasonable effectiveness of Character-level Language Models](https://nbviewer.jupyter.org/gist/yoavg/d76121dfde2618422139)
+
+[好玩的文本生成](https://www.msra.cn/zh-cn/news/features/ruihua-song-20161226)
+
+# Word Embedding
+
+review:
+
+- https://ruder.io/word-embeddings-1/
+- https://ruder.io/word-embeddings-softmax/index.html
+
+word2vec tutorial:
+
+- [skip-gram](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/) by Chris McCormick
+- [negative sample](http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/) by Chris McCormick
+
+word2vec paper:
+
+- [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/abs/1301.3781)
+- [Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546)
+
+embedding for downstream tasks:
+
+- [A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning](https://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf)(Collobert and Weston 2008)
+
+- [Natural Language Processing (almost) from Scratch](https://arxiv.org/abs/1103.0398)
+- http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
+
+distributional semantic model:
+
+- [From frequency to meaning: Vector space models of semantics](https://www.aaai.org/Papers/JAIR/Vol37/JAIR-3705.pdf)
+- [A Brief History of Word Embeddings](https://www.gavagai.io/text-analytics/a-brief-history-of-word-embeddings/)
+- https://rare-technologies.com/making-sense-of-word2vec/
+- https://ruder.io/secret-word2vec/index.html
+- [Glove: Global Vectors for Word Representation](https://nlp.stanford.edu/pubs/glove.pdf)
+- [Improving Distributional Similarity with Lessons Learned from Word Embeddings](https://transacl.org/ojs/index.php/tacl/article/viewFile/570/124)
+
+sentence embedding:
+
+[Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882)
+
+[A Convolutional Neural Network for Modelling Sentences](https://www.aclweb.org/anthology/P14-1062)
+
+[paragraph vector](https://cs.stanford.edu/~quocle/paragraph_vector.pdf)
+
+[Skip-Thought Vectors](https://arxiv.org/abs/1506.06726)
+
+[Semi-supervised Sequence Learning](https://arxiv.org/abs/1511.01432)
+
+[Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://arxiv.org/abs/1705.02364)
+
+https://github.com/Separius/awesome-sentence-embedding
+
+https://www.dataiku.com/product/plugins/sentence-embedding/
+
+[Are distributional representations ready for the real world?](https://arxiv.org/abs/1705.11168)
+
+more aboue embedding:
+
+[Enriching Word Vectors with Subword Information](https://arxiv.org/pdf/1607.04606v1.pdf)
+
+https://ruder.io/word-embeddings-2017/
+
+[A Survey of Cross-lingual Word Embedding Models](https://arxiv.org/abs/1706.04902)
+
+visual embedding:
+
+- https://distill.pub/2016/misread-tsne/
+
 # Machine Learning
 
 [Linear Regression](http://ufldl.stanford.edu/tutorial/supervised/LinearRegression/)
@@ -6,26 +81,56 @@
 
 [Softmax Regression](http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/)
 
+https://cs231n.github.io/linear-classify/
+
+[Linear model by Andrew Ng](http://cs229.stanford.edu/notes/cs229-notes1.pdf)
+
+[An overview of gradient descent optimization algorithms](http://ruder.io/optimizing-gradient-descent/)
+
+
+
 # Neural Network
 
 神经网络入门介绍 [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/index.html) by Nielsen
 
-LSTM 介绍 [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Colah
+RNN:
 
-Transformer:
-- Transformer 介绍 [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) by Jay Alammar
-- Transformer 论文 [Attention Is All You Need]()
-- Transformer Pytorch 实现 [](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
+- LSTM 介绍 [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Colah
+- https://distill.pub/2016/augmented-rnns/
 
-[Dilated CNN](https://arxiv.org/abs/1610.10099)
+CNN:
 
-[RecNN](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
+- http://colah.github.io/posts/2014-07-Conv-Nets-Modular/
+- http://colah.github.io/posts/2014-07-Understanding-Convolutions/
+- http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/
+- [Dilated CNN](https://arxiv.org/abs/1610.10099)
 
-[Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215)
+RecNN:
 
-Attention:
+- [RecNN](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
+
+# Seq2Seq
+
+From SMT to NMT
+
+- [Statistical phrase-based translation](https://www.aclweb.org/anthology/N03-1017/)
+- [Recurrent continuous translation models](https://www.aclweb.org/anthology/D13-1176.pdf)
+- [Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation](https://arxiv.org/abs/1406.1078)
+- [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215)
+- [On the properties of neural machine translation: Encoder–Decoder approaches](https://www.aclweb.org/anthology/W14-4012.pdf)
+
+Attention
+
+- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://arxiv.org/abs/1502.03044)
 - [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473v7)
 - [Effective Approaches to Attention-based Neural Machine Translation](https://nlp.stanford.edu/pubs/emnlp15_attn.pdf)
+- https://distill.pub/2016/augmented-rnns/
+
+Transformer:
+
+- Transformer 介绍 [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) by Jay Alammar
+- Transformer 论文 [Attention Is All You Need]()
+- Transformer [Pytorch 实现](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
 
 
 
@@ -33,70 +138,82 @@ Attention:
 
 基于 NLTK 库的自然语言处理实践教程 [The NLTK Book](http://www.nltk.org/book/)
 
-# Natural Language Processing based Neural
-
 基于神经网络的自然语言处理方法的历史演进 [A Review of the Neural History of Natural Language Processing](http://ruder.io/a-review-of-the-recent-history-of-nlp/) by Sebastian Ruder
 
 基于神经网络的自然语言处理常用方法简介 [A Primer on Neural Network Models for Natural Language Processing](http://u.cs.biu.ac.il/~yogo/nnlp.pdf) by Yoav Goldberg
 
 基于深度学习的自然语言处理技术最佳实践 [Deep Learning for NLP Best Practices](http://ruder.io/deep-learning-nlp-best-practices/index.html#attentionhttp://ruder.io/deep-learning-nlp-best-practices/index.html)
 
+[基于 Pytorch 深度学习和自然语言处理入门](https://nlp-pt.apachecn.org/)
 
-# Language Model
 
-单词级 n-gram 前向神经网络语言模型 [A Neural Probabilistic Language Model](http://jmlr.org/papers/volume3/bengio03a/bengio03a.pdf) (Bengio et al., 2001; 2003)
-
-字符级 RNN 语言模型介绍 [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
-
-[好玩的文本生成](https://www.msra.cn/zh-cn/news/features/ruihua-song-20161226)
-
-# Word Embedding
-
-http://ruder.io/word-embeddings-2017/
-
-word2vec paper:
-- [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/abs/1301.3781)
-- [Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546)
-
-word2vec tutorial:
-
-- [skip-gram](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/) by Chris McCormick
-- [negative sample](http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/) by Chris McCormick
-
-[Glove: Global Vectors for Word Representation](https://nlp.stanford.edu/pubs/glove.pdf)
-
-[paragraph vector](https://cs.stanford.edu/~quocle/paragraph_vector.pdf)
-
-[A Convolutional Neural Network for Modelling Sentences](https://www.aclweb.org/anthology/P14-1062)
-
-[Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882)
-
-[Skip-Thought Vectors](https://arxiv.org/abs/1506.06726)
-
-[A Survey of Cross-lingual Word Embedding Models](https://arxiv.org/abs/1706.04902)
 
 # Pretrained language models
 
-https://thegradient.pub/nlp-imagenet/
+[NLP's ImageNet moment has arrived](http://ruder.io/nlp-imagenet/)
+
+[The Illustrated BERT, ELMo, and others](http://jalammar.github.io/illustrated-bert/)
 
 [Semi-supervised Sequence Learning](https://arxiv.org/abs/1511.01432)
 
-# Multi-task learning
+[ELMo](https://arxiv.org/abs/1802.05365)
+
+[ULMFiT](https://arxiv.org/abs/1801.06146)
+
+[OpenAI-Transformer](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)
+
+[BERT](https://arxiv.org/abs/1810.04805)
+
+[BERT Word Embeddings](http://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/)
 
 http://ruder.io/multi-task/
 
-[A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning](https://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf)(Collobert and Weston 2008)
-
 [auxiliary task](http://ruder.io/multi-task-learning-nlp/)
 
-# Books&Blog
+https://github.com/huggingface/transformers
+
+https://github.com/hanxiao/bert-as-service
+
+# Text Classification
+
+[Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882)
+
+[A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1510.03820)
+
+http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/
+
+https://github.com/yoonkim/CNN_sentence
+
+http://albertxiebnu.github.io/fasttext/
+
+# Text embedding search
+
+https://github.com/facebookresearch/faiss
+
+https://engineering.fb.com/data-infrastructure/faiss-a-library-for-efficient-similarity-search/
+
+https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch
+
+https://hanxiao.io/2019/11/22/Video-Semantic-Search-in-Large-Scale-using-GNES-and-TF-2-0/
+
+# Tools
+
+fasttext: https://github.com/facebookresearch/fastText
+
+textcnn: https://github.com/dennybritz/cnn-text-classification-tf
+
+
+
+# Resources
 
 [深度学习 500 问](https://github.com/scutan90/DeepLearning-500-questions)
 
 [中文自然语言处理相关资料](https://github.com/crownpku/awesome-chinese-nlp)
 
-[中文自然语言处理](https://chinesenlp.xyz/#/zh/) by 滴滴人工智能实验室
+[中文自然语言处理各任务最新进展](https://chinesenlp.xyz/#/zh/) by 滴滴人工智能实验室
 
 http://ruder.io/
 
 http://www.hankcs.com/
+
+[liuhuanyong 常见自然语言处理任务项目](https://liuhuanyong.github.io/)