Word2Vec原理之层次Softmax算法

在这个人工智能半边天,主流话题一天一变的现代世界里,不乏进步,也不乏泡沫,但Word2Vec依然是很富极客精神的典型代表,作者思路很巧妙,工程层面又很实用主义。

词嵌入的假设,是通过一个词所在的上下文可以获得词的语义甚至语法结构,有相似上下文的词在向量空间中是邻近的点。

1 背景概念

Word2Vec要解决的根本问题,说到底是自然语言处理的问题。一般认为自然语言处理模型氛围两大派系,分别是形式文法和统计语言模型。

1.1 统计语言模型

统计语言模型在分词、词性标注、语义理解等NLP基本任务,以及语音识别、机器翻译等领域有广泛应用。关于统计语言模型,文献资料非常多,这篇博客也讲的很详细。简单地说,可以用统一地形式化表示:

\[p(S)=p(w_1,w_2,…,w_n)=\prod_{i=1}^{n} p(w_i|w_1,w_2,…,w_{i-1})\]

上述表示可以看做是用来计算词构成一个句子的概率模型,这就是统计语言模型的本质。但是概率的计算是个大问题,数据稀疏、参数过多,很难实际运用。前辈们研究了各种模型。例如n-gram、n-pos、决策树模型、最大熵模型等。

1.1.1 n-gram

基于马尔科夫假设,即认为每个词只与前面\(n-1\)个词(上下文)有关。例如3-gram:

\[p(S)=p(w_1) p(w_2|w_1) p(w_3|w_1,w_2)…p(w_n|w_1,w_2,…,w_{n-1}) \\
=p(w_1)p(w_2|w_1)p(w_3|w_1,w_2)…p(w_n|w_{n-2},w_{n-1})\]

至于\(n\)的选择,在NLP任务中,普遍认为只需使用2或3,而概率的计算实质为:

\[p(w_i|w_1,w_2,…,w_{i-1})=\frac {count(w_1,w_2,…,w_{i-1},w_i)} {count(w_1,w_2,…,w_{i-1})}\]

1.1.2 神经网络语言模型(NNLM)

Distributed Representation的代表,每个输入词都是一个实数向量,而模型包含输入层、投影层、隐藏层和输出层。这个直接读Bengio2003的论文《A Neural Probabilistic Language Model》。

1.1.3 Log-Bilinear模型

虽然与Word2Vec关系紧密,但与本文话题暂时不相干,所以暂不展开。可参考Hinton2007年的论文[5]

1.2 词向量

一般认为词向量作为概率语言模型的重要组成部分,其普及也主要是《A neural probabilistic language model》中用到的一种工具,简单说来任意词\(w\),可以用一个固定长度的实值向量\(v\)来表示,这个\(v\)就是\(w\)的词向量。

具体地,词如何用向量来表示?通常认为有两种基本思路,一种称为One-hot Representation,而另一种叫做Distributed Representation。

One-hot的表示方法,非常稀疏, 每个词就是一个很长的向量,向量的维度就是词表的大小,而只有该词对应位置的数字是1,其他都是0。这种表示方法无法捕捉词与词之间的相似关系,且容易发生维数灾难。

Distributed的表示方法,Hinton在1986年提出,大致思想是用较低维度的实数向量来表示词,而用词之间的距离(例如余弦距离)来表示相似度。普遍认为这种方法能够较好地捕捉词与词之间的语义相似度。

许多模型或方法可以用来获取词向量,除了word2vec使用的模型外,LSA矩阵分解模型、pLSA潜在语义分析模型、LDA文档生成模型,也均可用来估计词向量。

例如一个一个size为30的词向量,可能是类似如下的样子:

2 数据结构

2.1 Huffman编码

Huffman编码是一种用于无损数据压缩的熵编码(权编码)算法。由大卫·霍夫曼在1952年发明。该方法通过一种变长编码表来对符号进行编码,而变长编码表是通过评估符号出现概率的方法得到,目标是出现概率高的符号使用较短的编码,出现概率低的符号则使用较长的编码。这种方式能有效降低编码之后的字符串的平均长度。

在谈论数据结构的时候,Huffman编码又称为最优二叉树,表示一种带权路径长度最短的二叉树。带权路径长度,指的就是叶子结点的权值乘以该结点到根结点的路径长度。

因此最短长度问题被转化为构建一个由字符(每个字符都是叶子结点)的出现频率作为权值所产生的Huffman树的问题。

详细解释参见维基百科。Huffman树的结构类似下图:

625px-Huffman_tree_2.svg

2.2 构造Huffman树

算法描述:假设有n个权值,则构造出来的Huffman树有n个叶子结点。若n个权值分别为{\(w_1\),\(w_2\),…,\(w_n\)}。

  1. 将{\(w_1\),\(w_2\),…,\(w_n\)}当做\(n\)棵树(每棵树1个结点)组成的森林。
  2. 选择根结点权值最小的两棵树,合并,获得一棵新树,且新树的根结点权值为其左、右子树根结点权值之和。词频大的结点作为左孩子结点,词频小的作为右孩子结点。
  3. 从森林中删除被选中的树,保留新树。
  4. 重复2、3步,直至森林中只剩下一棵树为止。

3 模型概要

3.1 CBOW

CBOW(Continuous Bag-of-Words Model):在已知上下文,例如\(w_{t-2}\),\(w_{t-1}\),\(w_{t+1}\),\(w_{t+2}\)的前提下,如何预测当前词\(w_t\)。学习的目标函数是最大化对数似然函数:

\[\sum_{w\in C}logp(w|Context(w))\]

3.2 Skip-gram

Skip-gram(Continuous Skip-gram Model):在已知当前词\(w_t\)的前提下,预测其上下文,例如\(w_{t-2}\),\(w_{t-1}\),\(w_{t+1}\),\(w_{t+2}\)。目标函数形如:

\[\sum_{w\in C}logp(Context(w)|w)\]

3.3 模型比较

CBOW和Skip-gram均为三层神经网络(输入层、投影层和输出层),而上下文的词的具体个数是可定义的。窗口大小为5时,两个模型的网络结构如下:

同样用于计算概率值,从模型的计算方式看,skip-gram想要预测更多(上下文),一次会更比CBOW慢一些,但有观点认为对低频词效果更好一些。

4 算法策略

然而,上面的公式计算很不切实际,因为计算\(logp(w_{t+j}|w_{t})\)的梯度与\(W\)的大小直接相关。为了降低复杂度,Word2Vec使用了Hierarchical Softmax和Negative Sampling两种求解策略。普遍认为Hierarchical Softmax对低频词效果较好;Negative Sampling对高频词效果较好,向量维度较低时效果更好。

4.1 Hierarchical Softmax

作为一种计算高效的近似方法,Hierarchical Softmax被广泛使用。Morin和Bengio[1]首先将这种方法引入神经网络语言模型。该方法不用为了获得概率分布而评估神经网络中的\(W\)个输出结点,而只需要评估大约\(log_2(W)\)个结点。层次Softmax使用一种二叉树结构来表示词典里的所有词,\(V\)个词都是二叉树的叶子结点,而这棵树一共有\(V-1\)个非叶子结点。

对于每个叶子结点(词),总有一条从根结点出发到该结点的唯一路径。这个路径很重要,因为要靠它来估算这个词出现的概率。以下图为例(图来自Xin Rong,2014[6]),白色结点为词典中的词,深色是非叶子结点。图中画出了从根结点到词\(w_2\)的唯一路径,路径长度\(L(w_2)\)=4,而\(n(w,j)\)表示从根结点到词\(w_2\)的路径上的的第\(j\)个结点。

在层次Softmax模型中,叶子节点的词没有直接输出的向量,而非叶子节点其实都有响应的输出向量。

4.1.1 求目标词的概率

在模型的训练过程中,通过Huffman编码,构造了一颗庞大的Huffman树,同时会给非叶子结点赋予向量。我们要计算的是目标词\(w\)的概率,这个概率的具体含义,是指从root结点开始随机走,走到目标词\(w\)的概率。因此在途中路过非叶子结点(包括root)时,需要分别知道往左走和往右走的概率。例如到达非叶子节点\(n\)的时候往左边走和往右边走的概率分别是:
\[
p(n,left) =
\sigma({\theta_n}^T \cdot h)
\]

\[
p(n,right) =
1- \sigma({\theta_n}^T \cdot h) =
\sigma({-\theta_n}^T \cdot h)
\]

以上图中目标词为\(w_2\)为例,

\[
p(w_2) = p(n(w_2,1),left) \cdot p(n(w_2,2),left) \cdot p(n(w_2,3),right) \\
= \sigma({\theta_{n(w_2,1)}}^T \cdot h) \cdot \sigma({\theta_{n(w_2,2)}}^T \cdot h) \cdot
\sigma({-\theta_{n(w_2,3)}}^T \cdot h)
\]

到这里可以看出目标词为\(w\)的概率可以表示为:

\[p(w)=\prod_{j=1}^{L(w)-1}\sigma( sign(w,j)\cdot {\theta_{n(w,j)}}^Th )\]

其中\(\theta_{n(w,j)}\)是非叶子结点\(n(w,j)\)的向量表示(即输出向量);\(h\)是隐藏层的输出值,从输入词的向量中计算得来;\(sign(x,j)\)是一个特殊函数定义:
\[
sign(w,j)=
\begin{cases}
1, & 若n(w,j+1)是n(w,j)的左孩子 \\
-1,& 若n(w,j+1)是n(w,j)的右孩子
\end{cases}
\]

此外,所有词的概率和为1,即:

\[
\sum_{i=1}^np(w_i)=1
\]

4.1.2 参数更新公式

\[\theta^{(new)}_j = \theta^{(old)}_j – \eta(\sigma({\theta_j}^Th)-t_j)h\]

其中,\(j = 1,2,…,L(w)-1\)

4.2 Negative Sampling

Hierarchical Softmax的另一个替代方法叫做Noise Contrastive Estimation(NCE),Gutmann和Hyvarinen[2]介绍了这种方法,而Mnih和Teh[3]将该方法应用到了语言模型中。然而,Skip-gram模型主要关注如何学习出更高质量的矢量表示。因此可以在确保矢量质量的情况下,尽量地简化NCE。

这部分内容这里暂时不展开。

5 预处理技巧

以下根据源代码来看下两个初始化环节。

5.1 词典构建

拿到sentences以后首先是创建词典,即从文本序列中创建一个词典。词典数据是一个词对应一个出现次数,词典的规模是可以定义的,有一个参数叫做max_vocab_size即是用来限定最大词典量的。因为这是创建词典的过程,一般希望出现频率较高的词被保留,而频率较低的词被略去。所以统计的过程中,如果当前词汇数超过max_vocab_size,则需要按略去词频低于min_reduce的词,初始化的时候,min_reduce为1。随着实际词典的增大,min_reduce有可能会递增。

5.2 SubSampling

全部扫过一遍词典后,需要再次考虑删去总体频率较低的词(出现次数少于min_count的长尾词)。不过,最重要的还是Subsampling。

SubSampling有时候被称作DownSampling,也曾有人译作亚采样,实际是对高频词进行随机采样,关于随机采样的选择问题,考虑高频词往往提供相对较少的信息,因此可以将高于特定词频的词语丢弃掉,以提高训练速度。Mikolov在论文指出这种亚采样能够带来2到10倍的性能提升,并能够提升低频词的表示精度。

采样参数sample表示采样百分比,默认是1e-3,Google推荐的是1e-3至1e-5。

\[P(w_i) = 1 – \sqrt{t \over f(w_i)}\]

其中\(f(w_i)\)是词\(w_i\)的词频,t是阈值。而这个是Mikolov论文里的说法,实际Word2Vec的代码,以及后续gensim的实现,都采用了如下公式来表示词\(w_i\)被丢弃的概率:

\[P(w_i) = 1-(\sqrt{v_{w_i} \over {sample * N_W}} + 1)* {sample * N_W \over v_{w_i}}\]

或者这样写看起来简洁一些:

\[ P(w_i)=1-(\sqrt{sample*N_W \over v_{w_i}} + {sample*N_W \over v_{w_i}})\]

其中:\(N_W\)是参与训练的单词总数,包含重复单词,实质即词频累加;\(v_{w_i}\)是词\(w_i\)的词频。在gensim的实现中,对\(sample<1\)和\(sample \ge1\)的情况区分对待,具体可去看gensim版源码。

参考

[1] Frederic Morin and Yoshua Bengio. Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics, pages 246–252, 2005.
[2] Michael U Gutmann and Aapo Hyv¨arinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. The Journal of Machine Learning Research, 13:307–361, 2012.
[3] Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. arXiv preprint arXiv:1206.6426, 2012.
[4] Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR).
[5] A. Mnih and G. Hinton. Three new graphical models for statistical language modelling. Proceedings of the 24th international conference on Machine learning, pages 641–648, 2007
[6] Rong X. word2vec parameter learning explained[J]. arXiv preprint arXiv:1411.2738, 2014.

149 thoughts on “Word2Vec原理之层次Softmax算法

  1. Evelyn Serrell说道:

    This is a memo to the webmaster. I discovered your Word2Vec原理之层次Softmax算法 | 一灯@qiancy.com page by searching on Google but it was hard to find as you were not on the front page of search results. I know you could have more traffic to your site. I have found a website which offers to dramatically improve your website rankings and traffic to your site: http://hdmh.info/37 I managed to get close to 500 visitors/day using their service, you could also get many more targeted traffic from search engines than you have now. Their services brought significantly more traffic to my site. I hope this helps!

  2. find here说道:

    I just want to mention I am just very new to blogging and definitely loved this web page. Very likely I’m going to bookmark your site . You really come with terrific articles. Regards for revealing your website.

  3. MetroClick specializes in building completely interactive products like Photo Booth for rental or sale, Touch Screen Kiosks and Digital Signage, and experiences. With our own hardware production facility and in-house software development teams, we are able to achieve the highest level of customization and versatility for Photo Booths, Touch Screen Kiosks and Digital Signage. MetroClick, 121 Varick St, #301, New York, NY 10013, +1 646-843-0888

  4. kumahasiaweh说道:

    Hi there, I found your site by the use of Google at the same time as looking for a comparable subject, your site came up, it appears great. I’ve bookmarked it in my google bookmarks.

  5. faytech=说道:

    Faytech North America is a touch screen Manufacturer of both monitors and pcs. They specialize in the design, development, manufacturing and marketing of Capacitive touch screen, Resistive touch screen, Industrial touch screen, IP65 touch screen, touchscreen monitors and integrated touchscreen PCs. Contact them at http://www.faytech.us, 121 Varick Street,3rd Floor,New York, NY 10013,+1 646 205 3214

  6. dentysta gorzów说道:

    Hi are using WordPress for your site platform? I’m new to the blog world but I’m trying to get started and create my own. Do you require any html coding knowledge to make your own blog? Any help would be greatly appreciated!

  7. Great site. Lots of helpful info here. I am sending it to some buddies ans additionally sharing in delicious. And of course, thank you in your effort!

  8. you can try here说道:

    It truly is practically not possible to see well-updated viewers on this subject, regrettably you come across as like you are familiar with what exactly you’re covering! Thank You

  9. Fairly useful elements that you have said, thanks a lot for writing.

  10. dentysta gorzów说道:

    Hi there! I know this is kinda off topic however I’d figured I’d ask. Would you be interested in exchanging links or maybe guest writing a blog post or vice-versa? My website covers a lot of the same topics as yours and I feel we could greatly benefit from each other. If you’re interested feel free to send me an email. I look forward to hearing from you! Wonderful blog by the way!

  11. I have recently started a site, the information you provide on this site has helped me tremendously. Thank you for all of your time & work.

  12. What i do not realize is in fact how you’re no longer really a lot more smartly-appreciated than you might be right now. You’re very intelligent. You know thus considerably in the case of this subject, made me personally consider it from a lot of numerous angles. Its like women and men aren’t interested except it’s something to do with Woman gaga! Your own stuffs outstanding. At all times handle it up!

  13. dentysta gorzów说道:

    Fantastic goods from you, man. I have understand your stuff previous to and you’re just too magnificent. I really like what you’ve acquired here, really like what you’re stating and the way in which you say it. You make it entertaining and you still care for to keep it sensible. I can’t wait to read far more from you. This is actually a tremendous site.

  14. It happens to be perfect opportunity to create some desires for the long run. I’ve go through this write-up and if I could, I want to propose you handful worthwhile tips and advice.

  15. dig this说道:

    I merely intend to inform you that I am new to blogging and clearly loved your website. Likely I am most likely to remember your blog post . You literally have extraordinary article materials. Get Pleasure From it for expressing with us your url information

  16. I simply could not leave your web site before suggesting that I extremely loved the usual information a person provide in your visitors? Is gonna be again frequently in order to check up on new posts

  17. Utterly indited articles , regards for information .

  18. flowers Ireland说道:

    What’s up, I check your new stuff daily. Your writing style is awesome, keep up the good work!

  19. Exceedingly useful highlights you have said, thanks so much for setting up.

  20. dentysta gorzów说道:

    Very nice post. I just stumbled upon your blog and wished to say that I’ve really enjoyed surfing around your blog posts. In any case I’ll be subscribing to your rss feed and I hope you write again very soon!

  21. Hey, you used to write wonderful, but the last few posts have been kinda boring… I miss your super writings. Past several posts are just a bit out of track! come on!

  22. Thanks for sharing excellent informations. Your web-site is very cool. I am impressed by the details that you¡¦ve on this site. It reveals how nicely you perceive this subject. Bookmarked this web page, will come back for extra articles. You, my friend, ROCK! I found just the info I already searched everywhere and simply could not come across. What a great web-site.

  23. As soon as I noticed this site I went on reddit to share some of the love with them.

  24. I just could not leave your site prior to suggesting that I actually loved the usual info an individual supply for your guests? Is going to be back incessantly in order to check up on new posts

  25. Hi there! This article couldn’t be written any better! Reading through this article reminds me of my previous roommate! He constantly kept talking about this. I’ll send this information to him. Pretty sure he’ll have a great read. Thank you for sharing!

  26. I enjoy you because of your entire effort on this web page. Gloria loves working on investigation and it’s easy to understand why. I notice all of the lively way you provide useful steps by means of the website and encourage participation from other ones about this point then our child is without a doubt becoming educated a lot of things. Take advantage of the remaining portion of the new year. You’re performing a useful job.

  27. My brother recommended I might like this blog. He was entirely right. This post truly made my day. You cann’t imagine simply how much time I had spent for this info! Thanks!

  28. Terrific work! That is the type of info that are supposed to be shared across the internet. Shame on Google for no longer positioning this publish higher! Come on over and seek advice from my website . Thank you =)

  29. I got what you mean , regards for posting .Woh I am glad to find this website through google. “Being intelligent is not a felony, but most societies evaluate it as at least a misdemeanor.” by Lazarus Long.

  30. Mzlxyyzj.Com说道:

    If you are going for finest contents like I do, simply pay a visit this site everyday because it offers
    feature contents, thanks

  31. Hi mates, how is the whole thing, and what
    you want to say about this article, in my view its truly
    amazing designed for me.

  32. MichaelJemery.com is a site with many hypnosis downloads. Whether you are looking for free hypnosis downloads, self hypnosis download for mp3, video and any audio files, Michael Jemery has the downloads for you. You can download hypnosis from apps, audio, mp3 and even youtube !

  33. Some really fantastic info , Glad I detected this. “Doctrine is nothing but the skin of truth set up and stuffed.” by Henry Ward Beecher.

  34. site说道:

    I really hope to advise you that I am new to putting up a blog and undeniably valued your information. More than likely I am going to bookmark your blog post . You simply have superb article material. Be Thankful For it for giving out with us your very own internet webpage

  35. I really enjoy looking through on this website , it holds good posts . “Dream no small dreams. They have no power to stir the souls of men.” by Victor Hugo.

  36. Just a smiling visitor here to share the love (:, btw outstanding layout.

  37. bag leg说道:

    Cheap bag back, Buy Quality bag collection directly from China
    bag ipad Suppliers: Casual Shoulder Bag simple design Ladies Cross Body Bag Vintage Purse Bag
    Leather Cross Body Shoulder Messenger Bag
    China bag ipad Suppliers

  38. biocentro说道:

    Cursos masajes para toda la gente de todas las edades.

  39. Health & Fitness说道:

    I have been absent for some time, but now I remember why I used to love this site. Thanks , I¡¦ll try and check back more often. How frequently you update your site?

  40. you are actually a excellent webmaster. The site loading speed is incredible. It seems that you are doing any unique trick. In addition, The contents are masterpiece. you have performed a great process on this subject!

  41. Wow! This could be one particular of the most helpful blogs We’ve ever arrive across on this subject. Actually Wonderful. I’m also a specialist in this topic so I can understand your effort.

  42. I was recommended this blog by my cousin. I am not sure whether this post is written by him as nobody else know such detailed about my problem. You are amazing! Thanks!

  43. Travel & Leisure说道:

    Fantastic beat ! I wish to apprentice while you amend your web site, how could i subscribe for a blog website? The account aided me a acceptable deal. I had been a little bit acquainted of this your broadcast provided bright clear idea

  44. continue here说道:

    Emeryeps is a SEO(Search Engine optimzation) and Internet Marketing company. They help businesses to get traffic from various search engine and online community. They have seo experts and consultants with many years of SEO Experiences. No matter where your business is located, EmeryEPS.com can help your business to secure your highly convertible leads online.

  45. I as well as my pals were taking note of the great guides located on your web page and all of a sudden came up with a terrible suspicion I had not thanked the web blog owner for them. Those people appeared to be consequently warmed to study them and have now certainly been loving them. We appreciate you indeed being considerably kind and also for making a choice on some smart subject matter most people are really needing to understand about. My sincere regret for not expressing appreciation to sooner.

  46. I really wanted to make a note to be able to express gratitude to you for all of the superb secrets you are giving on this site. My incredibly long internet research has finally been honored with reputable suggestions to talk about with my best friends. I would mention that we site visitors actually are truly blessed to live in a very good community with many lovely individuals with very beneficial guidelines. I feel very much fortunate to have discovered your webpage and look forward to some more pleasurable times reading here. Thank you once more for a lot of things.

  47. Hey, you used to write wonderful, but the last several posts have been kinda boring¡K I miss your tremendous writings. Past several posts are just a little bit out of track! come on!

  48. F*ckin’ tremendous things here. I’m very glad to look your post. Thanks a lot and i’m looking forward to touch you. Will you kindly drop me a e-mail?

  49. Great amazing things here. I¡¦m very glad to peer your article. Thanks a lot and i’m looking ahead to touch you. Will you please drop me a e-mail?

  50. I was examining some of your posts on this website and I conceive this site is very informative ! Keep on putting up.

  51. Its like you read my mind! You appear to know a lot about this, like you wrote the book in it or something. I think that you can do with some pics to drive the message home a little bit, but other than that, this is wonderful blog. A fantastic read. I will certainly be back.

  52. I have recently started a website, the info you provide on this website has helped me greatly. Thank you for all of your time & work.

  53. I have been absent for some time, but now I remember why I used to love this blog. Thanks, I will try and check back more frequently. How frequently you update your site?

  54. Wow! This could be one particular of the most useful blogs We’ve ever arrive across on this subject. Basically Great. I am also an expert in this topic therefore I can understand your effort.

  55. I reckon something truly special in this website.

  56. Very interesting subject , thanks for posting . “The friendship that can cease has never been real.” by Saint Jerome.

  57. Enjoyed reading this, very good stuff, thankyou . “I will do my best. That is all I can do. I ask for your help-and God’s.” by Lyndon B. Johnson.

  58. Very interesting details you have mentioned , regards for posting . “I don’t know what you could say about a day in which you have seen four beautiful sunsets.” by John Glenn.

  59. It is appropriate time to make a few plans for the long run and it’s time to be happy. I have learn this put up and if I may just I want to counsel you some attention-grabbing issues or tips. Maybe you can write next articles relating to this article. I want to read even more things about it!

  60. I gotta bookmark this website it seems very helpful very beneficial

  61. My coder is trying to persuade me to move to .net from PHP. I have always disliked the idea because of the costs. But he’s tryiong none the less. I’ve been using Movable-type on several websites for about a year and am worried about switching to another platform. I have heard very good things about blogengine.net. Is there a way I can import all my wordpress posts into it? Any help would be really appreciated!

  62. I really like your writing style, wonderful info , thankyou for posting : D.

  63. suba me说道:

    A4YZmX Simply wanna say that this is extremely helpful, Thanks for taking your time to write this.

  64. It is appropriate time to make some plans for the long run and it’s time to be happy. I have read this post and if I could I wish to suggest you some fascinating things or advice. Perhaps you could write subsequent articles regarding this article. I wish to learn more things about it!

  65. I genuinely enjoy looking through on this site, it holds wonderful blog posts. “For Brutus is an honourable man So are they all, all honourable men.” by William Shakespeare.

  66. cara main poker说道:

    My coder is trying to convince me to move to .net from PHP. I have always disliked the idea because of the costs. But he’s tryiong none the less. I’ve been using WordPress on various websites for about a year and am concerned about switching to another platform. I have heard excellent things about blogengine.net. Is there a way I can transfer all my wordpress content into it? Any kind of help would be greatly appreciated!

  67. jeremy abrams说道:

    LAK1Oy You could definitely see your enthusiasm in the work you write. The world hopes for more passionate writers like you who aren at afraid to say how they believe. Always go after your heart.

  68. Quality posts is the key to invite the users to pay a visit the web page,
    that’s what this website is providing.

  69. movies reviews说道:

    You are my aspiration, I possess few web logs and rarely run out from post . аАа’аАТ‚аЂТ˜Tis the most tender part of love, each other to forgive. by John Sheffield.

  70. Muchos Gracias for your blog post.Really looking forward to read more. Want more.

  71. You have made some good points there. I checked on the internet to find out more about the issue and found most individuals will go along with your views on this web site.

  72. Thanks again for the blog.Really thank you!

  73. It as hard to find experienced people on this topic, however, you sound like you know what you are talking about! Thanks

  74. What as Happening i am new to this, I stumbled upon this I ave found It absolutely helpful and it has helped me out loads. I hope to contribute & aid other users like its helped me. Good job.

  75. It as not that I want to replicate your web-site, but I really like the design and style. Could you let me know which design are you using? Or was it tailor made?

  76. This is a good tip especially to those new to the blogosphere. Simple but very precise info Many thanks for sharing this one. A must read article!

  77. Home Improvement说道:

    Good write-up, I’m normal visitor of one’s site, maintain up the excellent operate, and It is going to be a regular visitor for a lengthy time.

  78. Very neat article.Really thank you! Much obliged.

  79. Way cool! Some very valid points! I appreciate you writing this article and the rest of the website is very good.

  80. Looking forward to reading more. Great blog post.Much thanks again. Keep writing.

  81. I really like what you guys are usually up too. This sort of clever work and coverage! Keep up the very good works guys I’ve incorporated you guys to my personal blogroll.

  82. Thanks for sharing, this is a fantastic article post.Thanks Again. Great.

  83. Im thankful for the blog post.Really thank you! Want more.

  84. شهرزاد说道:

    This is one awesome article post.

  85. Thanks so much for the article.Much thanks again. Much obliged.

  86. Thanks again for the blog article.Really looking forward to read more. Want more.

  87. Does your website have a contact page? I’m having trouble locating it but, I’d like to send you an email. I’ve got some suggestions for your blog you might be interested in hearing. Either way, great blog and I look forward to seeing it expand over time.

  88. Im grateful for the blog.Thanks Again. Great.

  89. Some genuinely fantastic blog posts on this website , thanks for contribution.

  90. Thanks for the blog article. Great.

  91. پنجره说道:

    Really interesting info! Perfect exactly what I was trying to find!

  92. Thanks again for the post.Much thanks again. Much obliged.

  93. Muchos Gracias for your blog.Really thank you! Much obliged.

  94. Fantastic article post.Really thank you! Fantastic.

  95. Awesome blog article.Really looking forward to read more.

  96. Health说道:

    I’m not sure where you are getting your information, but great topic. I needs to spend some time learning much more or understanding more. Thanks for fantastic info I was looking for this information for my mission.

  97. Thanks-a-mundo for the post.Thanks Again. Great.

  98. Im thankful for the article post.Really looking forward to read more. Keep writing.

  99. Very informative article.Really looking forward to read more. Much obliged.

  100. Some genuinely prize content on this website , saved to my bookmarks.

  101. Cam4com说道:

    Some truly nice stuff on this web site , I it.

  102. camdoll-s.us说道:

    read stuff from. Thanks for posting when you

  103. Business说道:

    Nice blog here! Also your web site loads up fast! What host are you using? Can I get your affiliate link to your host? I wish my web site loaded up as fast as yours lol

  104. sexcamly说道:

    I truly appreciate this post. I have been looking everywhere for this! Thank God I found it on Bing. You have made my day! Thank you again.

  105. Really appreciate you sharing this blog post.Really looking forward to read more. Cool.

  106. bitcoi-n.us说道:

    more attention. I all probably be back again to see more, thanks for the info!

  107. pretty practical stuff, overall I think this is worth a bookmark, thanks

  108. random cam chat说道:

    Muchos Gracias for your blog post.Thanks Again.

  109. Women说道:

    Excellent site. Lots of useful info here. I’m sending it to some pals ans also sharing in delicious. And of course, thanks to your effort!

  110. It as amazing designed for me to have a web site, which is good in support of my knowledge. thanks admin

  111. Thank you for your post. Awesome.

  112. عاشقانه说道:

    There may be noticeably a bundle to find out about this. I assume you made certain good factors in options also.

  113. Career And Jobs说道:

    Wow! This could be one particular of the most helpful blogs We have ever arrive across on this subject. Basically Fantastic. I am also a specialist in this topic therefore I can understand your hard work.

  114. It as not that I want to replicate your web-site, but I really like the layout. Could you tell me which style are you using? Or was it tailor made?

  115. It as especially a abundant as thriving as practical a part of details. I will live thankful that you just free this type of information as anyway as us all.

  116. so much and I am looking forward to touch you. Will you please drop me a mail? my homepage candy crush saga cheats

  117. This is a topic that as close to my heart Best wishes! Where are your contact details though?

  118. Thanks for sharing superb informations. Your web-site is very cool. I am impressed by the details that you’ve on this web site. It reveals how nicely you perceive this subject. Bookmarked this web page, will come back for more articles. You, my pal, ROCK! I found just the info I already searched all over the place and simply could not come across. What a great web site.

  119. Very interesting subject, regards for posting.

  120. You might try adding a video or a picture or two

  121. Fashion说道:

    Hello, Neat post. There’s an issue together with your web site in web explorer, could test this… IE nonetheless is the marketplace chief and a good portion of people will pass over your fantastic writing due to this problem.

  122. دانلود说道:

    Wow, great blog.Really looking forward to read more. Fantastic.

  123. Technology说道:

    Someone necessarily help to make significantly posts I would state. That is the first time I frequented your web page and up to now? I amazed with the research you made to make this particular publish extraordinary. Great process!

  124. Spot on with this write-up, I seriously feel this web site needs a great deal more attention. I all probably be back again to see more, thanks for the info!

  125. 迷你仓比较说道:

    Major thankies for the blog article. Will read on

  126. check说道:

    It’s an amazing piece of writing in favor of all the online people;
    they will get advantage from it I am sure.

  127. Trein说道:

    you will have an awesome weblog right here! would you prefer to make some invite posts on my weblog?

  128. Brooks说道:

    magnificent issues altogether, you just gained a emblem new reader. What may you recommend about your publish that you made some days ago? Any certain?

  129. This is a very good tip particularly to those new to the blogosphere. Brief but very accurate info Appreciate your sharing this one. A must read article!

  130. Everyone loves what you guys are usually up too. This sort of clever work and reporting! Keep up the great works guys I’ve included you guys to my blogroll.

  131. clickmeter说道:

    You are my inspiration, I own few web logs and occasionally run out from brand . Truth springs from argument amongst friends. by David Hume.

  132. Pets说道:

    As soon as I discovered this web site I went on reddit to share some of the love with them.

  133. revitoll.us说道:

    Your means of describing the whole thing in this post is really good, all be able to easily understand it, Thanks a lot.

  134. You made some good points there. I looked on the internet for the subject matter and found most persons will go along with with your site.

  135. dani说道:

    “Greetings! Very helpful advice within this post! It is the little changes which will make the greatest changes. Thanks for sharing!”

  136. You ave made some good points there. I looked on the web for more information about the issue and found most individuals will go along with your views on this website.

  137. provilluss.us说道:

    I think other website proprietors should take this web site as an model, very clean and excellent user genial style and design, as well as the content. You are an expert in this topic!

  138. Your style is really unique compared to other people I ave read stuff from. Thanks for posting when you ave got the opportunity, Guess I will just book mark this site.

  139. Acnezine说道:

    I wanjt to encourage conttinue your reat work, have a nice holiday weekend!

  140. site de rencontre gratuit en belgique How do i make firefox my main browser for windows live messenger?

  141. Wow, great blog article.Really thank you! Cool.

  142. Pot management说道:

    You ave made some really good points there. I looked on the web for more information about the issue and found most people will go along with your views on this web site.

  143. The most effective magic formula for the men you can explore as we speak.

  144. mihran说道:

    Really informative blog post.Really thank you! Really Cool.

  145. Thank you, I have just been looking for info about this subject for ages and yours is the best I have discovered so far. But, what about the bottom line? Are you sure about the source?

  146. Great – I should definitely pronounce, impressed with your site. I had no trouble navigating through all the tabs and related info ended up being truly easy to do to access. I recently found what I hoped for before you know it at all. Reasonably unusual. Is likely to appreciate it for those who add forums or anything, site theme . a tones way for your client to communicate. Excellent task.

发表评论

电子邮件地址不会被公开。