Word2Vec原理之负采样算法

这篇主要根据前人的文章内容,整理了一遍skip-gram模型下negative-sampling的基本推导思路,只作学习之用,并不完整。公式也都是来自于已有论文。

分布式假设

Word2Vec虽然一直火热,也引发了各种词嵌入,但实际上不可否认依然是缺乏严谨的可解释性。所谓的分布式假设,就是处于相似上下文中的词具有相似的含义,一般认为这种相似包含了语义层面的相似,但却很难使用自然语言的词法、句法或语义的层面的现有知识来充分地解释。这可能也是词向量的一个缺陷,不过另一方面,这种分布式假设的概念可以应用到很多其他的领域的数据分析,比如广告分析、app点击数据分析等,毕竟只要能替代词和语料的关系,并且有上下文和中心词的类比关系,就可以一试,所以应用的广度可以说无法估计。

网络的分层

三层网络模型不再赘述,具体对于输入层和投影层,回顾CBOW模型,投影层是2c个输入词向量求和,即\(x_w=\sum_{i=1}^{2c}v(Context(w)_i)\in R^m\),最后输出为一颗huffman树。Skip-gram的输入层就是中心词\(w\)的向量,投影层是一个恒等投影,即\(v(w)\)。另,有的文献中沿用了一般神经网络的说法称投影层为隐含层。

Skip-gram模型

目标是要最大化概率:
\[
\arg\max_\theta \prod_{W\in Corpus} \prod_{c\in C(w)}p(c|w;\theta)
\]
\(C(w)\)是词\(w\)的上下文集合,因此可以简化为:
\[
\arg\max_\theta \prod_{(w,c)\in D}p(c|w;\theta)
\]
\((w,c)\)是目标词和上下文构成的pair。

求参的方法是使用softmax,其中\(v_c\)和\(v_w\in R^d\),分别是\(c\)和\(w\)的向量表示,\(C\)是所有上下文(contexts)的集合。参数\(\theta\)由\(v_{c_i}\)和\(v_{w_i}\)构成,其中\(w\in V\),\(c\in C\),\(i\in1,…,d\),参数的规模是\(|C|\times|V|\times d\)。
\[
p(c|w;\theta) = { e^{v_c \cdot v_w} \over \sum_{c’ \in C}e^{v’_c \cdot v_w}}
\]

用取log的方式:

\[\arg \max_\theta \sum_{(w,c)\in D}logp(c|w) = \sum_{(w,c)\in D}(loge^{v_c \cdot v_w}-log\sum_{c’}e^{v_{c’}\cdot v_w})\]

对嵌入过程的假设是:最大化上述目标函数可以得到好的嵌入\(v_w, \forall w \in V\),在这个意义上,相似的词会有相似的向量。

由于\(\sum_{c’ \in C}e^{v’_c \cdot v_w}\)的计算量太大,可以使用Hierarchical Softmax替代Softmax,这个在上一篇博客中已经提及。这里主要讨论Mikolov提出的Negative-Sampling方法.

值得一提的是,尽管不是所有版本的word2vec都实现了模型与算法的各种组合,但CBOW和Skip-gram两种模型,结合层次softmax和负采样两种算法,实际上可以有四种不同的组合。

Negative-Sampling

Negative-Sampling(NEG)基于skip-gram模型,但实际上是优化另一个目标函数,Mikolov说NEG是从NCE(Noise Contrastive Estimation)简化而来,用于提高训练速度,也可以改善词向量的质量。相比于分层Softmax,NEG使用的方式是随机负采样,而不是Huffman树。

我们的目标是找到合适的参数来最大化观测数据来源于语料库的概率:
\[\begin{align}
&\arg\max_\theta \prod_{(w,c)\in D}p(D=1|w,c;\theta) \\
= & \arg\max_\theta log \prod_{(w,c)\in D}p(D=1|w,c;\theta)\\
= & \arg\max_\theta \sum_{(w,c)\in D}logp(D=1|w,c;\theta)
\end{align}
\]

\(p(D=1|w,c;\theta)\)可以使用softmax来定义:
\[
p(D=1|w,c;\theta)={1 \over {1+e^{-v_c \cdot v_w}}}
\]

因此,目标函数变成:

\[
\begin{align}
\arg \max_\theta \sum_{(w,c)\in D}log{1 \over {1+e^{-v_c \cdot v_w}}}
\end{align}
\]

这个目标函数存在一个问题,如果我们设定\(\theta\)使得每一对\((w,c)\)的\(p(D=1|w,c;\theta)=1\),那这个目标函数就无意义了。而只要设置\(\theta\)对所有\(v_c\)和\(v_w\),使得\(v_c=v_w\)且\(v_c \cdot v_w=K\),而K是一个足够大的数字,则这种情况很容易出现(在Goldberg[1]的实验中当\(K\approx 40\)时,概率就为1了)。因为为了避免所有向量都是相同的值,可以去掉某些\((w,c)\)的组合,即可以随机选择\((w,c)\)对中的一部分作为负例。目标函数被改成:

\[
\begin{align}
& \arg\max_\theta \prod_{(w,c)\in D}p(D=1|c,w;\theta) \prod_{(w,c)\in D’}p(D=0|c,w;\theta) \\
= & \arg \max_\theta \prod_{(w,c)\in D}p(D=1|c,w;\theta) \prod_{(w,c)\in D’}(1-p(D=1|c,w;\theta)) \\
= & \arg \max_\theta \sum_{(w,c)\in D}logp(D=1|c,w;\theta) + \sum_{(w,c)\in D’}log(1-p(D=1|c,w;\theta)) \\
= & \arg \max_\theta \sum_{(w,c)\in D}log{1 \over {1+e^{-v_c \cdot v_w}}}+ \sum_{(w,c)\in D’}log(1-{1 \over {1+e^{-v_c \cdot v_w}}}) \\
= & \arg \max_\theta \sum_{(w,c)\in D}log{1 \over {1+e^{-v_c \cdot v_w}}}+ \sum_{(w,c)\in D’}log{1 \over {1+e^{v_c \cdot v_w}}}
\end{align}
\]

考虑sigmoid函数\(\sigma(x)={1\over {1+e^{-x}}}\),上述等式可以改写为:

\[
\begin{align}
& \arg \max_\theta \sum_{(w,c)\in D}log{1 \over {1+e^{-v_c \cdot v_w}}}+ \sum_{(w,c)\in D’}log{1 \over {1+e^{v_c \cdot v_w}}} \\
=& \arg \max_\theta \sum_{(w,c)\in D}log\sigma(v_c \cdot v_w)+ \sum_{(w,c)\in D’}log\sigma(-v_c \cdot v_w)
\end{align}
\]

这个目标函数表面的含义也可以理解为要尽量增大正例的\((v_c \cdot v_w)\)数据对,而尽量降低负例的\((v_c \cdot v_w)\)数据对。词与词之间,若其上下文很相近,则他们本身也很相似。

D’的选取与具体采样方法有关。在Mikolov的原文[2]中,考虑一个\((w,c)\in D\)的数据对样本,以及\(k\)个负样本\((w_j)\in D’, D’=W_{negative}\)。目标函数的写法是:

\[
log\sigma(v_c \cdot v_w) + \sum_{w_j\in W_{negative}}log\sigma(-v_{w_j} \cdot v_c)
\]

[1] Goldberg Y, Levy O. word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method[J]. arXiv preprint arXiv:1402.3722, 2014.
[2] Mikolov T, Dean J. Distributed representations of words and phrases and their compositionality[J]. Advances in neural information processing systems, 2013.

148 thoughts on “Word2Vec原理之负采样算法

  1. windfringe说道:

    看不懂了。。

  2. Lanm说道:

    不明白的是,参数theta到底是用在什么地方更新的?

  3. Bernadine Tsuji说道:

    Great write-up, I?¦m regular visitor of one?¦s web site, maintain up the nice operate, and It is going to be a regular visitor for a lengthy time.

  4. check that说道:

    I just want to say I am just all new to weblog and definitely savored this page. Almost certainly I’m going to bookmark your blog . You surely come with perfect well written articles. Thank you for sharing with us your blog site.

  5. kumahasiaweh说道:

    Hello there, I found your blog by way of Google at the same time as looking for a comparable topic, your website got here up, it seems to be good. I have bookmarked it in my google bookmarks.

  6. MetroclicK"说道:

    MetroClick specializes in building completely interactive products like Photo Booth for rental or sale, Touch Screen Kiosks and Digital Signage, and experiences. With our own hardware production facility and in-house software development teams, we are able to achieve the highest level of customization and versatility for Photo Booths, Touch Screen Kiosks and Digital Signage. MetroClick, 121 Varick St, #301, New York, NY 10013, +1 646-843-0888

  7. Faytech North America is a touch screen Manufacturer of both monitors and pcs. They specialize in the design, development, manufacturing and marketing of Capacitive touch screen, Resistive touch screen, Industrial touch screen, IP65 touch screen, touchscreen monitors and integrated touchscreen PCs. Contact them at http://www.faytech.us, 121 Varick Street,3rd Floor,New York, NY 10013,+1 646 205 3214

  8. dentysta gorzów说道:

    Have you ever thought about creating an ebook or guest authoring on other sites? I have a blog based on the same ideas you discuss and would love to have you share some stories/information. I know my readers would appreciate your work. If you’re even remotely interested, feel free to shoot me an email.

  9. I gotta bookmark this internet site it seems invaluable very useful

  10. Noticeably helpful data that you have remarked, warm regards for putting up.

  11. chenkun说道:

    我认为skip-gram的softmax function 写的不对,softmax的分母不应该是整个vocabulary 吗,怎么在这里成了context的所有单词呢?

  12. Profile Website说道:

    It really is almost unthinkable to find well-aware viewers on this theme, nonetheless you come across as like you know what exactly you’re preaching about! Gratitude

  13. you’re in reality a good webmaster. The website loading velocity is amazing. It seems that you are doing any distinctive trick. In addition, The contents are masterpiece. you’ve done a fantastic task in this subject!

  14. Hi there very cool blog!! Man .. Excellent .. Wonderful .. I’ll bookmark your website and take the feeds additionally…I am glad to search out a lot of useful info right here in the post, we’d like develop more strategies in this regard, thanks for sharing.

  15. dentysta gorzów说道:

    Awesome Post: BLOGURL. You do a fantastic job. Thanks again. NAME

  16. Remarkably useful knowledge that you have mentioned, thanks for posting.

  17. official website说道:

    I really wish to reveal to you that I am new to writing and really enjoyed your site. Most likely I am most likely to remember your blog post . You seriously have extraordinary article information. Acknowledge it for sharing with us the best internet site page

  18. visit homepage说道:

    It really is ideal day to put together some desires for the near future. I have read through this post and if I may just, I desire to suggest to you you a few helpful instruction.

  19. I simply could not leave your site prior to suggesting that I extremely loved the standard info a person provide in your visitors? Is going to be back steadily to check out new posts

  20. Thank you for sharing excellent informations. Your web-site is very cool. I’m impressed by the details that you’ve on this website. It reveals how nicely you understand this subject. Bookmarked this web page, will come back for extra articles. You, my friend, ROCK! I found simply the info I already searched everywhere and simply couldn’t come across. What an ideal web-site.

  21. Flowers ireland说道:

    Howdy! This post couldn’t be written much better! Looking through this article reminds me of my previous roommate! He continually kept talking about this. I will forward this article to him. Pretty sure he will have a great read. Many thanks for sharing!

  22. flowers Ireland说道:

    Hi, I check your new stuff regularly. Your writing style is witty, keep up the good work!

  23. dentysta gorzów说道:

    The BLOGURL is definitely the freshest within this treasured topic. I concur together as well as your conclusions and might eagerly look forward to your future updates. Saying thanks received’t basically be passable, for the good lucidity within your writing. NAME

  24. florists Dublin说道:

    Hi there, I read your new stuff daily. Your writing style is witty, keep up the good work!

  25. Great blog here! Also your web site loads up very fast! What host are you using? Can I get your affiliate link to your host? I wish my website loaded up as fast as yours lol

  26. Thanks for sharing superb informations. Your site is so cool. I’m impressed by the details that you¡¦ve on this web site. It reveals how nicely you perceive this subject. Bookmarked this website page, will come back for extra articles. You, my pal, ROCK! I found just the info I already searched everywhere and just couldn’t come across. What an ideal web-site.

  27. We’re a group of volunteers and opening a new scheme in our community. Your website offered us with valuable info to work on. You’ve done a formidable job and our entire community will be thankful to you.

  28. You can certainly see your skills within the paintings you write. The arena hopes for even more passionate writers such as you who are not afraid to say how they believe. All the time follow your heart. “If the grass is greener in the other fellow’s yard – let him worry about cutting it.” by Fred Allen.

  29. F*ckin’ remarkable issues here. I am very happy to see your post. Thanks a lot and i’m having a look forward to contact you. Will you please drop me a mail?

  30. find out here说道:

    I merely wish to reveal to you that I am new to writing a blog and genuinely cherished your post. Probably I am inclined to remember your blog post . You seriously have impressive article material. Delight In it for share-out with us your main internet site article

  31. Whats Taking place i’m new to this, I stumbled upon this I’ve found It positively helpful and it has aided me out loads. I am hoping to give a contribution & aid other customers like its aided me. Great job.

  32. I truly enjoy looking through on this website , it holds good articles . “I have a new philosophy. I’m only going to dread one day at a time.” by Charles M. Schulz.

  33. Excellent goods from you, man. I’ve understand your stuff previous to and you are just extremely wonderful. I actually like what you’ve acquired here, really like what you are stating and the way in which you say it. You make it enjoyable and you still take care of to keep it wise. I cant wait to read much more from you. This is really a tremendous site.

  34. As soon as I found this web site I went on reddit to share some of the love with them.

  35. My brother suggested I might like this blog. He was entirely right. This post truly made my day. You cann’t imagine simply how much time I had spent for this info! Thanks!

  36. I like this blog so much, saved to fav. “Respect for the fragility and importance of an individual life is still the mark of an educated man.” by Norman Cousins.

  37. whoah this weblog is excellent i love reading your articles. Keep up the great paintings! You realize, a lot of people are looking round for this information, you can aid them greatly.

  38. Health & Fitness说道:

    Good write-up, I¡¦m normal visitor of one¡¦s site, maintain up the excellent operate, and It is going to be a regular visitor for a long time.

  39. Wow, awesome blog layout! How long have you been blogging for? you make blogging look easy. The overall look of your website is magnificent, as well as the content!

  40. Travel & Leisure说道:

    fantastic points altogether, you simply gained a new reader. What may you suggest about your submit that you simply made a few days ago? Any positive?

  41. You can certainly see your skills within the paintings you write. The arena hopes for more passionate writers such as you who are not afraid to mention how they believe. Always follow your heart.

  42. Automotive说道:

    Wow! Thank you! I continually wanted to write on my blog something like that. Can I include a part of your post to my website?

  43. biocentro.cl说道:

    Cursos masajes para toda la gente de todas las edades.

  44. Emery EPS说道:

    Emeryeps is a SEO(Search Engine optimzation) and Internet Marketing company. They help businesses to get traffic from various search engine and online community. They have seo experts and consultants with many years of SEO Experiences. No matter where your business is located, EmeryEPS.com can help your business to secure your highly convertible leads online.

  45. more info here说道:

    MichaelJemery.com is a site with many hypnosis downloads. Whether you are looking for free hypnosis downloads, self hypnosis download for mp3, video and any audio files, Michael Jemery has the downloads for you. You can download hypnosis from apps, audio, mp3 and even youtube !

  46. Definitely believe that which you said. Your favorite reason appeared to be on the web the simplest thing to be aware of. I say to you, I certainly get annoyed while people consider worries that they just don’t know about. You managed to hit the nail upon the top and also defined out the whole thing without having side effect , people could take a signal. Will likely be back to get more. Thanks

  47. Thanks a lot for sharing this with all people you actually recognize what you are speaking about! Bookmarked. Kindly also talk over with my website =). We will have a hyperlink exchange arrangement between us!

  48. Hmm it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I too am an aspiring blog blogger but I’m still new to everything. Do you have any suggestions for novice blog writers? I’d genuinely appreciate it.

  49. You are a very bright individual!

  50. I have not checked in here for a while since I thought it was getting boring, but the last few posts are great quality so I guess I’ll add you back to my everyday bloglist. You deserve it my friend 🙂

  51. Thanks, I’ve recently been searching for information approximately this subject for a long time and yours is the greatest I have came upon so far. However, what concerning the bottom line? Are you sure concerning the supply?

  52. I think this is among the most important information for me. And i’m glad reading your article. But want to remark on few general things, The web site style is ideal, the articles is really nice : D. Good job, cheers

  53. I like what you guys are up also. Such clever work and reporting! Carry on the excellent works guys I have incorporated you guys to my blogroll. I think it will improve the value of my website :).

  54. hi!,I love your writing very so much! proportion we keep up a correspondence extra about your article on AOL? I need a specialist on this area to unravel my problem. Maybe that’s you! Having a look forward to peer you.

  55. I was studying some of your content on this site and I think this site is real instructive! Continue putting up.

  56. Simply wanna comment that you have a very decent site, I love the design it actually stands out.

  57. Thankyou for helping out, excellent info .

  58. Enjoyed looking at this, very good stuff, thanks . “I will do my best. That is all I can do. I ask for your help-and God’s.” by Lyndon B. Johnson.

  59. I view something genuinely interesting about your site so I saved to favorites .

  60. Only wanna input that you have a very decent web site , I like the design and style it really stands out.

  61. Does your site have a contact page? I’m having trouble locating it but, I’d like to send you an e-mail. I’ve got some ideas for your blog you might be interested in hearing. Either way, great blog and I look forward to seeing it develop over time.

  62. F*ckin’ amazing issues here. I’m very glad to peer your article. Thank you a lot and i’m having a look ahead to contact you. Will you kindly drop me a e-mail?

  63. My developer is trying to convince me to move to .net from PHP. I have always disliked the idea because of the costs. But he’s tryiong none the less. I’ve been using WordPress on a number of websites for about a year and am anxious about switching to another platform. I have heard excellent things about blogengine.net. Is there a way I can import all my wordpress content into it? Any kind of help would be greatly appreciated!

  64. lucy ann说道:

    2ETHj3 When someone writes an paragraph he/she keeps the idea of a

  65. Trapp Pricing说道:

    Just wanna input on few general things, The website design and style is perfect, the content material is very good. “The way you treat yourself sets the standard for others.” by Sonya Friedman.

  66. It is perfect time to make a few plans for the longer term and it is time to be happy. I have learn this post and if I may just I desire to suggest you few attention-grabbing issues or tips. Maybe you can write subsequent articles regarding this article. I wish to read even more things about it!

  67. Woah! I’m really loving the template/theme of this website. It’s simple, yet effective. A lot of times it’s hard to get that “perfect balance” between superb usability and visual appearance. I must say you’ve done a amazing job with this. Also, the blog loads very fast for me on Opera. Exceptional Blog!

  68. hello!,I like your writing very much! percentage we keep up a correspondence extra approximately your article on AOL? I require a specialist in this house to resolve my problem. May be that is you! Having a look ahead to see you.

  69. jeremy abrams说道:

    n1XxQd Preserve аАа’аАТ‚аЂТ˜em coming you all do such a wonderful position at these Concepts cannot tell you how considerably I, for one particular appreciate all you do!

  70. Hi, I do believe this is a great website. I stumbledupon it 😉 I may come back once again since i have bookmarked it. Money and freedom is the greatest way to change, may you be rich and continue to guide other people.

  71. Does your site have a contact page? I’m having trouble locating it but, I’d like to shoot you an e-mail. I’ve got some creative ideas for your blog you might be interested in hearing. Either way, great website and I look forward to seeing it improve over time.

  72. Brooks说道:

    magnificent issues altogether, you just gained a emblem new reader. What may you recommend about your publish that you made some days ago? Any certain?

  73. this paragraph, in my view its actually amazing in support of me.

  74. JAPAN JERSEY ??????30????????????????5??????????????? | ????????

  75. online movies说道:

    It is really a nice and helpful piece of info. I am glad that you shared this useful information with us. Please keep us up to date like this. Thank you for sharing.

  76. This website was how do I say it? Relevant!! Finally I have found something that helped me. Thanks!

  77. Say, you got a nice blog article.Really looking forward to read more. Want more.

  78. Very good article! We are linking to this particularly great post on our website. Keep up the great writing.

  79. lushacre.com.sg I want to start a blog but would like to own the domain. Any ideas how to go about this?.

  80. Hi, I do think this is a great web site. I stumbledupon it 😉 I am going to revisit once again since I bookmarked it. Money and freedom is the greatest way to change, may you be rich and continue to guide others.

  81. very couple of internet sites that occur to become detailed below, from our point of view are undoubtedly properly worth checking out

  82. Home Improvement说道:

    You have brought up a very fantastic details , appreciate it for the post.

  83. Really enjoyed this article.Really thank you! Really Cool.

  84. Does your website have a contact page? I’m having a tough time locating it but, I’d like to send you an email. I’ve got some suggestions for your blog you might be interested in hearing. Either way, great blog and I look forward to seeing it grow over time.

  85. Really appreciate you sharing this post.Really looking forward to read more. Cool.

  86. read more说道:

    Im thankful for the article.Really thank you! Really Cool.

  87. Doriserele说道:

    Absolutely NEW update of SEO/SMM package “XRumer 16.0 + XEvil 3.0”:
    captcha solution of Google, Facebook, Bing, Hotmail, SolveMedia, Yandex,
    and more than 8400 another size-types of captchas,
    with highest precision (80..100%) and highest speed (100 img per second).
    You can connect XEvil 3.0 to all most popular SEO/SMM software: XRumer, GSA SER, ZennoPoster, Srapebox, Senuke, and more than 100 of other software.

    Interested? You can find a lot of introducing videos about XEvil in YouTube.
    See you later 😉

    XRumer201707

  88. شهرزاد说道:

    I truly appreciate this blog post.Really thank you! Fantastic.

  89. دانلود说道:

    Very good blog article.Really thank you!

  90. Major thankies for the article post.Really thank you! Great.

  91. I? not that much of a online reader to be honest but your blogs really nice, keep it up! I all go ahead and bookmark your site to come back later on. Cheers

  92. A round of applause for your blog article. Will read on…

  93. reading science fiction books is the stuff that i am always into. science fiction really widens my imagination*

  94. wow, awesome article post. Will read on…

  95. پنجره说道:

    Major thankies for the post. Will read on

  96. Thanks a lot for the blog. Cool.

  97. Just Browsing While I was surfing today I noticed a great article concerning

  98. ugg sale ugg boots cheap ugg outlet stores genuine ugg boots cheap uggs uk bottes ugg pas cher cheap ugg boots sale goedkope uggs kopen

  99. I really liked your post.Really thank you! Really Cool.

  100. A round of applause for your article post.Thanks Again. Really Cool.

  101. magnificent issues altogether, you simply gained a new reader. What would you recommend about your put up that you simply made some days ago? Any certain?

  102. Health说道:

    I got what you intend, thankyou for posting .Woh I am pleased to find this website through google. “It is a very hard undertaking to seek to please everybody.” by Publilius Syrus.

  103. Wow, amazing blog layout! How long have you been blogging for? you make blogging look easy. The overall look of your website is wonderful, as well as the content!. Thanks For Your article about sex.

  104. Very nice write-up. I definitely appreciate this website. Stick with it!

  105. Politics Forum说道:

    You made some clear points there. I did a search on the subject matter and found most people will approve with your site.

  106. c-am4.us说道:

    It is usually a very pleased day for far North Queensland, even state rugby league usually, Sheppard reported.

  107. Cam dolls说道:

    Louis Vuitton Outlet Well done ! Drinking water might have been more useful, but hey

  108. Business说道:

    I will right away clutch your rss feed as I can not in finding your e-mail subscription hyperlink or e-newsletter service. Do you’ve any? Kindly allow me realize so that I could subscribe. Thanks.

  109. sexcamly.us说道:

    Wow, superb blog layout! How long have you been blogging for? you make blogging look easy. The overall look of your site is wonderful, as well as the content!

  110. Thanks so much for the blog post.Thanks Again. Awesome.

  111. Way cool! Some very valid points! I appreciate you penning this article plus the rest of the site is extremely good.

  112. Thanks for sharing, this is a fantastic post.

  113. Wanted to drop a remark and let you know your Rss feed isnt working today. I tried including it to my Google reader account but got nothing.

  114. Women说道:

    you’re really a just right webmaster. The web site loading pace is amazing. It sort of feels that you are doing any unique trick. Moreover, The contents are masterwork. you have done a fantastic process on this topic!

  115. Some truly wonderful blog posts on this website , thanks for contribution.

  116. Career And Jobs说道:

    excellent points altogether, you just received a new reader. What could you suggest in regards to your submit that you just made a few days in the past? Any sure?

  117. Thanks so much for the article post.Thanks Again.

  118. You ave made some good points there. I checked on the internet for more information about the issue and found most people will go along with your views on this site.

  119. I have recently started a web site, the information you provide on this website has helped me greatly. Thank you for all of your time & work. “Everyone is responsible and no one is to blame.” by Will Schutz.

  120. I wanted to thank you for this great article, I certainly loved each and every small bit of it. I ave bookmarked your internet site to look at the latest stuff you post.

  121. Everyone loves what you guys are usually up too. This kind of clever work and coverage! Keep up the excellent works guys I’ve incorporated you guys to my own blogroll.

  122. Please let me know if this alright with you. Regards!

  123. Fashion说道:

    I do not even know how I ended up here, but I thought this post was great. I don’t know who you are but definitely you are going to a famous blogger if you aren’t already 😉 Cheers!

  124. Hi, I do think this is a great site. I stumbledupon it 😉 I’m going to return once again since i have saved as a favorite it. Money and freedom is the greatest way to change, may you be rich and continue to guide others.

  125. Technology说道:

    I really appreciate this post. I have been looking everywhere for this! Thank goodness I found it on Bing. You’ve made my day! Thanks again!

  126. Hi, Neat post. There is an issue with your website in internet explorer, might check this… IE nonetheless is the marketplace chief and a big component to other folks will miss your great writing because of this problem.

  127. 迷你仓说道:

    This is my first time pay a quick visit at here and i am in fact happy to read everthing at alone place.

  128. Woh I love your content , bookmarked !.

  129. That is a good tip particularly to those new to the blogosphere. Brief but very accurate information Many thanks for sharing this one. A must read post!

  130. Treinreizen说道:

    is incredible. It kind of feels that you are doing any unique trick.

  131. click-meter.us说道:

    I regard something genuinely special in this website.

  132. Revitol说道:

    Simply a smiling visitor here to share the love (:, btw outstanding style and design.

  133. Women's clothing说道:

    Normally I do not learn article on blogs, however I wish to say that this write-up very pressured me to try and do it! Your writing style has been amazed me. Thanks, quite nice article.

  134. Doriserele说道:

    Revolutional update of SEO/SMM package “XRumer 16.0 + XEvil”:
    captchas regignizing of Google, Facebook, Bing, Hotmail, SolveMedia, Yandex,
    and more than 8400 another subtypes of captchas,
    with highest precision (80..100%) and highest speed (100 img per second).
    You can connect XEvil 3.0 to all most popular SEO/SMM programms: XRumer, GSA SER, ZennoPoster, Srapebox, Senuke, and more than 100 of other software.

    Interested? There are a lot of demo videos about XEvil in YouTube.
    Good luck!

    XRumer201707

  135. This is very interesting, You are a very skilled blogger. I ave joined your feed and look forward to seeking more of your excellent post. Also, I ave shared your website in my social networks!

  136. loves can you say that about?) louis vuitton hlouis vuitton handbags replicabags replica it as back this fall in mouth watering chocolate. How can you go wrong

  137. Provillus说道:

    Thanks again for the post.Really looking forward to read more. Will read on

  138. You understand a whole lot its almost hard to argue with you

  139. Loving the info on this website , you have done outstanding job on the articles.

  140. Boilx说道:

    You ought to join in a contest for starters of the highest quality blogs online. I will recommend this page!

  141. Your style is so unique in comparison to other people I have read stuff from. I appreciate you for posting when you have the opportunity, Guess I all just book mark this site.

  142. wonderful points altogether, you simply received a logo new reader. What might you suggest in regards to your submit that you made some days ago? Any positive?

  143. adult说道:

    Woman of Alien Ideal work you might have completed, this website is absolutely interesting with fantastic details. Time is God as way of retaining everything from happening directly.

  144. Very interesting subject, thanks for putting up.

  145. 강남야구장说道:

    Hi there, I wish for to subscribe for this web site to get newest updates, thus where can i do it please help.

发表评论

电子邮件地址不会被公开。