SNE与t-SNE降维算法理解

1. SNE概要

数据降维,大体分为线性方法和非线性方法。其中线性方法例如PCA和LDA,而非线性方法又有保留局部特征、基于全局特征等方法。有人整理了一张分类图,下面这张图从网上引用而来:

相比于其他降维方法,t-SNE是近年比较火热的一种高维数据可视化技术,能够通过降维,将高维数据降维并给出二维或三维的坐标点,从而可以在人能够轻易理解的平面或立体空间内将数据可视化出来。这个方法是SNE的变种,SNE是Hinton在2002年提出来的方法。Stochastic Neighbor Embedding,好吧,又是embedding。目标是将高维数据映射到低维后,尽量保持数据点之间的空间结构,这样在高维空间里距离较远的点,在低维空间中依然保持较远的距离。在传统的方法中,PCA和MDS是线性技术,用于保持相距较远的数据点之间的低维表示。Maaten将t-SNE的降维结果与其他7种降维方法的结果,在5种不同的数据集中作了对比。

1.1 高维数据的相似度概率分布

SNE将数据点之间高维的欧氏距离转换为表示相似度的条件概率,即用条件概率\(p(j|i)\)表示点\(x_j\)到点\(x_i\)的相似度,这个含义可以理解为:若以\(x_i\)为中心的高斯分布来选取邻居,则\(x_i\)选择\(x_j\)作为自己邻居的概率是\(p(j|i)\)。若数据点相距较近,则\(p(j|i)\)较大,相反若数据点相距非常远,\(p(j|i)\)则可以接近无穷小。条件概率\(p(j|i)\)定义如下:

\[
p(j|i)= {exp({-\Vert x_i-x_j\Vert^2 \over 2\sigma_i^2}) \over \sum_{k\neq i}exp({-\Vert x_i-x_k\Vert^2 \over 2\sigma_i^2})}
\]

高斯模型的方差选择

上面的公式中,\(\sigma_i\)是以\(x_i\)为中心的高斯分布的方差。因为数据点的稠密程度不同,因此这个方差选用单个固定值是不合理的,例如在密度高的区域适合选用较小的方差值。可以认为任意一个特定的方差\(\sigma_i\)代表了一个一个数据点对于其他所有数据点的概率分布\(P_i\)。论文引入了一个固定值,困惑度(perplexity),用来二分搜索这个合适的\(\sigma_i\)。而perplexity本身是由用户指定的,其定义为:

\[Perp(P_i)=2^{H(P_i)}\]

其中,\(H(P_i)\)是概率分布\(P_i\)的熵:

\[H(P_i)=-\sum_j{p_{j|i}log_2{p_{j|i}}}\]

perplexity可以理解为一个选择有效邻居数目的平滑方法。论文给出了典型的取值范围是5到50。

1.2 映射到低维后的相似度概率分布

考虑到目标是衡量点对间的相似度,因此令\(p(i|i)\)为0。同样,对于低维点\(y_i\)(对应高维\(x_i\))和\(y_j\)(对应高维\(x_j\)),也可以估算类似的条件概率,一下表示为\(q(j|i)\),在这里,我们将高斯分布的所有方差\(\sigma_i\)设为\(1 \over \sqrt{2}\)。这样,下面的\(q(j|i)\)公式就被定义为用来描述图上的点\(y_j\)和点\(y_i\)之间的相似度:

\[
q(j|i)={exp({-\Vert y_i-y_j\Vert^2}) \over \sum_{k\neq i}exp({-\Vert y_i-y_k\Vert^2})}
\]

这就是\(y_i\)和\(y_j\)之间的相似概率,并同样设置\(q(i|i)=0\)。这个时候,理论上,如果低维点正确地表示了高维点,那\(p(j|i)\)和\(q(j|i)\)这两个概率应当相等。至于说用\(q(j|i)\)代表\(p(j|i)\)是否靠谱,论文中选择用KL散度(Kullback-Leibler divergence)来衡量。SNE使用梯度下降法在所有的数据点上最小化KL散度之和。损失函数如下:
\[
C = \sum_iKL(P_i\Vert Q_i)=\sum_i \sum_j p(j|i)log{p(j|i)\over q(j|i)}
\]

\(P_i\)表示在给定数据点\(x_i\)基础上,其他所以数据点的条件概率分布。
\(Q_i\)表示在给定低维映射数据点\(y_i\)基础上所有其他低维映射数据点的条件概率分布。

由于KL散度并不具有对称性,所以数据点对距离之间的不同类型的错误在权重上是不均匀的。简单来说,从\(P\)到\(Q\)的度量通常并不等于从\(Q\)到\(P\)的度量。在这里的场景下,可以理解为,当使用较分散的低维点来表示较聚集的高维数据点时(例如用较小的\(q(j|i)\)来表示一个较大的\(p(j|i)\)),损失值较大;而使用较聚集的低维点来表示分布较分散的高维数据点,则损失值较小(因为只损失了概率分布Q中的一部分概率质量)。因此,最小化SNE损失函数的意义,在于通过取得高维空间高斯分布的合理的方差值,在低维中尽可能多地保留数据的局部结构。没错,是局部结构。这意味着当高维数据数据点之间距离较远时,从损失函数看,若映射所得的低维点距离很近,损失反倒很小,是明显不合理的。

对损失函数对\(y_i\)求梯度后可以得到:
\[
\frac{\delta C}{\delta y_i}=2\sum_{j}(p(j|i)-q(j|i)+p(i|j)-q(i|j))(y_i-y_j)
\]

作者将这个梯度的物理意义理解为是一种合力,这个合力是由低维点\(y_i\)和其他所有点\(y_j\)之间的各种力共同构成的,且所有的力的方向是沿着\((y_i-y_j)\)。用\(y_i\)和\(y_j\)距离表示高维数据点之间,两两数据对可以看做一个弹簧,若这个距离太近了,则这种力表现为斥力;若距离太远,便表现为引力。弹簧的力量与长度和刚度成正比,而\((p(j|i)-q(j|i)+p(i|j)-q(i|j))\)所表征的偏差值与此有关。

2. t-SNE的改进

t-SNE重点解决了两个问题,一个是SNE的不对称问题(用联合概率分布替代条件概率分布),另一个是不同类别之间簇的拥挤问题(引入\(t\)分布)。

2.1 不对称问题

Maaten采用的方法是用联合概率分布来替代条件概率分布。高维数据下的联合概率分布为\(P\),低维环境下的联合概率分布为\(Q\),使得\(\forall i,j\),\(p_{ij}=p_{ji}\),\(q_{ij}=q_{ji}\)。

定义低维空间的联合概率:
\[
q_{ij}=\frac{\exp (-\left \| y_i-y_j \right \|^2)}{\sum_{k \neq l}\exp (-\left \| y_k-y_l \right \|^2)}
\]

按理说高维空间的概率应定义为:

\[
p_{ij}=\frac{\exp (-\left \| x_i-x_j \right \|^2/2\sigma^2)}{\sum_{k \neq l}\exp (-\left \| x_k-x_l \right \|^2/2\sigma^2)}
\]

然而这种情况下,一旦数据点\(x_i\)在距离群簇非常远,则\({\Vert x_i-x_j \Vert}^2\)的值会很大,而\(p_{ij}\)会相应变得非常小,也就是说\(x_i\)的位置很远这件事情对损失函数影响很小(没有足够的惩罚),那这个点在低维空间中将无法从其他点中区分出来。因此Maaten提出了对称的条件概率来重新定义上述联合概率\(p_{ij}\),对于数量为\(n\)的数据点,新的概率公式是:

\[
p_{ij}=\frac{p_{j|i}+p_{i|j}}{2n}
\]

至此,损失函数可修改为:
\[
C=KL(P||Q)=\sum_i \sum_j p_{ij}\log \frac{p_{ij}}{q_{ij}}
\]

对\(y_i\)的梯度也一下子简化了不少:

\[
\frac{\delta C}{\delta y_i}=4\sum_{j}(p_{ij}-q_{ij})(y_i-y_j)
\]

2.2 拥挤问题

拥挤问题,就是从高维空间映射到低维空间后,不同类别的簇容易挤在一起,无法较好地区分开。名字就表明了意图,t-sne解决这个问题的方式是t分布,准确地说是自由度为1的\(t\)分布,也就是柯西分布。\(t\)分布具有长尾特性,相比于正太分布,柯西分布在尾部趋向于0的速度更慢。因此t-sne在低为空间中引入这种分布,既能解决拥挤问题,又能解决优化问题。

因此,使用柯西分布重新定义\(q_{ij}\):

\[
q_{ij}=\frac{ (1 + \left \| y_i-y_j \right \|^2)^{-1}}{\sum_{k \neq l} (1 + \left \| y_k-y_l \right \|^2)^{-1}}
\]

同样的,梯度变为:

\[
\frac{\delta C}{\delta y_i}=4\sum_{j}(p_{ij}-q_{ij})(y_i-y_j)(1 + \left \| y_i-y_j \right \|^2)^{-1}
\]

至于为什么要选择这样的分布,由于柯西分布尾部更厚的特性特性,因此,\((1-{\Vert y_i-y_j \Vert}^2)^{-1}\)几乎就是与\(\Vert y_i-y_j \Vert\)直接成反比,而后者能表示在低维空间中相距较大的点对,接近平方反比定律,也就是说,能够使得相似度低的对象能够更好地拉开距离。

总而言之,可以看出t-SNE的终极目标是通过最小化数据点在高维空间和低维空间中的KL距离,来保持相邻的点在两个不同空间中的分布是相似性。在高维空间中,使用高斯分布来刻划两个点之间的相似度,而在低维空间中,使用t分布来计算。
论文提供了较多实验分析,很详细,在此略过,下图是其中MNIST数据集的实验效果。

3. 应用与误读

降维和探索数据,是比较常讲的两个词,我认为本质上说,是一种降维思想的技术方法,但实际用途,目前来说更多的是辅助探索数据。原因么,很难可视化的数据被可以用二维或三维的点来表示,所有的团簇和距离关系被拉到了人眼所能理解的笛卡尔坐标系下面,这种状态下,相比于机器模式识别,更是人眼的强项。

对于模型的不同参数作用于不同数据的理解,强烈推荐distill.pub上的这篇文章:How to Use t-SNE Effectively,另外,云栖社区上有个翻译版。这篇文章里罗列了大量的动态可视化的效果,来说明不同数据、不同参数对可视化效果的影响,也感叹随着模型灵活度的提高,分析结果的理解难度也随之加大。

具体,作者有如下有学习意义的结论或部分猜测:
1. Those hyperparameters really matter
2. Cluster sizes in a t-SNE plot mean nothing
3. Distances between clusters might not mean anything
4. Random noise doesn’t always look random
5. You can see some shapes, sometimes
6. For topology, you may need more than one plot

这篇文章对于理解模型各部件对数据具体可能带来什么影响,以及减少对数据的误读,有非常直观的实验意义。正如作者对文章url的命名:misread-tsne。

173 thoughts on “SNE与t-SNE降维算法理解

  1. Keturah Voisine说道:

    You really make it seem so easy with your presentation but I find this matter to be really something that I think I would never understand. It seems too complex and extremely broad for me. I am looking forward for your next post, I’ll try to get the hang of it!

  2. website link说道:

    I just want to say I’m newbie to blogs and truly savored you’re blog site. Almost certainly I’m likely to bookmark your blog . You certainly come with incredible articles. Many thanks for sharing with us your website page.

  3. kumahasiaweh说道:

    Hi there, I found your website by way of Google at the same time as looking for a related matter, your web site got here up, it appears to be like great. I have bookmarked it in my google bookmarks.

  4. MetroClick specializes in building completely interactive products like Photo Booth for rental or sale, Touch Screen Kiosks and Digital Signage, and experiences. With our own hardware production facility and in-house software development teams, we are able to achieve the highest level of customization and versatility for Photo Booths, Touch Screen Kiosks and Digital Signage. MetroClick, 121 Varick St, #301, New York, NY 10013, +1 646-843-0888

  5. Faytech North America is a touch screen Manufacturer of both monitors and pcs. They specialize in the design, development, manufacturing and marketing of Capacitive touch screen, Resistive touch screen, Industrial touch screen, IP65 touch screen, touchscreen monitors and integrated touchscreen PCs. Contact them at http://www.faytech.us, 121 Varick Street,3rd Floor,New York, NY 10013,+1 646 205 3214

  6. dentysta gorzów说道:

    Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something. I think that you could do with a few pics to drive the message home a bit, but other than that, this is excellent blog. An excellent read. I will certainly be back.

  7. I truly appreciate this post. I’ve been looking all over for this! Thank goodness I found it on Bing. You have made my day! Thanks again!

  8. It can be nearly impossible to find well-educated women and men on this subject, even though you seem like you understand the things that you’re writing on! Excellent

  9. additional hints说道:

    Seriously absorbing knowledge that you have stated, thanks a lot for posting.

  10. dentysta gorzów说道:

    Almost all of the things you say happens to be astonishingly appropriate and that makes me ponder why I had not looked at this in this light previously. This particular piece really did switch the light on for me as far as this issue goes. Nevertheless at this time there is actually one particular point I am not necessarily too cozy with so whilst I try to reconcile that with the actual core idea of the point, let me observe exactly what the rest of your readers have to say.Well done.

  11. HunterDum说道:

    分析王http://sport-analysis.com

  12. I will right away grasp your rss as I can not to find your email subscription hyperlink or e-newsletter service. Do you have any? Kindly let me know so that I may subscribe. Thanks.

  13. Enjoyed reading this, very good stuff, appreciate it. “Whenever you want to marry someone, go have lunch with his ex-wife.” by Francis William Bourdillon.

  14. I simply want to notify you that I am new to blogging and completely valued your page. Quite possibly I am prone to bookmark your blog post . You simply have amazing article material. Love it for expressing with us your favorite domain write-up

  15. address说道:

    It’s the best opportunity to construct some goals for the future. I have looked over this document and if I may possibly, I want to encourage you a few helpful tips.

  16. I just couldn’t depart your website before suggesting that I extremely enjoyed the standard information a person supply to your visitors? Is gonna be back frequently to inspect new posts

  17. magnificent points altogether, you simply won a new reader. What may you recommend about your submit that you made some days ago? Any positive?

  18. Very insightful knowledge you have stated, thanks so much for publishing.

  19. Flowers cork说道:

    Wow! This blog looks exactly like my old one! It’s on a completely different subject but it has pretty much the same layout and design. Outstanding choice of colors!

  20. dentysta gorzów说道:

    Hey there! I know this is somewhat off topic but I was wondering if you knew where I could get a captcha plugin for my comment form? I’m using the same blog platform as yours and I’m having problems finding one? Thanks a lot!

  21. Amazing! This blog looks just like my old one! It’s on a totally different topic but it has pretty much the same page layout and design. Great choice of colors!

  22. Its like you read my mind! You seem to know a lot about this, like you wrote the book in it or something. I think that you could do with some pics to drive the message home a bit, but other than that, this is magnificent blog. A great read. I will definitely be back.

  23. I have been absent for some time, but now I remember why I used to love this website. Thank you, I will try and check back more frequently. How frequently you update your site?

  24. see说道:

    I merely need to advise you that I am new to blog posting and certainly cherished your article. More than likely I am likely to remember your blog post . You undoubtedly have superb article content. Appreciate it for swapping with us the best site post

  25. I truly enjoy examining on this website , it holds wonderful articles . “Something unpredictable but in the end it’s right, I hope you have the time of your life.” by Greenday.

  26. F*ckin’ remarkable issues here. I’m very glad to see your post. Thank you so much and i am taking a look ahead to contact you. Will you kindly drop me a mail?

  27. Definitely, what a fantastic site and revealing posts, I definitely will bookmark your blog.Have an awsome day!

  28. Of course, what a magnificent site and instructive posts, I will bookmark your blog.Best Regards!

  29. I’ve been absent for a while, but now I remember why I used to love this web site. Thanks , I¡¦ll try and check back more often. How frequently you update your website?

  30. Thank you for the sensible critique. Me and my neighbor were just preparing to do a little research about this. We got a grab a book from our area library but I think I learned more from this post. I’m very glad to see such wonderful information being shared freely out there.

  31. Some really excellent articles on this internet site, appreciate it for contribution. “For today and its blessings, I owe the world an attitude of gratitude.” by Clarence E. Hodges.

  32. I wanted to put you that tiny note to help say thanks a lot yet again for those awesome principles you’ve contributed on this site. It has been so remarkably generous with you giving without restraint precisely what numerous people could possibly have made available for an e-book to get some money for their own end, specifically now that you might well have done it in case you decided. Those tactics additionally worked to be a great way to be aware that most people have the same fervor similar to my very own to find out a little more when it comes to this matter. I believe there are many more fun instances in the future for individuals who discover your site.

  33. Hello my family member! I want to say that this post is amazing, great written and come with almost all important infos. I’d like to look more posts like this.

  34. I think this is among the most significant information for me. And i’m glad reading your article. But wanna remark on few general things, The web site style is perfect, the articles is really excellent : D. Good job, cheers

  35. Hi there! I just wanted to ask if you ever have any problems with hackers?
    My last blog (wordpress) was hacked and I ended up losing several weeks of hard work due
    to no backup. Do you have any solutions to protect against hackers?

  36. Health & Fitness说道:

    Excellent read, I just passed this onto a colleague who was doing a little research on that. And he actually bought me lunch because I found it for him smile So let me rephrase that: Thanks for lunch!

  37. MichaelJemery.com is a site with many hypnosis downloads. Whether you are looking for free hypnosis downloads, self hypnosis download for mp3, video and any audio files, Michael Jemery has the downloads for you. You can download hypnosis from apps, audio, mp3 and even youtube !

  38. Well I really enjoyed reading it. This tip procured by you is very practical for proper planning.

  39. biocentro.cl说道:

    Cursos masajes para toda la gente de todas las edades.

  40. I have to show my thanks to this writer just for rescuing me from such a dilemma. After looking out throughout the the net and seeing suggestions which were not powerful, I was thinking my entire life was well over. Being alive without the answers to the difficulties you’ve solved by means of the site is a critical case, as well as ones which might have badly affected my career if I hadn’t discovered your web page. Your own competence and kindness in handling all the things was excellent. I’m not sure what I would have done if I had not encountered such a step like this. It’s possible to now look forward to my future. Thanks a lot very much for this expert and sensible guide. I won’t hesitate to suggest your blog to any individual who will need assistance on this subject matter.

  41. Travel & Leisure说道:

    We are a group of volunteers and starting a new scheme in our community. Your website provided us with valuable info to work on. You’ve done an impressive job and our entire community will be grateful to you.

  42. Automotive说道:

    Hello, i think that i saw you visited my website so i came to “return the favor”.I am attempting to find things to enhance my site!I suppose its ok to use some of your ideas!!

  43. hey there and thank you for your info – I have certainly picked up something new from right here. I did however expertise several technical issues using this site, as I experienced to reload the web site many times previous to I could get it to load properly. I had been wondering if your web host is OK? Not that I am complaining, but sluggish loading instances times will sometimes affect your placement in google and can damage your high-quality score if ads and marketing with Adwords. Anyway I’m adding this RSS to my email and could look out for a lot more of your respective interesting content. Ensure that you update this again very soon..

  44. website url说道:

    Emeryeps is a SEO(Search Engine optimzation) and Internet Marketing company. They help businesses to get traffic from various search engine and online community. They have seo experts and consultants with many years of SEO Experiences. No matter where your business is located, EmeryEPS.com can help your business to secure your highly convertible leads online.

  45. Only a smiling visitor here to share the love (:, btw great design .

  46. dentysta gorzow说道:

    Thanks, will check back BLOGURL shortly, have bookmarked you right now. NAME

  47. dentysta gorzów说道:

    An impressive share, I simply given the onto a colleague who had previously been doing a bit analysis for this: BLOGTITLE. And he in particular fact purchased me breakfast from I found it intended for him.. smile. So let me reword which usually: Thnx for the heal! However yeah Thnkx for spending enough time to debate this, I really feel strongly to fix it and love reading more with this topic. If attainable, as you become practical experience, would you thoughts updating your weblog a lot easier particulars? It is extremely a good choice for me. Massive thumb up of this BLOGURL! NAME

  48. I must express some thanks to the writer just for rescuing me from this particular condition. Right after checking through the search engines and getting concepts that were not helpful, I was thinking my entire life was over. Being alive devoid of the approaches to the difficulties you have fixed as a result of your guideline is a serious case, as well as the kind which could have badly damaged my career if I had not discovered the blog. Your good ability and kindness in controlling every aspect was helpful. I don’t know what I would’ve done if I had not come across such a stuff like this. I am able to now relish my future. Thanks for your time so much for your impressive and amazing guide. I won’t think twice to propose your web sites to any individual who needs to have direction about this situation.

  49. Very interesting details you have mentioned , regards for putting up. “A big man is one who makes us feel bigger when we are with him.” by John C. Maxwell.

  50. Keep up the superb piece of work, I read few blog posts on this website and I think that your site is real interesting and contains sets of great info .

  51. Wow, superb blog layout! How long have you been blogging for? you make blogging look easy. The overall look of your site is magnificent, as well as the content!

  52. I do consider all of the concepts you have offered on your post. They are really convincing and will certainly work. Still, the posts are very brief for starters. May just you please prolong them a bit from subsequent time? Thank you for the post.

  53. I’ve read several excellent stuff here. Certainly value bookmarking for revisiting. I surprise how a lot effort you place to make one of these magnificent informative site.

  54. Wow! This can be one particular of the most useful blogs We have ever arrive across on this subject. Actually Great. I’m also an expert in this topic so I can understand your effort.

  55. I dugg some of you post as I cogitated they were very beneficial very helpful

  56. There is visibly a bunch to realize about this. I suppose you made some nice points in features also.

  57. Fantastic website. Lots of helpful information here. I’m sending it to several pals ans additionally sharing in delicious. And naturally, thanks to your sweat!

  58. I loved as much as you will receive carried out right here. The sketch is attractive, your authored subject matter stylish. nonetheless, you command get bought an impatience over that you wish be delivering the following. unwell unquestionably come more formerly again since exactly the same nearly a lot often inside case you shield this increase.

  59. I consider something truly interesting about your blog so I saved to my bookmarks .

  60. Whats Taking place i’m new to this, I stumbled upon this I have discovered It positively useful and it has aided me out loads. I hope to contribute & aid other customers like its helped me. Good job.

  61. harga hp xiaomi说道:

    My programmer is trying to persuade me to move to .net from PHP. I have always disliked the idea because of the expenses. But he’s tryiong none the less. I’ve been using Movable-type on a variety of websites for about a year and am anxious about switching to another platform. I have heard good things about blogengine.net. Is there a way I can transfer all my wordpress posts into it? Any kind of help would be greatly appreciated!

  62. My coder is trying to persuade me to move to .net from PHP. I have always disliked the idea because of the costs. But he’s tryiong none the less. I’ve been using Movable-type on various websites for about a year and am nervous about switching to another platform. I have heard fantastic things about blogengine.net. Is there a way I can transfer all my wordpress content into it? Any help would be greatly appreciated!

  63. hi!,I like your writing very so much! percentage we communicate more about your article on AOL? I require an expert in this space to unravel my problem. May be that’s you! Taking a look ahead to peer you.

  64. pron best说道:

    pKcOeR if you are if you are in an apartment that is confined, then folding tables would be very well suited for you;;

  65. Howdy! This post could not be written much better! Looking at this article reminds me of my previous roommate! He continually kept preaching about this. I’ll send this article to him. Fairly certain he’ll have a good read. Thanks for sharing!

  66. Whoa! This blog looks just like my old one! It’s on a completely different subject but it has pretty much the same layout and design. Superb choice of colors!

  67. Good day! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us valuable information to work on. You have done a marvellous job!

  68. Woah! I’m really enjoying the template/theme of this site. It’s simple, yet effective. A lot of times it’s challenging to get that “perfect balance” between usability and appearance. I must say you’ve done a amazing job with this. In addition, the blog loads very quick for me on Internet explorer. Outstanding Blog!

  69. My developer is trying to convince me to move to .net from PHP. I have always disliked the idea because of the expenses. But he’s tryiong none the less. I’ve been using WordPress on several websites for about a year and am anxious about switching to another platform. I have heard excellent things about blogengine.net. Is there a way I can import all my wordpress posts into it? Any help would be greatly appreciated!

  70. I haven’t checked in here for a while as I thought it was getting boring, but the last few posts are great quality so I guess I’ll add you back to my everyday bloglist. You deserve it my friend 🙂

  71. new kia rio说道:

    Hi, I do believe this is a great website. I stumbledupon it 😉 I may revisit once again since I bookmarked it. Money and freedom is the greatest way to change, may you be rich and continue to guide others.

  72. suba me说道:

    kKZOln This blog is without a doubt interesting and informative. I have picked up many handy things out of this blog. I ad love to go back again soon. Thanks a lot!

  73. Hi, I do think this is an excellent web site. I stumbledupon it 😉 I’m going to revisit yet again since I book marked it. Money and freedom is the best way to change, may you be rich and continue to guide others.

  74. Brooks说道:

    magnificent issues altogether, you just gained a emblem new reader. What may you recommend about your publish that you made some days ago? Any certain?

  75. Does your site have a contact page? I’m having a tough time locating it but, I’d like to shoot you an e-mail. I’ve got some ideas for your blog you might be interested in hearing. Either way, great site and I look forward to seeing it develop over time.

  76. info movies说道:

    This is one awesome post.Thanks Again. Keep writing.

  77. It’а†s in reality a nice and helpful piece of info. I am satisfied that you just shared this helpful info with us. Please stay us up to date like this. Thank you for sharing.

  78. Hi! I’ve been following your weblog for a long time now and finally got the bravery to go ahead and give you a shout out from Porter Tx! Just wanted to say keep up the great job!

  79. movies说道:

    Im grateful for the article post.Much thanks again. Really Cool.

  80. hungry mate说道:

    Well I definitely enjoyed reading it. This subject procured by you is very effective for good planning.

  81. Just what I was searching for, thankyou for posting.

  82. There is noticeably a bunch to know about this. I consider you made various nice points in features also.

  83. Home Improvement说道:

    I like this post, enjoyed this one thanks for posting. “The world is round and the place which may seem like the end may also be only the beginning.” by George Baker.

  84. Im thankful for the blog.Thanks Again. Will read on

  85. Doriserele说道:

    Revolutional update of SEO/SMM software “XRumer 16.0 + XEvil 3.0”:
    captcha breaking of Google, Facebook, Bing, Hotmail, SolveMedia, Yandex,
    and more than 8400 another subtypes of captcha,
    with highest precision (80..100%) and highest speed (100 img per second).
    You can connect XEvil 3.0 to all most popular SEO/SMM software: XRumer, GSA SER, ZennoPoster, Srapebox, Senuke, and more than 100 of other programms.

    Interested? There are a lot of demo videos about XEvil in YouTube.
    See you later!

    XRumer201707

  86. Thanks a lot for the blog article.Much thanks again. Cool.

  87. read more at说道:

    Some really prize content on this site, saved to bookmarks.

  88. chaturbate说道:

    Very nice article. I definitely love this site. Continue the good work!

  89. Woah! I’m really enjoying the template/theme of this blog. It’s simple, yet effective. A lot of times it’s very difficult to get that “perfect balance” between user friendliness and visual appearance. I must say you’ve done a very good job with this. Also, the blog loads very quick for me on Safari. Exceptional Blog!

  90. شهرزاد说道:

    Thanks a lot for the post.Thanks Again. Keep writing.

  91. دانلود说道:

    It as difficult to find well-informed people for this topic, but you sound like you know what you are talking about! Thanks

  92. Thank you ever so for you article post.Really thank you! Awesome.

  93. to write some content for your blog in exchange for a link back to

  94. Great, thanks for sharing this blog post.Much thanks again. Really Great.

  95. You have noted very interesting details! ps nice internet site.

  96. Thanks so much for the blog.Really thank you! Really Great.

  97. پنجره说道:

    Very nice article, exactly what I wanted to find.

  98. Thanks for sharing, this is a fantastic article.Really thank you! Great.

  99. intporn free porn forums adult xxx port pussy vagina

  100. Im thankful for the blog. Really Cool.

  101. Health说道:

    Hello, you used to write excellent, but the last few posts have been kinda boring… I miss your tremendous writings. Past several posts are just a little out of track! come on!

  102. Whats up! I just wish to give an enormous thumbs up for the good data you have right here on this post. I will likely be coming back to your blog for extra soon.

  103. Really appreciate you sharing this blog post.Really thank you! Want more.

  104. Very interesting information!Perfect just what I was searching for! Being rich is having money being wealthy is having time. by Margaret Bonnano.

  105. My brother recommended I might like this blog. He was totally right. This post actually made my day. You cann at imagine simply how much time I had spent for this info! Thanks!

  106. http://c-am4.us说道:

    It is really a nice and helpful piece of information. I am glad that you shared this useful information with us. Please keep us up to date like this. Thanks for sharing.

  107. Yes, you are correct friend, on a regular basis updating website is in fact needed in support of SEO. Fastidious argument keeps it up.

  108. Business说道:

    Thanks, I have just been looking for information approximately this topic for a long time and yours is the greatest I’ve came upon till now. But, what concerning the bottom line? Are you certain concerning the source?

  109. Pretty! This has been an incredibly wonderful post. Many thanks

  110. Say, you got a nice blog post.Really thank you! Awesome.

  111. Where else may anybodаАааБТ“ fаАаб‚Т€Т“gure out that kin? аА аБТ•f info in

  112. repair service, routine maintenance and electricity conservation of economic roofing systems will probably be as cost-effective as is possible. And using this

  113. Hi there, just wanted to say, I enjoyed this article.
    It was inspiring. Keep on posting!

  114. gay video chat说道:

    Wow, amazing blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your web site is fantastic, as well as the content!

  115. I always spent my half an hour to read this weblog’s
    articles all the time along with a mug of coffee.

  116. Women说道:

    I really appreciate this post. I’ve been looking all over for this! Thank goodness I found it on Bing. You’ve made my day! Thx again!

  117. Im no pro, but I feel you just crafted an excellent point. You certainly understand what youre talking about, and I can really get behind that. Thanks for staying so upfront and so truthful.

  118. motivator说道:

    I cannot thank you enough for the blog post. Fantastic.

  119. robloxwiki说道:

    Woah! I’m really enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s hard to get that “perfect balance” between superb usability and visual appearance. I must say that you’ve done a superb job with this. In addition, the blog loads super fast for me on Internet explorer. Excellent Blog!

  120. Hi, I do think this is a great site. I stumbledupon it 😉 I will return yet again since I bookmarked it. Money and freedom is the best way to change, may you be rich and continue to guide other people.

  121. wow, awesome article post.Really looking forward to read more. Great.

  122. Career And Jobs说道:

    I like the efforts you have put in this, thank you for all the great articles.

  123. This is one awesome blog.Much thanks again. Really Great.

  124. Wow, wonderful blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your website is fantastic, let alone the content!

  125. What as up to every body, it as my first pay a visit of this web site; this website consists of amazing and genuinely good data designed for visitors.

  126. Does your website have a contact page? I’m having
    trouble locating it but, I’d like to send you an email. I’ve got some suggestions for your blog you might be interested in hearing.
    Either way, great blog and I look forward to seeing it grow over time.

  127. I’а†ve learn a few excellent stuff here. Certainly worth bookmarking for revisiting. I surprise how a lot attempt you set to make this kind of wonderful informative website.

  128. you might have an incredible blog here! would you like to make some invite posts on my weblog?

  129. The gang of four should match a final four digits
    of your plastic card account number.

  130. This blog is obviously awesome and also amusing. I have discovered many useful stuff out of it. I ad love to come back over and over again. Thanks!

  131. Fashion说道:

    Rattling superb information can be found on website . “We should be eternally vigilant against attempts to check the expression of opinions that we loathe.” by Oliver Wendell Holmes.

  132. دانلود说道:

    Usually I don at learn post on blogs, but I would like to say that this write-up very compelled me to check out and do so! Your writing taste has been surprised me. Thanks, very nice post.

  133. Does your website have a contact page? I’m having problems locating it but, I’d like to shoot you an email. I’ve got some creative ideas for your blog you might be interested in hearing. Either way, great site and I look forward to seeing it develop over time.

  134. I conceive this internet site has some real excellent info for everyone. “Glory is fleeting, but obscurity is forever.” by Napoleon.

  135. I love the efforts you have put in this, thank you for all the great articles.

  136. Technology说道:

    I like this web site very much, Its a very nice place to read and receive info . “The mark of a good action is that it appears inevitable in retrospect.” by Robert Louis Stephenson.

  137. I enjoy what you guys tend to be up too. This kind of clever work and exposure! Keep up the terrific works guys I’ve you guys to my own blogroll.

  138. Buy Yugioh Cards说道:

    This actually answered my drawback, thanks!

  139. Secret Flask说道:

    Recommeneded websites Here you all find some sites that we think you all appreciate, just click the links over

  140. 迷你仓说道:

    Thanks for sharing, this is a fantastic blog.Much thanks again.

  141. You ave made some really good points there. I looked on the internet to find out more about the issue and found most people will go along with your views on this website.

  142. Doriserele说道:

    Absolutely NEW update of SEO/SMM software “XRumer 16.0 + XEvil 3.0”:
    captcha breaking of Google, Facebook, Bing, Hotmail, SolveMedia, Yandex,
    and more than 8400 another subtypes of captchas,
    with highest precision (80..100%) and highest speed (100 img per second).
    You can connect XEvil 3.0 to all most popular SEO/SMM programms: XRumer, GSA SER, ZennoPoster, Srapebox, Senuke, and more than 100 of other programms.

    Interested? There are a lot of demo videos about XEvil in YouTube.
    See you later!

    XRumer201707

  143. Trein说道:

    Major thankies for the article.Really thank you! Cool.

  144. Treinreizen说道:

    That is a very good tip particularly to those fresh to the blogosphere. Short but very precise info Thanks for sharing this one. A must read article!

  145. clickmeter说道:

    This awesome blog is really awesome as well as diverting. I have picked helluva helpful advices out of this source. I ad love to come back again and again. Thanks a lot!

  146. Pets说道:

    Thanks for the sensible critique. Me & my neighbor were just preparing to do some research on this. We got a grab a book from our local library but I think I learned more clear from this post. I am very glad to see such magnificent info being shared freely out there.

  147. Heya i am for the first time here. I found this board and I find It
    really useful & it helped me out a lot. I hope to give something back and help others like you helped me.

  148. I went over this site and I think you have a lot of fantastic information, bookmarked (:.

  149. Women's clothing说道:

    It is lovely worth sufficient for me. Personally,

  150. Breast Actives说道:

    I think that you can do with some pics to drive the message home a bit,

  151. Thanks a lot for the blog article. Much obliged.

  152. Provillus说道:

    Wow, what a video it is! Actually nice quality video, the lesson given in this video is actually informative.

  153. This awesome blog is really awesome and besides amusing. I have discovered helluva handy advices out of this amazing blog. I ad love to return over and over again. Thanks a bunch!

  154. I truly appreciate this post.Really looking forward to read more. Want more.

  155. Thanks for the article.Really looking forward to read more. Awesome.

  156. Very neat blog.Much thanks again. Really Great.

  157. This website was… how do you say it? Relevant!!
    Finally I have found something which helped me.

    Cheers!

  158. Weed banks说道:

    Loving the article.. thanks for your insight Value the admission you delivered.. My web searches seem complete.. thank you. of course, analysis is paying off.

  159. No matter if some one searches for his necessary
    thing, so he/she wants to be available that in detail, thus that thing is
    maintained over here.

  160. spam说道:

    Right now it seems like Drupal could be the preferred blogging platform available at the moment. (from what I ave read) Is the fact that what you are using in your weblog?

  161. very nice submit, i certainly love this website, carry on it

  162. I was reading some of your blog posts on this internet site and I believe this website is very instructive! Retain putting up.

  163. Hello friends, nice post and nice arguments commented at this place, I am truly
    enjoying by these.

  164. Special reef说道:

    WOW just what I was searching for. Came here by searching for Coral Aquarium

  165. Digital Agency说道:

    I’m not positive the place you are getting your info, however great topic.
    I must spend a while finding out much more or working out more.
    Thank you for fantastic info I was looking for this info for
    my mission.

  166. It’s awesome to pay a quick visit this web page and reading the views
    of all mates regarding this post, while I am also keen of getting experience.

  167. Schweiz Aquarium说道:

    It’s awesome to go to see this website and reading the views of all mates regarding this
    post, while I am also eager of getting familiarity.

  168. Lihat artikel ini untuk informasi selengkapnya.

  169. Asian dating might very well be an excellent thing in your case so you
    may find yourself inside a married position in a month or two
    in case you start dating now. Whereas before seeking a bride from China would have been irregular,
    nowadays it’s nothing with the sort. But Norman and
    Samantha’s main problem is they are extreme opposites.

  170. hdmi to vga说道:

    Admiring the time and energy you put into your site and
    in depth information you provide. It’s nice to come across
    a blog every once in a while that isn’t the same out of date rehashed information. Fantastic
    read! I’ve bookmarked your site and I’m including your RSS feeds to my Google account.

  171. I like the helpful info you provide in your articles.
    I’ll bookmark your weblog and check again here regularly.

    I’m quite sure I’ll learn a lot of new stuff right here!
    Good luck for the next!

发表评论

电子邮件地址不会被公开。