ON MINING "COMMENT SEARCH", THE NEXT BIG MOTHER LODE OF "UGC"

THE NEXT GOLD RUSH

Steve Rubel hits it out of the park with his first prediction for 2006, in a post titled "2006 Trends Part I:  Comment Search".  His prediction is one with which I've been in violent agreement for some time.

As he puts it:

"...shockingly, every single blog search engine is missing out on the next great opportunity. They're focusing solely on searching blogs. They're building great tools for bloggers, but they're failing to recognize that what we all need is a way to search the entire conversation. This is where the blog search war will be fought in the months ahead.

The high end vendors like Intelliseek get this, but others don't. The gang of four - Technorati, PubSub, Google and Ice Rocket - continue to pile on the features for mining blogs, as they seemingly ignore what will be the next great land grab - mining comments.

Blog comments have perhaps more collective wisdom inside them than any other form of consumer generated content. However, as of today, there's essentially no way to mine them."

I agree.  Blog comments are the next "mother lode" of "user-generated content" (UGC). 

Steve articulates well what others have described before, including myself.  As I put it, not as eloquently back in June, as point number 6 in a seven part wish list to improve blogging:

"While we're at it, this "Outlook for Blogs" service should also allow the user to deal with user's Comments that the user posts on OTHER, external blogs.  Specifically, the "comments" service should:  (a)  automatically keep track of the comments that the user posts on other blogs, along with the CONTEXT of the original post/s that the user is commenting on...and, (b) allow these comments and context posts to be either POSTED or DISPLAYED on the blog of the user's choice via the "Outlook for Blogs" interface."

Others have described this wish even better.

Here's an excerpt from Zoli Erdos, from his post responding to Steve Rubel above, succinctly describing the problem in a post two months ago:

"If you’re a Technorati top 100 or even 500 blogger, most of the conversation happens around your own blog, in the form of comments and trackbacks from other blogs.  However, for the the rest of us, the other 20 million bloggers, chances are the conversation really takes place outside our own blog, and I for one certainly can’t keep track of all comments I left on other blogs.  An occasional Google search on my name reveals lots of these “half-conversations” where I left a comment, the blog owner or other readers responded, but I’ve never seen the response, since I forgot to go back and-re-read all those blog-post."

As he goes on:

"The current crop of tracking / linking services all have a top-down publisher-centric view, everything revolves around a blog and related posts, totally missing this other, “bottom-up” half of the conversation.  Don’t we all  need something that shows an integrated view of all conversations where we are participating per subject matter (blog title), whether we started it or someone else?"

Yahoo!'s Jeremy Zawodny put it another way back in October on comment tracking:

"It took me a while to figure this out, but every aggregator I've seen has completely fails to make it easy to stay engaged in a discussion taking place in comments on one or more blog posts. I typically comment on a blog post and never remember to go back to see if anyone else commented on what I said. That's not much of a "conversation," is it?...

So comment tracking/monitoring ends up as a very manual process full of repeat visits, which means it's very, very hard to scale."

As I've touched on recently, Memeorandum is one new service in 2005 that comes closest to fulfilling this need of automatically tracking conversations ACROSS blogs, but it does so with OTHER BLOG POSTS ONLY, not the comments.

There are partial solutions to just tracking comments like email notifications when a new comment is added to a comment discussion you're interested in, and/or RSS comment feeds, but those are not entirely satisfying solutions.  Nor are they as easy to use as they should be for mainstream users, both to subscribe, and more importantly to UNSUBSCRIBE. 

Also, they're not the basis of entirely new web businesses around blogs and all the user-generated content.

Thinking about it, it's an even potential bigger opportunity than traditional blogs themselves.

There are possibly several times more folks who COMMENT on blogs as those who MAINTAIN their own blogs.  These are the "Lurkers" of old (remember the good old days of message boards?) who occasionally come out and say something when they really feel strongly about.

Imagine if every person who comments had a PRE-SET user name that worked on all blogs in the system.  Then imagine is that user-name could be used, with the user's permission of course, to construct a "virtual blog" for that user on the fly, listing their comments across various blogs, WITH the under-lying context.  Voila...we'd have millions of new bloggers overnight with their own virtual blogs, WITHOUT them having to go through the EFFORT OF MAINTAINING A REGULAR BLOG AT ALL.

Obviously, there are a lot more ways to mine the rich mother lode of comments out there.  It'll be interesting to see how/if it gets done in 2006.

Comments

[Niko]

As a relatively new blogger it's been quite frustrating for me to notice how the discussion in the comments is disconnected from the discussion that happens across blogs.

The current system also seems to lower the signal to noise ratio of blog conversations, as people can't effectively participate in "one conversation" but everybody needs to build the context again in their own posts, creating multiple, or layered conversations about the same topic. Trackbacks can connect these discussions technically, but not contextually.

Until we get a better system, I'll continue using a simple and more or less effective way of following the comment discussions I participate in. I've got a bookmarklet for adding pages to watchthatpage.com and the service reminds me of changed pages once a day...

[Chris Yeh]

It's a pretty cumbersome solution, but what I've been doing is tagging all the posts I comment on with del.icio.us.

http://del.icio.us/chrisyeh/yehcomment

But it requires a lot of manual work for something which should be automated!

888