Link Collection- July 17, 2011

• Preview of Storm: The Hadoop of Realtime Processing – BackType Technology

  Curious to see what happens with Storm, post-Twitter acquisition of BackType

  “There are three broad use cases for Storm:

  1. Stream processing: This is the traditional realtime processing use case: process messages and update a variety of databases.

  2. Continuous computation: Storm can be used to do a continuous computation and stream out the results as they’re computed. For example, we used Storm the other day to compute trending users on Twitter off of the Twitter firehose. Every second, Storm streams out the 50 users with the most retweets in the last few minutes with perfect accuracy. We stream this information directly into a webpage which visualizes and animates the trending users in realtime.

  3. Distributed RPC: Distributed RPC is perhaps the most unexpected and most compelling use case for Storm. There are a lot of queries that are both hard to precompute and too intense to compute on the fly on a single machine.”

  tags: storm backtype oss real-time

• The challenges of streaming real-time data – O’Reilly Radar

  “You would be shocked at the ratio of engineers who can’t build event-driven, asynchronous data processing applications, to those who can, yet this is a big part of this space.”

  “…Jud Valeski: “Big data” as we talk about it today has been slayed by lots of cool abstractions (e.g. Hadoop) that fit nicely into the way we think about the stack we all know and love. “Big streams,” on the other hand, challenge the parallelization primitives folks have been solving for “big data.” There’s very little overlap, unfortunately. So, on the software solution side, better and more widely used frameworks are needed…”

  tags: real-time streaming event-driven

Posted from Diigo. The rest of my favorite links are here.