“"What's ubiquitous and cheap?" [Google’s Hal] Varian asks. "Data." And what is scarce? The analytic ability to utilize that data.”
The June issue of Wired has an excellent article by Steven Levy, entitled Secret of Googlenomics: Data-Fueled Recipe Brews Profitability. The article delves into the history and algorithms behind Google’s auction based ad system, highlighting the significance of engineering, mathematics, economics, and data mining in Google’s success.
On the economics front, the article explains Hal Varian’s role as Chief Economist at Google, including why Google needs a chief economist:
“The simplest reason is that the company is an economy unto itself. The ad auction, marinated in that special sauce, is a seething laboratory of fiduciary forensics, with customers ranging from giant multinationals to dorm-room entrepreneurs, all billed by the world's largest micropayment system.
Google depends on economic principles to hone what has become the search engine of choice for more than 60 percent of all Internet surfers, and the company uses auction theory to grease the skids of its own operations. All these calculations require an army of math geeks, algorithms of Ramanujanian complexity, and a sales force more comfortable with whiteboard markers than fairway irons.”
After reading the article, Varian’s economic view of data ubiquity and analytic scarcity really stuck with me. The quote I opened the post with isn’t directed at software availability or processing power. It refers to the scarcity of people qualified to churn abundant data into economic value.
What follows are some excerpts “about harnessing supply and demand”. The sub-headers and emphasis are mine.
Enter Econometricians
"The people working for me are generally econometricians—sort of a cross between statisticians and economists," says Varian, who moved to Google full-time in 2007 (he's on leave from Berkeley) and leads two teams, one of them focused on analysis.
"Google needs mathematical types that have a rich tool set for looking for signals in noise," says statistician Daryl Pregibon, who joined Google in 2003 after 23 years as a top scientist at Bell Labs and AT&T Labs. "The rough rule of thumb is one statistician for every 100 computer scientists."
Ubiquitous Data
“As the amount of data at the company's disposal grows, the opportunities to exploit it multiply, which ends up further extending the range and scope of the Google economy…
Keywords and click rates are their bread and butter. "We are trying to understand the mechanisms behind the metrics," says Qing Wu, one of Varian's minions. His specialty is forecasting, so now he predicts patterns of queries based on the season, the climate, international holidays, even the time of day. "We have temperature data, weather data, and queries data, so we can do correlation and statistical modeling," Wu says. The results all feed into Google's backend system, helping advertisers devise more-efficient campaigns.”
Continuous Analysis
“To track and test their predictions, Wu and his colleagues use dozens of onscreen dashboards that continuously stream information, a sort of Bloomberg terminal for the Googlesphere. Wu checks obsessively to see whether reality is matching the forecasts: "With a dashboard, you can monitor the queries, the amount of money you make, how many advertisers you have, how many keywords they're bidding on, what the rate of return is for each advertiser."”
Behavioral Based Insights
“Wu calls Google "the barometer of the world." Indeed, studying the clicks is like looking through a window with a panoramic view of everything. You can see the change of seasons—clicks gravitating toward skiing and heavy clothes in winter, bikinis and sunscreen in summer—and you can track who's up and down in pop culture. Most of us remember news events from television or newspapers; Googlers recall them as spikes in their graphs. "One of the big things a few years ago was the SARS epidemic," Tang says. Wu didn't even have to read the papers to know about the financial meltdown—he saw the jump in people Googling for gold. And since prediction and analysis are so crucial to AdWords, every bit of data, no matter how seemingly trivial, has potential value.”
Rise of the Datarati
“Varian believes that a new era is dawning for what you might call the datarati—and it's all about harnessing supply and demand. "What's ubiquitous and cheap?" Varian asks. "Data." And what is scarce? The analytic ability to utilize that data. As a result, he believes that the kind of technical person who once would have wound up working for a hedge fund on Wall Street will now work at a firm whose business hinges on making smart, daring choices—decisions based on surprising results gleaned from algorithmic spelunking and executed with the confidence that comes from really doing the math.”
Now, a few questions I think folks should consider:
- Who does that math in your organization?
- Does your analytics / active information strategy suffer from information processing richness and insight scarcity?
- Who are, or should be, your datarati?
Perhaps I’m just grumpy this week. Or, concerned for the future. Or, most likely, both. Nevertheless, I find conventional SOA lore more bothersome than usual. Specifically, the paired notions that the sole reason to implement services (or not) is re-use potential, and that the main architectural aspect of SOA is governing said services for re-use. Now, don’t misinterpret, there is true value in sharing services and governance is critical. However, SOA, or better said, services-architecture doesn’t begin and end with re-use potential and enforcement. 