January 5th, 2011

Data Quality for Real-time: Correctness and Tolerance

As part of my research agenda, I’m catching up on works by my thought-leading friends in the enterprise architecture field.  Today, I read Gene Leganza’s Top 15 Technology Trends EA Should Watch 2011 to 2013, published by Forrester’s EA Practice. [This is a client (for pay) report, which the good folks at Forrester shared with me.]

The themes of business visibility and responsiveness are present throughout the report, supported by several of the technology trends, including next generation business intelligence, business rules processing and policy-based SOA, smart technology management and event-driven patterns.

Tucked into the section [6-1] on Next-gen Business Intelligence (BI) is an extremely critical, and oft-overlooked, point on real-time processing and data quality:

“The shift from historical to real-time analytics will require that related processes such as data quality services also move to real time.”

The report continues to (correctly) state that “The complexity challenge [of Next-gen BI] will not be around the technologies per se but rather in the continued effort of gaining business consensus on data governance so that bad data is not driving strategic and operational decisions.”

I’ve written in the past how real-time operational adjustments can actually aid strategic decision-making, because the adjustments add a degree of correctness to the historical information base.  However, that’s only true when the operational decisions are based on good data. 

As Gene’s report points out, this is where things get tricky.  You need to re-think and reposition data quality procedures to achieve correctness in real-time operational decision-making. 

However, you need to be cognizant of the impact, in terms of delay, caused by adding data quality checks to real-time processing.  Obviously, the tolerance for delay will vary by business scenario, and perhaps even by transaction.

If the business scenario (or transaction) has a low tolerance for introducing delay, then you need to shift your tolerance view to impact of error.  If the outcome of the real-time decision falls within your error tolerance range, then proceed with the action.  If not, don’t force a real-time decision that your business might regret.

As you work through data governance policies for real-time, be sure to include tolerances for delay, correctness and decision (action or inaction) risk.

Print Friendly

Posted by brenda michelson at 4:28 pm in active information, change-friendly, information management | Permalink | Comments(2)
| Trackback URL

{ 1 comment… read it below or add one }

Francis Carden January 7, 2011 at 1:58 pm

Real-Time BI implies real-time measuring and there are few technologies out there that can do this at the coal-face (user desktops). It’s a rapidly growing space, the ability to capture in real-time what our users are doing. It’s been hard until now because 95% of the systems in use were never designed with this capability. OpenSpan has often been referred as a Next – Gen time and motion study capability when in reality, what’s closer to the truth is that we turn actual users / desktop applications into real-time emitters into BI tools. Enabling 20 year old to 1 day old on-premise or SAAS apps to emit events is what you have in your tool bag now. Detecting an error, as it’s occurring before a user has even completed a transaction – now THAT’s real-time!

Reply

Leave a Comment

{ 1 trackback }