Does your Data have a Confidence Score?
Every time you’ve applied for credit, the decision is made quickly, easily, and consistently based on a key piece of data – a credit score. Have you ever thought about credit scores, and what happened before they existed? Granting credit was a time-intensive processes plagued by bias, data gather errors, and inconsistent decisions. Does that sound like your big data & analytics programs today? According to research, up to 80% of time is spent finding, fixing, and integrating data. What’s more, 12% of the time is spent defending data and re-validating it. That doesn’t leave much time for the real point – analyzing and using data to make better decisions. Isn’t there a way this process could be expedited?
Yes, there is. What if every piece of data had a confidence score? Just like a credit score, that information confidence score would be well understood by decision-makers within your organization. Well, it’s not just a possibility, it’s a reality. The need for confidence has never been greater. With the explosion of big data, organizations are exponentially increasing the confidence problem. (More data) x (uncertainty) = a greater level of uncertainty. By 2015, 80% of all data will be uncertain. You can already see the cracks in the foundation today. 1 in 3 business leaders don’t trust the information they use to make important decisions.
Some progressive organizations have attempted to address data uncertainty already. But even the most advanced organizations tend to address only one aspect of confidence. Most commonly, organizations may understand data lineage and use that to approximate confidence. “Where did you get that data from” is still the most common way of saying “Should I have any confidence in that data”. But there are many more factors in determining confidence. System integrity – how many systems had the same data value vs. being in conflict? Governance – were policies followed? Correctness – is the data validated, verified, and standardized? Completeness – are records complete, and do we have a common view of master data records? Secure and protected – is the data safe from breach and data loss? Currency – is the data up to date?
In fact, those seven factors all play a role in information confidence. And research has revealed that they aren’t just qualitative. You can put a number to each of them, and how they affect each other. In other words, you can calculate an information confidence score. Just like a credit score, you can start to develop confidence levels for different usages of data. Doing social media sentiment analysis? No problem – your confidence needs to be at 590. Making decisions for long term company strategy? Well, now your confidence level needs to be 670. For any decision you need to make, you can determine how confident you need to be, and how confident you are. This is a significant breakthrough for organizations. The ‘black magic’ and ‘art’ of determining confidence are gone. It’s a science. And the days of saying “We don’t even know how to begin determining information confidence, so we won’t even bother” are over. A new era of data transparency and confidence has begun.
Calculating scores is one thing, making it part of your day to day business is quite another. So how do you put this into effect? Information confidence should be calculated when you encounter data – whether that is via integration, or whether you place data in a landing zone to assess its value. Information confidence should be calculated and then stored with the data itself – in other words, it is metadata. And when it is done as part of an Information Integration and Governance technology, it can be done seamlessly when data is integrated, and while it is being governed. By integrating information confidence into your overall data fabric, it becomes part of your applications and processes that are already utilizing the data.
Who owns data in your organization? If you’re like most, the answer might be “many people” or “I’m not sure”. While some organizations are investing in Chief Data Officers (CDO), the majority have not yet done so. And so the burden falls to you. Determine Information Confidence for your organization. Utilize this Information Confidence Calculator, take the results, and show them to any business leader who cares about or ‘owns’ customer data in your organization.
About David CorriganI’ve spent my entire career helping clients utilize emerging technology to solve their customer data problems. I've always enjoyed solving abstract problems. I've worked with hundreds of companies to utilize new technology, plan and drive to a roadmap, and evangelize and drive momentum for their information projects. During the day, I work on product strategy and marketing for @InfoTrellis, and I'm busy trying to disrupt the customer data and analytics market so that organizations can finally understand every single one of their customers. After hours, I like to take photographs, read, write, practice yoga, or watch soccer - Manchester United and Toronto FC are my teams of choice. Follow me on Twitter @DCorrigan or on LinkedIn at http://ca.linkedin.com/pub/david-corrigan/3/aa3/92.
- 360 view
- Big Data
- customer 360
- Customer big data
- Customer context matching
- Customer data
- customer intelligence management
- customer personalization
- Data Confidence
- Data Quality
- Data Warehousing
- Hadoop Systems & Analytics
- Information Governance
- Information Integration
- Information Lifecycle Management
- Master Data Management
- omnichannel personalization
- Privacy and Security
- Stream Computing
- Visual Context for Data
- Interesting read, & I've seen AI applied to form an identity graph for customers/prospects for better targeting.… twitter.com/i/web/status/1… 3 weeks ago
- Good read, and aligns with other stats that suggest that 67% of customers will churn after 1 bad experience - even… twitter.com/i/web/status/1… 3 weeks ago
- An interesting but also troubling read. There's huge upside to automation in industries like insurance, not to men… twitter.com/i/web/status/1… 1 month ago
- Have to retweet & elaborate - #graphbd is a great technology but the queries can be intense - part of the value of… twitter.com/i/web/status/1… 1 month ago
- RT @AllSight: "By putting intelligence before insight, you have the right context to get the right answers." - @DCorrigan #CIP #BigData #Da… 1 month ago