A Tale of Rework & Opportunity Lost: Ungoverned Big Data


Has this ever happened to you? You want to use some information for a business purpose, only to realize that you are not permitted to use the information in that way? Sure it has. It happens all the time. It happened to one company who we will call ABC. ABC hired a Chief Data Officer, Bill. Bill’s first task was to inventory all of ABC’s data, and how and where it was being used ( learn how to inventory your data and manage governance policies by clicking here). He also had to take stock of ABC’s existing governance program, to figure out all of the existing policies and whether they were being followed.

Bill’s first few months on the job were great. By taking inventory of data, he was able to help many other people – the CFO, VP Sales, and VP Operations, by connecting them with the data they needed. Then along came “the retention project”. Mary, the Chief Marketing Officer, had commissioned many different projects – some of them with cloud-based apps and outside vendors. She was getting great results, and “the retention project” was one such example. She got that off the ground with a 3rd party vendor and cloud solution to analyze web data trends and make predictions on customer churn events. She married that data with her customer database to make decisions on who should be retained. The project was then handed over to internal IT to support, and that’s when Bill got involved.

As Bill’s team audited the ‘retention project’, they noted that customer web view data was being used without obtaining their consent. By using that data to make offers to the customer, it violated a privacy law. That was a fairly cut and dry case of a governance rule violation, so Bill moved to shut down that data feed. Mary, predictably, wasn’t too pleased. She said shutting down that data flow essentially meant shutting down the project. Bill said ‘but it’s against the law’. And then Mary produced results that showed a 30% improvement in key client retention. Believe it or not, a vigorous debate ensured about the merits of continuing the campaign even though it was in violation of the law!

After thinking it over that night, Bill came up with a different approach. He and the governance team devised a rapid out-reach program to obtain consent from their client base. The consent request was included with another “business as usual” client outreach, and the thinking was it would improve the odds of gaining consent. In the interim, the data governance team had determined that mixing the client web-log data with external data and using it to determine an “average customer retention event” could possibly work, because it wouldn’t violate privacy laws. Bill’s team worked with Mary’s marketing analysts to test the analytics – and the result was the predictions were only -5% less accurate, which was an acceptable deviation. Project ‘retention’ was kept up and running, and yielded very impressive retention results during the interim period in which client consent was gathered.

ABC learned valuable lessons through ‘project retention’. First, they now collaborate on all data-centric projects with Bill and the data governance team to determine the right way to use data while staying in line with governance rules (watch a video on how to manage proactive governance in your organization here). Second, they established a proactive customer consent mechanism to gather a broader reaching set of consents for analytics and marketing purposes. And that’s really the moral of the story. The “bad guy” in this real-life scenario was a lack of process, which unwittingly pitted two employees into a zero-sum game, when they should have been working together towards the same goal. Violating privacy laws and governance rules are significant issues, but are actually the least of your concerns when it comes to governing big data. The real potential downside of ungoverned big data is rework and lost time – which translates into lost customers, lost revenue, and missed opportunities.

For more on how to govern big data proactively, read about Building Confidence in Big Data .

Share your input on whether governance, marketing, and big data & analytics is a zero-sum game in your organization or if you proactively work together to optimize your data usage.

You Can’t Forget What You Can’t Remember

In order to forget something, first you need to remember it.  That simple premise will cause organizations a great deal of pain as consumer privacy legislation takes effect.

The concern about consumer data privacy is at an all-time high.  70% of Europeans are concerned about the reuse of their personal data.[1]  86% of Americans are concerned with data collection from internet browsing and how it is used to generate personalized banner advertisements.[2]   Their primary concern is how that data may be used for other purposes, or packaged and resold to other entities.  With data breaches and issues such as the NSA’s collection of private data making headlines each week, it’s no wonder that consumer sensitivity is heightened.

This will present a very large problem for companies, because law makers are starting to take action.  The European Union announced changes to the 1995 Data Protection Directive to take effect starting in 2014.[3]  It contains one very logical and innocent looking directive – “the right to be forgotten” which means that upon request from a consumer, an organization must delete all of their personal data.  That sounds simple.  It’s actually a wildly complex problem, because of the premise above – you cannot forget what you cannot remember.  And most organizations aren’t particularly good at remembering their customers.

Click here to READ THE REST OF THIS BLOG ON http://www.ibmbigdatahub.com  – or click or go to http://ibm.co/1bEaJGS

Watch a video discussion on this topic here http://ibm.co/1gWKqwF

[1] Forrester Research.  EU Regulations And Public Opinion Shift The Scope Of Data Governance

by Henry Peyret, October 17, 2013

[2] Perfect Storm For Behavioral Advertising:

How The Confluence Of Four Events In 2009 May Hasten Legislation (And What This Means For Companies Which Use Behavioral Advertising) By:  Susan E. Gindin

[3] Forrester Research.  EU Regulations And Public Opinion Shift The Scope Of Data Governance

by Henry Peyret, October 17, 2013

No Security? No Data! Big problem ….

unlockProtecting and security sensitive big data is necessary to ensure data is shared for new forms of analysis.  Before the owners of that data will share it  (yes, political silos still exist, and yes individuals still feel they own data and can say no to sharing it), they want to ensure it is adequately protected.  Especially if they are the ones in the cross-hairs if that data is misused.

At the Data Governance Financial Services Conference last week in New York, I spoke on the issue of Confidence in Big Data.  And boy, did that topic ever resonate with the audience.  I spoke with a Chief Data Officer who said confidence was really the main issue she deals with – governance is all about confidently ensuring that her business users trust and protect their information.  A head of governance approached me to discuss confidence in customer data; they were struggling to ensure they were confident in accurately identifying customers and households as the basis for big data analytics.   There were a lot of common themes that came out of my discussions – customer data and big data, rapid integration of new data and business user self-service, how to visually display data confidence to business audiences …. but one issue dominated the conversations – privacy and security.

Ensuring privacy and security for big data, or any data for that matter, is always a top concern. Why?  Well, someone might go to jail if sensitive data is exposed.  Or face compliance fines.  That’s always a compelling reason to act.  But I heard something different at this conference.  One Chief Data Officer described it this way – “Imagine you want to buy a new car and safety and security is your top concern.  10 years ago you could always decide to add a security device or alarm after you buy the car.  But now, you want a system integrated with the ignition.  And for safety you want front and side curtain airbags – you’re never going to install those after the fact.  So the issue becomes a non-starter – you’ll only buy a car with the features already integrated.  The same thing is happening at our firm.  Security is a pre-requisite for big data.  If we can ensure data security for sensitive information, that project will be approved over one that lacks security.  It’s a non-starter for big data and analytics – no security, no data.”

That certainly makes sense.  Data security is as fundamental to sharing big data for new analysis as policing is to a healthy and thriving society and economy – it’s a fundamental pre-requisite.  And it offers an interesting twist on the reason to worry about privacy and security.  If you want to share big data freely, combine it in new an interesting ways in new technologies such as Hadoop or NoSQL, then you need to ensure it is protected.  Big data is by definition sensitive data – it’s important information about your customers, your products, your suppliers.  That data must be masked when it’s appropriate to do so (good rule of thumb – if the actual data value isn’t relevant for the analysis, mask it).  It must be monitored to ensure that internal users aren’t accessing it inappropriately.

Before embarking on a new big data and analytics project, make sure you’ve taken care of the fundamentals.  Make sure you can adequately protect and secure sensitive data before you ask a data owner to share it.

For tips on how to protect and secure big data, check out this ebook – Top Tips for Securing Big Data Environments


Get every new post delivered to your Inbox.