Evolving Integration and Governance for Big Data Requirements
I had the pleasure of Tweet-chatting with Jim Harris, James Kobielus, Tim Crawford, and Richard Lee on the topic of Evolving Integration & Governance for Big Data. We started the discussion by asking whether there will be a backlash to high-profile mistakes with big data & analytics, such as the recent OfficeMax issue when they sent a market mailing with the name “Mike Seay – Daughter Killer in Car Crash” in the envelop window. Everyone agreed there would be some backlash, both from mistakes such as these, as well as consumer reaction to the ‘big brother’ feeling that some big data campaigns evoke (maybe that company knows a little too much about me). Regulations will force companies to address fundamental issues, such as knowing the origin of data and its intended use. We all agreed – there would be some consumer backlash, an increase in regulations, and therefore orgs must be ready to respond with a more agile approach to big data security. All were in agreement that this should in no way slow down the adoption of big data. In fact, the ability to protect sensitive big data could become a major differentiator for some firms.
Next, we drilled into how to find data and establish a level of confidence in it. It simply takes too long to gather data on each big data project – today I heard estimated ranging between 40 and 80% of the time was consumed on that one task. That’s obscene. What’s worse – companies pay that same tax repeatedly by failing to leverage an integration & governance platform. Integration technology could surely help reduce that number dramatically, with the aid of automated discovery and classification, and self-service data integration capabilities. The issue of confidence brought out varied opinions. While some suggested that confidence was low, or always in question by business users, others asserted that confidence was high until proven otherwise, or at least high in reports that users have used previously (we trust what we’ve always assumed to be true). Confidence, while subjective and difficult to quantify, is very important to adoption of big data and analytics. If users lack confidence in data, they will lack confidence in the results.
Another topic which sparked debate was the best way to fix these issues of confidence and rapid big data discovery – can existing integration and governance technology evolve and adapt to new requirements, or does it need to be reimagined and reinvented? Everyone agreed that evolution was the desirable and logical path. Existing integration and governance technologies should adapt to big data scale, and adopt a wider variety of data types. They should also evolve to include new big data technologies to address those broader requirements. The conclusion was clear – there’s no need to reimagine and reinvent when the core products are fundamentally sound and built for big data – simple evolve them for these new requirements.
These tweet chats are hosted weekly under the hashtag #bigdatamgmt and I encourage you all to join the discussion. Also check out the latest blogs, videos, and infographics on big data integration & governance at ibmbigdatahub.com.
Recent Posts
Blogroll
Categories
- 360 view
- Big Data
- customer 360
- Customer big data
- Customer context matching
- Customer data
- customer intelligence management
- customer personalization
- Data Confidence
- Data Quality
- Data Warehousing
- Hadoop Systems & Analytics
- Information Governance
- Information Integration
- Information Lifecycle Management
- Master Data Management
- metadata
- omnichannel personalization
- Privacy and Security
- Stream Computing
- Uncategorized
- Visual Context for Data