Big Data Integration & Governance – Leveraging Your Existing Technology

In September 2013, IBM announced exciting innovations in Information Integration & Governance (IIG) that represented an evolution of these products for the new era of computing – big data, cloud, mobile, and social.  The key point was evolution – because this product line was designed from its onset to handle large volumes, was built to be modular and therefore accommodate new components and technologies, it was able to adapt to new big data requirements and adopt new big data technologies with ease.  Fortunately the product set was already designed to handle the key requirements of big data – massive scalability and parallel processing.

Some of the key things IBM announced were agile integration and governance, confidence in big data, and automated integration.  It’s been six months since those announcements, and these capabilities are really taking hold in the market. 

First let’s examine data confidence.  This has really become a top market issue in the past year.  Numerous companies that I’ve met with are more concerned with measuring confidence in data, and improving confidence in data for their big data & analytics projects.  The rise of the Chief Data Officer (CDO) gives evidence to this point – companies are charging CDOs with understanding and improving data so it may be utilized, and trusted, for analytics.  There are numerous press articles highlighting the importance of “confidence” and “metadata” and “lineage” with respect to big data projects.  Here’s one such article that appeared in Forbes – http://onforb.es/1hU1VOt.  But most importantly, clients are making this part of their big data projects – I’ve seen a noticeable shift in terms of utilizing metadata and lineage to approximate confidence levels in big data. 

IBM also announced advancements in automated integration, a capability it was the first to champion as ‘the next frontier for data integration’.  The hypothesis was simple – in the era of big data, you’ll have more sources and more repositories for specialized analytic workloads, therefore you’ll have more integration, so it should be made simple.  And that’s exactly what has played out in the market.  Companies have adopted IBM’s Data Click to speed integration between big data sources and targets, and to offer business department ‘self-service’ to access big data.   This has proven extremely valuable, as more companies look to centralize big data in a ‘data lake’, and technical users in business departments want access to that data to bring into their own repositories for ad hoc analysis. 

And IBM’s capabilities in Agile Governance have received a warm reception from the market.  The ability to apply appropriate levels of governance at different points in time is the key to quickly, yet safely, harnessing the power of big data.  Many organizations have adopted IBM’s agile data security capabilities to monitor, mask and protect sensitive big data. 

IBM’s Information Integration & Governance portfolio is the market leader in Information Governance, and it was designed from the beginning to handle the biggest and most complex requirements.  That original design placed it in a fortunate position – it was already ready for the era of big data in terms of performance and scalability, and simply had to adapt for newer requirements such as variety, and new technologies such as Hadoop and NoSQL.  So while competitors have played catch up and have reinvented their portfolio for big data, IBM has invested to move forward.  Be on the lookout for some exciting new innovations in IIG in 2014. 

For more information, go to IBM Big Data Hub.