In the information driven world of the future how enterprise data is managed and controlled will be more critical than ever. The technology landscape is changing rapidly as data platforms emerge and cloud based infrastructures get easier to leverage. The need for better information combined with this technology advancement is bringing forth an opportunity to truly solve enterprise data problems. The time is ripe to reconsider your data strategy to fully realize the deep value trapped inside silos of enterprise data. The challenge facing IT departments is the legacy quality issues associated with this data and the fact that it is scattered across the enterprise.
Typically organizations are suffering from several factors that make fully leveraging data difficult. Incomplete and dispersed data sets are stuck in application databases that may or may not allow full access. Database proliferation caused by reporting requirements has caused duplicate and incomplete data sets. Processes, applications and associated data that were configured for manual operations are not optimized for a more automated real time world. These are just a few of the master data issues that enterprises are facing today and progressive IT departments have to get ahead of these problems to continue to help business digitally transform.
A new way of thinking about data is in order.
Figure 1 shows a new data abstraction that is emerging to help solve enterprise data problems. Companies are beginning to move towards a three layered approach to data that will help resolve current master data woes. The bottom layer represents data creation, the traditional application and reporting databases that exist across the enterprise today. What is emerging is a consolidation layer, represented by the middle block which is focused on creating a high quality data “superset” derived from traditional data stores. This consolidated superset contains both legacy structured (ERP, PLM, etc) and unstructured (IoT, Social, etc) data that is associated to specific “360 views” of what the business is interested in comprehending. Some examples of these “360 views” or “Masters” are customers, intermediaries, devices and employees. This data superset is the single source of the truth for these views and forms the basis for more complete reporting, data analysis and artificial intelligence. This consolidation layer is optimized differently than traditional databases. It becomes a more passive data layer, allowing for the expansion and inclusion of data over time rather than being structured in advance for performance and transactional efficiency.
In order to prepare for digital transformation, IT departments will have to re-address master data quality and segregation issues and launch a new data platform strategy
As new data requests, analysis methods and modeling techniques emerge, information is extracted from this data superset and the resulting data combined into new insights which enable deeper reporting, real-time application updates and feedback loops to originating applications. The data superset is continually utilized and grown to solve the analysis needs of the business. This new central repository can also be used to stage data and transform it, producing a higher quality data store for enterprise use. Some recent industry examples of this cleansing process include a three step model where data is brought in raw, cleaned up based on business criteria, and then stored and associated with a “360 view”. After maturing, the resulting “gold” data can be used for analysis and restructured as needed to enable better reporting and application performance. This final restructuring of information and insights is represented by the top layer in Figure 1. The resulting visualization would be much more comprehensive than the lower level application reporting databases because the scope of information is representative of a much larger cross section of enterprise data.
Governance and appropriate security will become critical to success because each data element needs to be properly vetted, transformed and secured to ensure high quality and authorize appropriate access. A master data governance team will provide oversight as the approach to consolidation and transformation evolves. Failure to apply an adequate amount of analytical scrutiny to how data is brought into the platform and govern its evolution will result in a low quality data platform that can’t be fully trusted or is misused. Early experience indicates that a passive or un-managed approach to filling the platform results in loss of confidence, exposes critical data and delivers a very costly, protracted cleanup effort. A well designed governance structure that includes a change control board element is paramount to success and will ensure value is realized. A knowledge capture process ensuring appropriate documentation for reference will need to be implemented as the business becomes data obsessed and information becomes central to operations. Analytical outputs become a critical asset and must be treated more like engineering specifications kept under some level of configuration management control so that they can be built upon over time.
Platforms and tools are emerging from many cloud infrastructure providers to make this three layer architecture a reality. Recently, Hadoop open source components are being packaged for rapid configuration and deployment paired with security and visualization tools from cloud service providers. These packages are providing authentication, access, basic analytics and visualization in a la carte business model that scales with the data over time.
This is a very powerful risk sharing combination that’s making the architecture easier to realize and will help speed up industry adoption.
It’s important to understand the capabilities of your internal group and how the tools available in the marketplace will fit the skill sets of your team. Data ingestion and clean up can be accomplished through the use of open source tools or pre-packaged master data management platforms. Teams that have less depth of understanding of the methods used for data cleansing can leverage these services to be more successful faster. Knowing your technical team’s limitations and hiring appropriate experienced consultants to fill gaps is a good way to ensure that everything is properly considered. Scale up considerations should be carefully weighed out against the future data possibilities. A data puddle today can grow to a pond and a lake as more unstructured and transaction data becomes available from the IoT, service systems or external systems like social networks. Technical teams alone can severely underestimate the long term scale considerations and fall back to narrowly focused visions of enterprise data limiting the future potential of the data platform.
For progressive IT teams this is a huge opportunity to strategically contribute to the digital transformation of the enterprise. New thinking on data architecture will bring more depth to the current information that your business is using to operate and open the door to new internal and external data driven opportunities. These building blocks are best driven with a harmonized approach from a central team that is fully aligned with the business. IT professionals need to explain the future data platform strategy and what value it can bring to the organization. IT should join forces with other functions and help them embrace the digital future with an aligned plan to unlock enterprise data. This is an exciting time to be an IT leader and play a role in digitally transforming the company!