Why CDI Projects Fail – Part 2: Data Model Inflexibility

In a world where customer acquisition costs are soaring and customer retention — particularly retaining profitable ones — is getting harder for every enterprise, “knowing your customer” is not just a slogan but a business mandate. However, in order to truly know its customers, an enterprise needs to create a unified and comprehensive customer view from all its disparate data sources, including customer databases and customer information files, financial and order management systems, product catalogs, and external data services.

Once integrated, these unified customer views provide the entire enterprise with the ability to drive customer-centric strategies. However, using existing technology platforms to build and manage such unified customer views — across disparate data sources, applications and channels — can be a complex and costly exercise and often achieves only limited success.

One of the primary culprits driving the astronomical expense, and often failure, of customer master data hub projects is the lack of flexibility in the underlying data model of the solution. This is especially true for Customer Data Integration (CDI) vendors who have anchored their solution around existing applications such as Customer Relationship Management (CRM) applications or around a fixed industry data model such as retail banking or insurance.

This lack of flexibility becomes their Achilles’ heel, as significant customization is often required to enable the data model to reflect the existing systems and business requirements. With a heavily modified data model, the business services that sit on top of it break, along with any management logic that is bound to the fixed data structures. Further, with no flexibility in limiting the scope of customer data types that must be modeled within the customer hub, the project becomes much larger than necessary, which adds to the overall risk of failure.

Therefore, it is critical for companies to review the data model flexibility of various CDI solutions to ensure success of their projects both in the short and long term. The following provides some of the key factors to consider when determining if a data model is flexible:

Data Model: Brittle or Extensible?

Since a customer hub is highly specific to each enterprise — in fact to each business unit within the enterprise — the critical question is not whether an out-of-the-box data model is “rich.” In fact, like a pre-fixed menu at an expensive restaurant, it may well be too rich with extraneous attributes not appropriate for a scalable master hub. The essential question is: how readily does the solution permit the data model to be modified? Most CDI solutions offer a fixed albeit rich data model developed from their unique application perspective.

When these solutions attempt to adapt this to the true needs of a business outside their original design framework, the model either cannot provide the entity and relationship structures necessary or it requires extensive and expensive customization. Also, once complete, the resulting data model is highly inefficient due to the processing burdens associated with combining the existing application logic and the modified model.

Ideally, an adaptive CDI solution must offer an initial data model specific to each industry vertical — or allow any proven data model to be imported — that can be further modified to reflect the exact business entities needed to be modeled for the customer hub. In addition, it must supply all the supporting data management services, including extensive metadata management, to facilitate the model’s extension and changes over time. The net result is a relevant, highly specific data model that fits the project needs from inception without forcing an enterprise to standardize on a fixed data model.

Business Services: Customized or Composed?

A CDI solution based on a fixed data model may claim to offer a layer of abstract business services that can be used out-of-the-box for integration. However, these business services are “brittle” and need to be customized if the underlying data model is modified. Therefore, in an implementation, after a fixed data model is customized and de-normalized, there may be only a handful of relevant business services remaining. Worse, even these business services may require customization, and the more such a solution is customized the harder it is to migrate to future product upgrades.

Instead, an adaptive architecture must offer a set of granular data integration services, including a full set of APIs, in a services integration framework that may be used to compose only the relevant higher-order business services. These business services and the APIs are maintained for a smooth path to product upgrades. This approach offers a wide variety of methods within a service-oriented architecture for sharing all the data entities based on rules within the hub — at various levels of abstraction — with downstream systems.

Data Types: Which Ones to Model?

Each customer data type — master reference data, relationship data, and transaction data — has separate characteristics and challenges and therefore requires different treatment within the CDI solution. For example, reference data that exists in every system or data repository is often duplicated, conflicting and does not have a system-of-record. On the other hand, transaction data usually does have a system-of-record; consequently, there is little conflict during reconciliation.

Also, while reference data is a very small subset of customer data and hence has smaller volume, the volume for transaction data can be very large, requiring a substantial investment in infrastructure to store the replicated data from source systems. Relationship data can only be managed effectively after the underlying conflicts of reference data have been resolved. Further, and for proper management, relationship data and its groupings (like households) also require sophisticated visualization tools to display the complex relationships between entities.

A fixed data model approach offers an enticing, extensive application-level data model whereby all customer attributes — master data, relationship data and transaction data — are fully encapsulated in the model for the hub. This fully persistent approach requires many customer attributes to be duplicated and stored in the customer hub. Conversely, an adaptive approach must permit the data model to be segmented in such a way that only the core master data and relationship data attributes are modeled and stored in the customer master hub. The transaction data is addressed based on where it most appropriately resides given the characteristics of the underlying source system.

For example, if the source systems are batch oriented, have no real-time interfaces, or have system load limitations — yet already provide data extracts at regular intervals — the transaction data may reside in an operational data store that is accessed by the Hub dynamically. On the other hand, if source systems possess real-time integration capabilities and are not under severe system loads, the transaction data may be harvested dynamically and cached within the hub for low latency and high availability, removing the need to unnecessarily duplicate source system data in a separate repository. This flexibility can dramatically reduce the volume of data required to be stored in the customer data hub, which reduces the total cost of ownership while increasing the system’s ability to flexibly deliver data in any form, on demand.

Data Model Choice Affects Scalability

Lastly, data model flexibility also has a significant impact on the scalability and performance of the CDI solution. Because the fixed data model solution usually tunes the model a priori to support the native applications, the performance and scalability parameters available to your organization are already constrained by the vendor’s need to support its application architecture.

In contrast, to the flexible data model approach, once the model is configured, all the tuning and normalization is done to support the specific scalability and performance requirements of the master data hub project and related consuming applications — not just to support a single vendor’s application. This difference in approach to scalability can have significant effects across the entire data lifecycle of building, using, managing and extending a master data hub.

Recently, the myth that an adaptive solution requires a compromise on performance was dispelled with CDI Benchmarking Results, providing evidence of high performance and scalability — all within an adaptive, extensible architecture.

In summary, data model flexibility is a critical part of an adaptive CDI architecture. An enterprise looking to mitigate risk of their CDI implementation should consider the full effects of their data model choice across several factors outlined above. A wrong decision in this area can increase the project cost, reduce its manageability over time, or in the worst case, doom the entire initiative.