Big Data was a much-discussed concept across IT management last year. In fact, the effect of big data and the non-stop massive growth has been a very real challenge for CSP's data infrastructure and technology teams for quite some time.
Global tier 1 providers have been experiencing activity levels of billions of call data records per day to the point where custom database technologies are now being developed and modified each year. Peta-byte scale database systems holding years of CDRs and internet protocol detail records (IPDRs) are now considered the norm at many tier 1 providers.
When you consider the more stringent compliance regulations being imposed, the Big Data effect can become even more complex and challenging. A single global provider now faces a situation where their data centers and database solutions need to proactively configure, manage and retain specific customer records for a designated time period. Each country's regulations impose different timeframes, requiring different data types, in different formats as well as specific response times to access that data.
In the past year alone, we have seen Japan and India enforce government-led initiatives upon providers regardless of size and subscriber level. Fulfilling lawful intercept requires substantial investments by providers in storage, database and search technologies and of course trained resources with the right skill-set to deploy such technology solutions.
Lawful intercept involves both real-time monitoring and access to xDRs. For example, law enforcement will obtain a judicial warrant to tap the landlines, cellphones and email accounts of suspects, as well as to receive copies of their call, SMS/MMS and other communications records. Having this data readily available in the correct format and delivered in a timely manner creates ongoing pressure to make the data center run efficiently and cost effectively, not to mention the pressure of a legal fine.
Gaining scale
Today CSPs must think innovatively not only through the power of interoperability with other global partners, providers and suppliers but use technology, network architecture and standardization to achieve scale and better IT economics.
In a recent Gartner industry report - 'Credicts 2011: CSPs Must Rethink Business Paradigms to Meet Market Challenges?- it cited that in developed markets margins in mobile, fixed combined services are falling, with very few exceptions. This is due to IP substitution, lower ARPU from new customers, increasing competition, regulators getting a better view of noncompetitive price elements, and the need to invest in expensive broadband networks with long pay-back cycles.
Improving the customer experience and service level is absolutely a necessary part of staying ahead, and partnerships formed each day to provide new applications and mobile capabilities to drive new channels of revenue. However, all the front-end customer facing products and services that are rolled out will only be possible and even sustainable if the back-office and data infrastructure are capable of management and scale over time.
Let's consider the data problem for today's IT team. If you examine the life-cycle of a single customer's communication 'transactions?in say a given month or billing cycle, the core data (essentially CDRs and IPDRs) that are captured, collated and stored are immediately historical in nature, which means once the 'transaction?takes place, it does not undergo any further change. This is quite different when compared to a traditional transaction on an online transactional database system (OLTP) such as a retail purchase.
The customer communication data is then "moved" through a series of applications to enable various IT and business functions, such as network capacity planning, traffic analysis, user trending, revenue assurance, billing, digital advertising and more. Some of these OSS/BSS applications leverage the same database system and in other cases, the data is moved around with varying degrees of latency. Data sub-sets are formed depending on application function and requirements and data warehouses are fed with customer activity to perform trending analysis and determine the overall value of a customer at any point in time.
Given the growth in data volume and complexity across diverse systems and applications, organizations are forced to retain the data for as little time as possible largely because systems have simply bloated to the point where it negatively impacts performance. For a large provider, even the critical function of replicating data from a complex billing application to provide continuous uptime can cause significant technology challenges due to the volume of the data sets.
Smaller footprint
However, new database technologies are now coming to the forefront of solving this problem directly. Specialized databases now have the ability to significantly de-duplicate and reduce this data to a much smaller footprint. When you consider a 100-terabyte system reduced to a storage footprint of 5 terabytes, you can begin to imagine the cost savings not only in storage but also hardware capacity.
Data in this specialized database built for the purpose of storing data online for long time-periods can be queried and retrieved using standard SQL Query language or various business intelligence and analytics solutions. With a database solution focused on big data retention that enables significant cost reduction, CSPs can store critical customer activity data for much longer timeframes - beyond what is required by regulators. By having access to larger historic data sets, CSPs can gain greater insights to customer behavior patterns and even discover new opportunities to further segment the market and create new services with a wider range of price points.
For CSPs to gain completive advantage in the ever-changing markets, they need to look more inward than out and discover new technologies and new approaches to managing their "core" - essentially the customer and transaction data. By applying new data infrastructure capabilities to achieve greater cost efficiencies, CSPs can literally discover new service offerings and even new markets.
The ocean of Big Data faced by today's CSP does not have to reach boiling point - applying specialized database capabilities across the infrastructure will significantly reduce cost, manage scale and complexity and will certainly have a cooling effect.
Dr. Hossein Eslambolchi is chairman and CEO of Divvio Inc, a web portal that allows users to build personalized 'channels' of digital content by leveraging advanced adaptive learning technologies and Web 2.0 community behavior. He is also chairman of 2020 Venture Partners, which provides technology and operations consulting to private equity firms and venture capitalists.
The article first appeared in TM Forum CXO Community