Simplified and scalable data integration for Telco BSS

Telcos often face challenges integrating data spread across multiple BSS domains. The siloed nature of these domains means data is typically not transferable or compatible with other domains. In this post, we explore specific challenges faced by Telcos and how we could design a scalable data integration solution on AWS using the concepts outlined in the TM Forum’s information framework.

The importance of data in Telco BSS

Telco’s BSS solution plays an instrumental role in enabling and managing the overall customer experience throughout the customer lifecycle. It is responsible for the monetization of the Telco’s key asset, in other words the network. It consists of a collection of processes and sub-domains that help manage customer orders, promote service consumption, calculate subscription fees, and allow for additional services to be added.

The data in Telco BSS plays a significant role in enabling a Telco to function effectively. BSS sub-domains must be able to exchange and consume each other’s bulk data sets efficiently to provide the best quality customer experience and meet several regulatory requirements.

Let’s look at the key sub-domains and their significance with respect to data:

1. Customer and Partner management systems track customer interactions, data, and relationships to improve customer retention and increase sales. Order management systems handle taking, tracking, and fulfilling customer orders for products and services, such as provisioning and delivery. This sub-domain holds customer data as well as data related to orders such as shopping carts, delivery options, payment methods, and order status.

2. Channel Management provides ways to interact with customers through various channels such as retail, contact centers, self-service, and web portals.

3. Product Management Systems define offerings, pricing, lifecycles, and details of products and services available for sale. This sub-domain holds data related to product offerings, cross-product discounts, bundles, and pricing.

4. Revenue Management domain provides the capability to monetize the offered service. It includes key functionalities such as mediation, charging, billing, and payment solutions for processing usage data to apply rates, generate bills, and maintain customer payment and usage records. This sub-domain holds revenue related data such as balance, charges, transactions, payments, invoices, etc.

5. Analytics and Assurance domain provides the capability to combine data from various systems and derives actionable insights.

Figure 1. Illustration of the subdomains of a Telco BSS landscape

Challenges with data integration

There are several data integration patterns that co-exist in a Telco’s data architecture, as shown in the following diagram.

Figure 2. Illustration of current data integration patterns

First, there exist several manual data integrations that need human inputs through spreadsheets, forms, look-ups to other systems, and even approvals. This presents a unique challenge to the time-to-market of new products or offerings. As an example, publishing a new product or a tariff onto online channels may require enriching data from the product catalogue manually. This enrichment may require manually adding product descriptions, legal addendums to contract terms, or even some approvals.

Second, as a tactical solution for several urgent business and regulatory needs, 1:1 data integration has been built up and has remained in service due to its criticality. For example, consider data integrations that run urgent marketing campaigns as a tactical response to a competitor.

Third, as an initiative to streamline the data integration domain, Telcos have been implementing a centralized data store that helps feed the right data to the right systems. These are typically on-premises in nature and need a large team to procure hardware, manage, install, and develop. The team needs to upgrade these tools frequently to get the latest support. This may lead to a team focusing on infrastructure issues and tuning as opposed to actually performing the data integration work. Even after spending months optimizing, when running parallel jobs, the environment slows down impacting other jobs. As an example, a centralized data store, which continues to act as a feeder system or a data spine for other BSS systems, is often built using such traditional tools on-premises.

Additionally, duplication of data sets across these patterns is another concern. With ever-growing data sets, compliance requirements such as data sensitivity and retention policies make managing multiple data sets a complex challenge. The lack of re-usable data sets further cause a snowballing effect, in other words the creation of even more data sets for specific needs.

Building a system of trust, implementing Data Governance and Security across these patterns, is particularly challenging. CISOs need a unified view of who, why, where, how, and what is accessing the Telco’s data, and they typically initiate separate projects or compliance initiatives to achieve it. This increases the overall complexity of the data architecture.

In summary, these patterns result in a non-scalable, high-cost posture impacting a Telco’s core business agility and innovation.

The TM Forum Information Framework

TM Forum is a global telecommunications industry alliance of over 850 service providers and supplier companies, collaborating to lead positive change, enable growth, and promote success across the industry.

The Information Framework (SID) provides an information/data reference model along with a common vocabulary for implementing business processes. This business model is independent of platform, language, or protocol, and it can help identify business entities that are important to a Telco. It also provides the basis for a common data model and a common data dictionary, and it offers a foundation for function, application, component, and API development.

Using this framework, we can effectively standardize the data model that we would create in our central data integration layer. The following diagram is an example of how we could logically represent data sets related to Customer Payments.

Figure 3. Customer Domain – Customer Bill Collection ABE

Solution overview

A robust data integration solution shares a common set of capabilities. These fit into a layered, component-driven architecture that decouples functionality and allows incremental development. These layers can be categorized as follows:

• Ingestion: Taking in and collecting data from various sources
• Storage: Saving, organizing, and managing the collected data
• Processing: Analyzing, transforming, and enriching the data
• Cataloging: Documenting and indexing the data for discoverability
• Consumption: Enabling the use of the data through reporting, analytics, applications
• Governance: Establishing policies, security, and guidelines for appropriate data usage and management

The loose coupling between layers also allows the flexibility to change technologies and an ability to scale independently. However, the foundational, shared capabilities persist regardless of the technologies used in each layer.

Figure 4. Infographic highlighting key features and benefits of a serverless data solution

Due to the rapidly expanding volume of BSS data, storage cost-effectiveness, reliability, and scalability play a big role. Telcos need to consider a solution that is easy to use with no code/low code and can scale resources while providing robust access control and governance.

AWS provides several building blocks to enable different layers of data integration architecture. For ingestion and storage, Amazon Simple Storage Service (S3) offers a durable, highly available, and cost-efficient data lake with virtually unlimited storage capacity. For processing, AWS Glue handles extract, transform, and load (ETL) jobs through a fully managed service that can discover, prepare, and integrate different kinds of data sources at scale without needing to manage the server. Its no/low code capability further simplifies the data integration job. AWS Step Functions and Amazon DynamoDB allow for the capturing of additional inputs and parameters needed for data transformations. For consumption – Transformed datasets can be exposed to consumers through for SQL querying or for API access. These components integrate together but can also be used selectively to build solutions incrementally. By leveraging these services, the undifferentiated heavy lifting is handled by AWS, allowing teams to focus on value-added data workflows.

AWS Lake Formation provides central mechanism to manage access control for data. This mitigates Telco’s concern regarding data security related to sharing data among different BSS domains. Lake Formation uses the underlying services, such as Amazon S3 and Glue mentioned previously, and it provides the centralized capability to define security policies that restrict access to data at the database, table, column, row, and cell level using fine-grained access control. This enables least privilege access control, thereby bringing the Telcos full control of their data sets.

While we implement the ETL processes in AWS Glue, we align with the TM Forum’s Information Framework by using their prescribed logical data domains. In turn, we can reduce duplication in our data sets, promoting structure and re-use. This also helps driving initiatives such as the use of TM Forum’s Open API for standardizing Telco’s API layer.

Reference Architecture

Now, let’s consider a scenario where the Revenue Assurance team would like to do an end-to-end analysis of the data in the Order to Cash processes, namely a full reconciliation of CRM vs Billing vs Collections to identify revenue leakages. Let’s design this solution on AWS with a potential to scale this to the other domains, as shown in the following diagram.

Figure 5. Diagram of the AWS architecture for a Telco BSS data integration

1. The raw extracts from the CRM and Billing domains are uploaded using AWS Transfer Family for SFTP to S3 buckets or directly to Amazon S3 if these domains are on AWS. Besides batch data ingestion, data ingestion can also be done in near real-time using Amazon Kinesis, which provides easy integration for streaming data. Kinesis manages the underlying infrastructure, storage, networking, and configuration needed to capture and load data into Amazon S3. It also scales elastically without requiring intervention or associated developer overhead.

2. Once these files are received in the S3 bucket, it triggers a Step Function, which orchestrates human inputs/approvals. For example, a specific record such as a long running call for a large customer could require a separate notification and approval from a Revenue Assurance expert. These human inputs are captured and stored in DynamoDB tables for audit history and input to the transformation process.

3. The Step Function now triggers the corresponding AWS Glue Job to start transforming the data sets as per the format requested by the target domain. Furthermore, in alignment with the information framework, we could use the Customer Product Order ABE and Customer Bill ABE to drive reconciliation between what was ordered by the customer and what is being billed to identify potential revenue leakage.

4. AWS Glue now creates the final data sets, using the source files in the S3 bucket, the human inputs stored in DynamoDB, and outputs to another S3 bucket.

5. Lake Formation forms a fabric of data governance that helps to collect and catalog data in Amazon S3, clean and classify data using machine learning (ML) algorithms, and provide secure access to these data sets to the BSS systems/domains.

6. Athena provides the querying capability through APIs through the API Gateway for systems that need synchronous calls, for example Service Orchestration may need specific up-to-date CRM data.

7. The final extracts are now provided using Transfer Family for SFTP for on-premises applications, or Applications on AWS can directly access the S3 bucket through appropriate bucket policies. The Revenue Assurance team now has the necessary information to run their reconciliation processes.

8. Data Encryption, Security, Audit Trails, and Logging are implemented using Lake Formation, AWS Key Management Service (KMS), AWS Identity and Access Management (IAM), AWS CloudTrail, Amazon CloudWatch, and AWS Security Hub.

Conclusion

In this post we explained the challenges of Telcos managing and providing vast tailored data sets to BSS sub-domains, thereby effectively realizing the value of BSS data. AWS can help simplify and centralize these data integrations in a serverless way.

The benefits of the solution include scalability, flexibility with an open-source tooling, data integrity, and governance. The TM Forum’s Information Framework allows us to improve upon the structure and reusability of data sets, thus improving the efficiency of data engineering teams, and thereby driving direct business outcomes in areas such as product design, marketing, fraud detection, churn prediction, and revenue assurance. We have illustrated the AWS reference architecture with a layered approach for a scalable solution that can overcome the complexity of common data integration challenges in Telco BSS.