Using AWS Migration Hub network visualization to overcome application and server dependency challenges

When you embark on a cloud migration, it can be challenging to identify application and server dependencies. In some organizations, IT assets are maintained in a spreadsheet that is updated infrequently. Some organizations have configuration management database (CMDB) software, but it tends to be outdated or unusable. It lacks the infrastructure data that is essential for making technical decisions during migration.

Some common challenges:

• Planning the migration for servers shared by multiple applications.
• Correlating applications to servers and identifying their interdependencies.
• Understanding the frequency of server communications.

When you use a robust automation mechanism to gather data, you can scale discovery, identify a migration grouping of servers, and create a data-driven migration plan that mitigates risk and maximizes ROI.

AWS Migration Hub provides a single location to track the progress of application migrations across multiple AWS and partner solutions. Using Migration Hub allows you to choose the AWS and partner migration tools that best fit your needs. It provides visibility into the status of migrations across your portfolio of applications.

In this blog post, I’ll show you how to use Migration Hub network visualization to build a high-fidelity model of server dependencies and use it to identify migration waves and move groups. (A move group is a set of servers or applications to be migrated and tested together. Move groups are organized into migration waves to accomplish business goals.)

Using Migration Hub will help accelerate your discovery conversations with the technical and business owners of applications and enable data-driven migration planning. For example, it can be challenging to identify which applications to test when a move group is migrated. Migration Hub network visualization helps you identify the blast radius of a server or move group so you can plan application regression testing and required test coverage. In addition, Migration Hub integration with Amazon Athena can help you analyze detailed information collected in Migration Hub.

Migration Hub capabilities for dependencies visualization and analysis

1. Use Migration Hub network visualization to understand server dependencies. This helps visualize communication patterns among servers in your data center. Visualization helps identify server clusters where the chain of dependencies can be broken to form migration waves and groups. For example, legacy distributed applications have multiple servers that service a single application. In this scenario, grouping servers into applications will help visualize application-level dependencies. For more information, see Group Servers as Applications in the AWS Application Discovery Service User Guide.
2. Use Migration Hub integration with Amazon Athena for deep-dive analysis. For power users, AWS Discovery Agent provides detailed, fine-grained information, including process-level server data, source and destination IP addresses, network bytes, and a timestamp of communication that can help you understand server affinities. For example, if two servers communicate very frequently with sizeable data exchange, they should be placed in the same move group to ensure the best performance for end users after migration.

Prerequisites

To use Migration Hub, you must first set up data collection using the AWS Discovery Agent. You can find procedure for installation in the AWS Application Discovery Service User Guide. After you’ve collected data over a 2–4-week period, you can start your analysis and create a migration plan.

How to use Migration Hub network visualization

I will use an example of a customer who is planning the first wave of a migration. The customer has an on-premises test environment of several servers, which host marketing, analytics, manufacturing, ERP, and HR applications. The customer has installed the AWS Application Discovery Agent and turned-on data collection in Migration Hub. You can find procedure for installation at AWS Application Discovery Service User Guide. After two to four weeks, there is enough data in Migration Hub to produce a high-fidelity view of communication patterns. First, the customer will use Migration Hub network visualization to understand dependencies and arrive at move groups. Then, the customer will explore how the data can be analyzed using Athena.

The architecture of data collection and application dependency mapping:

Figure 1: Architecture diagram

The agents on the servers and VMs send data to Migration Hub.

Group servers as applications to track migration

In Migration Hub, you can group servers as applications to facilitate migration tracking. For the customer example, here are the steps:

1. In the left navigation pane of the Migration Hub console, choose Servers, and then choose the Server list
2. Choose servers that you want to belong to same application, and then choose Group as application.

Figure 2: Servers page

3. Choose Group as a new application, enter an application name, and then choose Group.

Figure 3: Group as application

You can also add servers and VMs to an existing application by choosing Add to an existing application.

4. In the left navigation pane, choose Applications. The application you just created is displayed in the list.

Figure 4: Applications page

5. Choose the application to view application details, add or remove servers, and update migration status.

Figure 5: Details page for Ofbiz application

For servers that are shared by multiple applications, you can use other mechanisms like tags to facilitate server grouping.

Visualize network dependencies

Dependency visualization at the server level enables data-driven discussions with on-premises data center personnel and application owners. When you augment Migration Hub data with application information, you can identify not just inter-application dependencies, but redundant assets that should be retired.

1. In the left pane of the Migration Hub console, choose Servers, and then choose the Server list
2. Choose one or more servers, and then choose Visualize network.

Figure 6: Visualize network

In the context of our customer example customer, you will see the servers and the communication dependencies between servers. An arrowhead on the lines indicates the direction of the dependency. You can change settings to modify the visual elements for clarity and detail.

In Figure 7, the orange servers are those with the AWS Discovery Agent installed. The labels under the server are the application name you entered when you grouped the servers. The lines and arrows represent the communication patterns provided by the agents over the collection period.

Figure 7: Visualized servers

Create migration waves and move groups

When it comes to creating a migration wave, one common heuristic is to identify the business functions of servers. This helps set the expectations of business stakeholders about change windows and regression testing. Using this method, migration waves for this customer might be manufacturing, marketing, financial, infrastructure, ERP, and talent management.

You can use the Migration Hub network visualization in Figure 7 to identify move groups in the four manufacturing servers (highlighted in aqua). Two of these servers have an incoming dependency from an analytics application. Based on this data, you have two choices:

1. Split the manufacturing migration wave into two move groups, as shown in Figure 8:

Figure 8: Manufacturing move groups

In this choice, move group 1 includes tst-db-mfg-01, tst-as-mfg-01, tst-ws-mfg-01 and move group 2 includes tst-db-mfg-02. The manufacturing application owners go through two migrations, one for each move group. The migration of each move group requires coordination with the analytics application owners. Smaller-scoped move groups reduce risk, but increase the coordination effort required.

1. Combine the manufacturing application into a single move group, as shown in Figure 9.

Figure 9: Single manufacturing move group

In this choice, the move group includes tst-db-mfg-01, tst-as-mfg-01, tst-ws-mfg-01, and tst-db-mfg-02. The manufacturing application owners go through only one migration. Regression testing for this move group requires participation from analytics application owners. Because these are test environment servers, the customer in my example scenario would lean toward this choice.

This is just one example of how Migration Hub network visualization can inform a migration project plan of a customer. Servers and applications can be chunked and iterated in other ways, such as by their technical function (web servers, databases, and application servers) or by operating systems such as Windows and Linux.

In addition to using data from Migration Hub, you should collect information about the qualitative aspects of applications and servers such as application criticality, resource availability for migration and testing, change windows through interviews with application owners and factor this information into your migration planning.

Use Amazon Athena

You might have situations where your applications share servers, which requires a many-to-many mapping between applications and servers. In this case, you can work with raw data collected by the AWS Discovery Agent to perform sophisticated queries to examine process-level data, performance, and dependencies.

Figure 10 shows the architecture for analyzing data collected by Migration Hub.

Figure 10: Analyzing data collected by Migration Hub

The services used in this architecture, Amazon Simple Storage Service (Amazon S3), Amazon Athena, and Amazon QuickSight are not part of Migration Hub. Although Migration Hub is free, you will incur charges in your AWS account for the use of these other services.

Athena queries can help you find more granular information about the interaction between the servers. You can use predefined queries to perform powerful deep-dive analysis of process-level inbound and outbound communications, system performance, and runtime software in these servers. For example, if you want to know the frequency of interaction between dependent servers, use the following query in Amazon Athena:

SELECT a.source_ip,
         a.source_hostname,
         a.destination_ip,
         b.host_name AS destination_hostname,
         a.frequency
FROM 
    (SELECT a.source_ip AS source_ip,
         b.host_name AS source_hostname,
         a.destination_ip AS destination_ip,
         a.frequency AS frequency
    FROM "application_discovery_service_database"."outbound_query_helper" a, "application_discovery_service_database"."hostname_ip_helper" b
    WHERE a.source_ip = b.ip_address
            AND source_ip != destination_ip ) a, hostname_ip_helper b
WHERE a.destination_ip = b.ip_address

Figure 11 shows the partial result set of the frequency of database interactions for WordPress and Ofbiz applications.

Figure 11: Partial result set

Cleanup

In Migration Hub, delete servers and applications in the respective tabs. In Amazon Athena, delete Migration Hub data source named “application_discovery_service_database” and the corresponding S3 bucket that stored from Application Discovery Agents.

Conclusion

The key to success in your migration planning is using high-fidelity data. In this post, I showed you how to group servers as applications, identify move groups based on network visualization, and use Amazon Athena to perform sophisticated queries. I also demonstrated the benefits of using network visualization with Migration Hub specifically for planning migrations. For more information, see the Migration Hub User Guide.

About the Author