How to analyze AWS Network Firewall logs using Amazon OpenSearch Service – Part 1

This two-part blog series demonstrates how to build network analytics and visualizations using data available through AWS Network Firewall logs. Network Firewall supports Amazon Kinesis Data Firehose as one of the logging destinations, and these logs can be streamed to Amazon OpenSearch Service as a delivery destination. Network Firewall logs contain several data points, such as source IPs, destination IPs, source and destination ports, netflow bytes, protocols, etc., for traffic that are dropped or alerted based on the rules configured. These are high in volume and become difficult to analyze directly from logs. To simplify this analysis, logs can be sent to Amazon OpenSearch Service where you can index log data, visualize, and analyze using dashboards.

In part 1, we walk you through steps to configure Amazon OpenSearch Service to receive logs from AWS Network Firewall using Amazon Kinesis Data Firehose.

Architecture overview

Figure 1 – Solution Architecture

Here is the functional flow of this architecture:

1. Network Firewall consistently inspects and monitors Network traffic to and from your VPC. Suricata Intrusion Prevention System (IPS) rules configured as a Network Firewall Stateful rule group detect threats and block attacks against known vulnerabilities, as well as create alert logs.
2. These logs are directly written to the Kinesis Data Firehose delivery stream through Direct PUT.
3. Kinesis Data Firehose transports log data to Amazon OpenSearch Service.
4. Amazon OpenSearch Service allows Kinesis Data Firehose to create and use the index through the domain level access policy and index-specific permission for Kinesis Data Firehose’s Service role.
5. Visualize and analyze Network Firewall logs in Amazon OpenSearch Service using Amazon OpenSearch Service Dashboards.

The following steps configure this architecture in your AWS account.

Prepare Amazon OpenSearch Service

1. First, choose the region of your choice where Amazon OpenSearch Service is supported, and create a new Amazon OpenSearch Service domain through → Amazon OpenSearch Service→ Create domain. If you prefer to use an existing domain, then you can skip this section.
2. Set a domain name. In this example, I am using the domain name anf-logs-domain.
3. For this example, I am using the deployment type Development and testing, latest version, 1-AZ, t3.medium.search and number of nodes=1. Note that t3.medium.search instance incurs costs. Refer here for the pricing details of search instances.
4. You can choose the Deployment type as required for your use case. Refer to Operational best practices for Amazon OpenSearch Service for best practices regarding operating Amazon OpenSearch Service domains and general guidelines that apply to many use cases.
   

   Figure 2 – Amazon OpenSearch domain creation

   

   Figure 3 – Amazon OpenSearch domain creation – nodes configuration

   a. Select Network as Public access. Make sure that fine-grain access control is enabled, select create master user, and set the master username and password. For testing and development purposes, we’re configuring the domain with Public Access and leveraging on Domain Access policies for IP restrictions and Basic Authentication using fine-grain access control. For production use cases, consider using VPC access. Select here to learn more about fine-grain access control in Amazon OpenSearch Service.

   

   Figure 4 – Amazon OpenSearch domain creation – FGAC configuration

5. You can use IP-based policies to restrict access to a domain for IP addresses or CIDR blocks. Configure Source IPs with IP ranges from which Amazon OpenSearch Service (for example, Dashboard URL) access will be allowed.
6. You can also authenticate your Amazon OpenSearch Service Dashboards using SAML authentication or Amazon Cognito authentication. We’re skipping these authentication methods for this testing and development purpose.
7. To do this, under Access policy, choose Configure domain level access policy → JSON, and update the policy with the following sample policy. This access policy allows access only to the required source IP ranges. Remember to update the following sample policy before using it:
   • Update aws:SourceIP with the IP address ranges from which the Amazon OpenSearch Service Dashboard URL will be accessed.
   • Change the region at the ARN of Resource with the region of the Amazon OpenSearch Service domain.
   • Change the account ID at the ARN of Resource with the account ID of the Amazon OpenSearch Service domain
8. For more information on access policies, see Identity and Access Management in Amazon OpenSearch Service.

{ 
   "Version": "2012-10-17", 
   "Statement": [ 
      { 
         "Effect": "Allow", 
         "Principal": { 
             "AWS": "*" 
         },
         "Action": [
              "es:ESHttp*" 
          ], 
          "Condition": {
              "IpAddress": { 
                  "aws:SourceIp": [ 
                     "203.0.113.0/24", 
                     "198.51.100.0/24" 
                  ] 
              } 
           }, 
           "Resource": "arn:aws:es:us-east-2:111122223333:domain/anf-logs-domain/*" 
        } 
    ]
 }

Figure 5 – Amazon OpenSearch domain creation – access policy configuration

9. Leave the rest of the configuration as default and select Create. Domain creation will take 15-20 minutes. Proceed to the next step after the domain becomes available.

10. Test access to the Amazon OpenSearch Service Dashboard URL. You can find this URL in the General information of the domain. Log in using the master username and password that you configured.

Prepare Kinesis Data Firehose

1. As a next step, create a Kinesis Data Firehose delivery stream to transport data from Network Firewall to Amazon OpenSearch Service. This delivery stream should be in the same account and region where Network Firewall will be.
2. Set the source as Direct PUT which allows applications to write directly to the delivery stream. Set the destination as Amazon OpenSearch Service.
3. Change the Delivery stream name if required. Under Transform records, set Data transformation to Disabled.
   

   Figure 6 – Amazon Kinesis Delivery stream configuration

4. Under Destination settings, browse and choose the Amazon OpenSearch Service domain. I am choosing anf-logs-domain
5. Set the required name for the index that will be created in the Amazon OpenSearch Service domain. I am specifying the index name as anf-index. This is the index at which both the alert and logs of the Network Firewall will be organized. You may choose to use a different index for different logs. If so, then you must configure the log destinations at Network Firewall to different Kinesis Data Firehose delivery streams.
   

   Figure 7 – Amazon Kinesis Destination settings

6. Under Backup settings, create/choose S3 bucket to collect all failed data.
   

   Figure 8 – Amazon Kinesis – Backup settings

7. Leave the rest of the configuration as default and select Create delivery stream.
8. Once the delivery stream is active, select the delivery stream name and go to Configuration → Permissions. Here, you find the role that Kinesis Data Firehose uses for all of the delivery stream needs. Select the IAM role and copy the ARN from its summary. This IAM role should be allowed to create and use the index on the Amazon OpenSearch Service domain side.

Amazon OpenSearch Service Index-specific permission for the Service role

1. For Kinesis Data Firehose to deliver data to an index in Amazon OpenSearch Service, the Service role of the Kinesis Data Firehose delivery stream must be allowed access through index-specific permission. To do so, log in to Amazon OpenSearch Service Dashboard and go to Security.
   

   Figure 9 – Amazon OpenSearch Security configuration

2. Create a role for access only to an index which will be used by Kinesis Data Firehose.
3. Create a new role under Security → Roles → Create role.
4. Give a name for the role and set the cluster permission to indices_all.
5. Then, set the index permission with the index name configured in Kinesis Data Firehose delivery stream (anf-index*) as pattern. Furthermore, set the permissions as indices_all.
   

   Figure 10 – Amazon OpenSearch Role creation

6. For this example, I am allowing permission to all tenants. You can restrict to required tenants.
   

   Figure 11 – Amazon OpenSearch Tenant permissions for the role

7. Select create role. Then, map this role to Kinesis Data Firehose Service role by selecting Mapped users → Manage mapping and set Backend role with the ARN of the Kinesis Data Firehose Service role copied earlier.
   

   Figure 12 – Amazon OpenSearch backend roles mapping

8. Test the Kinesis Data Firehose delivery stream by following the instructions in Test Using OpenSearch Service as the Destination.

Prepare Network Firewall

1. As next step, create AWS Network Firewall using AWS CloudFormation template. Click here to download and extract the zip file locally.
2. Make sure that you’re in the required account and region, and then Create AWS CloudFormation stack.
3. Choose Upload a template file and choose the AWS-Network-Firewall-demo.yaml that you extracted locally.
4. Select Next and set a name for the stack. Leave the rest as default until the Review page.
5. At the bottom of the Review page, select the check box I acknowledge that AWS CloudFormation might create IAM resources with custom names and select Create stack.
6. This stack will deploy resources as mentioned in the following architecture:

Figure 13 – AWS Network Firewall configuration

• All inbound traffic to the Protected subnet will be routed through Network Firewall endpoint (vpce-id) using Internet Gateway Route Table, where all Stateful and Stateless Firewall rules are applied for decision on traffic flow.
• All outbound traffic from Protected subnet to public will be routed through Network Firewall endpoint (vpce-id) using Protected Subnet Route Table, and sent to Internet Gateway based on the outcome of rules validation.

Change log destination to Kinesis Data Firehose

1. Network Firewall created through CloudFormation stack sends alert logs to . This destination must be changed to Kinesis Data Firehose.
2. Get the name of the Kinesis Data Firehose delivery stream (PUT-OPS-XXXXX) created earlier.
3. Go to Network Firewall, select Network Firewall → Firewall details → Logging → Edit.
4. Uncheck Alert under log type and select Save. This is to have a clean slate. Repeat the same step and select both Alert and Flow log types. Choose destinations for both log types to Kinesis Data Firehose and paste the name of the Kinesis Data Firehose delivery stream here. Select Save and confirm the log destination as shown in the following:
   

   Figure 14 – AWS Network Firewall log destination configuration

Configure rules

1. Network Firewall supports Intrusion Prevention System (IPS), which actively inspects traffic flow with real-time network. Moreover, it protects the application layer from vulnerability exploits and brute force attacks using signature-based detection.
2. Network Firewall Stateful rule group uses Suricata IPS rules engine. Download Suricata emerging user agent rules from here,and create Stateful rule group. These rules will alert on different types of ET user agent activities.
3. To do this, go to Network Firewall rule groups and Create Network Firewall rule group.
4. Set Rule group type to Stateful rule group, specify a name (EmergingUserAgentRuleGroup), and set Capacity to 500. Capacity can’t be exceeded or modified once set. Therefore, configure with enough room for your rule group to grow.
   

   Figure 15 – AWS Network Firewall Stateful rule group configuration

5. Choose Suricata compatible IPS rules as Stateful rule group option.
   

   Figure 16 – AWS Network Firewall Suricata IPS rules selection

6. Copy the content of the downloaded Suricata emerging user agent rules to the Suricata compatible IPS rules section.
   

   Figure 17 – AWS Network Firewall Suricata IPS rules configuration

7. Leave the rest of the configurations as default and select Create Stateful rule group.
8. Similarly, create another Suricata Stateful rule group TestmynidsDropRuleGroupwith Capacity=50 as mentioned in the following. This rule will drop packets and generate alerts when the content of the packet has a string value uid=0|28|root|29| and the traffic is classified as bad-unknown. You can test this rule from the Web server Amazon Elastic Compute Cloud (Amazon EC2) instance using curl --max-time 5 http://testmynids.org/uid/index.html.
   

   Figure 18 – AWS Network Firewall TestmynidsDropRuleGroup Stateful rule group configuration

   

   Figure 19 – AWS Network Firewall TestmynidsDropRuleGroup Suricata IPS rules selection

   

   Figure 20 – AWS Network Firewall TestmynidsDropRuleGroup Suricata IPS rules configuration

9. drop ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7; metadata:created_at 2010_09_23, updated_at 2010_09_23;)
10. Go to Firewall policies and select DemoFirewallPolicy.
11. Scroll down to Stateful rule groups → Select StatefulRuleGroup → Actions → Dissociate from policy.
12. Then, under the same Stateful rule groups (of DemoFirewallPolicy) → Actions → Add unmanaged Stateful rule group, and add both EmergingUserAgentRuleGroup and TestmynidsDropRuleGroup.

Conclusion

In this blog post, we demonstrated steps involved to send AWS Network Firewall logs to Amazon OpenSearch Service using Kinesis Data Firehose. We walked through how to setup Amazon OpenSearch Service Index-specific permission for Kinesis Data Firehose Service role. Furthermore, we demonstrated how to configure rules in Network Firewall.

In part 2 of this blog-post series, we cover steps to generate test alerts, validating them and configure dashboards in Amazon OpenSearch Service to visualize and analyze log data.

About the authors: