Automate IIS and HttpErr Logs to Amazon CloudWatch Using EC2 Systems Manager

When you have workloads or applications hosted on IIS Web Server, it’s important to monitor and analyze both IIS and HttpErr logs for abnormalities. IIS logs contain an entry for every request to the site. However, at times, you might not find the requests in IIS logs, even though IIS logging is enabled. There is a good chance that the request was rejected by HTTP.sys (Kernel mode driver for HTTP) before it was handed to the IIS worker process. Some of the common reasons for rejecting a request include the following:

The service is unavailable because the application pool is offline. (Http 503: Service Unavailable)
Parse error – Bad request. (Http 400)

In this blog post, we show you how to configure Windows EC2 instances to send HttpErr and IIS logs to Amazon CloudWatch using Amazon EC2 Systems Manager (SSM). Then we show you how to set up a CloudWatch alarm to notify you when the IIS application pool stops using Amazon Simple Notification Service (SNS).

Amazon EC2 Systems Manager is a management service that helps you automate management tasks such as collecting system inventory, applying operating system (OS) patches, automating the creation of Amazon Machine Images (AMIs), and configuring operating systems and applications at scale. Systems Manager lets you remotely and securely manage the configuration of your managed instances.

Configuring Windows EC2 for CloudWatch using EC2 Systems Manager

There are many ways to send instance metrics to CloudWatch. Let’s look at the steps you need to follow to integrate System Manager State Manager with CloudWatch.

Set up the configuration file for CloudWatch
Configure integration with CloudWatch
Create a CloudWatch metric filter and configure an alarm

Step 1: Set up the configuration file for CloudWatch

Download the sample JSON file for CloudWatch

Ensure that “isEnabled” is set to true

{
	"IsEnabled": true,
	"EngineConfiguration": {
	...
	}
}

In the JSON file, add the following configuration information after the IISLogs section.

{
	"Id": "HttpErrLogs",
	"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
	"Parameters": {
		"LogDirectoryPath": "C:\\Windows\\System32\\LogFiles\\HTTPERR",
		"TimestampFormat": "yyyy-MM-dd HH:mm:ss",
		"Encoding": "UTF-8",
		"Filter": "",
		"CultureName": "en-US",
		"TimeZoneKind": "UTC",
		"LineCount": "3"
	}
}

Add the following configuration to ensure that the log data is sent to CloudWatch. Modify or update the Region, LogGroup, and Logstream name.
```
{
	"Id": "HttpErrCloudWatchLogs",
	"FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
	"Parameters": {
		"AccessKey": "",
		"SecretKey": "",
		"Region": "us-east-2",
		"LogGroup": "WebServer",
		"LogStream": "{instance_id}-httpErr"
	}
}
```
Region: The Region where you want to send log data.
LogGroup: Name for your log group. This will appear on the Log Groups screen on the CloudWatch console.
LogStream: Destination log stream. This will appear on the Log Groups > Streams screen on the CloudWatch console.

Note: If you use Systems Manager State Manager, you don’t have to provide credentials in the configuration file.

Flow Control – Each data type must have a corresponding destination in the Flows section. For example, to send log details defined in the HttpErrLogs section to the destination defined in the HttpCloudWatchLog section, add “HttpErrLogs,HttpCloudWatchLog” to the Flows section

"Flows": {
	"Flows": [
		"HttpErrLogs,HttpErrCloudWatchLogs"
	]
}

Similarly, add the following to configure the IIS logs (for LogDirectoryPath, check the IIS logs directory and SiteID)

{
	"Id": "IISLogs",
	"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
	"Parameters": {
		"LogDirectoryPath": "C:\\inetpub\\logs\\LogFiles\\W3SVC1",
		"TimestampFormat": "yyyy-MM-dd HH:mm:ss",
		"Encoding": "UTF-8",
		"Filter": "",
		"CultureName": "en-US",
		"TimeZoneKind": "UTC",
		"LineCount": "3"
	}
},
{
	"Id": "IISCloudWatchLogs",
	"FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
	"Parameters": {
		"AccessKey": "",
		"SecretKey": "",
		"Region": "us-east-2",
		"LogGroup": "WebServer",
		"LogStream": "{instance_id}-iis"
	}
}
...
"Flows": {
	"Flows": [
		"IISLogs,IISCloudWatchLogs",
		"HttpErrLogs,HttpCloudWatchLogs"
	]
}
}

Complete the configuration using this SSM_HttpErr_Cloudwatch example file

Step 2: Configure integration with CloudWatch

You can either choose Run Command or State Manager features of Systems Manager to integrate with CloudWatch. In this blog, we will show you how to use State Manager to integrate with CloudWatch.

State Manager automates the process of keeping your managed instances in a defined state. You can use State Manager to ensure that your instances are bootstrapped with specific software at start up, joined to a Windows domain (Windows instances only), or patched with specific software updates.

Note: Amazon EC2 Systems Manager requires an IAM role for EC2 instances that will process commands. Please ensure you have the correct policy attached to the instance.

Open the AWS Management Console, and go to the EC2 console. In the EC2 console navigation pane, choose State Manager in the Systems Manager Services section and select Create an association.

Provide an associate name and select AWS-ConfigureCloudWatch as the Document.

For Targets, choose the instances to integrate with CloudWatch either by tag or manually. We recommend choosing the target by tag. In case you don’t see any instances here, please check the IAM role policy.
For Schedule, choose 30 minutes as the time interval for how often you want Systems Manager to apply this policy. This doesn’t affect the frequency when the SSM Agent sends data to CloudWatch.
For Parameters, ensure Status is Enabled and copy and paste your JSON contents into the properties.

(Optional) Select Advanced, Write to S3 to send command output to an Amazon S3 bucket.
Choose Create Association.

On the State Manager, select the association that you just created and then choose Apply Association Now.

For more information, see http://docs.thinkwithwp.com/AWSEC2/latest/WindowsGuide/send_logs_to_cwl_instances.html#ec2-configuration-cwl

After you configure integration, the SSM Agent sends all the logs you configured in your JSON file to CloudWatch. SSM creates a log file that can be found at: C:/ProgramData/Amazon/SSM/Logs. It contains the following entries if LogGroup and LogStream are created successfully. If you don’t see these entries, check the error log, which can be found in the same path.

In the CloudWatch Logs console, in the Log Groups list, you should see WebServer, and in the LogStream list you should see {instance_id}-httpErr.

Step 3: Create a CloudWatch metric filter and configure an alarm

Next, we create a metric filter and configure an alarm that is triggered when an IIS application pool stops. If an IIS application stops or goes offline, the following entries are logged in HttpErr logs with s-status as 503 and s-reason as AppOffline.

These logs are located at C:/Windows/System32/LogFiles/HTTPERR

In the CloudWatch console choose Logs in the navigation pane.
Select the WebServer log group and then choose the Create Metric Filter button.

Type “AppOffline” for the Filter pattern.
You can test your filter pattern by selecting Log Data to Test.
Choose Assign Metric and type “IIS AppPool” for the Filter Name.
For Metric Details, type “IIS” for Metrics Namespace and “AppPoolOffline” for the Metric Name. Choose Create Filter.
Next, let’s create an alarm.

Provide the Alarm name and a brief description. In the Whenever section, specify a threshold (AppPoolOffline > 0) for 2 consecutive periods (period of 1 minute). For Statistic, choose Standard. Be sure that Sample Count is chosen in the drop-down box. You could choose a higher resolution based on your requirements, but remember that there are cost implications.

For Actions, for Whenever this alarm select State is Alarm. For Send notification to create a New list to send notification to. Make sure to confirm the subscription.

Whenever an alarm threshold is breached (that is, when the App Pool stops), you will receive an email notification.

Conclusion

Businesses are moving toward automated IT. It’s common for applications to span across environments and locations. They can be in the AWS Cloud and in on-premises data centers. It’s a challenge to ensure that the infrastructure powering your applications is consistent.. To help ensure a consistent approach, you can use State Manager to create policies, reapply these policies to prevent configuration drift, and monitor the status of your intended state.

In this blog post, you have learned how to configure a Windows EC2 instance to send HttpErr and IIS logs to CloudWatch using Amazon EC2 Systems Manager. These steps can be applied to a fleet of Windows Instances running IIS to consolidate logs from all the instances centrally.

To learn more about Amazon EC2 Systems Manager and EC2 Systems Manager State Manager, go to – https://thinkwithwp.com/ec2/systems-manager/

About the Author

Pawan Puthran is a Senior Technical Account Manager at Amazon Web Services. He works with Enterprise Support customers, and he provides technical guidance and assistance to help them make the best use of AWS services. He loves to write blogs outlining his solutions on multiple AWS products.

AWS Cloud Operations Blog