AWS Compute Blog

Centralizing Windows Logs with Amazon Elasticsearch Services

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

Cloud administrators often rely on centralized logging systems to better understand their environments, learn usage patterns, and identify future problems so that they can pre-emptively prevent them from occurring, or troubleshoot them more effectively. Of course, these are just some of the functions a centralized logging strategy provides, all which help IT organizations become more productive.  However, delivering these insightful services in-house requires significant effort, including daily management and updates, backups, high availability, hardening, encryption in transit and at rest, and storage management.

 

Cloud admins can reduce this management effort by creating a near real-time event management solution running on AWS using Amazon Elasticsearch Services and Winlogbeat from Elastic as a Windows Agent. This blog post provides step by step instructions on how you can create such a management solution to analyze your Windows Logs based Amazon EC2 instances without the need of expensive third-party products.

Step by Step

Steps for this procedure include:

  1. Launch an Amazon Elasticsearch Service domain.
  2. Install and configure the Winlogbeat agent offered by Elastic.co in an Amazon EC2 running Windows instances.
  3. Customize Logs visualization on Kibana.

1. Launch an Amazon Elasticsearch Service domain.

  1. Open the AWS Console Elasticsearch Service Dashboard
  2. Click “Create a new domain”.
  3. Enter the Domain Name and select the version Elasticsearch 6.0 or earlier.
  4. Click Next.
  5. On Configure Cluster Page, choose the options that better fit your needs. In my case, it will be the following configuration:

Node Configuration

 – Instance Count: 1
– Instance Type: m4.large.elasticsearch

Storage

 – Storage Type: EBS
 – EBS Volume Type: General Purpose (SSD)
 – EBS Volume Size: 100 GB

 Snapshot Configuration

  – Automated snapshot start hour: 00:00 UTC (Default)

 

6. Click Next.

7. On Network Configuration page.

8. Select VPC Access (Recommended)

– Select the VPC, Subnet and Security Groups in which the Elasticsearch cluster will be launched. (The Amazon ElasticSearch Service uses TCP port 443 and must be released in the selected Security Group).

– Kibana Authentication will not be used in this demo, leave it blank.

– On Access Policy: Do not require signing request with IAM credential (Allows associated security groups within your VPC full access to the domain).

9. Click Next. 

10. Click Confirm and the Amazon Elasticsearch domain will be launched.

     While the Amazon Elasticsearch Service is launching, configure the agent on the Amazon EC2.

 

2. Install and configure the Winlogbeat agent offered by Elastic.co on Amazon EC2 Instances.

1.     Access Elasticsearch Winlogbeat and download the x64 installer.

Download Wingbeat Web page

2.     Extract the contents in the “C:\Program Files” directory and rename the extracted directory to Winlogbeat.

Powershell Console on the Winglobeat Directory

3.     Within the Winlogbeat directory (renamed earlier), there is a file called winlogbeat.yml, open it for editing.

4.     The winlogbeat.event_logs section should contain the name of the logs that will be sent to the Amazon Elasticsearch service.

File winlogbeat.event_logs describing the log name

*Note: Use the following command in PowerShell to access the name of all event logs available in the operating system:

       Get-WinEvent -Listlog * | Format-List -Property LogName > C:\Logs.txt

The output.elasticsearch section must contain the URL of the Amazon Elasticsearch Service domain. Port 443 must be specified as in the image below, otherwise the agent will attempt to use the default port of Elasticsearch (9300/tcp).

File winlogbeat.event_logs describing the Amazon Elasticsearch Service configuration

In the setup.kibana section, you must add the URL of the domain. Port 443 must be specified as in the image above (before /_plugin/kibana). The   URL must be entered exactly as in the image below (in quotation marks).

File winlogbeat.event_logs describing the Kibana configuration

* Note: The Amazon Elasticsearch Service and Kibana URLs are available in the administration console of the Elasticsearch domain just created in step 1.

Amazon Elasticsearch Service Console Overview tab

5.     After the winlogbeat.yml file is properly configured, install the agent.

6.     Open Powershell as administrator and navigate to the “C:\Program Files\ Winlogbeat” directory

Run the installation script: .\install-service-winlogbeat.ps1

Powershell Console running the command .\install-service-winlogbeat.ps1 inside the Winlogbeat directory

*Note: If script execution fails due to operating system policy restriction, run Setup with the following command:

PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-winlogbeat.ps1

7.     If successfully installed, the Service Status will appear as Stopped. Verify that the configuration file has no syntax error with the following command:

.\winlogbeat.exe test config -c .\winlogbeat.yml

Powershell Console running the command .\winlogbeat.ext test config -c .\winlogbeat.yml inside the Winlogbeat directory

8.     Run the command to import the Kibana dashboard. The output should read ‘Loaded dashboards’.

.\winlogbeat.exe setup –dashboards

Powershell Console running the command .\winlogbeat.exe setup -dashboards inside the Winlogbeat directory

9.      Without errors in the configuration file and dashboards imported, start the service with the following command:

Start-Service winlogbeat

 Powershell Console running the command Start-Service winlogbeat inside the Winlogbeat directory

10.  Verify that the agent has successfully connected to the Amazon Elasticsearch Service by analyzing the log generated in the agent in “C:\ProgramData\winlogbeat\Logs”  file and search for Connection to backoff with Established result.

Checking the log on C:\ProgramData\Winlogbeat\Logs in order to confirm the sucessful connection between Agent and Amazon Elasticsearch Service

3. Customize Logs visualization on Kibana

Logs will now start being sent to the Amazon Elasticsearch Service and you can create insights about the environment.

1.     Access Kibana through a browser of an Amazon EC2 Windows instance at the Kibana address entered in the administration console of the Amazon Elasticsearch Service domain.

Amazon Elasticsearch Service Console Overview tab

2.     Navigate to the Dashboard tab and you will see the Dashboard named: Winlogbeat Dashboard, click on it and a screen similar to the one below will be displayed.

On Kibana Dashboard accessing Winlogbeat Dashboard

You now have an AWS-managed log centralization system. Just set up your Amazon EC2 instances to send the logs to Amazon Elasticsearch Service and create the filters as needed.

Here is an example on how about create a filter for Logon Failure for Event ID 4625. With the Security logs configured as a shipping log in winlogbeat.yml, any unsuccessful logon attempt will generate an EventID 4625 and this will be filtered.

Click Add a filter and create a filter as in the image below and click Save.

On Kibana Dashboard accessing Winlogbeat Dashboard, adding a filter for event ID 4625

After the filter is created, click PIN FILTER as shown in the image below and then click Discover.

On Kibana Dashboard accessing Winlogbeat Dashboard, pinning a filter

It will now be possible to better understand each log and take the appropriate actions.

On Kibana Dashboard accessing Winlogbeat Dashboard with the results

 

To learn more about Microsoft Workloads on AWS, go to:

https://thinkwithwp.com/pt/solutionspace/microsoft-workloads/

 

About the Author

Photo of the Author

 

 

Marcio Morales is a Senior Solution Architect at Amazon Web Services. He works with AWS customers to provide guidance and technical assistance on running their Microsoft workloads on AWS.