On the Importance of Data Logs

By Chris Munns, Solutions Architect, AWS

As a Solutions Architect at Amazon Web Services, I work with a lot of startups, helping them to build, deploy, manage, and grow their infrastructures on the AWS platform. When I meet with a new company, one of the first questions I ask is “What are you doing with your logs?” More often than not, the answer is little to nothing. Most folks just make sure logs aren’t filling up their disk space. Some people are aggregating them but don’t monitor or track them. Most aren’t keeping their log data for more than a handful of days at best.Some of these companies however, are doing the right thing, and recognize the value in properly aggregating, analyzing, and storing this extremely important operational data. So what are you doing with your logs?

The Importance of Logs

Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Your operating systems generate them, as well as your web servers, your applications, your databases, etc.Amid the mess of confusing lines of seemingly random text can be hints about performance, security, bugs in code, access patterns, and other operational data. But AWS customers use log data for many nonoperational reasons as well, including generating product recommendations, performing A/B tests on site design, tracking user interactions, and delivering targeted ads to end users.Without the proper tools, finding these insights can be like searching for a needle in a haystack. But with the right practices in place, your logs can become the number one source of both operational and business information. Holding onto and making log information available to others inside your organization can be incredibly valuable. To paraphrase heavily, “there’s gold in them thar hills” (of logs)!

Types of Logs

Throughout your infrastructure are many different sources of logs:

Operating system logs: Some of the most basic information is about what the operating system is doing. Are processes dying? Are they being started? Is something causing host resource starvation? Is someone attempting to log into your hosts via SSH repeatedly? Are hardware components failing? Depending on your OS, your operating system logs can offer a wealth of information.

Web Server logs: Apache, Nginx, IIS, and others all provide a ton of logging information about requests and how the server itself is performing. Are there errors in your code that the language handler can’t handle? Is someone looking at your web server for data that isn’t there, or that shouldn’t be there? Is someone actively trying to scan your web server for vulnerabilities? How is your page performing? Lots of basic and not so basic information can be found in your web server logs, making them some of the most important.

Application logs: Application logging is a feature that you might have written into your own applications or is already written into an application you are using. Is your application having trouble connecting to your database? How quickly is it addressing requests? Why did your application crash? What caused users to see that error page? Often application logs can be the first place to go for troubleshooting issues that customers might be experiencing, making them an incredibly valuable resource to keep around.

Database logs: Slow queries? Stability issues? Corruption? All queries? Database logs potentially provide access to all of this information.

CDN logs: Most CDNs provide logs very similar to those your web server would give you. This can be very valuable in analyzing site usage and generating recommendations and more.

Amazon Web Service Logs

A number of different Amazon Web Services generate and provide logs. Often these logs are stored as files in an Amazon S3 bucket of your choosing, or made available via the service’s API or web console. Here are the AWS services that today provide access to logs:

Amazon S3 Access logs
Amazon CloudFront Access logs
Elastic Load Balancer (ELB) logs
Amazon Relational Database Service (RDS) logs
Amazon Elastic MapReduce (EMR) logs
Amazon Redshift logs
AWS Elastic Beanstalk logs
AWS OpsWorks logs (or this link)
AWS Import/Export logs
AWS Data Pipeline logs
AWS CloudTrail logs

The last one above, CloudTrail, records API calls for your account and delivers log files to you. You will definitely want to enable CloudTrail on your accounts.You’ll want to make sure that you keep an eye on the log data you are saving, either regularly purging it or rotating it to Amazon Glacier for even longer term storage depending on your business’s data archival policies. Use S3’s object lifecycle management to define this.

Tools for Log Analysis

Once you have all of your logs centralized and in a single place, what next? Having all this data and no way to analyze it doesn’t help increase operational understanding or intelligence. You need a tool that can help you with this mountain of information. Sometimes this is a tool purposefully built for log analysis, or it could be a something as simple as a data warehouse workflow using services like Amazon Redshift or Elastic MapReduce (EMR) and Data Pipeline to provide nightly log analysis reports. Many companies use a combination of these services over time as they find more and more uses cases for log information.Whatever the use, find a tool that offers you the flexibility, right cost, and scalability that your business needs and dive on in. Here are just a handful broken down by SaaS solutions, open source, and enterprise:

SaaS Solutions
– Boundary
– Cloudlytics
– Loggly
– Papertrail
– Splunk Storm
– SumoLogic

Open Source Solutions
– Graylog2
– LogStash + Kibana + ElasticSearch
– Log.io

Enterprise Solutions
– Logscape
– Splunk
– Tibco LogLogicFor an example of how to do some log analysis in AWS using EMR, check out these two great tutorials of ours:
–Analyze Log Data with Apache Hive, Windows PowerShell, and Amazon EMR
– Analyze Elastic Load Balancing Log Data

Getting Going

Logs and log analysis can be a big topic. Hopefully you’ve gotten the idea that if you aren’t keeping and tracking your logs today, you are missing out and should take the time and opportunity to start handling this super important and valuable data better! As you’ve read here, there are many different sources and origins for log information, and a huge number of tools you can use to analyze that data. Putting the data and tools together in a meaningful way can pay off well in both gained operational intelligence and potentially even a business opportunity.I hope you’ve enjoyed this post and continue to come back to keep on top of the guidance and advice we will be offering up here in the near future. As always, feel free to ping us at AWS for more information, help, or advice this and other topics. Happy hacking!