Unlocking sustainable power using Stem’s AI-driven clean energy platform on AWS

In response to the mounting impacts of climate-driven extreme weather, and continued global and regional mandates, industries and businesses worldwide are increasingly investing in sustainability, environmental, social, and governance (ESG), as well as Net-Zero strategies. This includes setting goals to reduce businesses’ carbon emissions by 2030 and accelerating investments in the adoption of clean energy technologies. Unlike the greenwashing strategies that worked in the past, today’s businesses (and their investors) specifically require data-backed solutions and proof of progress toward these goals. Furthermore, the grid must undergo a profound modernization to bring renewable energy sources online and unlock their full potential while proving environmental impact. For example, renewable energy generated from solar or wind must be available even if the sun isn’t shining and the wind isn’t blowing. Advanced storage technologies coupled with AI-driven software to properly manage renewable assets will be crucial to transforming the grid from an aging supplier of commodity electricity to an intelligent “system of systems” that produces optimized energy and environmental outcomes.

Stem is a global leader in AI-driven clean energy solutions and services with over a decade of energy experience helping to simplify the deployment and maximize the value of storage, solar, and EV charging assets for owners and operators. Stem’s unified clean energy management platform, Athena^®, uses historical and real-time data, advanced data analytics, machine learning (ML), and data-driven applications to offer more accurate modeling, as well as optimally size projects, to help reduce risk for its customers. From the beginning, Athena was built as an open, extensible platform, and it has been getting smarter as it processes more data and continuously learns from the over 32 gigawatts (GW) of solar, energy storage, and EV charging assets under Athena management across 50 countries. This real-world data advantage and making it actionable are central to the value that Stem’s Athena platform delivers to businesses.

Use-case

Today, Stem’s Athena platform processes terabytes of data each day. Its ML algorithms generate multiple forecasts – about weather, prices, solar generation, energy demand, and other factors – and analyze how energy assets can capture value at different times. Then, Athena formulates a strategy for maximizing that value and implements it in various ways: by bidding assets into wholesale markets, charging and discharging batteries, injecting solar energy into the grid, or storing it for later, thereby allowing EVs to charge now or overnight. Moreover, because Athena constantly monitors system performance, it adjusts to changing circumstances in real-time.

Stem energy experts design Athena’s algorithms to learn and adapt to each customer’s priorities and goals, whether it’s an independent power producer’s bidding strategy in a wholesale market, a water district’s need for backup power, or a facility’s goal to reduce greenhouse gas emissions. Athena automates the acquisition and curation of data, forecasting, optimizing, and performing real-time autonomous controls to maximize asset performance and ROI, including interfacing with wholesale markets and utilities, integrating and optimizing various assets and value streams, and complying with incentive policies and asset warranty requirements.

Data is at the heart of Stem’s value to its clients. Therefore, Stem must make sure of the quality of its data, that the data is accessible in real-time, and that the data can be adequately analyzed, in real-time, by Stem’s Athena’s platform.

Solution architecture

Stem’s Athena platform collects data from several thousand devices in the field that continuously stream data every second. This is currently over a terabyte of raw data per day and constantly growing. As Stem’s business grew and the Athena platform amassed greater volumes of data at faster intervals, Stem needed a storage system that could provide the unlimited storage of data for long-term archiving. Furthermore, they knew early on that its AI-based Athena platform would be best served by the elastic big data storage and elastic compute, which are only possible in a cloud-native environment.

Stem chose AWS as the cloud infrastructure provider given its proven track record in providing scalable infrastructure that’s easy to integrate. Over time, the technical stack components that Stem uses have changed, and many AWS offerings have grown and evolved to continue to support Stem’s infrastructure needs.

Figure: Stem’s Athena AI Solution architecture on AWS

Since 2017, Stem standardized the utilization of container technology for consistency and ease when deploying its services. Stem’s build process generates container images, which can then be deployed on developer workstations using Docker Compose and on production compute clusters using Kubernetes. Stem was an early adopter of Kubernetes and at the time had to deploy and manage its own Kubernetes clusters on Amazon Elastic Compute Cloud (Amazon EC2). When AWS started offering Amazon Elastic Kubernetes Service (Amazon EKS), Stem found immediate value in making the switch. Amazon EKS provided pre-packaged Kubernetes integrations into AWS services, such as Elastic Load Balancing, optimized networking drivers, and the ease of staying up-to-date on frequent security and feature releases. Besides the elastic scaling of Amazon EKS clusters on Amazon EC2, Stem is now evaluating AWS Fargate for the serverless elastic scalability of compute capacity on Amazon EKS. Stem uses Kubernetes Jobs to spawn hundreds of containers for AI and ML model training and simulations as needed, and the compute elasticity provided by Amazon EKS and Fargate makes it effortless.

Managing growing volumes of data

Over the years, Stem has amassed petabytes of data and relies on Amazon Simple Storage Service (Amazon S3) for the tiered storage of that data. By leveraging Amazon Kinesis to stream data and Amazon Kinesis Data Firehose to continuously persist it to Amazon S3, Stem has an efficient, no-code, and serverless method for archiving incoming data. Furthermore, this lets Stem focus on building stream processing components for other core value-added data processing tasks.

Stem’s operations team wanted the ability to do ad-hoc queries on raw data for various analyses, often on data for the lifetime of the systems deployed. Stem didn’t want them to bog down its operational databases with the ad-hoc query load. Moreover, operational databases only contained a short time span of high-resolution data. Since Stem archives data using near real-time streaming into Amazon S3, the team turned to Presto on Amazon EMR to support these query needs. This lets Stem use dedicated compute capacity to query data on Amazon S3 without impacting the performance of its operational data stores. Stem did experience issues with right-sizing the Presto cluster to support variable query load and subsequently switched to Amazon Athena (not to be confused with Stem’s Athena platform) for these queries. Stem was among the early adopters of AWS Athena and welcomed the ease of use of serverless queries on AmazonS3 data. The pain points shifted to the manual registration of numerous partitions that Stem had in Amazon S3 data. In later versions, AWS Athena introduced partition projection, which improved performance and eliminated the need for partition management in the Glue metadata store. Over the years, Stem has evaluated several technologies to replace AWS Athena, but none could beat the performance and cost-effectiveness for its needs. Today, in addition to long-term ad-hoc data mining, Stem uses AWS Athena for several near real-time analyses and monitoring data pipelines used by network operations.

In addition to Amazon S3, Stem uses several other data stores including Amazon ElastiCache for Redis for various forms of in-memory storage, Amazon DynamoDB as an operational data store of time series data, Amazon Aurora for relational data, and Amazon OpenSearch Service for logs and dashboard monitoring.

AWS technologies are helping Stem reliably manage big data and big compute for our AI-powered energy management solutions.

Conclusion

In this post, you learned how Amazon services are helping Stem support its data-driven intelligent energy platform, all while achieving the AWS design principles of sustainability. Amazon’s pursuit of serverless technologies enables right-sizing workloads to drive optimal resource utilization and reduce energy demand for the data center. Stem’s usage of these technologies not only aligns with Amazon’s sustainability goals, but also results in cost reductions and environmental impact for Stem.

If you would like to get started building your own solution, contact us at sustainability@amazon.com.