Amazon OpenSearch Service now scales to 1000 data nodes on a single cluster
Amazon OpenSearch Service now enables you to scale a single cluster to 1000 data nodes (1000 hot nodes and/or 750 warm nodes) and enables you to manage 25 petabytes of data (10 Petabytes in hot nodes and further 15 Petabytes in warm nodes). You no longer need to setup multiple clusters for workloads that require more than 200 data nodes or more than 3 Petabytes of data.
Today, for workloads of more than 3 to 4 petabytes of data, you need to create multiple clusters in OpenSearch Service. This may have required you to refactor your applications or business logic to work with your workload split across multiple clusters. In addition, every cluster requires its own configuration, management, and monitoring, adding to the operational overhead. With this launch, you can scale a single cluster up to 1000 nodes, or 25 petabytes of data, removing the operational overhead that comes with managing multiple clusters.
To scale a cluster beyond 200 nodes, you have to request an increase through Service Quota, after which you can modify your cluster configuration using the AWS Console, AWS CLI, or the AWS SDK. Depending on the size of the cluster, OpenSearch Service will recommend configuration pre-requisites across data nodes, cluster manager nodes, and coordinator nodes. For more information, refer to the documentation.
The new limits are available to all OpenSearch Service clusters running OpenSearch 2.17 and above in all AWS regions where Amazon OpenSearch Service is available. Please refer to the AWS Region Table for more information about Amazon OpenSearch Service availability.