Please subscribe to the product from AWS Marketplace and Activate the Glue connector from AWS Glue Studio .

Pre-requisite

• Elasticsearch domain
  • For Amazon OpenSearch Service, enable compatibility mode.
• Network reachability from the Glue job to Elasticsearch domain
  • For Amazon Elasticsearch Service users, you need to allow access from Glue jobs in the policy.
    • If you have your Elasticsearch domain with VPC access, you need to attach a Glue Network connection with VPC configuration to the Glue job.
    • If you have your Elasticsearch domain with public access and you want to allow access only from Glue jobs, you can add NAT Gateway in your subnet, and attach a Glue Network Connection using this subnet to the Glue job.
• AWS Secrets Manager secret (you can create the secret in following steps)

Create a new secret for Elasticsearch in AWS Secrets Manager

Create a secret in AWS Secrets Manager to store username/password.

1. On the Secrets Manager console, choose Store a new secret.
2. For Secret type, select Other type of secret.
3. Enter your key as es.net.http.auth.user and the value as your Elasticsearch username.
4. Enter your key as es.net.http.auth.pass and the value as your Elasticsearch password.
5. Leave the rest of the options at their default.
6. Choose Next.
7. Give a name to the secret elasticsearch_credentials.
8. Follow through the rest of the steps to store the secret.

Connection options

You can pass the following options to the connector.

• path (required): The Elasticsearch path in the format index/type.
• es.nodes (required): The Elasticsearch HTTPS endpoint.
• es.port(required): The Elasticsearch port number. For Amazon Elasticsearch Service, choose 443.
• es.nodes.wan.only (optional): Whether the connector is used against an Elasticsearch instance in a cloud/restricted environment over the WAN. For Amazon Elasticsearch Service, choose true.

See other available options here: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html 

References

For more details, please refer to Glue Elasticsearch connector guidance: https://docs.thinkwithwp.com/glue/latest/ug/tutorial-elastisearch-connector.html . For more details about this open-source Elasticsearch spark connector, please refer to this open-source connector online reference: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html .