AWS Database Blog

Automate CEV creation with AWS CloudFormation in Amazon RDS Custom for Oracle

Amazon Relational Database Service (Amazon RDS) Custom for Oracle is a managed database service for legacy, custom, and packaged applications that require access to the underlying operating system and database environment. A custom engine version (CEV) for Amazon RDS Custom for Oracle is a binary volume snapshot of a database engine and specific Amazon Machine Image (AMI). Amazon RDS Custom for Oracle uses the latest available AMI by default, but you can reuse an existing AMI. Amazon RDS Custom uses the installation files you uploaded to Amazon Simple Storage Service (Amazon S3) and the AMI to create your CEV for you. For more information about CEVs, refer to Working with custom engine versions for Amazon RDS Custom for Oracle.

In this post, we discuss the most common issues that you might encounter while creating a CEV for an RDS Custom for Oracle instance and provide a solution to overcome these issues using an AWS CloudFormation template to automate the CEV creation process.

Issues while creating a CEV

To create a CEV, you need a manifest file. A manifest is a collection of database installation files and one-off patches described in JSON format. This is an input JSON for the API call to create a CEV that contains all patches bundled together. A CEV manifest includes the following:

  • A list of installation files (required) that you uploaded to Amazon S3. Amazon RDS Custom applies the patches in the order in which they’re listed in the manifest. The following is a sample CEV manifest for Oracle Database 19c: 
    {
   "databaseInstallationFileNames": [
       "V982063-01.zip"
   ],
   "opatchFileNames": [
       "p6880880_190000_Linux-x86-64.zip"
   ],
   "psuRuPatchFileNames": [
       "p35042068_190000_Linux-x86-64.zip",
       "p35050341_190000_Linux-x86-64.zip"
   ],
   "otherPatchFileNames": [
       "p28730253_190000_Linux-x86-64.zip",
       "p29213893_1919000DBRU_Generic.zip",
       "p33125873_1919000DBRU_Linux-x86-64.zip",
       "p35220732_190000_Linux-x86-64.zip",
       "p35239280_190000_Generic.zip"
   ]
}
  • Installation parameters that set non-default values for the Oracle base, Oracle home, and the ID and name of the UNIX/Linux user and group (optional). You can’t modify the installation parameters for an existing CEV or an existing database instance. You also can’t upgrade from one CEV to another CEV when the installation parameters have different settings. The following is a sample CEV manifest for Oracle Database 19c with an installation parameters JSON field: 
    {
    "databaseInstallationFileNames": [
        "V982063-01.zip"
    ],
    "opatchFileNames": [
        "p6880880_190000_Linux-x86-64.zip"
    ],
    "psuRuPatchFileNames": [
        "p35042068_190000_Linux-x86-64.zip",
        "p35050341_190000_Linux-x86-64.zip"
    ],
    "otherPatchFileNames": [
        "p28730253_190000_Linux-x86-64.zip",
        "p29213893_1919000DBRU_Generic.zip",
        "p33125873_1919000DBRU_Linux-x86-64.zip",
        "p35220732_190000_Linux-x86-64.zip",
       "p35239280_190000_Generic.zip"
    ],
    "installationParameters": {
        "unixGroupName": "dba",
        "unixGroupId": 12345,
        "unixUname": "oracle",
        "unixUid": 12345,
        "oracleHome": "/home/oracle/oracle.19.0.0.0.ru-2020-04.rur-2020-04.r1.EE.1",
        "oracleBase": "/home/oracle"
    }
}

To create a CEV, complete the following steps:

  1. (Optional) Download the manifest templates.
  2. Download your database installation files and patches from Oracle Software Delivery Cloud.
  3. Upload your installation files to Amazon S3.
  4. Share your installation media in Amazon S3 across AWS accounts (optional).
  5. Prepare the CEV manifest.

You can then use this CEV to create the RDS Custom for Oracle instance.

The following are some of the common challenges encountered during the CEV creation process:

  • The process of manually configuring and creating a CEV is a time-consuming procedure.
  • Manual editing of manifest files often leads to typos and other errors, which causes CEV creation failure.
  • Oracle patches downloaded from Oracle Metalink don’t have valid checksums. This could lead to unsuccessful CEV creation.
  • You might neglect to include mandatory Oracle bug fixes for each of the major Release Updates.
  • CEV creation might stall due to invalid database objects while applying patches.
  • You might lack a fail fast mechanism to identify and address issues promptly.

Solution overview

We propose a solution to create a CEV by using a CloudFormation template, which helps avoid the aforementioned issues. We provide a CloudFormation template—you just need to download the template and follow the deployment steps to build the CEV. As of this writing, the template is developed to create the latest 19c version CEV.

This solution uses the following services and resources:

  • AWS CloudFormation – AWS CloudFormation helps you model and set up your AWS resources. You create a template that describes all the AWS resources that you want (such as EC2 instances, IAM roles, or CEVs). AWS CloudFormation takes care of provisioning and configuring those resources for you. In this solution, we deploy all the resources using a CloudFormation script.
  • Amazon Elastic Compute Cloud (Amazon EC2) – Amazon EC2 provides scalable compute capacity in the AWS Cloud. In this solution, we use Amazon EC2 to perform pre-work for creating the CEV.
  • AWS Identity and Access Management (IAM) – IAM helps you securely control access to AWS resources. With IAM, you can centrally manage permissions that control which AWS resources users can access.
  • Amazon S3 – Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance. In this solution, we use Amazon S3 to store downloaded Oracle patches.

The provided CloudFormation template performs the following high-level tasks:

  1. Read key input parameters for provisioning, such as EC2 instance, EC2 instance type, OTN credentials, and more.
  2. Provision an EC2 instance that downloads the patches and validate the checksum.
  3. Create an IAM role to interact with AWS services deployed in this solution.
  4. Create a CEV for Amazon RDS Custom for Oracle.

Prerequisites

This solution requires the following prerequisites before you run it:

  • An AWS account with IAM permissions to create and manage keys, Amazon RDS, Amazon EC2, Amazon S3, AWS CloudFormation, and Amazon Virtual Private Cloud (Amazon VPC) related resources
  • An S3 bucket
  • An Oracle support contract and license to access edelivery.oracle.com and download installers and patches
  • A VPC with either a public subnet or private subnet with access to the internet to download the Oracle patches and necessary packages for this solution
  • An AWS Region where Amazon RDS Custom is available

Deploy the solution

You can use this solution through the AWS Management Console or run it via the AWS Command Line Interface (AWS CLI). This solution assumes that you’re familiar with the process to deploy a CloudFormation template; for instructions, refer to Get started.

To deploy this solution in your account, complete the following steps:

  1. Clone the GitHub repository to your local machine or download the script from AWS Samples.
  2. Follow the GitHub readme to verify the prerequisites and deploy the CloudFormation stack.
  3. Provide the following input parameters for the stack:
    • S3Bucket – Enter the existing S3 bucket to store the downloaded Oracle patches.
    • S3Prefix – Specify if you have an S3 prefix (Yes/No) in your S3 bucket.
    • S3BucketPrefix – Enter the S3 bucket prefix for your folder structure; if not, leave it blank.
    • KMSKeyID – Enter the existing KMSKeyID used to encrypt.
    • EngineType – Specify your engine type. Amazon RDS custom for Oracle supports both NON-CDB (custom-oracle-ee) and CDB (custom-oracle-ee-cdb).
    • EngineVersion – Enter the name of your CEV.
    • LatestAmiId – This is the image ID is from Parameter Store, a capability of AWS Systems Manager. Leave it as is.
    • EC2SubnetID – Enter the existing subnet where the EC2 instance is created.
    • EC2SecurityGroup – Enter the existing security group for the EC2 instance.
    • DBVersion – Choose the DB version to download the Oracle patches and upload them to the S3 bucket.
    • OracleAccountUser – Enter your Oracle account user name to download the patches.
    • OracleAccountPassword – Enter your Oracle account password to download the patches.

    The following screenshot summarizes the parameters for our stack creation.

  4. On the Configure stack options page, choose Next
  5. On the Review page, validate the parameters
  6. Choose Create stack.
    The stack creation process can take approximately 2 to 3 hours to deploy.
  7. Verify the completion of the stack deployment.

When the deployment is complete, you should see the following resources on the AWS CloudFormation console:

  • IAM role – An IAM role called RDS-Custom-CEV-Automation-Role-xxxx is created to allow user interaction with AWS APIs from the EC2 instance.
  • IAM instance profile – An IAM instance profile called RDS-Custom-CEV-Automation-Instance-Profile-xxxx is created from the IAM role.
  • CEV for Amazon RDS Custom for Oracle – A CEV for Amazon RDS Custom for Oracle called 19.x-xxxx is created, which you can use to create an RDS Custom for Oracle instance.
  • IAM policy – An IAM policy called rds-cev-automation-1919-Mediaimportpolicy-xxxx is created to grant permissions for CEV creation.
  • EC2 instance – An EC2 instance (such as t3.medium) called RDS-Custom-CEV-Automation-xxxx is created and set up for downloading the Oracle patches and validating the checksum. Note the following configuration:
    • All the logs are in /home/ec2-user.
    • All the Oracle patches you downloaded are stored in /home/ec2-user/downloaded_patches.
  • AWS CloudFormation wait condition – An AWS CloudFormation wait condition called rds-cev-automation-1919-WaitCondition-xxxx is created to track the status of a configuration process in EC2 user-data.

When the stack creation is complete, navigate to the stack and choose the Resources tab to review all the resources that were created as part of this CloudFormation template.

Deployment of this CloudFormation template can take up to 3 hours to create a CEV. The CEV itself takes at least 2 hours.

Considerations

Note the following considerations:

  • This solution is intended to help you automate the manual steps needed to create a CEV for an RDS Custom for Oracle database.
  • Although this solution is non-invasive, make sure to test it in your development and test environments before implementing this in production.
  • Take extra precaution while deleting the CloudFormation stack. The deletion policy for the EC2 instance and the CEV is set to retain.
  • You’re responsible for the cost of the services that are deployed as part of this solution.

Clean up

To avoid ongoing costs, clean up the resources you no longer need that you created as part of this solution. You can use the AWS CloudFormation console or AWS CLI to delete the CloudFormation stack that you created, then remove the protection override for the RDS Custom instance and delete it.

Conclusion

In this post, we showed you to make the onboarding into Amazon RDS Custom for Oracle smoother by using a CloudFormation template that helps you create the desired CEV. This solution reduces manual intervention and failures to provide a better user experience.

If you have any questions or comments, then leave them in the comments section.

About the authors

Sharath Chandra Kampili is a Database Specialist Solutions Architect with Amazon Web Services. He works with AWS RDS team, focusing on commercial database engines like Oracle. Sharath works directly with AWS customers to provide guidance and technical assistance on the database projects, helping them improve the value of their solutions when using AWS.

Pavan Vukkisila is a Database Administrator in RDS DBS Managed Commercial Engines with Amazon Web Services. At AWS, he works primarily with Amazon RDS Oracle and RDS Custom for Oracle. He is focused on designing, developing new features on RDS Oracle and RDS Custom to solve customer problem.