Using Amazon EBS elastic volumes with Oracle databases (part 3): databases using Oracle ASM

October 2024: This post was reviewed for accuracy.

In parts 1 and 2 of this blog series, we discuss the elastic volumes feature for Amazon EBS and how it works with Oracle database storage layouts. We discuss elastic volumes with databases that use operating-system file systems and that use Logical Volume Managers (LVM) for database storage management. In this post, we discuss the storage layout on Amazon EC2 for Oracle databases that use Oracle Automated Storage Management (Oracle ASM). We demonstrate how you can scale the database storage without impact on availability. We also look at some of the advantages of using elastic volumes with Oracle databases.

Storage operations for databases using Oracle ASM

In this section, we briefly discuss the storage layout on Amazon EC2 for Oracle databases that use Oracle ASM for storage management. Then we discuss how Oracle database storage modifications like increasing the storage or changing the IOPS provisioned were done before the elastic volumes feature was introduced. We also discuss the associated challenges. Finally, we demonstrate how to address some of these challenges using elastic volumes with an example.

Storage layout for databases using Oracle ASM

Oracle ASM is a volume manager that is used to manage storage for Oracle databases. It includes a file system designed specifically for databases. Oracle ASM distributes data across disks to ensure uniform performance. It can also rebalance the data automatically after storage configuration changes like addition or removal of disks.

When using Oracle ASM, you create ASM disk groups that contain one or more ASM disks. An ASM disk is a storage device like an EBS volume. The ASM disk groups are exposed as file interfaces to the Oracle database instances for storing database files like data, control, redo log files, and so on. Oracle ASM uses an Oracle ASM instance. This instance is a special Oracle instance used to configure and manage the disk groups. It also provides file layout information to the Oracle database instance. Once the database instance obtains the necessary extents map, it will perform storage I/O operations directly without going through an Oracle ASM instance.

The following diagram depicts a database layout using Oracle ASM. There are two ASM disk groups—DATA and RECO, with five and three ASM disks (EBS volumes) respectively.

Oracle database storage operations without elastic volumes

To increase the storage or IOPS provisioned for your database, you create new ASM disks (EBS volumes). You add them to the ASM disk group using the following steps:

• Create a new EBS volume and attach it to the EC2 instance
• Create a primary partition on the device
• Create an ASM disk and make it available to Oracle ASM by using the oracleasm createdisk command
• Add the ASM disk to the ASM disk group using the ALTER DISKGROUP … ADD DISK command

After any storage configuration change, Oracle ASM automatically initiates a rebalance operation to distribute the data evenly across the disks. The power setting controls the speed of the rebalancing operation . You can set the default power limit by using the ASM_POWER_LIMIT database initialization parameter, or you can specify it using the POWER clause of the ALTER DISKGROUP statement.

Oracle database storage operations using elastic volumes

To modify an EBS volume, use the modify-volume command from the AWS CLI or the Modify Volume option from the AWS Management Console. Specify the new volume size and IOPS.

If you modify only the IOPS provisioned without changing the volume size, no changes are required at the operating system or Oracle ASM level. If you modify the EBS volume size, then you need to resize the ASM disks using the ALTER DISKGROUP … RESIZE ALL command.

When you modify the size or IOPS of an EBS volume, the data is automatically spread across multiple backend devices. This approach helps to avoid hot spots and ensure that you get the IOPS provisioned.

Example: increasing the storage for a database that uses Oracle ASM

In this section, we demonstrate how to increase the storage provisioned for an Oracle database that uses Oracle ASM for storage management, without any downtime. For this demonstration, we use an Oracle 19c database running on Red Hat Enterprise Linux. We have attached six EBS volumes of 50 GiB each to the EC2 instance, which are presented as ASM disks to Oracle ASM. We have created an ASM disk group DATA with a total size 300 GB from these six ASM disks. During this demonstration, we increase the storage provisioned to the database from 300 GiB to 600 GiB without downtime.

To demonstrate that the resize was performed without any database downtime, we have created a database stored procedure called evtestproc. This procedure inserts records into a table called evtesttab at 10-second intervals. We run this procedure while we perform the resize operation. We can confirm that the resize was done without any database downtime by verifying that records were inserted into the evtesttab table at 10-second intervals without any gap. For the definition of the evtestproc stored procedure, see part 1 of this blog series.

Step 1: Verifying the current setup

From the AWS Management Console, verify the size of the EBS volumes. Currently, it is 50 GB as seen in the following screenshot for one of the six EBS volumes.

Next, we query the v$asm_diskgroup view. We see that the DATA disk group has a total size of 300 GB and contains six ASM disks (backed by the six EBS volumes) of 50 GB each, as shown in the following screenshot.

Select substr(g.name,1,10) disk_group_name,round(g.total_mb/1024) total_dg_size_gb,
g.type redundancy_type,substr(d.path,1,10) disk_path,
round(d.os_mb/1024) os_disk_size_gb,
round(d.total_mb/1024) asm_disk_size_gb
from v$asm_diskgroup g, v$asm_disk d
where g.group_number=d.group_number
and g.name='DATA';

We have created a big file tablespace, 250 GB in size, called EVTestTablespace as shown in the following screenshot.

create bigfile tablespace EVTestTablespace datafile '+DATA' size 250G;
select file_name,round(bytes/1024/1024/1024) size_gb from dba_data_files
where tablespace_name='EVTESTTABLESPACE';

From the following screenshot, we can see that a little over 250 GB has been used from the 300 GB provisioned for the ASM disk group called DATA.

select round(total_mb/1024) diskgroup_size_gb,round((total_mb-free_mb)/1024) used_size_gb
from v$asm_diskgroup
where name='DATA';

Step 2: Setting up the stored procedure

Start the evtestproc stored procedure to insert records into the evtesttab table while we increase the storage provisioned to the database.

begin
  evtestproc();  //PLSQL procedure to insert records into the EVTESTTAB table at 10-second intervals
end;

We query the table from a different session to verify that records are being inserted.

select * from evtesttab order by counter;

Step 3: Resizing the EBS volumes

We now increase the size of the EBS volumes from 50 GB to 100 GB using the AWS CLI.

$ aws ec2 modify-volume --region eu-west-1 --volume-id vol-0fc2598d4bab322bd --size 100
{
    "VolumeModification": {
        "VolumeId": "vol-0fc2598d4bab322bd",
        "ModificationState": "modifying",
        "TargetSize": 100,
        "TargetIops": 100,
        "TargetVolumeType": "io2",
        "TargetMultiAttachEnabled": false,
        "OriginalSize": 50,
        "OriginalIops": 100,
        "OriginalVolumeType": "io2",
        "OriginalMultiAttachEnabled": false,
        "Progress": 0,
        "StartTime": "2024-08-08T08:11:58+00:00"
    }
}

We do this for each of the six EBS volumes. After a short while, we check the status of the volume modification request using the AWS CLI. Now you can see that the volume has entered the optimizing state, and the new size is reflected in the AWS Management Console, as seen in the following screenshot.

$ aws ec2 describe-volumes-modifications --region eu-west-1 --volume-id vol-0fc2598d4bab322bd 
{
    "VolumesModifications": [
        {
            "VolumeId": "vol-0fc2598d4bab322bd",
            "ModificationState": "optimizing",
            "TargetSize": 100,
            "TargetIops": 100,
            "TargetVolumeType": "io2",
            "TargetMultiAttachEnabled": false,
            "OriginalSize": 50,
            "OriginalIops": 100,
            "OriginalVolumeType": "io2",
            "OriginalMultiAttachEnabled": false,
            "Progress": 100,
            "StartTime": "2024-08-08T08:11:58+00:00",
            "EndTime": "2024-08-08T08:17:03+00:00"
        }
    ]
}

Note: If you need to determine the EBS volume that is mapped to an ASMLIB disk, see the following example:

Use the following script to find the devices mapped to the ASMLIB disks:

[oracle@ip-10-0-0-5 ~]$ for asmlibdisk in `ls /dev/oracleasm/disks/*`
  do
    echo "ASMLIB disk name: $asmlibdisk"
    asmdisk=`kfed read $asmlibdisk | grep dskname | tr -s ' '| cut -f2 -d' '`
    echo "ASM disk name: $asmdisk"
    majorminor=`ls -l $asmlibdisk | tr -s ' ' | cut -f5,6 -d' '`
    device=`ls -l /dev | tr -s ' ' | grep -w "$majorminor" | cut -f10 -d' '`
    echo "Device path: /dev/$device"
  done

ASMLIB disk name: /dev/oracleasm/disks/DATA1
ASM disk name: DATA1
Device path: /dev/nvme1n1p1
ASMLIB disk name: /dev/oracleasm/disks/DATA2
ASM disk name: DATA2
Device path: /dev/nvme6n1p1
ASMLIB disk name: /dev/oracleasm/disks/DATA3
ASM disk name: DATA3
Device path: /dev/nvme7n1p1
ASMLIB disk name: /dev/oracleasm/disks/DATA4
ASM disk name: DATA4
Device path: /dev/nvme8n1p1
ASMLIB disk name: /dev/oracleasm/disks/DATA5
ASM disk name: DATA5
Device path: /dev/nvme9n1p1
ASMLIB disk name: /dev/oracleasm/disks/DATA6
ASM disk name: DATA6
Device path: /dev/nvme10n1p1

Once you have identified the path of the mapped device, you can determine its EBS volume ID by matching the device path to the path mentioned in the EBS volume console page. For NVMe EBS volumes, use the steps listed in this document to identify the EBS device.

Use lsblk command to confirm the new size of the EBS volumes at the OS level:

[root@ip-10-0-0-5 ~]# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme1n1      259:1    0   100G  0 disk 
└─nvme1n1p1  259:10   0    50G  0 part 
nvme6n1      259:9    0   100G  0 disk 
└─nvme6n1p1  259:17   0    50G  0 part 
nvme7n1      259:12   0   100G  0 disk 
└─nvme7n1p1  259:18   0    50G  0 part 
nvme8n1      259:13   0   100G  0 disk 
└─nvme8n1p1  259:19   0    50G  0 part 
nvme9n1      259:15   0   100G  0 disk 
└─nvme9n1p1  259:20   0    50G  0 part 
nvme10n1     259:16   0   100G  0 disk 
└─nvme10n1p1 259:21   0    50G  0 part   

Step 4: Resizing the disk partitions on the EBS volumes

The partition must be resized before Oracle ASM recognizes the new size. You can use the following steps to resize the partition online:

Use the growpart command to resize the partition as shown.

[root@ip-10-0-0-5 ~]# growpart /dev/nvme1n1 1
CHANGED: partition=1 start=2048 old: size=104853504 end=104855551 new: size=209713119 end=209715166

We repeat the preceding steps to resize the partitions on all the six EBS volumes.

Verify that the partition has been extended. Use the lsblk command. The partition size should now be equal to the volume size.

[root@ip-10-0-0-5 ~]# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme1n1      259:1    0   100G  0 disk 
└─nvme1n1p1  259:10   0   100G  0 part 
nvme6n1      259:9    0   100G  0 disk 
└─nvme6n1p1  259:17   0   100G  0 part 
nvme7n1      259:12   0   100G  0 disk 
└─nvme7n1p1  259:18   0   100G  0 part 
nvme8n1      259:13   0   100G  0 disk 
└─nvme8n1p1  259:19   0   100G  0 part 
nvme9n1      259:15   0   100G  0 disk 
└─nvme9n1p1  259:20   0   100G  0 part 
nvme10n1     259:16   0   100G  0 disk 
└─nvme10n1p1 259:21   0   100G  0 part

After growing the partitions, resize the diskgroup to use the new disks’ size using the below statement:

ALTER DISKGROUP DATA RESIZE ALL;

Finally, we query the ASM views and verify that the ASM disks are reflecting the new size (100 GB), as shown in the following screenshot.

select substr(g.name,1,10) disk_group_name,round(g.total_mb/1024) total_dg_size_gb,
g.type redundancy_type,substr(d.path,1,10) disk_path,
round(d.os_mb/1024) os_disk_size_gb,
round(d.total_mb/1024) asm_disk_size_gb
from v$asm_diskgroup g, v$asm_disk d
where g.group_number=d.group_number
and g.name='DATA';

Note: If you are resizing the disks to a value greater than 2TB and you encountered ORA-15099 error, make sure that the COMPATIBLE.ASM or COMPATIBLE.RDBMS disk group attribute is set to 12.1 or greater.

Step 5: Increasing the database storage

We now increase the database storage available by resizing the big file tablespace to 500 GB, as shown in the following screenshots.

SQL> alter  tablespace EVTestTablespace resize 500g;
SQL> select file_name,round(bytes/1024/1024/1024) size_gb from dba_data_files
where tablespace_name='EVTESTTABLESPACE';

From the following screenshot, we can see that a little over 500 GB has been used from the 600 GB provisioned for the DATA ASM disk group.

SQL> select round(total_mb/1024) diskgroup_size_gb,round((total_mb-free_mb)/1024) used_size_gb
from v$asm_diskgroup
where name='DATA';

Step 6: Verifying that there was no database downtime while the storage was resized

We query the evtesttab table to verify that the PL/SQL procedure execution was uninterrupted. The query also verifies that records were inserted at 10-second intervals without any gaps, as shown in the following screenshot.

SQL>  select * from evtesttab order by counter;

Using this example, we demonstrate how to increase the storage allocated to an Oracle database that uses Oracle ASM for storage management, without any impact on database availability. You can also change the IOPS provisioned for the database using elastic volumes without any impact on database availability or performance. In addition, you can change the EBS volume type (for example, from io2 to gp3) using elastic volumes without any impact on database availability or performance.

512B and 4K sector size challenge

One of the points that should be considered while managing ASM disks on EC2 is the sector size of the underlying EBS storage. EBS advertises 512-byte or 4,096-byte (4K) physical sectors to the operating system. The advertised sector size of EBS volumes depends on three factors – the instance’s hypervisor, operating system, and volume configuration. If any one of these does not support 4K physical sectors, EBS will default to advertising 512-byte sectors. When changing the EC2 instance class or upgrading the OS, it is critical to consider potential impacts on the EBS volume sector size. Failure to account for sector size changes may result in errors when mounting ASM disk groups. Use the following commands to verify the physical and logical sector sizes:

cat /sys/block/nvme3n1/queue/physical_block_size 
cat /sys/block/nvme3n1/queue/logical_block_size

Advantages of using elastic volumes

• Ease of maintenance – You can use elastic volumes to increase the storage provisioned easily, without downtime or performance impact, by using simple API calls. Similarly, you can use elastic volumes to adjust the IOPS provisioned for your database easily, without downtime or performance impact, by using simple API calls. You can also automate these operations.
• Predictable spiky workloads – Some workloads have spikes that are predetermined and easily predictable, like month-end processing. We can use elastic volumes to dial up the IOPS provisioned for the database during the spikes and dial it down afterward. By doing this, you can meet the performance requirements of the database and keep costs to a minimum at the same time.
• Cost reduction – EBS io2 volumes offer the highest performance but are expensive compared to gp3 volumes, which offer a balance between price and performance. If you have a database that is used mostly during weekdays, you can dial down the IOPS provisioned during the weekend to save costs. Similarly, you can change the volume type from io2 to gp3 during the weekend to save costs.

Let’s look at the potential cost savings using the following example.

Suppose that you have a database that is deployed in the US-East (Ohio) Region, and it has a storage requirement of 1 TB and a peak IOPS requirement of 30,000. The database is mostly accessed during the weekdays (Monday–Friday) and has minimal usage during the weekend (Saturday–Sunday).

If you use only io1 volumes throughout the month for database storage, the EBS charges are $2075 per month as calculated in the following table.

*As of this writing, in the US-East (Ohio) Region.

The same configuration, with gp2 volumes instead of io1 volumes, costs only $100 per month as calculated in the following table.

*As of this writing, in the US-East (Ohio) Region .

The database is hardly accessed during the weekends. Thus, we can use elastic volumes to switch to gp2 volumes during the weekends (two days) and switch back to io1 volumes during the weekdays (five days) to save costs.

In this scenario, by switching between io1 and gp2 volume types, the monthly EBS charges reduce to $1512 as calculated in the following table. This is a savings of 27 percent each month compared to using only io1 volumes ($2075 per month).

• Debugging production issues – Suppose that you run into database performance issues in production (say, after an application patch release or an application version upgrade). In this case, you can use elastic volumes to temporarily increase the IOPS provisioned for your database until you debug and identify the problem. After the issue is resolved, you can dial down the IOPS to the normal level. You can perform this operation easily and without any application or database downtime.
• Performance testing – You can use the same QA environments for functional and performance testing. For example, you can design the QA environments to use gp3 volumes to optimize storage costs. For performance testing, you can simply change the storage type from gp3 to io2 for the duration of the performance testing activity. You can switch back to gp3 again once done. This approach eliminates the need for you to clone your functional testing environment for performance testing and saves costs at the same time.

Summary

Using the elastic volumes feature of Amazon EBS, you can increase the volume size, change the IOPS, or change the volume type while the volume is in use. In this three-part blog post, we show you how to scale the storage of your Oracle databases running on Amazon EC2 without impact on database availability or performance. We also discuss the various advantages of using elastic volumes feature with Oracle databases, including cost savings.

You can find more information on elastic volumes in the Amazon EBS documentation.

About the Authors

Ashok Shanmuga Sundaram is a partner solutions architect with the Global System Integrator (GSI) team at Amazon Web Services. He works with the GSIs to provide guidance on enterprise cloud adoption, migration and strategy.

Ejaz Sayyed is a partner solutions architect with the Global System Integrator (GSI) team at Amazon Web Services. His focus areas include AWS database services and, database and data warehouse migrations on AWS.

Nael Haddad is a senior product manager with Elastic Block Store (EBS) team at Amazon Web Services. He is driving new features associated with multiple EBS product offerings.

Audit History

Last reviewed and updated in October, 2024 by

Tomris Postaci is a Senior Database Specialist Solutions Architect with Amazon Web Services. She focuses on helping customers to design, deploy, and modernize various database workloads.

Manoj Duvva is a Technical Account Manager at Amazon Web Services. In his role, Manoj collaborates with enterprise customers to architect, implement, and scale cloud-based applications to meet their business objectives. He is a subject matter expert in Amazon RDS for Oracle. Manoj is passionate about cloud computing, databases, automation, and artificial intelligence.

Ibrahim Emara is a Solutions Architect at AWS.