Using Amazon EBS elastic volumes with Oracle databases (part 2): databases using LVM

August, 2024: This post has been reviewed and updated for accuracy.

In part 1 of this blog series, we discuss the elastic volumes feature. We also discuss Oracle database storage layouts for simple databases that use a single Amazon EBS volume without LVM for database storage. In this post, part 2 of the series, we discuss the storage layout for Amazon EC2 for Oracle databases that use LVM for storage management. In addition, we demonstrate how you can scale the database storage without an impact on availability.

Storage operations for databases using LVM

In this section, we briefly discuss the storage layout for Amazon EC2 for Oracle databases that use LVM for storage management. Then we discuss how Oracle database storage modifications like increasing the storage provisioned or changing the IOPS provisioned were done before elastic volumes were introduced. We also cover the associated challenges. Finally, we demonstrate how to address some of these challenges with elastic volumes by using an example.

Storage layout for databases using LVM

For larger databases that require multiple EBS volumes for database storage, you can use an LVM to manage the storage. In this scenario, you create volume groups and add the EBS volumes to the volume groups. Then you create logical volumes from the volume groups and create your file systems on top of the logical volumes. The following diagram shows this database storage layout using LVM.

Oracle database storage operations without elastic volumes

To increase the storage or IOPS provisioned for systems using multiple EBS volumes and LVM for storage management, you create new EBS volumes. You then add them to volume groups using the following steps:

Create a new EBS volume and attach it to the EC2 instance.
Create a physical volume using the pvcreate command.
Add the new physical volume to the volume group using the vgextend command.
Extend the logical volume using the lvextend command.
Resize the file system.

Although this increases the total storage capacity and IOPS available due to the addition of the new EBS volume, the data is not evenly distributed across your EBS volumes. This unevenness can result in hot spots. You have to manually rebalance or redistribute your data (Oracle data files) across the EBS volumes.

Oracle database storage operations using elastic volumes

To modify an EBS volume, use the modify-volume command from the AWS CLI or the Modify Volume option from the AWS Management Console. Specify the new volume size and IOPS. If you modify only the IOPS provisioned without changing the volume size, no changes are required at the operating system or LVM level. If you modify the EBS volume size, perform the following additional steps:

Resize the physical volumes using the pvresize command.
Resize the logical volumes using the lvresize command.
Resize the file system.

When you modify the size or IOPS of an EBS volume, the data is automatically spread across multiple backend devices. This approach helps avoid hot spots and ensure that you get the IOPS provisioned.

Example: increasing the storage for a database that uses LVM

In this section, we demonstrate how to increase the storage provisioned for an Oracle database that uses LVM for storage management, without any downtime. For this demonstration, we use an Oracle 19c database running on Red Hat Enterprise Linux (RHEL) that uses LVM for storage management. Two EBS volumes, each with 100 GiB of storage, are presented as physical volumes to the LVM. From these volumes, a logical volume of 200 GiB is created for Oracle database storage. During this demonstration, we increase the storage provisioned to the database from 200 GiB to 400 GiB without downtime.

To demonstrate that the resize is performed without any database downtime, we created a database stored procedure called evtestproc. This procedure inserts records into a table called evtesttab at 10-second intervals. We run this procedure while we perform the resize operation. We can confirm that the resize was done without any database downtime by verifying that records were inserted into the evtesttab table at 10-second intervals without any gap. For the definition of the evtestproc stored procedure, see part 1 of this blog series.

Step 1: Verifying the current setup

From the AWS Management Console, verify the size of the EBS volumes. Currently the size is 100 GiB, as seen in the following screenshot.

Verify that the volumes are attached to the instance using the lsblk -o +SERIAL command, NVMe EBS volumes have the EBS volume ID set as the serial number in the device identification. Use the Identify the EBS device document for other operating systems.

[ec2-user@ip-172-31-55-62 ~]$ lsblk
[ec2-user@ip-10-0-0-142 app]$ sudo lsblk -o +SERIAL
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT SERIAL
nvme1n1     259:0    0  150G  0 disk /u01       vol087a9c2d908bcfe48
nvme0n1     259:1    0  100G  0 disk            vol09124ae5b8874dfdd
├─nvme0n1p1 259:5    0    1M  0 part
├─nvme0n1p2 259:6    0  200M  0 part /boot/efi
└─nvme0n1p3 259:7    0 99.8G  0 part /
nvme5n1 259:11 0 100G 0 disk vol0e3d5b52043f057ab nvme6n1 259:12 0 100G 0 disk vol0063b8ee291c0a0d9

To continue, install the lvm package if it’s not already installed. Verify the installation using the following command.

[ec2-user@ip-10-0-0-142 ~]$ rpm -qa |grep -i lvm
lvm2-lockd-2.03.14-14.el8.x86_64
lvm2-libs-2.03.14-14.el8.x86_64
lvm2-dbusd-2.03.14-14.el8.noarch
lvm2-2.03.14-14.el8.x86_64

If lvm in not installed, install it using the yum command as shown following.

[root@ip-172-31-55-62 ~]# yum install lvm2*

Create the physical volumes using LVM.

[root@ip-172-31-55-62 ec2-user]# pvcreate /dev/xvdg
  Physical volume "/dev/sdg" successfully created

[root@ip-172-31-55-62 ec2-user]# pvcreate /dev/xvdh
  Physical volume "/dev/sdh" successfully created

Note: Be aware of some device name considerations: Depending on the block device driver of the kernel, your device might be attached with a different name than you specified. Amazon EBS volumes are exposed as NVMe block devices on instances built on the AWS Nitro System. The block device driver can assign NVMe device names in a different order than what you specified for the volumes in the block device mapping. Use the steps in How to mount Linux volume and keep mount point consistency for EBS persistence.

Then we create the volume group using LVM.

[ec2-user@ip-10-0-0-142 ~]$ sudo vgcreate EV-Datafile-VG /dev/nvme5n1 /dev/nvme6n1
  Volume group "EV-Datafile-VG" successfully created

Create the logical volume using LVM.

[ec2-user@ip-10-0-0-142 ~]$ sudo lvcreate -l +100%FREE -n EV-Data-LV EV-Datafile-VG
  Logical volume "EV-Data-LV" created.

Verify the configuration using the pvdisplay, vgdisplay, and lvdisplay commands as shown.

[ec2-user@ip-10-0-0-142 ~]$ sudo pvdisplay
  --- Physical volume ---
  PV Name               /dev/nvme5n1
  VG Name               EV-Datafile-VG
  PV Size               100.00 GiB / not usable 4.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              25599
  Free PE               0
  Allocated PE          25599
  PV UUID               SrcjkF-X5lO-cAwB-aWbn-W7pU-0xeR-axTbuX

  --- Physical volume ---
  PV Name               /dev/nvme6n1
  VG Name               EV-Datafile-VG
  PV Size               100.00 GiB / not usable 4.00 MiB      
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              25599
  Free PE               0
  Allocated PE          25599
  PV UUID               rAMjzF-SE9e-qklL-MMS6-4ThD-zQvi-VhnaMq

 [ec2-user@ip-10-0-0-142 ~]$ sudo vgdisplay
  --- Volume group ---
  VG Name               EV-Datafile-VG
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               199.99 GiB
  PE Size               4.00 MiB
  Total PE              51198
  Alloc PE / Size       51198 / 199.99 GiB
  Free  PE / Size       0 / 0
  VG UUID               FkHKqr-hHDR-Znxs-5VwS-Mi2s-bbKF-utvDkP

[ec2-user@ip-10-0-0-142 ~]$ sudo lvdisplay EV-Datafile-VG
  --- Logical volume ---
  LV Path                /dev/EV-Datafile-VG/EV-Data-LV
  LV Name                EV-Data-LV
  VG Name                EV-Datafile-VG
  LV UUID                IVlLK5-yWCA-80je-d8Zn-lSyA-7Ed2-D086h4
  LV Write Access        read/write
  LV Creation host, time ip-10-0-0-142.eu-west-1.compute.internal, 2024-08-07 20:40:14 +0000
  LV Status              available
  # open                 0
  LV Size                199.99 GiB
  Current LE             51198
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0

[root@ip-172-31-55-62 ec2-user]# vgdisplay
  --- Volume group ---
  VG Name               EV-Datafile-VG
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               199.99 GiB
  PE Size               4.00 MiB
  Total PE              51198
  Alloc PE / Size       0 / 0   
  Free  PE / Size       51198 / 199.99 GiB
  VG UUID               GJw617-s5hP-eVnp-7MCv-VHNS-g0vs-wEjT3u

[root@ip-172-31-55-62 ~]# lvdisplay EV-Datafile-VG
  --- Logical volume ---
  LV Path                /dev/EV-Datafile-VG/EV-Data-LV
  LV Name                EV-Data-LV
  VG Name                EV-Datafile-VG
  LV UUID                KhYS7n-fj5i-oEU3-2EPD-zSi6-2Tta-2XqPwe
  LV Write Access        read/write
  LV Creation host, time ip-172-31-55-62, 2018-06-13 18:22:09 +0000
  LV Status              available
  # open                 0
  LV Size                199.99 GiB
  Current LE             51198
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

For storing the data files, create a directory called customdf as shown following.

[ec2-user@ip-10-0-0-142 oracle]$ sudo mkdir -p /u01/app/oracle/oradata/cdb1/pdb1/customdf

Create a file system on the logical volume and mount it at /u01/app/oracle/oradata/cdb1/pdb1/customdf/ as shown following.

[ec2-user@ip-10-0-0-142 oracle]$ sudo mkfs.ext4 /dev/EV-Datafile-VG/EV-Data-LV 
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 52426752 4k blocks and 13107200 inodes
Filesystem UUID: 4d95dfca-e6aa-4ec2-a3ae-91126a7b99f0
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

[ec2-user@ip-10-0-0-142 oracle]$ sudo mount /dev/EV-Datafile-VG/EV-Data-LV /u01/app/oracle/oradata/cdb1/pdb1/customdf/

[root@ip-172-31-55-62 ec2-user]# mount /dev/EV-Datafile-VG/EV-Data-LV /u01/app/oracle/oradata/cdb1/pdb1/customdf/

Note: Both Ext4 and XFS support inline resizing. For this example, we use Ext4, because it’s a simpler choice.

To test create a big file tablespace called EVTestTablespace with a 180-GB data file called evtestdf.dbf, using SQL*Plus as shown following.

SQL> CREATE BIGFILE TABLESPACE EVTestTablespace DATAFILE '/u01/app/oracle/oradata/cdb1/pdb1/customdf/EVTESTDF.dbf' SIZE 180G;
Tablespace created.

You can verify the disk utilization at the OS level using the df command as shown following.

[grid@ip-10-0-0-142 ~]$ df -h | grep u01
/dev/nvme1n1                               150G   24G  127G  16% /u01
/dev/mapper/EV--Datafile--VG-EV--Data--LV  196G  181G  5.8G  97% /u01/app/oracle/oradata/cdb1/pdb1/customdf

Now, verify the size and data file location of the newly created tablespace as shown in the following screenshot.

Step 2: Setting up the stored procedure

To insert records into the evtesttab table while we increase the storage provisioned to the database, start the evtestproc stored procedure.

begin
  evtestproc();  //PLSQL procedure to insert records into the EVTESTTAB table at 10-second intervals
end;

We query the table from SQL Workbench to verify that records are being inserted.

Step 3: Resizing the EBS volume

We now increase the size of the EBS volumes 100 GB to 200 GB using the AWS CLI.

$ aws ec2 modify-volume --region eu-west-1 --volume-id vol-0e3d5b52043f057ab --size 200
{
    "VolumeModification": {
        "VolumeId": "vol-0e3d5b52043f057ab",
        "ModificationState": "modifying",
        "TargetSize": 200,
        "TargetIops": 6000,
        "TargetVolumeType": "io2",
        "TargetMultiAttachEnabled": false,
        "OriginalSize": 100,
        "OriginalIops": 6000,
        "OriginalVolumeType": "io2",
        "OriginalMultiAttachEnabled": false,
        "Progress": 0,
        "StartTime": "2024-08-07T22:28:33+00:00"
    }
}

$ aws ec2 modify-volume --region eu-west-1 --volume-id vol-0063b8ee291c0a0d9 --size 200
{
    "VolumeModification": {
        "VolumeId": "vol-0063b8ee291c0a0d9",
        "ModificationState": "modifying",
        "TargetSize": 200,
        "TargetIops": 6000,
        "TargetVolumeType": "io2",
        "TargetMultiAttachEnabled": false,
        "OriginalSize": 100,
        "OriginalIops": 6000,
        "OriginalVolumeType": "io2",
        "OriginalMultiAttachEnabled": false,
        "Progress": 0,
        "StartTime": "2024-08-07T22:28:59+00:00"
    }
}

After a short while, we check the status of the volume modification request using the AWS CLI. You can see that the volume has entered the optimizing state. The AWS Management Console shows the new size, as shown in the following screenshot.

$aws ec2 describe-volumes-modifications --region eu-west-1 --volume-id vol-0e3d5b52043f057ab
{
    "VolumesModifications": [
        {
            "VolumeId": "vol-0e3d5b52043f057ab",
            "ModificationState": "optimizing",
            "TargetSize": 200,
            "TargetIops": 6000,
            "TargetVolumeType": "io2",
            "TargetMultiAttachEnabled": false,
            "OriginalSize": 100,
            "OriginalIops": 6000,
            "OriginalVolumeType": "io2",
            "OriginalMultiAttachEnabled": false,
            "Progress": 0,
            "StartTime": "2024-08-07T22:28:33+00:00"
        }
    ]
}

$aws ec2 describe-volumes-modifications --region eu-west-1 --volume-id vol-0063b8ee291c0a0d9
{
    "VolumesModifications": [
        {
            "VolumeId": "vol-0063b8ee291c0a0d9",
            "ModificationState": "optimizing",
            "TargetSize": 200,
            "TargetIops": 6000,
            "TargetVolumeType": "io2",
            "TargetMultiAttachEnabled": false,
            "OriginalSize": 100,
            "OriginalIops": 6000,
            "OriginalVolumeType": "io2",
            "OriginalMultiAttachEnabled": false,
            "Progress": 0,
            "StartTime": "2024-08-07T22:28:59+00:00"
        }
    ]
}
}

Step 4: Resizing the physical volumes, logical volumes, and file system

To resize the physical volumes, use the pvresize command.

[ec2-user@ip-10-0-0-142 ~]$ sudo pvresize /dev/nvme5n1
  Physical volume "/dev/nvme5n1" changed
  1 physical volume(s) resized or updated / 0 physical volume(s) not resized
[ec2-user@ip-10-0-0-142 ~]$ sudo pvresize /dev/nvme6n1
  Physical volume "/dev/nvme6n1" changed
  1 physical volume(s) resized or updated / 0 physical volume(s) not resized

Verify that the new size is reflected using the pvdisplay command.

[ec2-user@ip-10-0-0-142 ~]$ sudo pvdisplay /dev/nvme5n1
  --- Physical volume ---
  PV Name               /dev/nvme5n1
  VG Name               EV-Datafile-VG
  PV Size               <200.00 GiB / not usable 3.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              51199
  Free PE               25600
  Allocated PE          25599
  PV UUID               SrcjkF-X5lO-cAwB-aWbn-W7pU-0xeR-axTbuX

[ec2-user@ip-10-0-0-142 ~]$ sudo pvdisplay /dev/nvme6n1
  --- Physical volume ---
  PV Name               /dev/nvme6n1
  VG Name               EV-Datafile-VG
  PV Size               <200.00 GiB / not usable 3.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              51199
  Free PE               25600
  Allocated PE          25599
  PV UUID               rAMjzF-SE9e-qklL-MMS6-4ThD-zQvi-VhnaMq

We now resize the logical volume using the lvresize command. The –l +100%FREE option of the lvresize command allocates all the newly added space to the logical volume.

[ec2-user@ip-10-0-0-142 ~]$ sudo lvresize -l +100%FREE /dev/EV-Datafile-VG/EV-Data-LV
  Size of logical volume EV-Datafile-VG/EV-Data-LV changed from 199.99 GiB (51198 extents) to 399.99 GiB (102398 extents).
  Logical volume EV-Datafile-VG/EV-Data-LV successfully resized.

Verify that the new volume size now appears using the lvdisplay command.

[ec2-user@ip-10-0-0-142 ~]$ sudo lvdisplay EV-Datafile-VG
  --- Logical volume ---
  LV Path                /dev/EV-Datafile-VG/EV-Data-LV        
  LV Name                EV-Data-LV
  VG Name                EV-Datafile-VG
  LV UUID                IVlLK5-yWCA-80je-d8Zn-lSyA-7Ed2-D086h4
  LV Write Access        read/write
  LV Creation host, time ip-10-0-0-142.eu-west-1.compute.internal, 2024-08-07 20:40:14 +0000
  LV Status              available
  # open                 1
  LV Size                399.99 GiB
  Current LE             102398
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

We finally resize the file system (ext4) using the resize2fs command.

[ec2-user@ip-10-0-0-142 ~]$ sudo resize2fs /dev/EV-Datafile-VG/EV-Data-LV 
resize2fs 1.45.6 (20-Mar-2020)
Filesystem at /dev/EV-Datafile-VG/EV-Data-LV is mounted on /u01/app/oracle/oradata/cdb1/pdb1/customdf; on-line resizing required
old_desc_blocks = 25, new_desc_blocks = 50
The filesystem on /dev/EV-Datafile-VG/EV-Data-LV is now 104855552 (4k) blocks long.

Verify that the file system has been resized using the df command.

[ec2-user@ip-10-0-0-142 ~]$ df -h | grep u01
/dev/nvme1n1                               150G   24G  127G  16% /u01
/dev/mapper/EV--Datafile--VG-EV--Data--LV  393G  181G  195G  49% /u01/app/oracle/oradata/cdb1/pdb1/customdf

Step 5: Increasing the database storage

We now increase the database storage available by resizing the big file tablespace to 360 GB, using SQL*Plus as shown following

SQL> ALTER TABLESPACE EVTestTableSpace RESIZE 360G;

Tablespace altered.

The database storage is now increased to 360 GB, as shown in the following screenshot.

Step 6 – Verify that there was no database downtime while the storage was resized

To verify that the PL/SQL procedure execution was uninterrupted, we query the evtesttab table. This query also verifies that records were inserted at 10-second intervals without any gaps, as shown in the following screenshot.

Using this example, we demonstrate how to increase the storage allocated to an Oracle database that uses LVM for storage management, without any impact on database availability. You can also change the IOPS provisioned for the database or change the EBS volume type (for example, from io1 to gp2) using elastic volumes. You can do so without any impact on database availability or performance.

In the next post, we discuss the storage layout on Amazon EC2 for Oracle databases that use Oracle Automated Storage Management (Oracle ASM). We also demonstrate how you can scale the database storage without impact on database availability.

About the Authors

Ashok Shanmuga Sundaram is a partner solutions architect with the Global System Integrator (GSI) team at Amazon Web Services. He works with the GSIs to provide guidance on enterprise cloud adoption, migration and strategy.

Ejaz Sayyed is a partner solutions architect with the Global System Integrator (GSI) team at Amazon Web Services. His focus areas include AWS database services and, database and data warehouse migrations on AWS.

Nael Haddad is a senior product manager with Elastic Block Store (EBS) team at Amazon Web Services. He is driving new features associated with multiple EBS product offerings.

Audit History

Last reviewed and updated in August, 2024 by

Tomris Postaci is a Senior Database Specialist Solutions Architect with Amazon Web Services. She focuses on helping customers to design, deploy, and modernize various database workloads.

Manoj Duvva is a Technical Account Manager at Amazon Web Services. In his role, Manoj collaborates with enterprise customers to architect, implement, and scale cloud-based applications to meet their business objectives. He is a subject matter expert in Amazon RDS for Oracle. Manoj is passionate about cloud computing, databases, automation, and artificial intelligence.

Ibrahim Emara is a Solutions Architect at AWS.

AWS Database Blog