Accelerating Migration Evaluator discovery for VMware environment

1. Introduction

An AWS Migration Assessment is a complimentary service that helps potential and existing AWS customers carefully plan and prepare for their migration to the AWS cloud. It includes creating a data-driven directional business case for migrating workloads to AWS. AWS Migration Evaluator (ME) is a data collection tool used in these assessments to collect on-premises server inventory and utilization information over a defined collection period (by default, 7 days).

Some customers are not able to use the Migration Evaluator tooling for various reasons – compliance, lack of required credentials or inability to provision a collector server, for example. To serve these customers, an AWS Migration Assessment can leverage data from third-party monitoring and inventory tools such as RVtools or LiveOptics. However, these tools don’t gather historical utilization data, they provide point-in-time utilization data only, which makes the assessment a less accurate model of overall utilization.

This blog post outlines how to gather historical utilization data from VMware’s vCenter Server without using the Migration Evaluator collector or a third-party tool. This approach is suitable for customers who can’t use our tooling, or who want to accelerate their migration assessment.

2. Solution Overview

Using PowerShell, it is possible to create a script that collects the historical data needed to run a Migration Evaluator assessment from a virtual infrastructure running on VMware vSphere and managed by a vCenter Server. It uses the VMware PowerCLI module to collect from vCenter the following properties and metrics for each managed VM:

1. Provisioning properties:
   1. Number of vCPU Cores
   2. Memory
   3. Attached Storage
   4. Operating System
   5. Host information (Physical cores, Memory and CPU model)
2. Utilization metrics – Peak and Average per day for previous days (# days is chosen by the user, default 7 days) for
   1. CPU
   2.  Memory

By default, the script retrieves historical CPU and Memory utilization metrics according to the customer-defined Statistics Collection Intervals in vCenter, with the defaults being:

The script then generates a Microsoft Excel file with all the data required to complete an Migration Evaluator assessment.

2.1 Prerequisites

1. Windows system that has network access to collect data from vCenter(s)
2. Windows PowerShell v5.0 or greater installed
3. VMware PowerCLI PS module
   PS C:\> Install-Module VMware.PowerCLI
4. ImportExcel PS module
   PS C:\> Install-Module -Name ImportExcel
5. vCenter Information
   1. vCenter IP or FQDN
      1. Protocol used to connect (http or https)
      2. Port if it is not a default one
   2. vCenter user account credentials with Read-Only system role, or a custom role with the permissions required to read statistics.
   3. Statistics Collection should be enabled in vCenter, please refer to: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vcenterhost.doc/GUID-5D85751A-2CE8-4FD5-83C8-90A17060631E.html

2.2 Script Execution Walkthrough

Once all modules are installed, save the script found in the appendix to the Windows system to which you installed PowerCLI as vmware_data_collector.ps1. You can then run it from PowerShell by entering .\vmware_data_collector.ps1 and providing the required parameters.

The script accepts the following parameters:

The PowerShell command will look like depending on your inputs

PS C:\temp>.\vmware_data_collector.ps1 -address "vcenter.domain.com" -username "admin"-password "password123"
or
PS C:\temp>.\vmware_data_collector.ps1 -address 10.23.112.235 -username "admin"-password "password123" -collectionDays 14 -protocol https -disableSSL
or
PS C:\temp>.\vmware_data_collector.ps1 -address "vcenter.domain.com" -username "admin" -password "password123" -collectionDays 28 -protocol http -port 591 -disableSSL -enableLogging

If you prefer to use the default values for all optional parameters, you can simply execute the script PS C:\temp>.\vmware_data_collector.ps1. By doing so, you will be prompted to enter only the address, username, and password.

The script will gather provisioning and utilization data from each VM while showing a progress bar as shown in Figure 1.

Figure 1 – vmware_data_collector.ps1 script execution in PowerShell

It will then generate an Excel file with a filename based on the current date and time, in the format VMWARE_Inventory_And_Usage_Workbook_yyyy-MM-dd_HH-mm-ss.xlsx.

The output file matches the Migration Evaluator collector export format, and you can review the file to ensure it aligns with your security requirements. If you would like to anonymize servers’ names, hostnames and IPs, you can refer to this blog post for guidance. You can also tag servers in the Environment column (such as Production or Development) in the Asset Ownership sheet, or tag servers with their SQL editions (if any) in the Database Type column of the Virtual Provisioning sheet. Server tagging helps to more accurately estimate the compute and licensing costs for your workloads.
Once tagging is complete you can upload the file using the “Self-reported files” section of the Migration Evaluator console, selecting the file format ‘Inventory and Utilization Export’ (see Figure 2). The assigned Migration Evaluator Solutions Architect will analyze the file, and will provide your deliverables.

Figure 2 – Export file upload to the Migration Evaluator console

3. Conclusion

In this post, I introduced a simple way for a Migration Evaluator customer to collect the needed data from vCenter to build a directional business case based on Migration Evaluator recommendations. This approach eliminates the need to install additional tools or manually input data, facilitating the process for customers.

To request a complimentary Migration Evaluator assessment, please speak to your Account Management team at AWS or simply request it directly here.

Appendix

<#
.SYNOPSIS
    Collects VMware vCenter inventory and usage data.

.DESCRIPTION
    This script connects to a VMware vCenter server and collects inventory and usage data,
    outputting the results to an Excel workbook.

.PARAMETER filterVMs
    Controls VM filtering: 'Y' to show only powered on VMs (default), 'N' to show all VMs, default is Y

.PARAMETER enableLogging
    Enables debug logging to a file when specified.

.PARAMETER disableSSL
    Disables SSL certificate validation when connecting to vCenter.

.PARAMETER adress
    The IP address or FQDN of the vCenter server.

.PARAMETER username
    The username for connecting to vCenter.

.PARAMETER password
    The password for connecting to vCenter.

.PARAMETER collectionDays
    Number of days to collect data for. Default is 7 days.

.PARAMETER protocol
    The protocol to use for connection: 'http' or 'https'. Default is https.

.PARAMETER port
    The port number to connect to vCenter. Defaults to 443 for HTTPS or 80 for HTTP.

.EXAMPLE
    .\vmware_data_collector.ps1 -address "vcenter.domain.com" -username "admin" -password "password"

.EXAMPLE
    .\vmware_data_collector.ps1 -address "vcenter.domain.com" -username "admin" -password "password" -collectionDays 14 -protocol https -port 443 -disableSSL

.NOTES
    File Name      : vmware_data_collector.ps1
    Prerequisite   : PowerCLI, ImportExcel module
#>

param(
    [Parameter(Mandatory=$true)]
    [string]$address,

    [Parameter(Mandatory=$true)]
    [string]$username,

    [Parameter(Mandatory=$true)]
    [string]$password,

    [Parameter(Mandatory=$false)]
	[ValidateRange(1, 365)]  
	[int]$collectionDays = 7,

    [Parameter(Mandatory=$false)]
    [ValidateSet('Y','N', IgnoreCase = $true)]
    [string]$filterVMs = 'Y',

    [Parameter(Mandatory=$false)]
    [ValidateSet('http','https', IgnoreCase = $true)]
    [string]$protocol = 'https',
    
	[Parameter(Mandatory=$false)]
	[ValidateRange(1, 65535)]
	[int]$port = 0,  # We'll set the actual default based on protocol	
    
    [Parameter(Mandatory=$false)]
    [switch]$enableLogging,

    [Parameter(Mandatory=$false)]
    [switch]$disableSSL
)


#output file
$output = "VMWARE_Inventory_And_Usage_Workbook_" + (Get-Date).ToString("yyyy-MM-dd_HH-mm-ss") + ".xlsx"
#log file if debugging is enabled

$logFile = ".\vm_details_$(Get-Date -Format 'yyyyMMdd_HHmmss').log"
#Disabling SSL check for invalid server certificate, similar to disable SSL check on the ME collector 
if ($disableSSL) {
    Write-Host "Disabling SSL certificate validation..."
    Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false| Out-Null
}

#disabling the VMWare Customer Experience Improvement Program
Set-PowerCLIConfiguration -Scope User -ParticipateInCEIP $false -Confirm:$false

Import-Module ImportExcel

#check for debug enabled
function Write-DebugLog {
    param([string]$message)
    if ($enableLogging) {
        Add-Content -Path $logFile -Value $message
    }
}

#selecting the port to connect to vcenter
if ($port -eq 0) {
    $port = if ($protocol -eq "https") { 443 } else { 80 }
}

Write-Host "Connecting to vCenter Server..."
#connect to the vcenter
Connect-VIServer $address -Protocol $protocol -User $username -Password $password -Port $port

$allvms = @()
$dailyStats = @()
$allhosts = @()

# Try to get the StatIntervals
$statIntervals = Get-StatInterval
# Check if $statIntervals is null or empty and set default values if necessary
if (-not $statIntervals) {
    # Define default constants
    $pastDay = 300      # 5 minutes
    $pastWeek = 1800    # 30 minutes
    $pastMonth = 7200   # 2 hours
    $pastYear = 86400   # 1 day
} else {
    # If data is returned, assign the actual values from the StatIntervals
    $pastDay = ($statIntervals | Where-Object { $_.Name -eq "Past Day" }).SamplingPeriodSecs
    $pastWeek = ($statIntervals | Where-Object { $_.Name -eq "Past Week" }).SamplingPeriodSecs
    $pastMonth = ($statIntervals | Where-Object { $_.Name -eq "Past Month" }).SamplingPeriodSecs
    $pastYear = ($statIntervals | Where-Object { $_.Name -eq "Past Year" }).SamplingPeriodSecs
}
#retrieving list of VMs depending on command line filterVMs

if ($filterVMs -eq 'Y') {
    $vms = Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"}
    Write-Host "Retrieving only powered on VMs."
} else {
    $vms = Get-VM
    Write-Host "Retrieving all VMs regardless of power state."
}
$totalVMs = $vms.Count
$currentVM = 0

#collecting hosts' information
$hosts = Get-VMHost
foreach ($esxi in $hosts) {
    Write-DebugLog "Collecting host information for $($esxi.Name)..."

    $hoststat = New-Object PSObject -Property @{
        "HostName"      = $esxi.Name
        "NumCPUs"       = $esxi.NumCpu
        "MemoryMB"      = $esxi.MemoryTotalMB
        "ProcessorType" = $esxi.ProcessorType
    }
	$allhosts += $hoststat	
}

#collecting virtual machines' information
foreach ($vm in $vms) {
	#create a progress bar
	$currentVM++
	$percentComplete = [math]::Round(($currentVM / $totalVMs) * 100, 2)
	Write-Progress -Activity "Processing VMs" -Status "Progress: $currentVM of $totalVMs" -PercentComplete $percentComplete

    try {
		#Re-check VMs' state to verify if state was changed (powered off or terminated) while connecting to vCenter 
		$currentVMState = Get-VM -Name $vm.Name -ErrorAction Stop
		if ($currentVMState.PowerState -eq "PoweredOff" -and $filterVMs -eq 'Y') {
            Write-DebugLog "Skipping VM $($vm.Name) as it is no longer powered on."
            continue
        }
		#collecting VMs provisioning data
		Write-DebugLog "Processing VM: $($vm.Name)"
		Write-DebugLog "Collecting basic information..."
		$vmstat = New-Object PSObject -Property @{
			"ServerName" = $vm.Name  
			"OperatingSystem" = $vm.Guest.OSFullName
			"NumCPUs" = $vm.NumCpu
			"MemoryVM" = $vm.MemoryMB 
			"StorageGB" = 0 
			"HostName"= $vm.VMhost
		}
		#collect the total storage of the vmdks
		Write-DebugLog "Calculating total storage for VM: $($vm.Name)"
		$disks = Get-HardDisk -VM $vm
		foreach ($disk in $disks) {
			$vmstat.StorageGB += [math]::Round($disk.CapacityGB, 2)  # Add disk size in GB with 2 decimals
		}
		Write-DebugLog "VM Details: $($vmstat | Out-String)"

		#collect CPU and memory utilization over the inputted period of time
		for ($i = 0; $i -lt $collectionDays; $i++) {
			$dayStart = (Get-Date).AddDays(-$i).Date  # Start of the day
			$dayEnd = $dayStart.AddDays(1).AddSeconds(-1)  # End of the day
			if ($i -le 1) {
				$intervalSecs = $pastDay 
			}
			elseif ($i -ge 2 -and $i -le 7) {
				$intervalSecs = $pastWeek
			}
			elseif ($i -gt 7 -and $i -le 30) {
				$intervalSecs = $pastMonth
			}
			else {
				$intervalSecs = $pastYear
			}
			Write-DebugLog "Retrieving peak CPU and memory usage for VM $($vm.Name) for $dayStart"
			
			# Get peak CPU and memory usage for the specific day
			$statcpu = Get-Stat -Entity $vm -Start $dayStart -Finish $dayEnd -MaxSamples 20000 -Stat cpu.usage.average -IntervalSecs $intervalSecs
			$statmem = Get-Stat -Entity $vm -Start $dayStart -Finish $dayEnd -MaxSamples 20000 -Stat mem.consumed.average -IntervalSecs $intervalSecs
			
			# Calculate peak and average values
            $CpuPeakValue = ($statcpu | Measure-Object -Property Value -Maximum | Select-Object -ExpandProperty Maximum)
            $CpuAvgValue= ($statcpu | Measure-Object -Property Value -Average | Select-Object -ExpandProperty Average)
            $MemoryPeakValue = ($statmem | Measure-Object -Property Value -Maximum | Select-Object -ExpandProperty Maximum)
            $MemoryAvgValue = ($statmem | Measure-Object -Property Value -Average | Select-Object -ExpandProperty Average)
            $cpuPeak = [math]::Round($CpuPeakValue, 2)
            $memPeak = [math]::Round($MemoryPeakValue / (1024 * $vm.MemoryMB), 2)
            $cpuAvg = [math]::Round($CpuAvgValue, 2)
            $memAvg = [math]::Round($MemoryAvgValue / (1024 * $vm.MemoryMB), 2)


			$dailyStat = New-Object PSObject -Property @{
				"ServerName" = $vm.Name 
				"Date" = $dayStart
				"CpuPeak" = $cpuPeak
				"CpuAvg" = $CpuAvg
				"MemPeak" = $memPeak
				"MemAvg" = $memAvg
			}
			$dailyStats += $dailyStat
		}
		$allvms += $vmstat
	}catch{
		Write-DebugLog "Error processing VM $($vm.Name): $_"
		Write-DebugLog "Error details: $($_.Exception.Message)"
	}
}

# Exporting provisioning and utilization metrics to output inventory file
try {
	
	$src_provisioning = $allvms
	$vms_provisioning = $null
	$phs_provisioning = $null
	$ast_ownership = $null
	$vms_utilization = $null
	$i=1
	$name_uid         = @{}
	
    $allhosts | ForEach-Object {
		# Physical Provisioning
		$name = $_."HostName"
		$ph_data = @(
			[PSCustomObject]@{
				"Unique Identifier"         = $(New-Guid)
				"Human Name"                = $name
                "pCpu Cores" 				= $_."NumCPUs"
                "Memory MB" 				= $_."MemoryMB"
                "Total Storage Size GB" = ""
                "Cpu String" 				= $_."ProcessorType"
                "Operating System" = ""
                "Database Type" = ""
                "Address" = ""
                "Remote Storage Size GB" = ""
                "Remote Storage Type" = ""
                "Local Storage Size GB" = ""
                "Local Storage Type" = ""
                "Location" = ""
                "Make" = ""
                "Model" = ""
				}
			)
		$phs_provisioning  += $ph_data
		} 
    $phs_provisioning | Export-Excel -Path $output -AutoSize -WorksheetName "Physical Provisioning"

	$src_provisioning | ForEach-Object {
		# Virtual Provisioning
		$uid                            = $(New-Guid)
		$name                           = $_."ServerName"
		$vm_data = @(
		  [PSCustomObject]@{
			"Unique Identifier"         = $uid
			"Human Name"                = $name
			"vCpu Cores"                = $_."NumCPUs"
			"Memory MB"                 = $_."MemoryVM"
			"Total Storage Size GB"     = $_."StorageGB"
			"Operating System"          = $_."OperatingSystem"
            "Database Type" = ""
			"Hypervisor Name"           = $_."HostName"
            "Address" = ""
			"Remote Storage Size GB"  ="" 
            "Remote Storage Type" = ""
            "Local Storage Size GB" = $_."StorageGB"
            "Local Storage Type" = ""
			}
		)
		$vms_provisioning  = $vms_provisioning + $vm_data

		# Asset Ownership
		$ao_data = @(
		  [PSCustomObject]@{
			"Unique Identifier"         = $uid
			"Human Name"                = $name
            "Environment" = ""
            "Application"= ""
            "SLA" = ""
            "Department" = ""
            "Line of Business" = ""
            "In Scope" = ""
			}
		)
		$ast_ownership += $ao_data
		$name_uid[$name]   = $uid
		$i++
	}
	$vms_provisioning | Export-Excel -Path $output -AutoSize -WorksheetName "Virtual Provisioning"
	$ast_ownership   | Export-Excel -Path $output -AutoSize -WorksheetName "Asset Ownership"
}

catch {
	Write-Error "[ERROR]provisioning data conversion failed!"
	Exit
} 

try {
	# Utilization
	$src_utilization  = $dailyStats
	$src_utilization | ForEach-Object {
		$name                                = $_."ServerName"
		$date                                = [DateTime]::Parse($_.Date, [cultureinfo]::GetCultureInfo('en-us'))
		$ut_data = @(
		  [PSCustomObject]@{
			"Unique Identifier"	             = $name_uid[$name]
			"Human Name"                     = $name
			"Cpu Utilization Peak (P95)"     = $_.CpuPeak / 100
			"Memory Utilization Peak (P95)"  = $_.MemPeak
			"Storage Utilization Peak (P95)" = 1
			"Cpu Utilization Avg (P95)"      = $_.CpuAvg / 100
			"Memory Utilization Avg (P95)"   = $_.MemAvg
			"Storage Utilization Avg (P95)"  = 1
			"Time On Percentage"             = 1
			"Time In-Use Percentage"         = 1
			"Time Stamp Start"               = $date.ToString("yyyy-MM-ddT00:00:00.0000000")
			"Time Stamp End"                 = $date.AddDays(1).ToString("yyyy-MM-ddT00:00:00.0000000")
			}
		)
		$vms_utilization  = $vms_utilization + $ut_data
	}

	$vms_utilization  | Export-Excel -Path $output -AutoSize -WorksheetName "Utilization"
}
catch {
	Write-Error "[ERROR]utilization data conversion failed!"
	Exit
}
Write-Host "Processed $vmCount VMs."
if ($EnableDebugLogging) {
    Write-Host "Debug information has been written to $logFile"
}
Write-Host "Export File has been generated, filename: $output"
Disconnect-VIServer -Confirm:$false
#Script end

About the Author