The need to learn how the whole VCF thing works is extremely needed now more than ever since is the only way to manage from technical and supportability point of view for certain type of customer/environment.
So the idea was to start building the management domain and do the bring up of VCF to understand the process.
What you need to know is that the nested esxi nodes that you are building for doing the bring up needs to match specific requirements:
Certificate matching host FQDN
Regenerate the Self-Signed Certificate on All Hosts
Step 1 – Log in to the ESXi host using an SSH client such as Putty.
Step 2 – Regenerate the self-signed certificate by executing the following command:
/sbin/generate-certificatesStep 3 – Restart the hostd and vpxa services by executing the following command:
rebootStep 4 – Repeat this procedure for all remaining hosts.
NTP configuration aligned across nodes and cloud builder
If like me your idea is to try also the vSAN ESA configuration this is enabling an extra step on the build up process that is basically checking the HCL and if the host configuration is matching the HCL of VMware regarding vSAN ESA.
In this case you need to REMOVE paravirtual SCSI controller in the host and the CD/DVD Rom drive.
But the important thing reported also on williamlam blog is how the HCL check is working, and how you can prevent issue if you are using nested esxi hosts:
https://williamlam.com/2023/11/custom-vsan-hcl-json-for-vmware-cloud-foundation-vcf-5-1-and-vsan-esa-using-nested-esxi.html
So basically what I did is configuring in a proper way the HCL file as explained in the previous blog.
I’ve adapted a little bit the script used for generating the hcl file.
# Author: William Lam
# Description: Dynamically generate custom vSAN ESA HCL JSON file connected to standalone ESXi host
$vmhost = Get-VMHost
$supportedESXiReleases = @("ESXi 8.0 U3")
Write-Host -ForegroundColor Green "`nCollecting SSD information from ESXi host ${vmhost} ... "
$imageManager = Get-View ($Vmhost.ExtensionData.ConfigManager.ImageConfigManager)
$vibs = $imageManager.fetchSoftwarePackages()
$storageDevices = $vmhost.ExtensionData.Config.StorageDevice.scsiTopology.Adapter
$storageAdapters = $vmhost.ExtensionData.Config.StorageDevice.hostBusAdapter
$devices = $vmhost.ExtensionData.Config.StorageDevice.scsiLun
$pciDevices = $vmhost.ExtensionData.Hardware.PciDevice
$ctrResults = @()
$ssdResults = @()
$seen = @{}
foreach ($storageDevice in $storageDevices) {
$targets = $storageDevice.target
if($targets -ne $null) {
foreach ($target in $targets) {
foreach ($ScsiLun in $target.Lun.ScsiLun) {
$device = $devices | where {$_.Key -eq $ScsiLun}
$storageAdapter = $storageAdapters | where {$_.Key -eq $storageDevice.Adapter}
$pciDevice = $pciDevices | where {$_.Id -eq $storageAdapter.Pci}
# Convert from Dec to Hex
$vid = ('{0:x}' -f $pciDevice.VendorId).ToLower()
$did = ('{0:x}' -f $pciDevice.DeviceId).ToLower()
$svid = ('{0:x}' -f $pciDevice.SubVendorId).ToLower()
$ssid = ('{0:x}' -f $pciDevice.SubDeviceId).ToLower()
$combined = "${vid}:${did}:${svid}:${ssid}"
if($storageAdapter.Driver -eq "nvme_pcie" -or $storageAdapter.Driver -eq "pvscsi") {
switch ($storageAdapter.Driver) {
"nvme_pcie" {
$controllerType = $storageAdapter.Driver
$controllerDriver = ($vibs | where {$_.name -eq "nvme-pcie"}).Version
}
"pvscsi" {
$controllerType = $storageAdapter.Driver
$controllerDriver = ($vibs | where {$_.name -eq "pvscsi"}).Version
}
}
$ssdReleases=@{}
foreach ($supportedESXiRelease in $supportedESXiReleases) {
$tmpObj = [ordered] @{
vsanSupport = @( "All Flash:","vSANESA-SingleTier")
$controllerType = [ordered] @{
$controllerDriver = [ordered] @{
firmwares = @(
[ordered] @{
firmware = $device.Revision
vsanSupport = [ordered] @{
tier = @("AF-Cache", "vSANESA-Singletier")
mode = @("vSAN", "vSAN ESA")
}
}
)
type = "inbox"
}
}
}
if(!$ssdReleases[$supportedESXiRelease]) {
$ssdReleases.Add($supportedESXiRelease,$tmpObj)
}
}
if($device.DeviceType -eq "disk" -and !$seen[$combined]) {
$ssdTmp = [ordered] @{
id = [int]$(Get-Random -Minimum 1000 -Maximum 50000).toString()
did = $did
vid = $vid
ssid = $ssid
svid = $svid
vendor = $device.Vendor
model = ($device.Model).trim()
devicetype = $device.ApplicationProtocol
partnername = $device.Vendor
productid = ($device.Model).trim()
partnumber = $device.SerialNumber
capacity = [Int]((($device.Capacity.BlockSize * $device.Capacity.Block) / 1048576))
vcglink = "https://williamlam.com/homelab"
releases = $ssdReleases
vsanSupport = [ordered] @{
mode = @("vSAN", "vSAN ESA")
tier = @("vSANESA-Singletier", "AF-Cache")
}
}
$controllerReleases=@{}
foreach ($supportedESXiRelease in $supportedESXiReleases) {
$tmpObj = [ordered] @{
$controllerType = [ordered] @{
$controllerDriver = [ordered] @{
type = "inbox"
queueDepth = $device.QueueDepth
firmwares = @(
[ordered] @{
firmware = $device.Revision
vsanSupport = @( "Hybrid:Pass-Through","All Flash:Pass-Through","vSAN ESA")
}
)
}
}
vsanSupport = @( "Hybrid:Pass-Through","All Flash:Pass-Through")
}
if(!$controllerReleases[$supportedESXiRelease]) {
$controllerReleases.Add($supportedESXiRelease,$tmpObj)
}
}
$controllerTmp = [ordered] @{
id = [int]$(Get-Random -Minimum 1000 -Maximum 50000).toString()
releases = $controllerReleases
}
$ctrResults += $controllerTmp
$ssdResults += $ssdTmp
$seen[$combined] = "yes"
}
}
}
}
}
}
# Retrieve the latest vSAN HCL jsonUpdatedTime
$results = Invoke-WebRequest -Uri 'https://partnerweb.vmware.com/service/vsan/all.json?lastupdatedtime' -Headers @{'x-vmw-esp-clientid'='vsan-hcl-vcf-2023'}
# Parse out content between '{...}'
$pattern = '\{(.+?)\}'
$matched = ([regex]::Matches($results, $pattern)).Value
if($matched -ne $null) {
$vsanHclTime = $matched|ConvertFrom-Json
} else {
Write-Error "Unable to retrieve vSAN HCL jsonUpdatedTime, ensure you have internet connectivity when running this script"
}
$hclObject = [ordered] @{
timestamp = $vsanHclTime.timestamp
jsonUpdatedTime = $vsanHclTime.jsonUpdatedTime
totalCount = $($ssdResults.count + $ctrResults.count)
supportedReleases = $supportedESXiReleases
eula = @{}
data = [ordered] @{
controller = @($ctrResults)
ssd = @($ssdResults)
hdd = @()
}
}
$dateTimeGenerated = Get-Date -Uformat "%m_%d_%Y_%H_%M_%S"
$outputFileName = "custom_vsan_esa_hcl_${dateTimeGenerated}.json"
Write-Host -ForegroundColor Green "Saving Custom vSAN ESA HCL to ${outputFileName}`n"
$hclObject | ConvertTo-Json -Depth 12 | Out-File -FilePath $outputFileName
The result should be something like below
{
"timestamp": 1725519818,
"jsonUpdatedTime": "September 3, 2024, 1:29 AM PDT",
"totalCount": 2,
"supportedReleases": [
"ESXi 8.0 U3"
],
"eula": {},
"data": {
"controller": [
{
"id": 28347,
"releases": {
"ESXi 8.0 U3": {
"nvme_pcie": {
"1.2.4.15-1vmw.803.0.0.24022510": {
"type": "inbox",
"queueDepth": 85,
"firmwares": [
{
"firmware": "1.3",
"vsanSupport": [
"Hybrid:Pass-Through",
"All Flash:Pass-Through",
"vSAN ESA"
]
}
]
}
},
"vsanSupport": [
"Hybrid:Pass-Through",
"All Flash:Pass-Through"
]
}
}
}
],
"ssd": [
{
"id": 1446,
"did": "7f0",
"vid": "15ad",
"ssid": "7f0",
"svid": "15ad",
"vendor": "NVMe",
"model": "VMware Virtual NVMe Disk",
"devicetype": "NVMe",
"partnername": "NVMe",
"productid": "VMware Virtual NVMe Disk",
"partnumber": "397bdbaddec3ad9e000c296416b8f79c",
"capacity": 131072,
"vcglink": "https://williamlam.com/homelab",
"releases": {
"ESXi 8.0 U3": {
"vsanSupport": [
"All Flash:",
"vSANESA-SingleTier"
],
"nvme_pcie": {
"1.2.4.15-1vmw.803.0.0.24022510": {
"firmwares": [
{
"firmware": "1.3",
"vsanSupport": {
"tier": [
"AF-Cache",
"vSANESA-Singletier"
],
"mode": [
"vSAN",
"vSAN ESA"
]
}
}
],
"type": "inbox"
}
}
}
},
"vsanSupport": {
"mode": [
"vSAN",
"vSAN ESA"
],
"tier": [
"vSANESA-Singletier",
"AF-Cache"
]
}
}
],
"hdd": []
}
}
After that you can upload the file directly on the Cloud Builder under /opt/vmware/bringup/tmp/
Apply also the command also showed below:
chmod 644 /opt/vmware/bringup/tmp/nested-esxi-vsan-esa-hcl.json
chown vcf_bringup:vcf /opt/vmware/bringup/tmp/nested-esxi-vsan-esa-hcl.json
After this is possible to start the bring-up process and if the workbook is filled in well and the rest of the things mentioned above are in place you should be able to deploy VCF on the Nested ESXi hosts.
Feel free to comment and thanks if you read this article
