Syed Jahanzaib Personal Blog to Share Knowledge !

June 18, 2022

VMWARE – VCSA 7.0 Migration/Cloning

Filed under: VMware Related — Tags: , — Syed Jahanzaib / Pinochio~:) @ 10:30 PM

Moving VCSA 7.0.0 (Build 16189207) from one ESXI host to other ESXI host
(without VMOTION)


Recently we added one new esxi server (server02) with some beefy hardware specs including enterprise ssd storage & wanted to move old vcenter hosted to this new esxi host to get benefits of good speed.

Since we donot have shared storage, therefore I used the CLONE method which worked surprisingly accurate without any hurdle.

  • Login to your Vcsa admin panel (https://your-vcsa-ip)
  • Select the vCenter Server virtual machine from the Inventory (source esxi server01 where VCSA is currently installed)
  • Right-click the VCSA virtual machine and click Clone
  • Select the Destination ESXi host server02 (in my case it was new ssd base esxi server), & follow on the instructions

Once the Clone process is done

  • Power off vCenter Server on the source host (server01 old server)
  • Power on the vCenter Server virtual machine on the destination ESXi host (server02 – new esxi server)

VCSA takes some time to start all of its services, Be patience ! (Follow this guide for VCSA Troubleshooting)

Once done, open the VCSA admin panel, & Login.

Alhamdolillah, in my case it was quite straight forward process! your mileage may vary 😀

Backup Note:

Backup is your friend 🙂 Make sure to take regular backups of VCSA using its management panel that is https://your-vcsa-ip:5480/

We take its backup vis windows shared folder using SMB option. you can  use SMB/FTP or other methods of your choice.

When you will click on BACKUP NOW, it will ask you to enter all the details like smb/ftp etc location etc, you need to configure it first !

Syed Jahanzaib

January 11, 2021

Vmware VCenter inaccessible Datastore

Filed under: VMware Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 11:03 AM

Recently one of our VCenter 6.7 got crashed & services were not accessible, I spent hours but couldn’t restore it. To save time in further troubleshooting we removed the VC from the esxi server . Few Esxi servers were managed by this Vcenter. I logged in to each ESXI  server & in Actions I selected “Disconnect it from Vcenter“. Afterwards when new Vcenter (VCSA-7) got installed , all esxi were added successfully , But one of the ESXI server was showing some errors, therefore I removed it from the Vcenter, & when I tried to add it again to VC , following error appeared

Datastore ‘M5-11.10–8TB-raid10’ conflicts with an existing datastore in the datacenter that has the same URL (ds:///vmfs/volumes/5d810e33-e56c55bf-71be-0894ef440178/), but is backed by different physical storage.

At VCenter I was seeing below

When I right clicked on this data store, DELETE/MOUNT/UNMOUNT option were greyed out as well. How can I remove this inaccessible data store?

From the Vcenter, I Browsed that inaccessible datastore  , & clicked on VMs tab, it was showing one VM which was moved to another esxi host in the past, I edited that old VM on that esxi server , and removed the mounted ISO (which was pointing to the affected vm esxi server)  , afterwards, the inaccessible datastore disappeared automagically, and the Esxi got re-added in VC again smoothly.

Sharing is caring !

Syed Jahanzaib

November 28, 2019

Virtualization: Quick Notes for myself

vmware esxi

VCSA 6.7 | 503 errors while accessing vcsa portal

503 Error- Solution # 1

While accessing VCSA via web browser, we faced following error


Login to VCSA by SSH, & issue below cmd’s

shell.set --enabled true
service-control --status --all

if you see services in STOPPED: section, then issue below cmd to start all services

service-control --start --all

Now try to access the VCSA via browser & it should work fine.

503 Error- Solution # 2

After logging to VCSA via SSH , and issue df -h, you see SEAT partition full or if you login to the management panel you see following,

If you are seeing seat partition getting near full, its better to extend that disk via ESXI server & then extend it using VCSA cmd’s. Also you can truncate the DB to remove old data logs.


Login to VCSA by SSH, & issue below cmd’s

shell.set --enabled true
# Login to Database
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

This will display the top 30 largest tables within the vCenter Server database

SELECT nspname || '.' || relname AS "relation", pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND C.relkind <> 'i' AND nspname !~ '^pg_toast' ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 30;

You will see tables as per following

VCDB=# SELECT nspname || '.' || relname AS "relation", pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND C.relkind <> 'i' AND nspname !~ '^pg_toast' ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 30;
relation | total_size
vc.vpx_event_arg_8 | 145 MB
vc.vpx_event_arg_2 | 143 MB
vc.vpx_event_arg_1 | 143 MB
vc.vpx_event_arg_92 | 143 MB
vc.vpx_event_arg_10 | 143 MB
vc.vpx_event_arg_11 | 143 MB
vc.vpx_event_arg_16 | 143 MB
vc.vpx_event_arg_13 | 142 MB
vc.vpx_event_arg_3 | 142 MB
vc.vpx_event_arg_90 | 142 MB
vc.vpx_event_arg_9 | 142 MB
vc.vpx_event_arg_4 | 142 MB
vc.vpx_event_arg_91 | 142 MB
vc.vpx_event_arg_15 | 142 MB
vc.vpx_event_arg_14 | 142 MB
vc.vpx_event_arg_87 | 142 MB
vc.vpx_event_arg_5 | 142 MB
vc.vpx_event_arg_12 | 142 MB
vc.vpx_event_arg_88 | 142 MB
vc.vpx_event_arg_89 | 142 MB
vc.vpx_event_arg_6 | 142 MB
vc.vpx_event_arg_21 | 142 MB
vc.vpx_event_arg_22 | 141 MB
vc.vpx_event_arg_23 | 141 MB
vc.vpx_event_arg_17 | 141 MB
vc.vpx_event_arg_7 | 141 MB
vc.vpx_event_arg_19 | 140 MB
vc.vpx_event_arg_24 | 140 MB
vc.vpx_event_arg_18 | 140 MB
vc.vpx_event_arg_20 | 140 MB
(30 rows)

Now  by seeing the table names, you can Truncate the large tables individually

For example:

truncate table vc.vpx_event_arg_8 cascade;

Keep cleaning as per disk space requirements, then Exit the vCenter Server Appliance database by running this command:

Verify the space is reclaimed by running the df -h command.

Preferably restart the VCSA services or restart the VCSA server so that alerts can clear up quickly

Read below vmware guide for more information

VCSA DB Tables Automatic cleanup via SCRIPTING ! Workaround using BASH SCRIPTING

First, Connect to VCSA via putty, & set shell

shell.set --enabled true

Now create two files, one is the bash file that will execute an sql query, & then the second file which will hold the sql query.

First the SQL query file

vi /root/dbcleanup.sql

& paste following data in it.

rec record;
FOR rec IN
FROM pg_tables
WHERE tablename ~ '^vpx_event_[0-9].*'
ORDER BY tablename
|| quote_ident(rec.schemaname) || '.'
|| quote_ident(rec.tablename) || ' CASCADE';

rec record;
FOR rec IN
FROM pg_tables
WHERE tablename ~ '^vpx_event_arg_[0-9].*'
ORDER BY tablename
|| quote_ident(rec.schemaname) || '.'
|| quote_ident(rec.tablename) || ' CASCADE';

Now the BASH script that will actually trigger the above dbcleanup.sql query file

vi /root/

and paste following lines in it

#> /var/mail/root
echo "$DT - DB process started ..." >> /var/log/db.log
# to get the PGPASSWORD, pull it from this cmd > "cat /etc/vmware-vpx/embedded_db.cfg"
export PGPASSWORD='w@x!8q6FB1HvmU9='
/opt/vmware/vpostgres/current/bin/psql -U vc -d VCDB postgres -f /root/dbcleanup.sql -h /var/run/vpostgres
echo "$DT - DB process ended ..." >> /var/log/db.log

(-h /var/run/vpostgres syntax was necessary to execute the psql CMD, it took 1 day to resolve, without this, you will get psql connect error)

Press :wq to save & exit VI editor.

Now schedule it to run every hour or as you like

crontab -e 

Press i to go in edit mode and paste following

# to execute the table cleanup script for every 5 minute,
*/5 * * * * /root/
# to execute the table cleanup script every HOUR,
@hourly /root/

Now press Escape, and then :wq which will result in save & exit the vi editor.

Congrats ! all done. 

This was written in particular to VCSA ver 6.7 only !



ESXI Server: ‘SEL Fullness’ message

Under ESXI, or VCENTER , I was seeing following alert ..

System Management Software 1 SEL Fullness: Log almost full

Solution # 1

In vSphere 6.7 , Navigate to following and reset event log

Monitor > Tasks and Events > Hardware Health > SYSTEM EVENT LOG > RESET EVENT LOG

Solution # 2

Enable SSH on that esxi host, and connect to it via ssh client & issue below cmd’s

localcli hardware ipmi sel clear
/etc/init.d/sfcbd-watchdog restart restart

For few minutes it will show disconnected in vcenter, after few minutes, Refresh the Webclient & related errors will be cleared.


Updating ESXI 6.5 to 6.7u3 from the CLI

Recently I did some major updates in our virtual infrastructure including ESXi (ver 6.5.0, 10719125) ,  Vcenter & Veeam ( . Things were updated in following order

  • Veeam B&R upgraded to ver
  • Vcenter upgraded to ver
  • ESXI hosts upgraded to ver 6.5.0, 10719125

For ESXI update from 6.5 to 6.7u3. In the past I always use Installer CD/USB to upgrade from older esxi to new version, but for this approach, I have to compromise on my holidays or sit very late in office. This time time I took another approach and upgraded all the esxi hosts one by one on sundays remotely from the home using offline bundle installer & esxi CLI method. This is how I accomplished it.

I first downloaded the 6.7 update offline bundle from the Lenovo site (since we have all the IBM/Lenovo brand servers therefore I selected this option to avoid any hardware driver issue).

Afterwards I uploaded this offline bundle zip file to Esxi datastore, then logged in to esxi host via SSH, and issued

esxcli software vib install -d /vmfs/volumes/5d0cf64f-a83e7c86-6a4d-40f2e922c64a/

Note: make sure to change datastore and filename as required.

It took few minutes, once I saw SUCCESS message, I completed the process by simply rebooted the ESXI host by cmd


Better approach is to update rather than install

esxcli software vib update -d /vmfs/volumes/5d0cf64f-a83e7c86-6a4d-40f2e922c64a/

Difference between VIB update and VIB install

Excerpt from “;

To install or update a .zip file, use the -d option. To install or update a .vib file use the -v option.

Using the update command is the recommended method for patch application. Using this command applies all of the newer contents in a patch, including all security fixes. Contents of the patch that are a lower revision than the existing packages on the system are not applied.

Using the install command overwrites the existing packages in the system with contents of the patch you are installing, including installing new packages and removing old packages. The install command may downgrade packages on the system and should be used with caution. If required, the install command can be used to downgrade a system (only for image profiles) when the –allow-downgrade flag is set.

The install method has the possibility of overwriting existing drivers. If you are using 3rd party ESXi images, VMware recommends using the update method to prevent an unbootable state.

Check esxi version from CLI

esxcli system version get
Product : VMware ESXi
Version : 6.7.0
Build : Releasebuild-14320388
Update : 3
Patch : 73


Will update more

Syed Jahanzaib

April 11, 2019

Vcenter 6.5: Cannot complete operation due to concurrent modification by another operation

Filed under: VMware Related — Tags: , , , — Syed Jahanzaib / Pinochio~:) @ 9:39 AM


We have few ESXI machines managed by Vcenter (all have same 6.5 version). Today when we tried to upgrade compatibility on one vm guest using Vcenter, it gave following error.

Cannot complete operation due to concurrent modification by another operation

After some troubleshooting, it came to my knowledge that there was a pending snapshot made by Veeam B&R software, that was causing the issue. After removal of this snapshot, the compatibility upgraded worked fine, & later we moved this VM from one esxi to another dueto resource strains.


Vcenter error and snapshot removal solved it


In one another encounter, whenever we tried to edit the guest VM setting, it gave error “Invalid configuration for device ‘1’.” , for this particular case we simply remove the affected guest VM from the inventory & re-added it and the problem got solve.

May 29, 2018

Vmware ESXI Upgrade Path Notes

Filed under: VMware Related — Syed Jahanzaib / Pinochio~:) @ 1:34 PM

Few days back, we did major upgrades in our data center including esxi upgrades and some network related rack replacement. We were running few vmware esxi 5.0 servers along with vcenter 5.0. To upgrade all servers with 6.5 , we follows below path …

  • Added new 6.5v Vcenter * recommended*
    [OR upgrade existing 5.x Vcenter to 6.5 VC]
  • Now Upgrade each esxi server to 5.5 one by one
  • Now Upgrade each esxi server to 6.5 one by one
  • Now add all esxi servers into new VC 6.5 & all Done 🙂

I had the esxi ISO in CD, therefore I shutdown the all VM guests, and then …

  • Rebooted the esxi server from the ESXI ISO CD
  • It auto detect the previously install esxi, and I just selected UPGRADE

& everything went smoothly.

VIB error when upgrade from 5.5 to 6.5

esxi vib error

When I tried to upgrade from 5.5 to 6.5 it gave some VIB errors as showed in the image above. (It was taken with nokia 3310 🙂

To resolve conflicting VIB, I enabled SSH on the esxi host, logged in via ssh, and issueed following

~ # esxcli software vib list |grep Broad

brcm 550.2.5.11-000000 Broadcom VMwareAccepted 2018-05-26
misc-cnic-register 1.78.75.v55.1-1OEM.550.0.0.1331820 Broadcom VMwareCertified 2018-05-26
net-bnx2 2.2.4f.v55.3-1OEM.550.0.0.1331820 Broadcom VMwareCertified 2018-05-26
net-bnx2x 1.78.80.v55.3-1OEM.550.0.0.1331820 Broadcom VMwareCertified 2018-05-26
net-cnic 1.78.76.v55.7-1OEM.550.0.0.1331820 Broadcom VMwareCertified 2018-05-26
net-tg3 3.135b.v55.1-1OEM.550.0.0.1331820 Broadcom VMwareCertified 2018-05-26
scsi-bnx2fc 1.78.76.v55.1-1OEM.550.0.0.1331820 Broadcom VMwareCertified 2018-05-26
scsi-bnx2i 2.78.76.v55.1-1OEM.550.0.0.1331820 Broadcom VMwareCertified 2018-05-26

# In my case the conflicting driver was net-bnx2x so I just removed it using below command,

~ # esxcli software vib remove -n net-bnx2x

Exit, & Rebooted, afterwards upgrade the esxi 6.5 worked well.

Datastore Conflicts with an existing datacenter that has the same URL

When I removed one esxi from the VC 6.5 & and tried to add it again, it gave following error


After inspection, it came to my knowledge that one of guest from another esxi server is using this data store , I simply moves this to another esxi, and then the host got added successfully !

Syed JahanzaiB

August 18, 2014

Vmware ESXI: You cannot use the vSphere client to edit the settings of virtual machines of version 10 or higher

Filed under: VMware Related — Tags: , — Syed Jahanzaib / Pinochio~:) @ 10:38 AM

Few days back, At remote location, when I converted a physical Linux machine into virtual machine (based on ESXI 5.5 , machine ver 10) I received following error when tried to edit its properties to add new interface card.

Editing virtual machine settings fails with the error: You cannot use the vSphere client to edit the settings of virtual machines of version 10 or higher ...



I had the option to downgrade it using v-converter client, but Time was really short as whole network was down and old physical machine was also out of order, so I used following hack to add the interface quickly and make it online.


  • Turn OFF the required Guest,
  • Remove the guest from the inventory (Right-click -> remove from inventory)
  • Browse your ESXI datastore where guest files are placed,
  • Now Download the .vmx file from your the location where your guest files are placed (Example guest’s name.vmx file)
  • Open it in any text editor (Example NOTEPAD PLUS+),
  • Change the following … 

virtualHW.version = “10”



virtualHW.version = “8”

As showed in the image below …


  • Save this file and upload back to original location.
  • Add the guest back to your inventory by right clicking the vmx file and selecting “Add to inventory”

Now try to edit the guest properties, and this time you will be able to do it.

There were some other workarounds too but in that particular situation, I found this method the most quickest and above all it worked well 🙂


Syed Jahanzaib

December 6, 2013

ESXI 5.5 static mac address “conflicts with VMware reserved MACs”

Filed under: VMware Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 11:36 AM



You may see above error in ESXI 5.5 after changing dynamically generated MAC address to STATIC mac address dueto application MAC bind restriction.

In new ESXI 5.5 ,  new policies have been added where the statically assigned MAC addresses can only be in the range 00:50:56:xx:xx:xx series, If you try to change it to something else and then power on the guest machine, you may see above error.

Few days before, a friend of mine upgraded his ESXI from ver 5.0 to 5.5. One of his guest application was binded with the MAC address and the new esxi 5.5 doesn’t allow to use that specific series mac address. So I googled and found following solution that worked (at least for me 😉 )

To RESOLVE this issue, follow this.

1- Enable SSH in ESXI Server Configuration

2- Turn off the target guest machine and change the mac address as per your requirements for the required interface.

3- Turn off the V-Sphere ESXI client.

3- Login to ESXI server via SSH using any ssh client like PUTTY

4- Goto your data store / guest machine folder and open the VMX file.


For example I have guest machine with ‘123‘ name. So I used following

~ # cd /vmfs/volumes/
/vmfs/volumes # ls

52a18cdd-49376389-86aa-000c29d1de32  61031d71-0233e8da-be74-f942274c16c3
52a18ce5-9d0863e6-e50b-000c29d1de32  8901537a-ad66db83-fd1f-38ac926cce01
52a18ce7-bd9d6e2a-dacf-000c29d1de32  datastore1
/vmfs/volumes #

/vmfs/volumes # cd datastore1/
/vmfs/volumes/52a18ce5-9d0863e6-e50b-000c29d1de32 # ls

/vmfs/volumes/52a18ce5-9d0863e6-e50b-000c29d1de32 # cd 123
/vmfs/volumes/52a18ce5-9d0863e6-e50b-000c29d1de32/123 # ls

123-flat.vmdk               vmware-14.log
123.nvram                     123.vmxf                      vmware-15.log
123.vmdk                      vmware-11.log                 vmware-16.log
123.vmsd                      vmware-12.log                 vmware.log
123.vmx                       vmware-13.log

As showed in the image below . . .


Now open the VMX file of the guest machine.

for example

vi 123.vmx

now press I and add this line anywhere.

ethernet0.checkMACAddress = "false"

(Change the ethernet0 to match your local ethernet number)

Now save and exit it by pressing :wq

Now start V-Sphere ESXI client and start the machine as you normally do 🙂

Congrats you are UP with new MAC address activated 😀


Following is an sample of working .vmx file for static mac address.

ethernet0.networkName = "LAN"
ethernet0.addressType = "static"
ethernet0.present = "TRUE"
ethernet0.checkMACAddress = "false"
ethernet0.address = "00:0C:29:Ha:Ha:Ho"

Syed Jahanzaib

September 23, 2013

Howto power on Vmware Workstation guest machine

Filed under: VMware Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 1:20 PM


Some one asked me howto power on Vmware Workstation guest machine via CLI.  you can use vmrun tool to power on the machine by using target vm guest .vmx file. I decided to write the simple method for my reference too, because I also use it heavily in my virtual lab.

To achieve it, You can create a shortcut on your desktop so that you can quickly start your required workstation base virtual machine by clicking a single icon on your desktop. OR if you want to power it automatically after windows starts, you can create a simple shortcut or batch file and add it windows STARTUP.


You have install vmware workstation in H:\vmware_workstation and your guest vm is in H:\temp_images_ubuntu-proxy , then simply use the following shortcut.

D:\vmware_workstation\vmrun start  D:\vmware_images\ubuntu-proxy\ubuntu.vmx

Tip: vmrun is available in workstation folder.

Also read following for more references.



Syed Jahanzaib

August 26, 2013

Mystry of ethernet not appearing in Ubuntu under VM after mac change

Filed under: Linux Related, VMware Related — Tags: — Syed Jahanzaib / Pinochio~:) @ 8:41 AM


Recently I was doing an remote installation of Ubuntu Server as a guest VM in ESXI 5.1 in INDIA/PUNE. It was OS requirement to change the default dynamic mac created by VM to some other fixed mac address because of an application requirement which was bind with the specific mac address. After changing mac address to required mac address e.g: 00:0h:c1:96:1c:98 via esxi client in guest vm networking properties , when I rebooted Ubuntu, the Ethernet adapter didn’t not appeared in the networking interfaces list.If i manualy try to bring up the interface by ifconfig eth0 up or ifconfig eth1 up , I receive following error.

SIOCSIFADDR: no such device
eth0: ERROR while getting interface flags: No such device
Bind socket to interface: no such device
Failed to bring up eth0

When I revert it to default (dynamic) it works ok.

After investigating deeply, it revealed that it occurred because if we clone a virtual machine , it creates a new MAC address for each network adapter. The guest operating system detects this is a new network adapter that has been inserted, but retains configuration for the previous network adapter (name and mac).

So here is how I fixed it.

Login to your Ubuntu server (which is installed under vm) with root access ,

Open the file /etc/udev/rules.d/70-persistent-net.rules in nano.

nano /etc/udev/rules.d/70-persistent-net.rules

You will see something like below . . . (After cloning of mac)

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x1022:0x2000 (pcnet32)  # Entry Ethernet adapter before cloning
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:11:3b:6a:8c", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x1022:0x2000 (pcnet32)  # Entry Ethernet adapter after cloning
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0h:c1:96:1c:98", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"


Now simply remove the first entry which is old interface entry with mac 00:0c:11:3b:6a:8c,

and in second entry which is eth1, change it to eth0

so after editing the file will look like following.

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.
# Syed Jahanzaib . .

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0h:c1:96:1c:98", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Save the file and and exit.

Now simply reboot the Ubuntu server, and then check your interface by


Hopefully it will appear in the list.


TIP: To avoid such issues, always change the mac address before OS installation :)~  / Zaib


Syed Jahanzaib

April 29, 2013

Older Posts »

%d bloggers like this: