Virtualization: Quick Notes for myself

---

---

VCSA 6.7 | 503 errors while accessing vcsa portal

503 Error- Solution # 1

While accessing VCSA via web browser, we faced following error

Solution#1:

Login to VCSA by SSH, & issue below cmd’s

shell.set --enabled true
shell
service-control --status --all

if you see services in STOPPED: section, then issue below cmd to start all services

service-control --start --all

Now try to access the VCSA via browser & it should work fine.

503 Error- Solution # 2

After logging to VCSA via SSH , and issue df -h, you see SEAT partition full or if you login to the management panel you see following,

If you are seeing seat partition getting near full, its better to extend that disk via ESXI server & then extend it using VCSA cmd’s. Also you can truncate the DB to remove old data logs.

Solution#1:

Login to VCSA by SSH, & issue below cmd’s

shell.set --enabled true
shell
# Login to Database
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

This will display the top 30 largest tables within the vCenter Server database

SELECT nspname || '.' || relname AS "relation", pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND C.relkind <> 'i' AND nspname !~ '^pg_toast' ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 30;

You will see tables as per following

VCDB=# SELECT nspname || '.' || relname AS "relation", pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND C.relkind <> 'i' AND nspname !~ '^pg_toast' ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 30;
relation | total_size
---------------------+------------
vc.vpx_event_arg_8 | 145 MB
vc.vpx_event_arg_2 | 143 MB
vc.vpx_event_arg_1 | 143 MB
vc.vpx_event_arg_92 | 143 MB
vc.vpx_event_arg_10 | 143 MB
vc.vpx_event_arg_11 | 143 MB
vc.vpx_event_arg_16 | 143 MB
vc.vpx_event_arg_13 | 142 MB
vc.vpx_event_arg_3 | 142 MB
vc.vpx_event_arg_90 | 142 MB
vc.vpx_event_arg_9 | 142 MB
vc.vpx_event_arg_4 | 142 MB
vc.vpx_event_arg_91 | 142 MB
vc.vpx_event_arg_15 | 142 MB
vc.vpx_event_arg_14 | 142 MB
vc.vpx_event_arg_87 | 142 MB
vc.vpx_event_arg_5 | 142 MB
vc.vpx_event_arg_12 | 142 MB
vc.vpx_event_arg_88 | 142 MB
vc.vpx_event_arg_89 | 142 MB
vc.vpx_event_arg_6 | 142 MB
vc.vpx_event_arg_21 | 142 MB
vc.vpx_event_arg_22 | 141 MB
vc.vpx_event_arg_23 | 141 MB
vc.vpx_event_arg_17 | 141 MB
vc.vpx_event_arg_7 | 141 MB
vc.vpx_event_arg_19 | 140 MB
vc.vpx_event_arg_24 | 140 MB
vc.vpx_event_arg_18 | 140 MB
vc.vpx_event_arg_20 | 140 MB
(30 rows)

Now  by seeing the table names, you can Truncate the large tables individually

For example:

truncate table vc.vpx_event_arg_8 cascade;

Keep cleaning as per disk space requirements, then Exit the vCenter Server Appliance database by running this command:
\q

Verify the space is reclaimed by running the df -h command.

Preferably restart the VCSA services or restart the VCSA server so that alerts can clear up quickly

Read below vmware guide for more information

https://kb-uat.vmware.com/s/article/2119809

---

VCSA DB Tables Automatic cleanup via SCRIPTING ! Workaround using BASH SCRIPTING

First, Connect to VCSA via putty, & set shell

shell.set --enabled true
shell

Now create two files, one is the bash file that will execute an sql query, & then the second file which will hold the sql query.

First the SQL query file

vi /root/dbcleanup.sql

& paste following data in it.

DO
$$
DECLARE
rec record;
BEGIN
FOR rec IN
SELECT *
FROM pg_tables
WHERE tablename ~ '^vpx_event_[0-9].*'
ORDER BY tablename
LOOP
EXECUTE 'TRUNCATE TABLE '
|| quote_ident(rec.schemaname) || '.'
|| quote_ident(rec.tablename) || ' CASCADE';
END LOOP;
END$$;

DO
$$
DECLARE
rec record;
BEGIN
FOR rec IN
SELECT *
FROM pg_tables
WHERE tablename ~ '^vpx_event_arg_[0-9].*'
ORDER BY tablename
LOOP
EXECUTE 'TRUNCATE TABLE '
|| quote_ident(rec.schemaname) || '.'
|| quote_ident(rec.tablename) || ' CASCADE';
END LOOP;
END$$;

Now the BASH script that will actually trigger the above dbcleanup.sql query file

vi /root/dbcleanup.sh

and paste following lines in it

#!/bin/bash
#> /var/mail/root
DT=`date`
echo "$DT - DB process started ..." >> /var/log/db.log
# to get the PGPASSWORD, pull it from this cmd > "cat /etc/vmware-vpx/embedded_db.cfg"
export PGPASSWORD='w@x!8q6FB1HvmU9='
/opt/vmware/vpostgres/current/bin/psql -U vc -d VCDB postgres -f /root/dbcleanup.sql -h /var/run/vpostgres
DT=`date`
echo "$DT - DB process ended ..." >> /var/log/db.log

(-h /var/run/vpostgres syntax was necessary to execute the psql CMD, it took 1 day to resolve, without this, you will get psql connect error)

Press :wq to save & exit VI editor.

Now schedule it to run every hour or as you like

crontab -e 

Press i to go in edit mode and paste following

# to execute the table cleanup script for every 5 minute,
*/5 * * * * /root/dbcleanup.sh
# to execute the table cleanup script every HOUR,
@hourly /root/dbcleanup.sh

Now press Escape, and then :wq which will result in save & exit the vi editor.

Congrats ! all done. 

This was written in particular to VCSA ver 6.7 only !

Syed.Jahanzaib

 

---

ESXI Server: ‘SEL Fullness’ message

Under ESXI, or VCENTER , I was seeing following alert ..

System Management Software 1 SEL Fullness: Log almost full

Solution # 1

In vSphere 6.7 , Navigate to following and reset event log

Monitor > Tasks and Events > Hardware Health > SYSTEM EVENT LOG > RESET EVENT LOG

Solution # 2

Enable SSH on that esxi host, and connect to it via ssh client & issue below cmd’s

localcli hardware ipmi sel clear
/etc/init.d/sfcbd-watchdog restart
services.sh restart

For few minutes it will show disconnected in vcenter, after few minutes, Refresh the Webclient & related errors will be cleared.

Alhamdolillah!

---

Updating ESXI 6.5 to 6.7u3 from the CLI

Recently I did some major updates in our virtual infrastructure including ESXi (ver 6.5.0, 10719125) ,  Vcenter & Veeam (9.5.4.2866) . Things were updated in following order

• Veeam B&R upgraded to ver 9.5.4.2866
• Vcenter upgraded to ver 6.7.0.40000
• ESXI hosts upgraded to ver 6.5.0, 10719125

For ESXI update from 6.5 to 6.7u3. In the past I always use Installer CD/USB to upgrade from older esxi to new version, but for this approach, I have to compromise on my holidays or sit very late in office. This time time I took another approach and upgraded all the esxi hosts one by one on sundays remotely from the home using offline bundle installer & esxi CLI method. This is how I accomplished it.

I first downloaded the 6.7 update offline bundle from the Lenovo site (since we have all the IBM/Lenovo brand servers therefore I selected this option to avoid any hardware driver issue).

https://vmware.lenovo.com/content/custom_iso/6.7/6.7u3/

Afterwards I uploaded this offline bundle zip file to Esxi datastore, then logged in to esxi host via SSH, and issued

esxcli software vib install -d /vmfs/volumes/5d0cf64f-a83e7c86-6a4d-40f2e922c64a/Lenovo_Offline_Bundle_VMware_ESXi_6.7.0.update03_14320388_LNV_20190920.zip

Note: make sure to change datastore and filename as required.

It took few minutes, once I saw SUCCESS message, I completed the process by simply rebooted the ESXI host by cmd

reboot

Better approach is to update rather than install

esxcli software vib update -d /vmfs/volumes/5d0cf64f-a83e7c86-6a4d-40f2e922c64a/Lenovo_Offline_Bundle_VMware_ESXi_6.7.0.update03_14320388_LNV_20190920.zip

Difference between VIB update and VIB install

Excerpt from “https://communities.vmware.com/thread/435959”

To install or update a .zip file, use the -d option. To install or update a .vib file use the -v option.

Using the update command is the recommended method for patch application. Using this command applies all of the newer contents in a patch, including all security fixes. Contents of the patch that are a lower revision than the existing packages on the system are not applied.

Using the install command overwrites the existing packages in the system with contents of the patch you are installing, including installing new packages and removing old packages. The install command may downgrade packages on the system and should be used with caution. If required, the install command can be used to downgrade a system (only for image profiles) when the –allow-downgrade flag is set.

The install method has the possibility of overwriting existing drivers. If you are using 3rd party ESXi images, VMware recommends using the update method to prevent an unbootable state.

Check esxi version from CLI

esxcli system version get
Product : VMware ESXi
Version : 6.7.0
Build : Releasebuild-14320388
Update : 3
Patch : 73

 

---

Will update more

Regards
Syed Jahanzaib