Tuesday, October 30, 2018

VDI Troubleshooting Tips

More and more organizations are virtualizing desktops than ever before. VMWare and Citrix have managed to attract organizations by simplifying the deployment and maintained of VDI’s. Security teams have also pushed for deploying VDI’s since it gets data off the edge and can be stored in a secure, centralized datacenter.

Moreover, many organizations have benefited by implementing VDI since it provides huge savings on the company’s high energy bills. Traditional PC’s can consume a lot of energy, whereas VDI end users generally use a thin-client that uses much lesser energy.

However, VDI does come with its fair share of problems that could be hard to tackle since it has various moving parts. It can be summed up into the 2 major categories -

1)    Connection Issues

When troubleshooting a connectivity issue, it is vital to narrow down the problem and eliminate as many potential causes as possible. Connection issues can be broken down into the following –

a)     Network Connectivity – Check the underlying network. For example, if a user complains on network issues, connect you laptop and see if you are able to connect to the network. Use basic tools such as ping and tracert to identify the point of failure.

In case the user is accessing the VDI remotely, identify if the issue is on the gateway or the VPN.

This also includes firewalls – Ensure the correct firewall ports are open.

b)    Application and Network Latency –
High Application and Network Latency usually cause sluggish behavior for the users. In case its Application latency, its important to understand the cause of the application latency. It could be the underlying infrastructure of the application server causing problems for your VDI environment.

If it’s a network latency issue, it could be related to bandwidth contention and it is important to identify the bottlenecks, such as the WAN links immeadiately.

2)    Infrastructure Issues
A lot of the infrastructure issues are caused due to the ever-growing needs of the end-users. However, it can also be noted that the main culprit of an infrastructure issue is inadequate planning before setting up the VDI environment.

a)     Resource provisioning –
VDI desktops require to be provisioned with adequate CPU, Memory and Storage resources. Under-provisioning VM’s can cause performance issues, whereas over-provisioning can cause wastage of resources and could cause other issues such as CU ready.

Rightsizing is more critical for persistent VDI’s since they are harder to change and maintain since they are static.

b)    Process Monitoring –
It is critical to monitor the processes running in VM and understand which processes cause issues within your VDI. There could be defunct or zombie processes that need to be terminated immediately, which may cause severe end-user issues.

VDI troubleshooting can be simplified using the diagram below. The below diagram gives us an idea of where the problem lies in the infrastructure.

Tools such as Uila can help identify the bottlenecks causing VDI issues and pin-point the user in the right direction.

Monday, October 29, 2018

Interpreting CPU ready and fixing problems caused due to CPU ready

What is CPU ready?

CPU ready is the percentage of time the Virtual Machine is ready but cannot get physical CPU resources from the hypervisor to run its processes. Basically, higher the CPU ready time, the worse off it is for your vSphere environment. CPU ready is a very critical metric and is generally associated with heavy performance degradations.

vCPU and Physical Cores

In order to understand CPU ready, it is important for us to understand the concept of vCPU and Physical Core. Each vCPU is seen as a single physical CPU core by the ESXI host. vCPU’s are given timeslots by the hypervisor, so that it can utilize the available physical cores.

Lets assume that 1vCPU = 1 Physical Core. It is quite common to provision multiple VM’s where the sum of their vCPU levels exceeds the total number of Physical CPU’s.

Ie. Let’s take an example of a host that has 10 Physical Cores and the VM’s provisioned inside it.


Here the sum of the vCPU is 4 + 4 + 4 = 12 vCPU, however the total number of Physical CPU available is 10. Since each of the VM’s require 4 vCPU’s, it is not possible to have 3 VM’s use CPu at the same time.

VM A and VM B would use the physical core initially, and when one of them is done, the hypervisor will schedule VM C to take up 4 Physical cores.

The time period that VM C has to wait in order to get the CPU resources is called CPU ready.

CPU Oversubscription and Host Over-Commitment

CPU Oversubscription and host over-commitment are the main causes of CPU ready time. First of all, you should understand, that it is quite normal to have more vCPU’s than the pCPU’s available. Otherwise, what’s the point of Virtualization!

However, the ratio of vCPU to pCPU must be at a sweet spot. It is recommended that the ratio should be 1:3 or less in order to minimize the CPU ready time. It is also recommended to keep CPU ready time to less than 5%. Anything greater, can cause severe production issues.

Will DRS (Distributed Resource Scheduler) help?
Short answer – No! DRS only takes into account CPU and Memory utilization into account while moving resources.

Ok, so how do I look for CPU ready?

For quick troubleshooting, you can use vSphere client to monitor the CPU ready time for individual VM’s.

For a wholistic view, a tool like Uila can easily help graphically identifying CPU ready and the hosts that are over committed. For example –

You can also quickly identify the oversubscription on the host –

So next time, if you have any CPU performance issues but your CPU utilization is low, keep in mind that it could be CPU ready.

Tuesday, October 23, 2018

Upgrading to vSphere 6.7 U1

vSphere comprises of 2 components – VCSA (vCenter server appliance) and ESXI.

VMware over the years have made it simple to upgrade the vCenter server appliance. In the past, when using a vCenter installed on Windows, it was always easier to create a brand new vCenter rather than update the existing server.

Now you can easily upgrade vCenter through VAMI(vCenter Appliance Management Web Interface). 

Upgrading VCSA

We first start with the upgrade of the VCSA. The VCSA can be updated through the VAMI (vCenter Appliance Management Web Interface).

1)    Access the VAMI. It can be accessed through port 5480

It will look as follows -

You can see from the screenshot above that 6.7 U1 is available.

2)    Click on “Stage and Install

3)    Now you’re up and running on VCSA 6.7 U1. Literally a 2 step process!

Upgrading your ESXI host to 6.7 U1

Now that VCSA is upgraded, you can now update the ESXI hosts.

1)    Go to Update manager –

2)    Create a new baseline

3)    Name the Baseline

4)    Attach the ESXI image and finish

5)    Now we attach our baseline and remediate.

6)    Once it is remediated, you will notice that the ESXI host is on 6.7 U1

Now it’s time to play with vSphere 6.7 U1. After plating with it for a couple of days, I will write a blog on the new features.

Being an outstanding leader – 5 L’s of leadership (From Pat Gelsinger’s keynote at VMUG Wisconsin)

I have been wanting to write this article since my last visit to Milwaukee for the VMUG Wisconsin Usercon a couple of week...