Category Archives: vSphere 6

How to change vRO 7.x Active-Active cluster to Active-StandBy?

A relatively simple looking process, but at times, it might make you wonder in which order I should be doing this. Lately while talking with a customer, I realized that in vRO documentation, hints of this process are given but not really in the form of step by step instructions. I then went ahead and tested following steps. I am sure these would be useful to people who might be looking at changing their vRO active-active cluster to active-standby.

1) Take a full backup of vRO Db if you are using External DB, if you are using embedded vPostgresDB as vRO DB then skip to step2
2) Take snapshot of all the vRO nodes.
3) Go into vRO control center of vRO node1, set the number of active nodes to 1. (Under vRo Control center-> Orchestrator Cluster Management -> Number of active nodes. )
4) Repeat step 3) for all the remaining vRO nodes in this cluster.
5) Restart vRO node1
6) Repeat step 5) for all the remaining vRO nodes in this cluster (one by one)

At the end of this, you will see in ‘vRO Control center -> Cluster Management’ that you have now only 1 node as Active and rest of those are stand by. Which node becomes active depends upon the order of restart of the nodes. (This can be seen from control center of any nodes in cluster)

Please note: Do not forget to remove snapshots taken in step 2) when you are done with this process.


How to collect log bundle of vRA using command line

as my previous post about collecting vRO log bundle using command line

This post addresses step by step instructions for collecting log bundle from vRA infra.

==> Take log bundle from vRA Appliance only

  1. login into vRA appliance using SSH Client
  2. cd /tmp
  3. run command ‘vcac-support’
  4. this will create a tgz file
  5. use WinSCP to take this file out and upload that to VMware FTP Server if you have an open SR.

==> Take log bundle from vRA Cluster (all nodes)

  1. login into vRA appliance using SSH Client
  2. cd /tmp
  3. run command ‘vcac-config log-bundle’
  4. this will create a tgz file
  5. use WinSCP to take this file out and upload that to VMware FTP Server if you have an open SR.

Rename vSphere datastore which is being used as Storage Path in vRA 7.2

Just for curiosity sake, This morning I have decided to see how vRA managed machines behave when I rename corresponding datastore name.

Let’s say I have a vSphere cluster being used as Compute Resource in vRA, and this cluster is being backed by a datastore named ‘datastore1’, and there are set of VMs residing on this datastore and are being managed by vRA 7.2, obviously in this case there’s a reservation in place which has got ‘datastore1’ selected as storage path.

Now, as I have just renamed datastore1 to a new name ‘NewDatastore1’ and triggered manual Inventory Data Collection. if I check back in reservation, the storage path name has been changed as per new name of the datastore.

and the corresponding database entries got updated without throwing any errors, which proves that rename of the datastore even if it is being used as StoragePath in vRA gets handled by quite well by vRA. a Simple data collection updates relevant entries.


vRO workflow execution tokens and state from Server.log

Lately I was working with a customer who requested that they need vRealize Orchestrator workflow execution token and state to be fetched from logs, since they have vRealize LogInsight in place which they wanted to make use of it.

This is when I looked in the server.log file and found following bits. in this case I created a workflow with name TestWorkflow1 and executed it directly using vRO client. Took token ID of the execution and filtered server.log file with it.

Execution Token ID: ff8080815c713c36015c9041dcd21f24

/var/log/vco/app-server # grep -i ff8080815c713c36015c9041dcd21f24 server.log
2017-06-10 04:27:34.486+0000 [http-nio-] INFO {} [Execution] Invoking workflow handler.ff8080815c713c36015c9041dcd21f24
2017-06-10 04:27:34.510+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [WorkflowHandler] Starting workflow ‘TestWorkflow1’ (ff8080815c713c36015c9041dcd21f24)…
2017-06-10 04:27:34.520+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [SCRIPTING_LOG] __item_stack:/item1
2017-06-10 04:27:34.539+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [SCRIPTING_LOG] __item_stack:/item0
2017-06-10 04:27:34.553+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [WorkflowHandler] End of workflow ‘TestWorkflow1’ (ff8080815c713c36015c9041dcd21f24), state: completed

Execution Token ID: ff8080815c713c36015c9052e0c21f2e

/var/log/vco/app-server # grep -i ff8080815c713c36015c9052e0c21f2e server.log
2017-06-10 04:46:09.606+0000 [http-nio-] INFO {} [Execution] Invoking workflow handler.ff8080815c713c36015c9052e0c21f2e
2017-06-10 04:46:09.628+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [WorkflowHandler] Starting workflow ‘TestWorkflow1’ (ff8080815c713c36015c9052e0c21f2e)…
2017-06-10 04:46:09.644+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [SCRIPTING_LOG] __item_stack:/item1
2017-06-10 04:46:09.677+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [SCRIPTING_LOG] __item_stack:/item0
2017-06-10 04:46:09.687+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [WorkflowHandler] End of workflow ‘TestWorkflow1’ (ff8080815c713c36015c9052e0c21f2e), state: completed

last line of both the log snippets are having state: completed which comes out as Failed in those execution which fails.

I must also mention the other easiest way I found to achieve this is documented step by step on following URL, where you literally go ahead and create a new workflow to get Execution Details of another workflows by just simply inputting name of the workflow.

I am working on my next post where I will explore vRLI and try to fetch history of execution tokens along with their states for a particular workflow.


Replace vRealize Business Standard 7 Self Signed Certificates

Replace vRealize Business Standard 7.0 Self-Signed SSL Certificates. Business Management tab in vRA reports SSLHandshakeException.


  • Business Management Tab in vRA reporting error Untrusted Certificate Chain
  • vRA VAMI starts to report following services status as Blank
    • vcbm-service (com.vmware.vcbm.vcbm)
    • pricing-api (com.vmware.vcbm.pricing)

Did this in a vRA 7.1 – vRB 7.0 based simple deployment

Where I ran into a situation where vRA self signed certificate was already expired and both the above listed symptoms where being faced.

To get full resolution to this, I went ahead and began process of replacing certificates with steps listed below:

  • take snapshot of vRA appliance and vRB appliance
  • go to vRB appliance VAMI page
  • under vRealize Automation tab in vRB VAMI, supply administrator password and click on Unregister button, let it come throw with a message that unregister is done successfully.
  • Now, on Administration tab -> click on SSL
  • Choose the Mode as Generate Self-signed certificate
    1. Supply common name: FQDN of vRB appliance
    2. Organization Name
    3. Organization Unit
    4. Country Code
    5. certreplacement
  • Click on Replace Certificate and wait for success message
  • Go back into vRealize Automation tab, register vRA FQDN with name of the default tenant, administrator user and password associated with this account.
  • This will indeed fix the service registration issues in vRA VAMI page, which was unregistered earlier.
  • But Business Management tab will still continue to show SSLHandShakeError, to resolve this make sure you go in vRA SSH console and run command ‘service vcac-server restart’, give it about 10 to 15 minutes and check state of the services in vRA which should come out as Registered and Business Management tab is also looking OK.

Additional Reference:


Change Lease to Never Expire in vRA 7.1

working with vRA 7.1

  1. created a blueprint with Minimum Lease as 10 Days and left Maximum lease as empty.
  2. created a deployment out of it with 10 days lease and now I have a need to change the lease of this same deployment to Never Expire.
  3. Action entitlement is there to do this and I went ahead with Change Lease, a dialog box popped up with Lable “Change the lease for a machine. Leave empty for indefinite” with Expiration date and time boxes.
  4. since the requirement was to set lease to Never expire, left Expiration date empty and tried to submit the request.
  5. Ended up seeing following error “An unexpected error occurred while validating your request.”
  6. Since deployment was created from Blueprint which was configured with no Max Lease duration, in vRA 7.1 if you try to submit above dialog box with some date which is 20 years from now also, that works perfectly fine.

Only point of concern is the lable on Change Lease dialog box which says Leave empty for indefinite, no matter what you do that doesn’t work in vRA 7.1

I have explored same part in vRA 7.2 where Change Lease dialog box lable is changed and if I try step 1 through 5 as above example, it works perfectly well. So I would say this issue is resolved in vRA 7.2


Unallocate IP from IP Range in vRA 6.2.x

This came as a requirement from one of the clients. They are using vRA 6.2.2 where they have migrated few Deployments (vSphere VMs) from one Reservation to Other and ended up in a situation that old Reservation based Network profile which was having IP Range configured from where these set of VMs were holding IPs allocation remained as it was and at the same time same set of VMs which are now moved to new reservation from where they picked up new IPs from the local IP range of that reservation.

Simple ask from customer was to make sure that old IP Range based IPs which were allocated in the name of moved VMs should be released so those can be used by new deployments

Little research on IaaS sql DB and found a way to resolve this.

Used following select statement to see if particular VM Name is having multiple network IP allocated againsts it.

select * from staticipv4address where virtualmachineid in (select virtualmachineid from virtualmachine where virtualmachinename in (”));

if above statement returns 2 or more entries for one VM, execute following update statement to set IP address to be unallocated which is not being used.

update staticipv4address set virtualmachineid = NULL, StaticIPv4AddressState = 1 where virtualmachineid in (select virtualmachineid from virtualmachine where virtualmachinename in (”)) and IPv4Address=”;

I did make sure to keep Backup of IaaS DB in case if that’s needed.
There will not be a need of vRA service outage.


VMware vForum Online 2016

Attend vForum Online 19th April, 2016 where featured speakers are VMware CEO Pat Gelsinger and VMware CTO Chris Wolf, talking SDDC, End User Computing and the Hybrid cloud.

There will be 8 Hands-On-Labs to test-drive a variety of VMware Solutions. Certain unique product demos by VMware system engineers and unlimited access to eBooks, tech-tips and other product related resources.

For more information and registration, Click Here


Delete Firewall rules in VCSA 6.0

Let’s say we have vCSA 6.0 appliance in place, where some firewall rules are created.

Managing this is easy using vSphere Web Client -> Administration -> System Configuration -> Nodes -> <your vCSA node> -> Manage -> Settings -> Firewall
(List of rules would appear like following screenshot)

Screen Shot 2016-02-10 at 10.51.08 PM
Click on ‘Edit’ button
Screen Shot 2016-02-10 at 11.10.59 PM
here there are buttons to Add new rule, Edit an existing rule, Re-order rules using up/down arrow, and the last button is to Delete.

Now if you want to see list of firewall rules using vCSA console, command is (Following output)

Screen Shot 2016-02-10 at 10.56.32 PM

and from this list if you want to delete one of the rule, remember rules are being displayed in the order they are there in GUI output, where very first rule record is index number 0, second record is index number 1 and so on

And to delete one of the record from the list, use following command.
api com.vmware.appliance.version1.networking.firewall.addr.inbound.delete –position 0
This will delete very first record from the list. and the second record which is there in the above screenshot will become the index position 0. see following screenshot

Screen Shot 2016-02-10 at 11.19.40 PM

Now, if you want to delete All the rules in firewall in just one go.

api com.vmware.appliance.version1.networking.firewall.addr.inbound.delete –all true
this will make sure that all the rules in vCSA 6.0 firewall gets deleted.

(Note: in my commands, I have actually used double dash without any space in between which is visible in screenshots but the Text I have typed in blogpost is not making it very clear)


vSphere 5.5 or higher and Reliable Memory Technology

like we know ECC (Error-Correcting Code) is a great feature we have to tackle soft errors happening in RAM, without causing OS to fail, but ECC protects us from single soft error in a memory block at a time, so if there are multiple soft errors in a single memory block or hard errors on one or multiple cells in main memory, this will surely cause OS Kernel to panic. This will then result into longer downtime in terms of going through variety of steps to identify faulty module to replace it. Or resetting OS (in case of multiple soft errors) with all it’s application services. If this was a virtualised environment with VMware ESXi  or any such other hypervisors then we know there will be multiple VMs running and the all go down along with the hypervisor.

This is where Reliable Memory Technology (RMT) comes into picture which is basically a hardware feature which works along with supported OS (like ESXi 5.5 or higher). This will make sure that if during ongoing operations, any multi-bit soft error in a DIMM or hard error occurs, it will be detected by RMT and will take corrective actions in such a way so it won’t trigger OS Kernel to panic.

For Example, let’s say there was a hard error in one of the DIMM, system will detect it and mark faulty cell and some cells around it as non-usable. so that current OS operations will continue but after next reboot, OS will not see those faulty cells because hardware is not even presenting those anymore.

RMT proves to be really great when it comes to minimise downtimes due to Memory Fault related kernel panics, and also avoiding to replace whole memory module due to hard errors in DIMM.

if you have vSphere 5.5 or higher with Enterprise or Enterprise plus edition license, and if you hardware has it, then in your ESXi host you will be able to see Reliable memory using following command.


List of References:
Tech. White Paper from Dell about RMT
Third-party blog post1
Third-party blog post2