Author Archives: Narendra

Error – Blueprint component Machine – Correct the highlighted errors

Soon after changing vRA IaaS service account password and reflecting it at all the required places using VMware KB https://kb.vmware.com/kb/2099949 and we realize that all is well.

But we end up with some strange issuses like the one I am going to be explaining here. This was in vRA 7.3, all was working fine. But as per Enterprise Password policy (even for service account 🙁 ) they ended up changing service account password. as I said in first paragraph, reflecting this is an easy task as long as we follow steps in KB 2099949 but what about Custom property values stored in Property groups which are encrypted.

what we started facing was, at the time we try to submit a request form for majority of blueprints, we started facing this error.

Blueprint Component Machine – Correct the highlighted error

correcthighlightederror1correcthighlightederror2

if we then go and look into component machine form, we don’t see any field highlighted, but error still persist.

this is when, go into Blueprint design, explore attached Property Groups, among those property groups, there would be one/more of the custom properties with encrypted value (like password field), just go ahead and retype values for those encrypted fields. Save the blueprint again. And raise a request form again, make sure you fill in all the required fields and submit the form. Error’s gone. 🙂

Facebooktwittergoogle_pluslinkedinmail

How to increase size of vRO 7.x appliance root partition?

Basically vRO appliance comes with 4.5 GB of root partition and 2.5 GB of Swap parition. This makes your vRO appliance with /dev/sda of 7 GB in size, and I have couple of my customers  who were requesting how to increase the size of this root partition.

Here’s what I have don to achieve this.

  • Take a full backup of both the vRO nodes and vRO DB.
  1. vRO VM -> edit settings -> Hard Disk1 -> Add more space as needed (I have increased it to 5 GB)

– make sure you take snapshot of the vRO VM right after executing step 1.

  1. vRO VM -> edit settings -> vm options -> boot options -> Force BIOS setup check. This bring in the bios screen on next boot.
  2. Mount the gparted live boot iso 64 bit version under vRO VM -> edit settings -> hardware -> CD/DVD drive -> datastore iso. Ensure “Connect on power on selected” (I have used gparted-live-0.29.0-1-i686.iso)
  3. Reboot the vRO appliance and change the boot order (step 2 would help getting directly into BIOS setup), make CDROM as first boot device. Save the settings and let the appliance boot
  4. Once the gparted grub menu is available then select “other modes of gparted live” -> “Safe Graphics” and follow defaults
  5. Once gparted loaded select devices /dev/sda
  6. It will display partitions on this device, select swap partition (/dev/sda2) and click on Button ‘Resize/Move’, drag swap partition as a whole to end of /dev/sda and place it there. Save the settings (Move)
  7. now select root partition (/dev/sda1) and click on ‘Resize/Move’, pickup right edge of /dev/sda1 and drag towards swap partition, occupying all the free space. Save the settings. (Resize)
  8. restart the VM. set CDDVD drive of the VM to Client Device in Edit settings.
  9. let the VM start up normally and check the filesystem.
  10. Please repeat all these steps on vRO appliance Node2 as well.

 

========= In my case, I have achieved following ==========

#df -h

Filesystem      Size  Used Avail Use% Mounted on

/dev/sda1       9.4G  2.3G  6.7G  25% /

udev            4.0G  104K  4.0G   1% /dev

tmpfs           4.0G  8.0K  4.0G   1% /dev/shm

/dev/sdc1        20G  173M   19G   1% /heap

/dev/sdb1       7.9G  202M  7.3G   3% /storage/log

/dev/sdb2       2.0G  108M  1.8G   6% /storage/db

====================================================

Thought this might help others who are trying to ahieve same on vRO appliance.

Facebooktwittergoogle_pluslinkedinmail

How to change vRO 7.x Active-Active cluster to Active-StandBy?

A relatively simple looking process, but at times, it might make you wonder in which order I should be doing this. Lately while talking with a customer, I realized that in vRO documentation, hints of this process are given but not really in the form of step by step instructions. I then went ahead and tested following steps. I am sure these would be useful to people who might be looking at changing their vRO active-active cluster to active-standby.

1) Take a full backup of vRO Db if you are using External DB, if you are using embedded vPostgresDB as vRO DB then skip to step2
2) Take snapshot of all the vRO nodes.
3) Go into vRO control center of vRO node1, set the number of active nodes to 1. (Under vRo Control center-> Orchestrator Cluster Management -> Number of active nodes. )
4) Repeat step 3) for all the remaining vRO nodes in this cluster.
5) Restart vRO node1
6) Repeat step 5) for all the remaining vRO nodes in this cluster (one by one)

At the end of this, you will see in ‘vRO Control center -> Cluster Management’ that you have now only 1 node as Active and rest of those are stand by. Which node becomes active depends upon the order of restart of the nodes. (This can be seen from control center of any nodes in cluster)

Please note: Do not forget to remove snapshots taken in step 2) when you are done with this process.

Facebooktwittergoogle_pluslinkedinmail

How to collect log bundle of vRA using command line

as my previous post about collecting vRO log bundle using command line

This post addresses step by step instructions for collecting log bundle from vRA infra.

==> Take log bundle from vRA Appliance only

  1. login into vRA appliance using SSH Client
  2. cd /tmp
  3. run command ‘vcac-support’
  4. this will create a tgz file
  5. use WinSCP to take this file out and upload that to VMware FTP Server if you have an open SR.

==> Take log bundle from vRA Cluster (all nodes)

  1. login into vRA appliance using SSH Client
  2. cd /tmp
  3. run command ‘vcac-config log-bundle’
  4. this will create a tgz file
  5. use WinSCP to take this file out and upload that to VMware FTP Server if you have an open SR.
Facebooktwittergoogle_pluslinkedinmail

Collection log bundle from vRO appliance using command line

Lately I have come across cases where vRO appliance 7.x doesn’t allow us to take log bundle.

Making an attempt to collect it using vRO Control Center fails with Timed out error. Same thing happens when we try using VAMI page.

in order to get log bundle successfully, use following CLI method.

  1. login into vRO appliance using SSH Client
  2. cd /tmp
  3. run command ‘vamisupport’
  4. this will create a tgz file
  5. use WinSCP to take this file out and upload that to VMware FTP Server if you have an open SR.
Facebooktwittergoogle_pluslinkedinmail

Rename vSphere datastore which is being used as Storage Path in vRA 7.2

Just for curiosity sake, This morning I have decided to see how vRA managed machines behave when I rename corresponding datastore name.

Let’s say I have a vSphere cluster being used as Compute Resource in vRA, and this cluster is being backed by a datastore named ‘datastore1’, and there are set of VMs residing on this datastore and are being managed by vRA 7.2, obviously in this case there’s a reservation in place which has got ‘datastore1’ selected as storage path.

Now, as I have just renamed datastore1 to a new name ‘NewDatastore1’ and triggered manual Inventory Data Collection. if I check back in reservation, the storage path name has been changed as per new name of the datastore.

and the corresponding database entries got updated without throwing any errors, which proves that rename of the datastore even if it is being used as StoragePath in vRA gets handled by quite well by vRA. a Simple data collection updates relevant entries.

Facebooktwittergoogle_pluslinkedinmail

vRO workflow execution tokens and state from Server.log

Lately I was working with a customer who requested that they need vRealize Orchestrator workflow execution token and state to be fetched from logs, since they have vRealize LogInsight in place which they wanted to make use of it.

This is when I looked in the server.log file and found following bits. in this case I created a workflow with name TestWorkflow1 and executed it directly using vRO client. Took token ID of the execution and filtered server.log file with it.

Execution Token ID: ff8080815c713c36015c9041dcd21f24

/var/log/vco/app-server # grep -i ff8080815c713c36015c9041dcd21f24 server.log
2017-06-10 04:27:34.486+0000 [http-nio-127.0.0.1-8280-exec-4] INFO {} [Execution] Invoking workflow handler.ff8080815c713c36015c9041dcd21f24
2017-06-10 04:27:34.510+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [WorkflowHandler] Starting workflow ‘TestWorkflow1’ (ff8080815c713c36015c9041dcd21f24)…
2017-06-10 04:27:34.520+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [SCRIPTING_LOG] __item_stack:/item1
2017-06-10 04:27:34.539+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [SCRIPTING_LOG] __item_stack:/item0
2017-06-10 04:27:34.553+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [WorkflowHandler] End of workflow ‘TestWorkflow1’ (ff8080815c713c36015c9041dcd21f24), state: completed

Execution Token ID: ff8080815c713c36015c9052e0c21f2e

/var/log/vco/app-server # grep -i ff8080815c713c36015c9052e0c21f2e server.log
2017-06-10 04:46:09.606+0000 [http-nio-127.0.0.1-8280-exec-9] INFO {} [Execution] Invoking workflow handler.ff8080815c713c36015c9052e0c21f2e
2017-06-10 04:46:09.628+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [WorkflowHandler] Starting workflow ‘TestWorkflow1’ (ff8080815c713c36015c9052e0c21f2e)…
2017-06-10 04:46:09.644+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [SCRIPTING_LOG] __item_stack:/item1
2017-06-10 04:46:09.677+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [SCRIPTING_LOG] __item_stack:/item0
2017-06-10 04:46:09.687+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [WorkflowHandler] End of workflow ‘TestWorkflow1’ (ff8080815c713c36015c9052e0c21f2e), state: completed

last line of both the log snippets are having state: completed which comes out as Failed in those execution which fails.

I must also mention the other easiest way I found to achieve this is documented step by step on following URL, where you literally go ahead and create a new workflow to get Execution Details of another workflows by just simply inputting name of the workflow.
https://www.vcoteam.info/articles/learn-vco/176-how-to-retrieve-workflow-execution-details.html

I am working on my next post where I will explore vRLI and try to fetch history of execution tokens along with their states for a particular workflow.

Facebooktwittergoogle_pluslinkedinmail

Replace vRealize Business Standard 7 Self Signed Certificates

Replace vRealize Business Standard 7.0 Self-Signed SSL Certificates. Business Management tab in vRA reports SSLHandshakeException.

BusinessManagement

  • Business Management Tab in vRA reporting error Javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Untrusted Certificate Chain
  • vRA VAMI starts to report following services status as Blank
    • vcbm-service (com.vmware.vcbm.vcbm)
    • pricing-api (com.vmware.vcbm.pricing)

Did this in a vRA 7.1 – vRB 7.0 based simple deployment

Where I ran into a situation where vRA self signed certificate was already expired and both the above listed symptoms where being faced.

To get full resolution to this, I went ahead and began process of replacing certificates with steps listed below:

  • take snapshot of vRA appliance and vRB appliance
  • go to vRB appliance VAMI page
  • under vRealize Automation tab in vRB VAMI, supply administrator password and click on Unregister button, let it come throw with a message that unregister is done successfully.
  • Now, on Administration tab -> click on SSL
  • Choose the Mode as Generate Self-signed certificate
    1. Supply common name: FQDN of vRB appliance
    2. Organization Name
    3. Organization Unit
    4. Country Code
    5. certreplacement
  • Click on Replace Certificate and wait for success message
  • Go back into vRealize Automation tab, register vRA FQDN with name of the default tenant, administrator user and password associated with this account.
  • This will indeed fix the service registration issues in vRA VAMI page, which was unregistered earlier.
  • But Business Management tab will still continue to show SSLHandShakeError, to resolve this make sure you go in vRA SSH console and run command ‘service vcac-server restart’, give it about 10 to 15 minutes and check state of the services in vRA which should come out as Registered and Business Management tab is also looking OK.

Additional Reference: http://pubs.vmware.com/vrealizebusinessstd-7.0/topic/com.vmware.ICbase/PDF/vRealizeStd-Install-7.0.pdf

Facebooktwittergoogle_pluslinkedinmail

How to reset administrator@vsphere.local account password in vRA 7.x?

Have you forgotten your administrator@vsphere.local account password which you created at the time of initial configuration of your vRA 7.x environment, here’s how you reset it.

1) Login into Master vRA Appliance VAMI using appliance root user
2) go to vRA Settings -> SSO
3) in password fields for administrator user account, enter new password that you want to set (Please remember you don’t need old password at all), also confirm the new password in confirm password field
4) Save settings
5) wait for 10 to 15 minutes (Once it comes with Settings saved successfully message), go into Services tab and wait until you see all the services reporting Status as Registered, periodically refresh the page using button provided

if this is a single node vRA appliance based environment, please skip to step 10) directly.

6) now proceed to second vRA node, login in VAMI
7) Navigate to vRA Settings -> Cluster
8) Provide the FQDN and credentials of the master node and hit join the cluster
9) Again give it about 10 to 15 minutes and check status of all the services to become Registered

– Please repeat step 6 to 9 for all the additional vRA appliances in vRA nodes cluster.

10) Clear cache in the browser and try to login with administrator@vsphere.local to default tenant

your administrator account password has been reset successfully.

Additional bits to consider:
If administrator@vsphere.local account was used at places like configuring vCAC plugin in vRO, make sure you update password at all those places.

Facebooktwittergoogle_pluslinkedinmail

vRA 7.x Infrastructure Tab reporting Server Error 401 – Unauthorized: Access is denied due to invalid credentials

the reason I am creating this blog post is purely because the message was mis-leading but the actual reason was different fot this error in Infrastructure Tab.

One of the customer retported that they cannot see IaaS-Service regsitred in vRA VAMI -> Services, it’s being reported as blank.

at the same time in vRA portal, under Infrastructure tab Server Error 401 – Unauthorized: Access is denied due to invalid credentials was being displayed.

InfraTab

Further checks in IaaS web server host where ModelManagerData repository was residing, IIS Application Pool called RepositoryAppPool was stopped.

AppPool

attempt to start it was failing with some weird error message.

Service account password was not modified and should be working without much trouble.

That’s when something interesting got pointed in Repository.log

[UTC:2017-04-07 02:27:49 Local:2017-04-07 10:27] [Error]: [sub-thread-Id=”1″  context=””  token=””] Failed to start repository service. Reason: System.Security.Cryptography.CryptographicException: There is not enough space on the disk.
at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)
at System.Security.Cryptography.X509Certificates.X509Utils._LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle& pCertCtx)
at System.Security.Cryptography.X509Certificates.X509Certificate.LoadCertificateFromBlob(Byte[] rawData, Object password, X509KeyStorageFlags keyStorageFlags) at VMware.Cafe.RegistrationData.ConvertCertStringToCertificate(String certRawData) at DynamicOps.Repository.Runtime.CafeClientAbstractFactory.InitializeFromDb(String coreModelConnectionString) at DynamicOps.Repository.Runtime.Common.RepositoryRuntime.Initialize().

and that certainly poited to right direction now, a quick look on system drive of the IaaS Web server host revelaed that there’s no space left on c: drive, and further investigation into this, and realized that IIS logs which were piling up since last 2 years were occupying 31 Gb all together, keeping just last 30 days worth log files and moved rest of the files on a different shared location was the first activity carried out.

Further steps taken are as below:

  1. Restart of the IaaS web server node after making enough space on c: drive
  2. restart of vcac-server in vRA appliacnes

These two actions have made things come back to normal.

Facebooktwittergoogle_pluslinkedinmail