Rename vSphere datastore which is being used as Storage Path in vRA 7.2

Just for curiosity sake, This morning I have decided to see how vRA managed machines behave when I rename corresponding datastore name.

Let’s say I have a vSphere cluster being used as Compute Resource in vRA, and this cluster is being backed by a datastore named ‘datastore1’, and there are set of VMs residing on this datastore and are being managed by vRA 7.2, obviously in this case there’s a reservation in place which has got ‘datastore1’ selected as storage path.

Now, as I have just renamed datastore1 to a new name ‘NewDatastore1’ and triggered manual Inventory Data Collection. if I check back in reservation, the storage path name has been changed as per new name of the datastore.

and the corresponding database entries got updated without throwing any errors, which proves that rename of the datastore even if it is being used as StoragePath in vRA gets handled by quite well by vRA. a Simple data collection updates relevant entries.

Facebooktwittergoogle_pluslinkedinmail

vRO workflow execution tokens and state from Server.log

Lately I was working with a customer who requested that they need vRealize Orchestrator workflow execution token and state to be fetched from logs, since they have vRealize LogInsight in place which they wanted to make use of it.

This is when I looked in the server.log file and found following bits. in this case I created a workflow with name TestWorkflow1 and executed it directly using vRO client. Took token ID of the execution and filtered server.log file with it.

Execution Token ID: ff8080815c713c36015c9041dcd21f24

/var/log/vco/app-server # grep -i ff8080815c713c36015c9041dcd21f24 server.log
2017-06-10 04:27:34.486+0000 [http-nio-127.0.0.1-8280-exec-4] INFO {} [Execution] Invoking workflow handler.ff8080815c713c36015c9041dcd21f24
2017-06-10 04:27:34.510+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [WorkflowHandler] Starting workflow ‘TestWorkflow1’ (ff8080815c713c36015c9041dcd21f24)…
2017-06-10 04:27:34.520+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [SCRIPTING_LOG] __item_stack:/item1
2017-06-10 04:27:34.539+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [SCRIPTING_LOG] __item_stack:/item0
2017-06-10 04:27:34.553+0000 [WorkflowExecutorPool-Thread-2] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9041dcd21f24} [WorkflowHandler] End of workflow ‘TestWorkflow1’ (ff8080815c713c36015c9041dcd21f24), state: completed

Execution Token ID: ff8080815c713c36015c9052e0c21f2e

/var/log/vco/app-server # grep -i ff8080815c713c36015c9052e0c21f2e server.log
2017-06-10 04:46:09.606+0000 [http-nio-127.0.0.1-8280-exec-9] INFO {} [Execution] Invoking workflow handler.ff8080815c713c36015c9052e0c21f2e
2017-06-10 04:46:09.628+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [WorkflowHandler] Starting workflow ‘TestWorkflow1’ (ff8080815c713c36015c9052e0c21f2e)…
2017-06-10 04:46:09.644+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [SCRIPTING_LOG] __item_stack:/item1
2017-06-10 04:46:09.677+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [SCRIPTING_LOG] __item_stack:/item0
2017-06-10 04:46:09.687+0000 [WorkflowExecutorPool-Thread-3] INFO {administrator@vsphere.local:TestWorkflow1:527daf33-2aac-423c-83b3-3c407c138fdc:token=ff8080815c713c36015c9052e0c21f2e} [WorkflowHandler] End of workflow ‘TestWorkflow1’ (ff8080815c713c36015c9052e0c21f2e), state: completed

last line of both the log snippets are having state: completed which comes out as Failed in those execution which fails.

I must also mention the other easiest way I found to achieve this is documented step by step on following URL, where you literally go ahead and create a new workflow to get Execution Details of another workflows by just simply inputting name of the workflow.
https://www.vcoteam.info/articles/learn-vco/176-how-to-retrieve-workflow-execution-details.html

I am working on my next post where I will explore vRLI and try to fetch history of execution tokens along with their states for a particular workflow.

Facebooktwittergoogle_pluslinkedinmail

Replace vRealize Business Standard 7 Self Signed Certificates

Replace vRealize Business Standard 7.0 Self-Signed SSL Certificates. Business Management tab in vRA reports SSLHandshakeException.

BusinessManagement

  • Business Management Tab in vRA reporting error Javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Untrusted Certificate Chain
  • vRA VAMI starts to report following services status as Blank
    • vcbm-service (com.vmware.vcbm.vcbm)
    • pricing-api (com.vmware.vcbm.pricing)

Did this in a vRA 7.1 – vRB 7.0 based simple deployment

Where I ran into a situation where vRA self signed certificate was already expired and both the above listed symptoms where being faced.

To get full resolution to this, I went ahead and began process of replacing certificates with steps listed below:

  • take snapshot of vRA appliance and vRB appliance
  • go to vRB appliance VAMI page
  • under vRealize Automation tab in vRB VAMI, supply administrator password and click on Unregister button, let it come throw with a message that unregister is done successfully.
  • Now, on Administration tab -> click on SSL
  • Choose the Mode as Generate Self-signed certificate
    1. Supply common name: FQDN of vRB appliance
    2. Organization Name
    3. Organization Unit
    4. Country Code
    5. certreplacement
  • Click on Replace Certificate and wait for success message
  • Go back into vRealize Automation tab, register vRA FQDN with name of the default tenant, administrator user and password associated with this account.
  • This will indeed fix the service registration issues in vRA VAMI page, which was unregistered earlier.
  • But Business Management tab will still continue to show SSLHandShakeError, to resolve this make sure you go in vRA SSH console and run command ‘service vcac-server restart’, give it about 10 to 15 minutes and check state of the services in vRA which should come out as Registered and Business Management tab is also looking OK.

Additional Reference: http://pubs.vmware.com/vrealizebusinessstd-7.0/topic/com.vmware.ICbase/PDF/vRealizeStd-Install-7.0.pdf

Facebooktwittergoogle_pluslinkedinmail

How to reset administrator@vsphere.local account password in vRA 7.x?

Have you forgotten your administrator@vsphere.local account password which you created at the time of initial configuration of your vRA 7.x environment, here’s how you reset it.

1) Login into Master vRA Appliance VAMI using appliance root user
2) go to vRA Settings -> SSO
3) in password fields for administrator user account, enter new password that you want to set (Please remember you don’t need old password at all), also confirm the new password in confirm password field
4) Save settings
5) wait for 10 to 15 minutes (Once it comes with Settings saved successfully message), go into Services tab and wait until you see all the services reporting Status as Registered, periodically refresh the page using button provided

if this is a single node vRA appliance based environment, please skip to step 10) directly.

6) now proceed to second vRA node, login in VAMI
7) Navigate to vRA Settings -> Cluster
8) Provide the FQDN and credentials of the master node and hit join the cluster
9) Again give it about 10 to 15 minutes and check status of all the services to become Registered

– Please repeat step 6 to 9 for all the additional vRA appliances in vRA nodes cluster.

10) Clear cache in the browser and try to login with administrator@vsphere.local to default tenant

your administrator account password has been reset successfully.

Additional bits to consider:
If administrator@vsphere.local account was used at places like configuring vCAC plugin in vRO, make sure you update password at all those places.

Facebooktwittergoogle_pluslinkedinmail

vRA 7.x Infrastructure Tab reporting Server Error 401 – Unauthorized: Access is denied due to invalid credentials

the reason I am creating this blog post is purely because the message was mis-leading but the actual reason was different fot this error in Infrastructure Tab.

One of the customer retported that they cannot see IaaS-Service regsitred in vRA VAMI -> Services, it’s being reported as blank.

at the same time in vRA portal, under Infrastructure tab Server Error 401 – Unauthorized: Access is denied due to invalid credentials was being displayed.

InfraTab

Further checks in IaaS web server host where ModelManagerData repository was residing, IIS Application Pool called RepositoryAppPool was stopped.

AppPool

attempt to start it was failing with some weird error message.

Service account password was not modified and should be working without much trouble.

That’s when something interesting got pointed in Repository.log

[UTC:2017-04-07 02:27:49 Local:2017-04-07 10:27] [Error]: [sub-thread-Id=”1″  context=””  token=””] Failed to start repository service. Reason: System.Security.Cryptography.CryptographicException: There is not enough space on the disk.
at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)
at System.Security.Cryptography.X509Certificates.X509Utils._LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle& pCertCtx)
at System.Security.Cryptography.X509Certificates.X509Certificate.LoadCertificateFromBlob(Byte[] rawData, Object password, X509KeyStorageFlags keyStorageFlags) at VMware.Cafe.RegistrationData.ConvertCertStringToCertificate(String certRawData) at DynamicOps.Repository.Runtime.CafeClientAbstractFactory.InitializeFromDb(String coreModelConnectionString) at DynamicOps.Repository.Runtime.Common.RepositoryRuntime.Initialize().

and that certainly poited to right direction now, a quick look on system drive of the IaaS Web server host revelaed that there’s no space left on c: drive, and further investigation into this, and realized that IIS logs which were piling up since last 2 years were occupying 31 Gb all together, keeping just last 30 days worth log files and moved rest of the files on a different shared location was the first activity carried out.

Further steps taken are as below:

  1. Restart of the IaaS web server node after making enough space on c: drive
  2. restart of vcac-server in vRA appliacnes

These two actions have made things come back to normal.

Facebooktwittergoogle_pluslinkedinmail

Add PowerShell Host fails in vRO 7.x

While making an attempt to add PowerShell host in vRO 7.x fails with an error

Initial Error: ‘Add a PowerShell host/item8’, state: ‘failed’, business state: ‘null’, exception: ‘Clients credentials have been revoked (18) (Dynamic Script Module name : addPowerShellHost#12)’
workflow: ‘Add a PowerShell host’ (EF8180808080808080808080808080803D80808001270557368849c62c352aa82)
|  ‘attribute’: name=errorCode type=string value=Clients credentials have been revoked (18) (Dynamic Script Module name : addPowerShellHost#12)

investigation into this came up with some very basic issues.

vRO was configured with External MS SQL Db which was being authorized by particular AD account credentials, and that account itself was locked out. This might have happened due to multiple wrong password logon attempts and that created communication between vRO and DB server to fail.

this was tracked by going in vRO configurator page where it was throwing error:

Error! Error occured while retrieving nodes configuration. org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
Error in both the vRO node is: couldn’t connect to database server.

but when checked with DB admin, they said DB is healthy enough and vRO node was able to reach DB server via ping as well. But when tried to test connection with service account, it was found to be locked. Unlocking same resolve everything, and was able to add powershell host also successfully.

Facebooktwittergoogle_pluslinkedinmail

Network and Security Inventory data collection fails in vRA 7.1

One of the customer reporting that their vRA 7.1 has started reporting deployment failures, and they were suspecting that this is happening due to Network and Security inventory data collection failures can be seen in vRA Infrastrastructure -> Compute Resource tab under all the Compute Resources.

Customer also revealed that they recently changed vRA -> External vRO -> NSX plugin configuration user credentails with a different username than the one in use earlier. and they were under the impression that due to this probably they started noticing Inventory data collection for network and security is failing now.

looking into Infrastracture -> monitoring -> logs

Error logs can be seen are as bellow:
Workflow ‘vSphereVCNSInventory’ failed with the following exception:
vRealize Orchestrator returned an error: Not Found.


DEM Worker at the same time was reporting errors as listed bellow:

2017-04-11T02:31:47.382Z CUA44494VPA100 vcac: [component=”iaas:DynamicOps.DEM.exe” priority=”Error” thread=”2768″] [sub-thread-Id=”52″  context=””  token=””]
false
Workflow ‘vSphereVCNSInventory’ failed with the following exception:
System.Net.WebException: vRealize Orchestrator returned an error: Not Found.
at DynamicOps.VcoModel.Common.VcoClient.DecodeJsonResponse(IRestResponse response)    at DynamicOps.VcoModel.Common.VcoInventoryReader.ReadInventory(VcoInventoryItemToken inventoryToken, String queryObject)    at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)
at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)


while Server.log of Orchestrator node was reporting following:

2017-04-11 01:51:05.155-0400 [http-nio-0.0.0.0-8281-exec-2] WARN  {} [SDKFinder] Unable to execute ‘fetchRelation’ for type : EdgePage : java.lang.NumberFormatException: For input string: “389,636,1012,2014,2020”
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor409.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at ch.dunes.vso.sdk.DirectInvoker.invoke(DirectInvoker.java:57) at ch.dunes.vso.sdk.SDKPluginFactoryInvoker.fetchRelation(SDKPluginFactoryInvoker.java:81) at ch.dunes.vso.sdk.SDKFinder.fetchRelation(SDKFinder.java:1123)
at ch.dunes.vso.sdk.SDKFinder._findRelation(SDKFinder.java:1098)
at ch.dunes.vso.sdk.SDKFinder.findRelation(SDKFinder.java:1016)
at ch.dunes.vso.sdk.ModulesFactory.findRelation(ModulesFactory.java:1606) at com.vmware.o11n.sdk.EnhancedScriptingSDK.findRelation(EnhancedScriptingSDK.java

in this environment, this was going on since last 1 year, which cusotmer failed to notice, and I found that they are using vRO-NSX plugin version 1.0.4 which is affected by a known issue.

VMware KB https://kb.vmware.com/kb/2148554
where even if Network and Security inventory collection fails, we don’t have to worry about it because this is happening due to vCO-NSX plugin version 1.0.4 which is in use currently in this environment, solution to this is included in vRO-NSX plugin 1.1 as mentioned in the quoted KB

As long as your vRO end point data collection is successful, it’s going to still let you use all the NSX components in your blueprint and deployments should not fail. If you still find deployments failing, I would suggest to open a Support Request with VMware.

Facebooktwittergoogle_pluslinkedinmail

Change Lease to Never Expire in vRA 7.1

working with vRA 7.1

  1. created a blueprint with Minimum Lease as 10 Days and left Maximum lease as empty.
  2. created a deployment out of it with 10 days lease and now I have a need to change the lease of this same deployment to Never Expire.
  3. Action entitlement is there to do this and I went ahead with Change Lease, a dialog box popped up with Lable “Change the lease for a machine. Leave empty for indefinite” with Expiration date and time boxes.
  4. since the requirement was to set lease to Never expire, left Expiration date empty and tried to submit the request.
  5. Ended up seeing following error “An unexpected error occurred while validating your request.”
    Lease
  6. Since deployment was created from Blueprint which was configured with no Max Lease duration, in vRA 7.1 if you try to submit above dialog box with some date which is 20 years from now also, that works perfectly fine.

Only point of concern is the lable on Change Lease dialog box which says Leave empty for indefinite, no matter what you do that doesn’t work in vRA 7.1

I have explored same part in vRA 7.2 where Change Lease dialog box lable is changed and if I try step 1 through 5 as above example, it works perfectly well. So I would say this issue is resolved in vRA 7.2

Facebooktwittergoogle_pluslinkedinmail

Unallocate IP from IP Range in vRA 6.2.x

This came as a requirement from one of the clients. They are using vRA 6.2.2 where they have migrated few Deployments (vSphere VMs) from one Reservation to Other and ended up in a situation that old Reservation based Network profile which was having IP Range configured from where these set of VMs were holding IPs allocation remained as it was and at the same time same set of VMs which are now moved to new reservation from where they picked up new IPs from the local IP range of that reservation.

Simple ask from customer was to make sure that old IP Range based IPs which were allocated in the name of moved VMs should be released so those can be used by new deployments

Little research on IaaS sql DB and found a way to resolve this.

Used following select statement to see if particular VM Name is having multiple network IP allocated againsts it.

select * from staticipv4address where virtualmachineid in (select virtualmachineid from virtualmachine where virtualmachinename in (”));

if above statement returns 2 or more entries for one VM, execute following update statement to set IP address to be unallocated which is not being used.

update staticipv4address set virtualmachineid = NULL, StaticIPv4AddressState = 1 where virtualmachineid in (select virtualmachineid from virtualmachine where virtualmachinename in (”)) and IPv4Address=”;

I did make sure to keep Backup of IaaS DB in case if that’s needed.
There will not be a need of vRA service outage.

Facebooktwittergoogle_pluslinkedinmail

VMware vForum Online 2016

Attend vForum Online 19th April, 2016 where featured speakers are VMware CEO Pat Gelsinger and VMware CTO Chris Wolf, talking SDDC, End User Computing and the Hybrid cloud.

There will be 8 Hands-On-Labs to test-drive a variety of VMware Solutions. Certain unique product demos by VMware system engineers and unlimited access to eBooks, tech-tips and other product related resources.

For more information and registration, Click Here

Facebooktwittergoogle_pluslinkedinmail