Category Archives: vRealize Orchestrator

vRA 7.x Infrastructure Tab reporting Server Error 401 – Unauthorized: Access is denied due to invalid credentials

the reason I am creating this blog post is purely because the message was mis-leading but the actual reason was different fot this error in Infrastructure Tab.

One of the customer retported that they cannot see IaaS-Service regsitred in vRA VAMI -> Services, it’s being reported as blank.

at the same time in vRA portal, under Infrastructure tab Server Error 401 – Unauthorized: Access is denied due to invalid credentials was being displayed.


Further checks in IaaS web server host where ModelManagerData repository was residing, IIS Application Pool called RepositoryAppPool was stopped.


attempt to start it was failing with some weird error message.

Service account password was not modified and should be working without much trouble.

That’s when something interesting got pointed in Repository.log

[UTC:2017-04-07 02:27:49 Local:2017-04-07 10:27] [Error]: [sub-thread-Id=”1″  context=””  token=””] Failed to start repository service. Reason: System.Security.Cryptography.CryptographicException: There is not enough space on the disk.
at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)
at System.Security.Cryptography.X509Certificates.X509Utils._LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle& pCertCtx)
at System.Security.Cryptography.X509Certificates.X509Certificate.LoadCertificateFromBlob(Byte[] rawData, Object password, X509KeyStorageFlags keyStorageFlags) at VMware.Cafe.RegistrationData.ConvertCertStringToCertificate(String certRawData) at DynamicOps.Repository.Runtime.CafeClientAbstractFactory.InitializeFromDb(String coreModelConnectionString) at DynamicOps.Repository.Runtime.Common.RepositoryRuntime.Initialize().

and that certainly poited to right direction now, a quick look on system drive of the IaaS Web server host revelaed that there’s no space left on c: drive, and further investigation into this, and realized that IIS logs which were piling up since last 2 years were occupying 31 Gb all together, keeping just last 30 days worth log files and moved rest of the files on a different shared location was the first activity carried out.

Further steps taken are as below:

  1. Restart of the IaaS web server node after making enough space on c: drive
  2. restart of vcac-server in vRA appliacnes

These two actions have made things come back to normal.


Add PowerShell Host fails in vRO 7.x

While making an attempt to add PowerShell host in vRO 7.x fails with an error

Initial Error: ‘Add a PowerShell host/item8’, state: ‘failed’, business state: ‘null’, exception: ‘Clients credentials have been revoked (18) (Dynamic Script Module name : addPowerShellHost#12)’
workflow: ‘Add a PowerShell host’ (EF8180808080808080808080808080803D80808001270557368849c62c352aa82)
|  ‘attribute’: name=errorCode type=string value=Clients credentials have been revoked (18) (Dynamic Script Module name : addPowerShellHost#12)

investigation into this came up with some very basic issues.

vRO was configured with External MS SQL Db which was being authorized by particular AD account credentials, and that account itself was locked out. This might have happened due to multiple wrong password logon attempts and that created communication between vRO and DB server to fail.

this was tracked by going in vRO configurator page where it was throwing error:

Error! Error occured while retrieving nodes configuration. org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
Error in both the vRO node is: couldn’t connect to database server.

but when checked with DB admin, they said DB is healthy enough and vRO node was able to reach DB server via ping as well. But when tried to test connection with service account, it was found to be locked. Unlocking same resolve everything, and was able to add powershell host also successfully.


Network and Security Inventory data collection fails in vRA 7.1

One of the customer reporting that their vRA 7.1 has started reporting deployment failures, and they were suspecting that this is happening due to Network and Security inventory data collection failures can be seen in vRA Infrastrastructure -> Compute Resource tab under all the Compute Resources.

Customer also revealed that they recently changed vRA -> External vRO -> NSX plugin configuration user credentails with a different username than the one in use earlier. and they were under the impression that due to this probably they started noticing Inventory data collection for network and security is failing now.

looking into Infrastracture -> monitoring -> logs

Error logs can be seen are as bellow:
Workflow ‘vSphereVCNSInventory’ failed with the following exception:
vRealize Orchestrator returned an error: Not Found.

DEM Worker at the same time was reporting errors as listed bellow:

2017-04-11T02:31:47.382Z CUA44494VPA100 vcac: [component=”iaas:DynamicOps.DEM.exe” priority=”Error” thread=”2768″] [sub-thread-Id=”52″  context=””  token=””]
Workflow ‘vSphereVCNSInventory’ failed with the following exception:
System.Net.WebException: vRealize Orchestrator returned an error: Not Found.
at DynamicOps.VcoModel.Common.VcoClient.DecodeJsonResponse(IRestResponse response)    at DynamicOps.VcoModel.Common.VcoInventoryReader.ReadInventory(VcoInventoryItemToken inventoryToken, String queryObject)    at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)
at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)

while Server.log of Orchestrator node was reporting following:

2017-04-11 01:51:05.155-0400 [http-nio-] WARN  {} [SDKFinder] Unable to execute ‘fetchRelation’ for type : EdgePage : java.lang.NumberFormatException: For input string: “389,636,1012,2014,2020”
at sun.reflect.GeneratedMethodAccessor409.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke( at java.lang.reflect.Method.invoke( at ch.dunes.vso.sdk.DirectInvoker.invoke( at ch.dunes.vso.sdk.SDKPluginFactoryInvoker.fetchRelation( at ch.dunes.vso.sdk.SDKFinder.fetchRelation(
at ch.dunes.vso.sdk.SDKFinder._findRelation(
at ch.dunes.vso.sdk.SDKFinder.findRelation(
at ch.dunes.vso.sdk.ModulesFactory.findRelation( at com.vmware.o11n.sdk.EnhancedScriptingSDK.findRelation(

in this environment, this was going on since last 1 year, which cusotmer failed to notice, and I found that they are using vRO-NSX plugin version 1.0.4 which is affected by a known issue.

VMware KB
where even if Network and Security inventory collection fails, we don’t have to worry about it because this is happening due to vCO-NSX plugin version 1.0.4 which is in use currently in this environment, solution to this is included in vRO-NSX plugin 1.1 as mentioned in the quoted KB

As long as your vRO end point data collection is successful, it’s going to still let you use all the NSX components in your blueprint and deployments should not fail. If you still find deployments failing, I would suggest to open a Support Request with VMware.