Add PowerShell Host fails in vRO 7.x

While making an attempt to add PowerShell host in vRO 7.x fails with an error

Initial Error: ‘Add a PowerShell host/item8’, state: ‘failed’, business state: ‘null’, exception: ‘Clients credentials have been revoked (18) (Dynamic Script Module name : addPowerShellHost#12)’
workflow: ‘Add a PowerShell host’ (EF8180808080808080808080808080803D80808001270557368849c62c352aa82)
|  ‘attribute’: name=errorCode type=string value=Clients credentials have been revoked (18) (Dynamic Script Module name : addPowerShellHost#12)

investigation into this came up with some very basic issues.

vRO was configured with External MS SQL Db which was being authorized by particular AD account credentials, and that account itself was locked out. This might have happened due to multiple wrong password logon attempts and that created communication between vRO and DB server to fail.

this was tracked by going in vRO configurator page where it was throwing error:

Error! Error occured while retrieving nodes configuration. org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
Error in both the vRO node is: couldn’t connect to database server.

but when checked with DB admin, they said DB is healthy enough and vRO node was able to reach DB server via ping as well. But when tried to test connection with service account, it was found to be locked. Unlocking same resolve everything, and was able to add powershell host also successfully.

Facebooktwittergoogle_pluslinkedinmail

Network and Security Inventory data collection fails in vRA 7.1

One of the customer reporting that their vRA 7.1 has started reporting deployment failures, and they were suspecting that this is happening due to Network and Security inventory data collection failures can be seen in vRA Infrastrastructure -> Compute Resource tab under all the Compute Resources.

Customer also revealed that they recently changed vRA -> External vRO -> NSX plugin configuration user credentails with a different username than the one in use earlier. and they were under the impression that due to this probably they started noticing Inventory data collection for network and security is failing now.

looking into Infrastracture -> monitoring -> logs

Error logs can be seen are as bellow:
Workflow ‘vSphereVCNSInventory’ failed with the following exception:
vRealize Orchestrator returned an error: Not Found.


DEM Worker at the same time was reporting errors as listed bellow:

2017-04-11T02:31:47.382Z CUA44494VPA100 vcac: [component=”iaas:DynamicOps.DEM.exe” priority=”Error” thread=”2768″] [sub-thread-Id=”52″  context=””  token=””]
false
Workflow ‘vSphereVCNSInventory’ failed with the following exception:
System.Net.WebException: vRealize Orchestrator returned an error: Not Found.
at DynamicOps.VcoModel.Common.VcoClient.DecodeJsonResponse(IRestResponse response)    at DynamicOps.VcoModel.Common.VcoInventoryReader.ReadInventory(VcoInventoryItemToken inventoryToken, String queryObject)    at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)
at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)


while Server.log of Orchestrator node was reporting following:

2017-04-11 01:51:05.155-0400 [http-nio-0.0.0.0-8281-exec-2] WARN  {} [SDKFinder] Unable to execute ‘fetchRelation’ for type : EdgePage : java.lang.NumberFormatException: For input string: “389,636,1012,2014,2020”
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor409.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at ch.dunes.vso.sdk.DirectInvoker.invoke(DirectInvoker.java:57) at ch.dunes.vso.sdk.SDKPluginFactoryInvoker.fetchRelation(SDKPluginFactoryInvoker.java:81) at ch.dunes.vso.sdk.SDKFinder.fetchRelation(SDKFinder.java:1123)
at ch.dunes.vso.sdk.SDKFinder._findRelation(SDKFinder.java:1098)
at ch.dunes.vso.sdk.SDKFinder.findRelation(SDKFinder.java:1016)
at ch.dunes.vso.sdk.ModulesFactory.findRelation(ModulesFactory.java:1606) at com.vmware.o11n.sdk.EnhancedScriptingSDK.findRelation(EnhancedScriptingSDK.java

in this environment, this was going on since last 1 year, which cusotmer failed to notice, and I found that they are using vRO-NSX plugin version 1.0.4 which is affected by a known issue.

VMware KB https://kb.vmware.com/kb/2148554
where even if Network and Security inventory collection fails, we don’t have to worry about it because this is happening due to vCO-NSX plugin version 1.0.4 which is in use currently in this environment, solution to this is included in vRO-NSX plugin 1.1 as mentioned in the quoted KB

As long as your vRO end point data collection is successful, it’s going to still let you use all the NSX components in your blueprint and deployments should not fail. If you still find deployments failing, I would suggest to open a Support Request with VMware.

Facebooktwittergoogle_pluslinkedinmail

Change Lease to Never Expire in vRA 7.1

working with vRA 7.1

  1. created a blueprint with Minimum Lease as 10 Days and left Maximum lease as empty.
  2. created a deployment out of it with 10 days lease and now I have a need to change the lease of this same deployment to Never Expire.
  3. Action entitlement is there to do this and I went ahead with Change Lease, a dialog box popped up with Lable “Change the lease for a machine. Leave empty for indefinite” with Expiration date and time boxes.
  4. since the requirement was to set lease to Never expire, left Expiration date empty and tried to submit the request.
  5. Ended up seeing following error “An unexpected error occurred while validating your request.”
    Lease
  6. Since deployment was created from Blueprint which was configured with no Max Lease duration, in vRA 7.1 if you try to submit above dialog box with some date which is 20 years from now also, that works perfectly fine.

Only point of concern is the lable on Change Lease dialog box which says Leave empty for indefinite, no matter what you do that doesn’t work in vRA 7.1

I have explored same part in vRA 7.2 where Change Lease dialog box lable is changed and if I try step 1 through 5 as above example, it works perfectly well. So I would say this issue is resolved in vRA 7.2

Facebooktwittergoogle_pluslinkedinmail

Unallocate IP from IP Range in vRA 6.2.x

This came as a requirement from one of the clients. They are using vRA 6.2.2 where they have migrated few Deployments (vSphere VMs) from one Reservation to Other and ended up in a situation that old Reservation based Network profile which was having IP Range configured from where these set of VMs were holding IPs allocation remained as it was and at the same time same set of VMs which are now moved to new reservation from where they picked up new IPs from the local IP range of that reservation.

Simple ask from customer was to make sure that old IP Range based IPs which were allocated in the name of moved VMs should be released so those can be used by new deployments

Little research on IaaS sql DB and found a way to resolve this.

Used following select statement to see if particular VM Name is having multiple network IP allocated againsts it.

select * from staticipv4address where virtualmachineid in (select virtualmachineid from virtualmachine where virtualmachinename in (”));

if above statement returns 2 or more entries for one VM, execute following update statement to set IP address to be unallocated which is not being used.

update staticipv4address set virtualmachineid = NULL, StaticIPv4AddressState = 1 where virtualmachineid in (select virtualmachineid from virtualmachine where virtualmachinename in (”)) and IPv4Address=”;

I did make sure to keep Backup of IaaS DB in case if that’s needed.
There will not be a need of vRA service outage.

Facebooktwittergoogle_pluslinkedinmail

VMware vForum Online 2016

Attend vForum Online 19th April, 2016 where featured speakers are VMware CEO Pat Gelsinger and VMware CTO Chris Wolf, talking SDDC, End User Computing and the Hybrid cloud.

There will be 8 Hands-On-Labs to test-drive a variety of VMware Solutions. Certain unique product demos by VMware system engineers and unlimited access to eBooks, tech-tips and other product related resources.

For more information and registration, Click Here

Facebooktwittergoogle_pluslinkedinmail

Simplified exam booking process

on 05/03/2016, PeasonVue and VMware announced a new process of Booking or Managing your VMware Certification exams.

As part of this, now you don’t have to request Authorization for Exam and then separately launch PearsonVue portal to book the exam. There’s no need to even maintain two accounts across these two websites.

All you got to do is, GO in your http://mylearn.vmware.com account. And then simply click on ‘Getting Started’ Tab. That’s where under Exam Authorization section, you will find a link to launch ‘Pearson Vue: Manage Exam’, which is single sign-on with your current mylearn portal account. See following screenshot

Screen Shot 2016-03-06 at 4.25.58 PM.png

Ref: http://www.pearsonvue.com/vmware/

Facebooktwittergoogle_pluslinkedinmail

Delete Firewall rules in VCSA 6.0

Let’s say we have vCSA 6.0 appliance in place, where some firewall rules are created.

Managing this is easy using vSphere Web Client -> Administration -> System Configuration -> Nodes -> <your vCSA node> -> Manage -> Settings -> Firewall
(List of rules would appear like following screenshot)

Screen Shot 2016-02-10 at 10.51.08 PM
Click on ‘Edit’ button
Screen Shot 2016-02-10 at 11.10.59 PM
W
here there are buttons to Add new rule, Edit an existing rule, Re-order rules using up/down arrow, and the last button is to Delete.

Now if you want to see list of firewall rules using vCSA console, command is

api.com.vmware.appliance.version1.networking.firwall.addr.inbound.list (Following output)

Screen Shot 2016-02-10 at 10.56.32 PM

and from this list if you want to delete one of the rule, remember rules are being displayed in the order they are there in GUI output, where very first rule record is index number 0, second record is index number 1 and so on

And to delete one of the record from the list, use following command.
api com.vmware.appliance.version1.networking.firewall.addr.inbound.delete –position 0
This will delete very first record from the list. and the second record which is there in the above screenshot will become the index position 0. see following screenshot

Screen Shot 2016-02-10 at 11.19.40 PM

Now, if you want to delete All the rules in firewall in just one go.

api com.vmware.appliance.version1.networking.firewall.addr.inbound.delete –all true
this will make sure that all the rules in vCSA 6.0 firewall gets deleted.

(Note: in my commands, I have actually used double dash without any space in between which is visible in screenshots but the Text I have typed in blogpost is not making it very clear)

Facebooktwittergoogle_pluslinkedinmail

vSphere 5.5 or higher and Reliable Memory Technology

like we know ECC (Error-Correcting Code) is a great feature we have to tackle soft errors happening in RAM, without causing OS to fail, but ECC protects us from single soft error in a memory block at a time, so if there are multiple soft errors in a single memory block or hard errors on one or multiple cells in main memory, this will surely cause OS Kernel to panic. This will then result into longer downtime in terms of going through variety of steps to identify faulty module to replace it. Or resetting OS (in case of multiple soft errors) with all it’s application services. If this was a virtualised environment with VMware ESXi  or any such other hypervisors then we know there will be multiple VMs running and the all go down along with the hypervisor.

This is where Reliable Memory Technology (RMT) comes into picture which is basically a hardware feature which works along with supported OS (like ESXi 5.5 or higher). This will make sure that if during ongoing operations, any multi-bit soft error in a DIMM or hard error occurs, it will be detected by RMT and will take corrective actions in such a way so it won’t trigger OS Kernel to panic.

For Example, let’s say there was a hard error in one of the DIMM, system will detect it and mark faulty cell and some cells around it as non-usable. so that current OS operations will continue but after next reboot, OS will not see those faulty cells because hardware is not even presenting those anymore.

RMT proves to be really great when it comes to minimise downtimes due to Memory Fault related kernel panics, and also avoiding to replace whole memory module due to hard errors in DIMM.

if you have vSphere 5.5 or higher with Enterprise or Enterprise plus edition license, and if you hardware has it, then in your ESXi host you will be able to see Reliable memory using following command.

ESXCLI HARDWARE MEMORY GET.

List of References:
Tech. White Paper from Dell about RMT
Third-party blog post1
Third-party blog post2

Facebooktwittergoogle_pluslinkedinmail