In this blog post, we will finally work our way up to what I’ve tried to accomplish with a combination of PowerShell, Cisco UCS Director’s northbound APIs, and the concept of parallel processing with workflows in UCS Director.
The Assumptions and Caveats
First, let’s talk about the assumptions for this PowerShell script and UCS Director workflow. My current network architecture is one that models itself after one of Cisco’s validated designs for data centers. This design has a network spine (provided by the Nexus 7000 switch line) and network leaves (provided by the Nexus 5000 switch lines). As part of the configuration of this design, Cisco FabricPath is enabled across the switching lines.
With the implementation of FabricPath, the process to create a simple L2 VLAN across each switch is relatively simple and lends itself to be easily capable of taking simple variables (mostly in the form of which switch the operation needs to be performed). The only main difference between the two switches that forces me to have to create two different VLAN creation scripts is due to the Nexus 7000 family. This only forces me to have to specify the VDC (virtual device context) before running the small script to create a VLAN and enable FabricPath mode on that VLAN.
Also, as part of my architecture and current usage of these switches is the fact that not all the L2 VLANs were created with Cisco UCS Director. In response to this, I created two different workflows. We’ll call them: Nexus List Add and Nexus List Rollback Setup. Nexus List Add is used when the L2 VLAN is detected to NOT already exist on the switch. This will run a block of NX-OS CLI that will create the VLAN based on a passed VLAN name and VLAN ID variable. From there, FabricPath mode is enabled and then we save the running-config to startup. Regarding Nexus List Rollback Setup, instead of going through and trying to create a VLAN that already exists on the switch, we register a rollback task with UCS Director (for consistent VLAN removal across the entire FabricPath domain) and set ownership of the VLAN in UCS Director for the customer in question. This forces UCS Director to know about the VLAN and which customer owns the VLAN in question.
One last caveat for the PowerShell environment on my Cisco PowerShell Agent servers. I’ll admit that I’ve been rather lazy since deploying these out originally and the PowerShell version on them is still version 4.0. This causes issues with large JSON responses by some of the UCS Director northbound APIs. In version 4.0, there was a maximum size of the response in which you could use the ConvertFrom-Json cmdlet. I was forced to use some borrowed code that adjusts some of the JSON parameters in the system for my runspace and then creates a custom PowerShell object. Unfortunately, this process adds a lot of overhead. I’ve recently found that by upgrading my environment to PowerShell 5.0, the issues go away and I can get away from the custom code to create the PoSH object.
The code is available at the following GitHub location: My Example Code
I wrote two specific functions in my PoSH code to be reusable through this script. They are called Get-UCSDURIs and Connect-NestedPSSession. For reference, I’ve previously blogged about why I needed to create Connect-NestedPSSession (Using Invoke-WebRequest with the Cisco PowerShell Agent). Get-UCSDURIs was created so that I could go through my entire switching stack and generate all the specific URIs for calling the two Nexus workflows and filling in all the variables. Since I have 2 Nexus 7000s and 10 Nexus 5000s in this configuration, I need to generate a combination of 12 URIs to send to the UCS Director northbound API, userAPISubmitWorkflowServiceRequest.
In Get-UCSDURIs, I also do a quick lookup of the workflow inputs by using the northbound API userAPIGetWorkflowInputs. The reason for this is that even if the input names on the workflows are the same, in UCS Director, they are given a specific number on the variable name to make them unique (example below from one of the JSON returns on Nexus List Add).
A total of three parameters are passed to the API when executing userAPISubmitWorkflowServiceRequest. The first, param0, is a string of the name of the workflow you wish to execute. The second is a lengthy string that’s passed as param1 in the userAPISubmitWorkflowServiceRequest URI. Most of the code in Get-UCSDURIs focuses on creating this parameter. Since all this information is put into the URI (and not using any sort of request body), I could not create this as a JSON literal. I had to create this as a large string object, which is why the code looks the way it does with multiple uses double quotes and back quotes. Lastly, we send a UCS Director service request ID (or SRID) to param2. In my case, I usually send the SRID of -1. This means that each workflow call is independent of themselves and does not register rollback tasks with the parent workflow. I handle rollbacks in a different way later, since I also want to use parallel processing in removing everything that I created.
The Main Script Body
I start the main script body by passing in multiple arguments from UCS Director. Many of these are specific to my environment, but the gist is that I need to pass in which of my site’s this needs to be created, what VLAN ID is being requested, what the requested VLAN name is going to be, which UCS Director group do I want to assign ownership to, and which parent UCSD SR ID I want to assign the rollback tasks too (however, in the prior paragraph, this is almost always -1, but I wanted the function to be usable for anything else too, not just this specific use case).
I’m also passing in specifics for the PowerShell Agent at the site in question (the PS Agent) and the username and password I want to use to initiate the nested remote PowerShell session using Connect-NestedPSSession.
With UCS Director and the northbound APIs, there is no Basic authorization header to work with. UCS Director creates an API key for each login that is accessible through the GUI. To be able to use that key, you need to store it into a hashtable in which one of the keys is labeled X-Cloupia-Request-Key. I do this by creating an empty hashtable first, the using the Add method on that object and insert the key/value pair into the hashtable. For the most part, this is the only thing required for the northbound APIs I’m using in this script.
After setting the header, I now have everything I need to start initiating northbound API calls to UCS Director. Before I start initiating any specific workflow, I need to go through a couple of lists to check to see what needs to be worked with and to determine which subsequent workflow (Nexus List Add or Nexus List Rollback Setup). I use the northbound API, userAPIGetTabularReport, to get these lists of networking equipment (MANAGED-NETWORK-ELEMENTS-T52) and the VLANs that have already been pulled from the switch inventories and in the UCS Director databases (VLANS-T52).
After running through these APIs and parsing through the responses (and filtering down to very specific information), we then begin the process of cross checking to see if the VLAN exists on the equipment. Depending upon whether it exists or doesn’t AND the model type of the equipment being checked, the switch information gets placed into one of four lists. Each is labeled with either a 5k or 7k string and whether it’s a rollback or not. These four lists are then processed and the URIs are generated using the Get-UCSDURI function. Lastly, all four URI string returns are smashed together into one large, comma-separated list which should contain, depending upon the site, either 10 lines or 12 lines of URIs to process.
We start the execution process, but taking our URI list and initiating each call using a foreach loop. I store the SR ID into a hashtable that I can use for keeping an eye on the workflows I just initiated. Now, to keep an eye on each one of these API requests, we utilize the northbound API, userAPIGetWorkflowStatus. This returns a numerical code to tell us what the status of the workflow is. If the status code returned is one in which the workflow has reached some sort of stop, I remove the SR ID from the hashtable. Also, I put the SR ID into a string called $sr_list. This list is going to be used to store all the workflow SR IDs and use it for rollback purposes. I will also return this list to UCS Director. Once all the SR IDs have been removed from the hashtable, my while loop will shut down and the script will finish.
The assumption is that all my API calls have completed (without critical stop) and that a Layer 2 VLAN has been pushed out, in a parallel processed way.
I haven’t mention much on this concept up to this point, having focused on the code instead. The reason that I wanted to perform this in parallel is that each switch acts independently of each other. While FabricPath requires you to create a VLAN on all the switches for traffic purposes, there’s no major error checking that forces you to have it all created, at once.
Taking this idea, you can see that instead of waiting in a long sequential order (switch1 -> switch2 -> … -> switch10), I can initiate all 10 requests, at the same time. This is what I mean by parallel processing. Before I created this script, my entire process of creating a L2 VLAN was taking upwards of 10-12 minutes. In a cloud world, this seemed extremely high, especially for something as simple as a L2 VLAN.
After this implementation, I have execution times as low as three minutes at one site (has less VLANs to parse through) and as high as five minutes (due to 12 switches with 1500 VLANs on them; this is roughly around 175,000 lines of raw JSON information). Oh, and I forgot to mention I also ADDED new functionality to the overall workflow (these efficiencies were just one part of the major fixes requested). I added automatic additions of vSphere created distributed portgroups and adding those to a customer’s network policy in their virtual data center. I also added the ability to go through UCS Central and find any customer service profiles and add the VLAN to the vNICs of each of those service profiles. Included in all of this were also permissions granting to our VMware hosted environment for those distributed portgroups. So, I added some major automated assumptions AND drastically cut the time of execution.
I hope that either you read through the entire series or at least got something new and useful out of this blog series. I was amazed just how little was being published from either Cisco SMEs or from the community with regards to PowerShell and Cisco UCS Director. I hope that you, the reader, have realized just how powerful the PowerShell capabilities can be and how easy you can extend systems with no direct support in Cisco UCS Director.