Building a Fling
One of the fun things working for VMware is that anybody can create a Fling. If you have an idea and the desire to program it, you can publish it. This is a great way for our developers to try out an idea.
One of the problems that can arise is when, like all of us working on this Fling, you’re not a professional programmer and you don’t always know the correct way to do things. This lack of knowledge can manifest in many ways, and often only shows up when the product is tested at scale.
The Problem
One problem that happened on this Fling is that we incorrectly used the APIs for export. This resulted in a problem with API Pagination. API pagination is a way to return small parts of a dataset in multiple API calls, instead of returning the entire dataset in a single call. Most APIs will automatically paginate when the dataset grows too large. If all of your testing is done with small datasets, you will likely never hit a paginated return. We don’t have a huge SDDC that we test against when developing the Fling – we have a 3-node cluster with a modest number of virtual machines, maybe a hundred groups and a few hundred firewall rules.
We didn’t implement pagination. If a customer has more than a thousand groups to export, only the first thousand will be exported. Then, when you go to import firewall rules, any rules dependent on group #1001 and higher will fail to import. A customer just hit this problem. I implemented a cursor in the group export function to resolve it.
Paginated Results
If an API dataset in VMC on AWS is paginated, it will return an initial set of data along with a cursor. A cursor is a pointer to the next set of data. If the dataset is paginated, the bottom of the JSON results will look like this – it tells you the total number of results, how they’re sorted, and gives you the cursor to get the next set of records.
],
"result_count": 1188,
"sort_by": "display_name",
"sort_ascending": true,
"cursor": "00041000"
}The Fix
I implemented pagination in the SDDC CGW Group export function. First, the function does an initial API call to retrieve a set of groups. The new code starts on line 10. It checks the first result set for the presence of a cursor. It then passes the cursor to the API to retrieve the next dataset. The while loop will continue calling the API with subsequent cursors until there are no cursors in the result set. It then appends the result set to the previous result set, so in the end we get all of the groups in a single variable.
def exportSDDCCGWGroups(self):
"""Exports the CGW groups to a JSON file"""
myURL = self.proxy_url + "/policy/api/v1/infra/domains/cgw/groups"
response = self.invokeVMCGET(myURL)
if response is None or response.status_code != 200:
return False
json_response = response.json()
cgw_groups = json_response['results']
# After grabbing an intial set of results, check for presence of a cursor
while "cursor" in json_response:
myURL = self.proxy_url + "/policy/api/v1/infra/domains/cgw/groups?cursor=" + json_response['cursor']
response = self.invokeVMCGET(myURL)
if response is None or response.status_code != 200:
return False
json_response = response.json()
cgw_groups.extend(json_response['results'])Testing
How do I test? I don’t have an SDDC with enough objects to create a cursor. I could change the API call to force it to give me paginated data, say in chunks of 20, but that’s not the same thing as testing against an SDDC that has a few thousand groups in it.
I needed to implement a testbed function to automatically create test data. I wasn’t initially sure whether I should include it as an actual function in the Fling itself, or whether I just used it for pre-release testing. But I decided the ability to create and delete large numbers of groups might be useful for others to do scale testing on their SDDCs, so I added it.
I created 3 testbed commands – create-cgw-groups, delete-cgw-groups, and delete-all-cgw-groups
This command create 1,500 test Compute Gateway groups, starting with the default cgw-test-group-0000.
python sddc_import_export.py -o testbed --test-name create-cgw-groups --num-objects 1500This command will create 1,500 test Compute Gateway groups, starting with cgw-test-group-0050
python sddc_import_export.py -o testbed --test-name create-cgw-groups --num-objects 1500 --start-num 50This command will delete 1,500 test Compute Gateway groups, starting with cgw-test-group-0000. You can also pass the –start-num argument to change the start number
python sddc_import_export.py -o testbed --test-name delete-cgw-groups --num-objects 1500This command will delete ALL CGW GROUPS – use with caution.
python sddc_import_export.py -o testbed --test-name delete-all-cgw-groupsSummary
The pagination fix and testbed functions are all available in version 1.5.6 of the Fling, published today.