Be efficient - virtualize and automate your test infrastructure

Author: Leni Kirilov, PowerCLI QE team, VMware Bulgaria

I met my friend Michael, a software quality engineer, who is validating the functional correctness and completeness of “Cake Wizard”. Cake Wizard is an online store, consisting of a database of items for sale, accounts, online payments, etc. It is an enterprise application developed with JEE.

In the process of validating the product, Michael needed a dedicated test environment. Michael told me he created a complete set of automation tools for executing the tests, gathering the results, and reverting the environment to its initial state. He was really proud of his accomplishment. He also integrated download and installation of product builds with a Git source control system for the test cases, Jenkins/Hudson for scheduling jobs on the test run process, and a custom test runner to execute JUnit… I was impressed! Michael had done a really great job by automating so many things. But he mentioned that he had many problems with automating the process. There were even lost test run results due to bugs.

I suggested a different approach – automation & virtualization. Michael looked confused, not to say surprised. “But virtualization is used only in production systems for better load-distribution, resource utilization and cloud services… what does it have to do with automation testing?” I smiled…

Nowadays, most testing looks like this: you get a physical environment, execute tests on it, and revert it to a clean state by using images. This approach is convenient because you don’t have to configure everything all over again. But there are some negatives as well.

- When you modify the hardware configuration, you must update the image, because it’s hardware specific;

- If the hardware configuration fails, you can lose all your data and test results, which is a potential risk for the project.

- The approach doesn’t scale well because a single hardware configuration can be at most a single test environment.

Virtualization has the same advantages as images-driven test environment setup and more:

- Better utilization – you can run more than one virtual machine (VM) on a single piece of hardware as long as there are enough resources. Provide your Quality Engineers their own environment to play with!

- Revert to a previous state – cleanup is easy because VMs are designed with that in mind. You can easily debug a problem by using snapshots. They are quite fast, too.

- Hardware failure – it’s irrelevant for the virtualized infrastructure. Since VMs are just files, you can back them up, in order to minimize loss, in case of server crashes

- Pool of resources – no need to reconfigure and recreate images when you buy new hardware. You just add it to the pool of resources.

- Maintenance & automation – with minimal effort you can replicate VMs by using virtualization and the already available automation scripts. You can even create your own automation scripts!

My friend Michael heard my arguments but was still skeptical. He had already created his environment and he asked me: “What’s the easiest way to virtualize it? What are the first steps?”

Let’s see how easy we can use VMware virtualization to:
- automate setting up a test environment,
- set up the test software
 - collect test results

[Set up and configure virtual test environment]

You need virtualization infrastructure, so let’s install VMware ESXi Hypervisor. It provides tools to run and manage virtual machines. Follow these instructions

You start your virtualized test environment by creating a empty virtual machine (VM). You install the OS (Guest OS) and applications you need just once and save the result VM. You can use it to clone and thus multiply the VM as many times as you need.

Let’s install Windows Server 2008 and a JBoss Application Server 6 on our VM and name it “Manually created VM”.

Manually configuring a VM is OK only in the process of learning the VM specifics. As Quality Engineers, we want to automate as much as possible in our work.

“How do we automate a virtual test environment and what is the best tool?”

The preferred tool for virtualization automation is VMware vSphere PowerCLI. It is closely developed with the VMware infrastructure suite (called vSphere). We need only some of the many features which PowerCLI provided, to create our test environment. The product platform is Microsoft PowerShell (running on .Net) – powerful yet very user friendly CLI (command line interpreter) which provides great console commands(called cmdlets) for managing other Windows products like SQL Server, IIS, Windows, MS Office etc. Even though PowerCLI is based on Microsoft technology, you can use it to install and configure Windows, as well as Red Hat Guest OS on your VM. Let’s see it in action:

Download and install latest PowerCLI version from here: http://www.vmware.com/go/powercli

To start using PowerCLI you should connect to your local VMware ESXi Server (which you should have installed and configured beforehand).

Connect-VIServer 192.168.1.1 -User ‘administrator’ -Password ‘pass1234’

#For later processing get your VM and store it in a variable.

$vm = Get-VM –Name “Test_Environment_VM_1”

Start-VM  $vm # you can start using your VM for configuration

It’s highly recommended to install VMware Tools on your VM – it makes some of the trickier operations possible, improves VM’s performance…
Let’s get all network interface cards of the OS installed on the VM and then configure them.

# get all network cards. GuestUser is the username required for login to the GuestOS

$nics = Get-VMGuestNetworkInterface $vm -GuestUser ‘administrator’ -GuestPassword ‘pass1234′

We have dedicated the 192.168.2.* subnetwork for our test infrastructure, so let’s update the VMs IP and DNS addresses:

Set-VMGuestNetworkInterface $nics[0] -Ip 192.168.2.2 –Dns 192.168.1.2 -GuestUser ‘administrator’ -GuestPassword ‘pass1234′

If you need a more advanced configuration for your Guest OS, you can take a look at the OSCustomization cmdlets, which give you the ability to configure user accounts, active directory, workgroups as the VM is created. There are examples in the documentation.

After you finish configuring your networking, you should create a snapshot that saves the state of your VM. This is useful when you want to revert this VM to a previous state. Let’s create a snapshot for future use:

New-Snapshot –VM $vm –Name “After network configuration – 2012.02.20”

[Create environment configuration files]

It’s good to keep various data about your environment. We create a hashtable called $env1. It can store the names of virtual machines, the servers’ IP addresses, DB table names, etc.

You can use the NIC objects to build this configuration file for future test executions or to form a report. The $nics variable contains all the information you need such as IP, DNS, Subnetmask, etc:

$env1[“IpAddress”] = $nics[0].Ip

You can save the hashtable $env1 to a file and use it as configuration for your “execute test” script. This way you can easily switch between environments with various properties.

$env1 | Export-CliXml “d:\CakeWizard\Testing\Environments\env1_win2008.xml”

#to use the configuration you import it using this call:

$env1Config = Import-CliXml “d:\CakeWizard\Testing\Environments\env1_win2008.xml”

Now you can combine all these one-liners in a short script that configures VM’s networking and creates a report/config file with the current IPs that are in use.

Simple, huh?

“What about applications and integration with other systems like Source Control Repositories or test execution systems like Hudson/Jenkins?”

[Set up tests and test framework]

For even greater customization, you can execute custom scripts in the Guest OS! By copying files to the Guest OS and executing them, you can automate the following action:
- downloading and installing the software under testing;
- syncing sources with tests and frameworks;
- preparing reports;
- managing Microsoft products via their PowerShell API;
- pretty much everything you can think of!

You can copy different executable files, ZIP files with scripts, DLLs… everything you want, to the VM’s local file system. Let’s sync the tests from our source control repository. I’ve written a short script to create workspace and sync everything with it.

Copy-VMGuestFile -LocalToGuest -VM $vm -Source “d:\CakeWizard\tools\sources_sync.bat” –Destination “c:\CakeWizard\tools\perforce_sync.bat” -GuestUser “administrator” -GuestPassword ‘pass1234′

#execute the script we’ve copied

Invoke-VMScript -ScriptText “c:\CakeWizard\tools\sources_sync.bat” -VM $vm -GuestUser ‘administrator’ -GuestPassword ‘pass1234′

The ScriptText parameter can point to another script or be the script itself. So that makes it pretty easy to be included in Hudson/Jenkins job.

The script you execute returns live results to your screen while working inside the VM. This is great for monitoring the progress of a long test run or analyzing test run results. The best thing about the Invoke-VMScript cmdlet is that it can execute many types of scripts: Powershell, BAT for Windows and Bash for Linux systems!

Now that we’ve prepared everything, it’s good time to create another snapshot. Let’s name it “Ready for test execution” or “Clean state”.

[Install product]

Let’s upgrade to the latest build version of our product. We have a Powershell script that does the job. Let’s see how to execute it:

Invoke-VMScript -ScriptText “c:\CakeWizard\Tools\upgradeProduct.ps1″ -VM $vm -GuestUser ‘administrator’ -GuestPassword ‘pass1234′

[Create report]

You can generate a test run execution report after the test execution. Our report holds information about the test id, test status, log files and execution times.

Invoke-VMScript -ScriptText “c:\CakeWizard\Tools\generateReport.bat” -VM $vm -GuestUser ‘administrator’ -GuestPassword ‘pass1234′ 

# locally copy a report

Copy-VMGuestFile -GuestToLocal -VM $vm -Source “c:\Tests\Results\Report\*.*” -Destination “d:\TestResults\Report\*.*” -GuestUser “administrator” -GuestPassword ‘pass1234′

[Clean up environment / Revert to snapshot]

When you run a test (again, you can use Invoke-VMScript to trigger execution), you might pollute your brand new Test_Environment_VM_1. Wouldn’t it be great, if you could always go back to a clean state of your test environment? This way you can be comfortable with running even the deadliest, most insane, build-crashing tests. Our new script for creating a new test environment could do the trick, but there’s an even easier way to fix the ruined VM – snapshots!

# reverting the vm to a previous state

Set-VM  –VM $vm –Snapshot “Clean state”

“Set up once – test multiple versions!” 

You can easily customize these one-line scripts for your own use cases. By combining all of the scripts, you will automate your virtual environment! You are just a step away from provisioning multiple test environments!

[Copy VM to create a new virtual test environment]

A VM is just a set of files, so we can copy a VM by copying its files. Let’s create a new environment called “Test_Environment_VM_2”.

#change directory to ‘vmstores:’. It gives you access to the datastore tree.

#navigate to your datastore and Test_Environment_VM_1 location and copy it

cd vmstores: 

Copy-Item “.\Test_Environment_VM_1” -Destination “Test_Environment_VM_2” –Recurse

#now let’s register the new VM

New-VM –VMFilePath “[datastore_name] .\Test_Environment_VM_2\Test_Environment_VM_2.vmx” –Name “Test_Environment_VM_2”| Start-VM

And you are ready with your new environment and John can help you with testing “Cake Wizard”

Note – there is an easier way to clone a virtual machine – check VM templates, a feature available in vCenter Server.

Virtualization makes your testing process quite agile and scalable – you can create more VMs on which to run tests, than the number of the physical machines you can buy or power up. Even your developer colleagues can use the VMs for developing the software in the first place!

You can also easily revert a VM to a clean state, without losing any time for configuration.

You can set up and configure the VM so that most of the things that you can do with a physical machine are possible (except for testing specific/special hardware devices).

You will no longer fear of hardware failures, because your VM can be automatically migrated with no downtime.

You can automate all these tasks with the highly-rated tool – VMware vSphere PowerCLI.

Well, Michael, I think that now you understand the advantages of virtualization in testing. As quirky as this concept sounds, you’ll find that the basics are not that different from the classical approach. Do you have any other questions, we can discuss over a cup of coffee?