Author:
Leni Kirilov, PowerCLI QE team, VMware Bulgaria
I met my friend Michael, a software quality engineer, who is
validating the functional correctness and completeness of “Cake Wizard”.
Cake Wizard is an online store, consisting of a database of items for
sale, accounts, online payments, etc. It is an enterprise application
developed with JEE.
In the process of validating the product, Michael needed a dedicated
test environment. Michael told me he created a complete set of
automation tools for executing the tests, gathering the results, and
reverting the environment to its initial state. He was really proud of
his accomplishment. He also integrated download and installation of
product builds with a Git source control system for the test cases,
Jenkins/Hudson for scheduling jobs on the test run process, and a custom
test runner to execute JUnit… I was impressed! Michael had done a
really great job by automating so many things. But he mentioned that he
had many problems with automating the process. There were even lost test
run results due to bugs.
I suggested a different approach – automation & virtualization.
Michael looked confused, not to say surprised. “But virtualization is
used only in production systems for better load-distribution, resource
utilization and cloud services… what does it have to do with automation
testing?” I smiled…
Nowadays, most testing looks like this: you get a physical
environment, execute tests on it, and revert it to a clean state by
using images. This approach is convenient because you don’t have to
configure everything all over again. But there are some negatives as
well.
- When you modify the hardware configuration, you must update the image, because it’s hardware specific;
- If the hardware configuration fails, you can lose all your data and test results, which is a potential risk for the project.
- The approach doesn’t scale well because a single hardware configuration can be at most a single test environment.
Virtualization has the same advantages as images-driven test environment setup and more:
- Better utilization – you can run more than one virtual
machine (VM) on a single piece of hardware as long as there are enough
resources. Provide your Quality Engineers their own environment to play
with!
- Revert to a previous state – cleanup is easy because VMs are
designed with that in mind. You can easily debug a problem by using
snapshots. They are quite fast, too.
- Hardware failure – it’s irrelevant for the virtualized
infrastructure. Since VMs are just files, you can back them up, in order
to minimize loss, in case of server crashes
- Pool of resources – no need to reconfigure and recreate images when you buy new hardware. You just add it to the pool of resources.
- Maintenance & automation – with minimal effort you can
replicate VMs by using virtualization and the already available
automation scripts. You can even create your own automation scripts!
My friend Michael heard my arguments but was still skeptical. He had
already created his environment and he asked me: “What’s the easiest way
to virtualize it? What are the first steps?”
Let’s see how easy we can use VMware virtualization to:
- automate setting up a test environment,
- set up the test software
- collect test results
[Set up and configure virtual test environment]
You need virtualization infrastructure, so let’s install VMware ESXi
Hypervisor. It provides tools to run and manage virtual machines.
Follow these
instructions
You start your virtualized test environment by creating a empty
virtual machine (VM). You install the OS (Guest OS) and applications you
need just once and save the result VM. You can use it to clone and thus multiply the VM as many times as you need.
Let’s install Windows Server 2008 and a JBoss Application Server 6 on our VM and name it “Manually created VM”.
Manually configuring a VM is OK only in the process of
learning the VM specifics. As Quality Engineers, we want to automate as
much as possible in our work.
“How do we automate a virtual test environment and what is the best tool?”
The preferred tool for virtualization automation is VMware vSphere PowerCLI.
It is closely developed with the VMware infrastructure suite (called
vSphere). We need only some of the many features which PowerCLI
provided, to create our test environment. The product platform is
Microsoft PowerShell (running on .Net) – powerful yet very user friendly
CLI (command line interpreter) which provides great console
commands(called cmdlets) for managing other Windows products like SQL
Server, IIS, Windows, MS Office etc. Even though PowerCLI is based on
Microsoft technology, you can use it to install and configure Windows,
as well as Red Hat Guest OS on your VM. Let’s see it in action:
Download and install latest PowerCLI version from here:
http://www.vmware.com/go/powercli
To start using PowerCLI you should connect to your local VMware ESXi
Server (which you should have installed and configured beforehand).
Connect-VIServer 192.168.1.1 -User ‘administrator’ -Password ‘pass1234’
#For later processing get your VM and store it in a variable.
$vm = Get-VM –Name “Test_Environment_VM_1”
Start-VM $vm # you can start using your VM for configuration
It’s highly recommended to install VMware Tools on your VM –
it makes some of the trickier operations possible, improves VM’s
performance…
Let’s get all network interface cards of the OS installed on the VM and then configure them.
# get all network cards. GuestUser is the username required for login to the GuestOS
$nics = Get-VMGuestNetworkInterface $vm -GuestUser ‘administrator’ -GuestPassword ‘pass1234′
We have dedicated the 192.168.2.* subnetwork for our test infrastructure, so let’s update the VMs IP and DNS addresses:
Set-VMGuestNetworkInterface $nics[0] -Ip 192.168.2.2 –Dns 192.168.1.2 -GuestUser ‘administrator’ -GuestPassword ‘pass1234′
If you need a more advanced configuration for your Guest OS, you can take a look at the
OSCustomization
cmdlets, which give you the ability to configure user accounts, active
directory, workgroups as the VM is created. There are examples in the
documentation.
After you finish configuring your networking, you should create a
snapshot that saves the state of your VM. This is useful when you want
to revert this VM to a previous state. Let’s create a snapshot for
future use:
New-Snapshot –VM $vm –Name “After network configuration – 2012.02.20”
[Create environment configuration files]
It’s good to keep various data about your environment. We create a hashtable called $env1. It can store the names of virtual machines, the servers’ IP addresses, DB table names, etc.
You can use the NIC objects to build this configuration file for future test executions or to form a report. The $nics variable contains all the information you need such as IP, DNS, Subnetmask, etc:
$env1[“IpAddress”] = $nics[0].Ip
You can save the hashtable $env1 to a file and use it as
configuration for your “execute test” script. This way you can easily
switch between environments with various properties.
$env1 | Export-CliXml “d:\CakeWizard\Testing\Environments\env1_win2008.xml”
#to use the configuration you import it using this call:
$env1Config = Import-CliXml “d:\CakeWizard\Testing\Environments\env1_win2008.xml”
Now you can combine all these one-liners in a short script that configures VM’s networking and creates a report/config file with the current IPs that are in use.
Simple, huh?
“What about applications and integration with other systems like Source
Control Repositories or test execution systems like Hudson/Jenkins?”
[Set up tests and test framework]
For even greater customization, you can execute custom scripts in the
Guest OS! By copying files to the Guest OS and executing them, you can
automate the following action:
- downloading and installing the software under testing;
- syncing sources with tests and frameworks;
- preparing reports;
- managing Microsoft products via their PowerShell API;
- pretty much everything you can think of!
You can copy different executable files, ZIP files with scripts,
DLLs… everything you want, to the VM’s local file system. Let’s sync the
tests from our source control repository. I’ve written a short script
to create workspace and sync everything with it.
Copy-VMGuestFile -LocalToGuest -VM $vm -Source “d:\CakeWizard\tools\sources_sync.bat” –Destination “c:\CakeWizard\tools\perforce_sync.bat” -GuestUser “administrator” -GuestPassword ‘pass1234′
#execute the script we’ve copied
Invoke-VMScript -ScriptText “c:\CakeWizard\tools\sources_sync.bat” -VM $vm -GuestUser ‘administrator’ -GuestPassword ‘pass1234′
The ScriptText parameter can point to another script or be the script
itself. So that makes it pretty easy to be included in Hudson/Jenkins
job.
The script you execute returns live results to your screen while
working inside the VM. This is great for monitoring the progress of a
long test run or analyzing test run results. The best thing about the Invoke-VMScript cmdlet is that it can execute many types of scripts: Powershell, BAT for Windows and Bash for Linux systems!
Now that we’ve prepared everything, it’s good time to create another snapshot. Let’s name it “Ready for test execution” or “Clean state”.
[Install product]
Let’s upgrade to the latest build version of our product. We have a
Powershell script that does the job. Let’s see how to execute it:
Invoke-VMScript -ScriptText “c:\CakeWizard\Tools\upgradeProduct.ps1″ -VM $vm -GuestUser ‘administrator’ -GuestPassword ‘pass1234′
[Create report]
You can generate a test run execution report after the test execution.
Our report holds information about the test id, test status, log files
and execution times.
Invoke-VMScript -ScriptText “c:\CakeWizard\Tools\generateReport.bat” -VM $vm -GuestUser ‘administrator’ -GuestPassword ‘pass1234′
# locally copy a report
Copy-VMGuestFile -GuestToLocal
-VM $vm -Source “c:\Tests\Results\Report\*.*” -Destination “d:\TestResults\Report\*.*” -GuestUser “administrator” -GuestPassword ‘pass1234′
[Clean up environment / Revert to snapshot]
When you run a test (again, you can use Invoke-VMScript to trigger
execution), you might pollute your brand new Test_Environment_VM_1.
Wouldn’t it be great, if you could always go back to a clean state of
your test environment? This way you can be comfortable with running even
the deadliest, most insane, build-crashing tests. Our new script for
creating a new test environment could do the trick, but there’s an even
easier way to fix the ruined VM – snapshots!
# reverting the vm to a previous state
Set-VM –VM $vm –Snapshot “Clean state”
“Set up once – test multiple versions!”
You can easily customize these one-line scripts for your own use cases.
By combining all of the scripts, you will automate your virtual
environment! You are just a step away from provisioning multiple test
environments!
[Copy VM to create a new virtual test environment]
A VM is just a set of files, so we can copy a VM by copying its files.
Let’s create a new environment called “Test_Environment_VM_2”.
#change directory to ‘vmstores:’. It gives you access to the datastore tree.
#navigate to your datastore and Test_Environment_VM_1 location and copy it
cd vmstores:
Copy-Item “.\Test_Environment_VM_1” -Destination “Test_Environment_VM_2” –Recurse
#now let’s register the new VM
New-VM –VMFilePath “[datastore_name] .\Test_Environment_VM_2\Test_Environment_VM_2.vmx” –Name “Test_Environment_VM_2”| Start-VM
And you are ready with your new environment and John can help you with testing “Cake Wizard”
Note – there is an easier way to clone a virtual machine – check VM templates, a feature available in vCenter Server.
Virtualization makes your testing process quite agile and scalable – you
can create more VMs on which to run tests, than the number of the
physical machines you can buy or power up. Even your developer
colleagues can use the VMs for developing the software in the first
place!
You can also easily revert a VM to a clean state, without losing any time for configuration.
You can set up and configure the VM so that most of the things that you
can do with a physical machine are possible (except for testing
specific/special hardware devices).
You will no longer fear of hardware failures, because your VM can be automatically migrated with no downtime.
You can automate all these tasks with the highly-rated tool – VMware vSphere PowerCLI.
Well, Michael, I think that now you understand the advantages of virtualization in testing.
As quirky as this concept sounds, you’ll find that the basics are not
that different from the classical approach. Do you have any other
questions, we can discuss over a cup of coffee?
