Test Automation is an old idea dating back to the 1970s and 80s, where software developers attempted to reduce the need for manual retesting of progressive releases of a software product, by running a script that tries to match expected outcomes with the actual ones. As the Lead Tester for Calce’s QA Engineering team, I consider Test Automation as an important efficiency strategy for our clients.
One nagging myth to dispel about Test Automation is that it is not an automatic way of doing a tester’s job! Instead, it’s just a way of extending a testers reach of what they do in their minds – which is to conceptualize the functionality and exception pathways in a given software product from the point of view of an end user. Automation allows testers to focus on new functionality and exceptions, thus improving engineering efficiency in incremental product development initiatives that span a significant period of time. The codification of test cases in an imperative language also encourages superior testing diligence – in that one is forced to minutely consider all the test scenarios when scripting a given test case.
Testers still have their work cut out when maintaining an automated test suite for a growing software product. They must carefully think-through the functional changes and exception scenarios, and codify them during every development sprint. This activity ideally aught to replace the need for writing manual test cases on Excel spreadsheets; but in practice all test scenarios might not be scriptable due to the inherent limitations in available test automation frameworks, so a certain percentage of manual test coverage will likely remain.
We at Calcey strongly believe that testing, whether automated or manual, requires human ingenuity to be effective. We consider automation as an important weapon in our quality assurance arsenal. As such, we leverage the following test automation technologies in our ongoing product development efforts:
- Selenium Web Driver with Gallio Automation Platform
- Selenium Web Driver with Robot Framework
- Selenium Web Driver with Cucumber Framework
As you can see, Selenium Web Driver is the test automation server of choice for all our projects. It is the server component that actually performs the GUI actions of the application by directly invoking the Browser automation API commands. It is the workhorse demon or “phantom tester” . The scripting language and execution frameworks used vary depending on the technologies with which the product is developed. Test cases written in Visual Studio/C# can be run with the Gallio Test Runner, which works seamlessly with Selenium Web Driver. Our preferred Test Runner for Python/django projects is the Robot Framework, which has an easy-to-use tabular test-data syntax and a keyword-driven testing approach.
Cucumber is a tool for running automated acceptance tests written in behavior-driven development (BDD) style. BDD is a popular Agile testing method that describes test scenarios in plain natural language. This means that the “tests” (which are plain text feature descriptions) are typically written before any imperative language scripting, and placed into “feature files”. Cucumber then interprets these tests into a specified programming language (such as Ruby, Java or .NET).
Calcey also uses Jenkins as its Continuous Integration tool, working in conjunction with test automation. Jenkins is our preferred open source tool to perform CI. The basic functionality of Jenkins is to monitor a version control system and to start and monitor a build system if changes occur. Jenkins monitors the whole build process and provides reports and notifications to alert maintainers on success or failures. We setup a scheduler in Jenkins to read the committed regression test scripts from GIT (our code repository), which runs the scripts, evaluates the results and shows it in our Jenkins dashboard. With every release we run regression suites to ensure the new functionalities don’t break existing ones. The QA team monitors the results and reports any failed scenarios as bugs on JIRA.
Many of our clients like CompareNetworks, Westfield Malls and Stanford University have benefitted significantly from the cost-efficiencies brought about through Test Automation. Test automation is our recommended strategy for reducing QA bandwidth bottlenecks in complex product development projects, where the product evolves continuously over several months or years. Also, Test Automation offers a challenge and an alternative to manual testing, making the lives of QA Engineers a whole lot more interesting!