TestQuest provides test automation solutions to the embedded systems market, covering wireless, PDA, TV/set-top boxes as well as PCs and servers. The original solution for phones and PDAs relied on hardware interfaces that drive the device under test and capture the results by direct interaction with the hardware. While this solution provides the best possible replication of a user interacting with a device it also requires either a hardware board or dedicated device so that the necessary electrical contacts can be made. This solution is appropriate for Hardware Vendors and operators but it restricted the appeal of TestQuest Pro to third party developers who might not wish to dedicate a device or devices to testing.

TestQuest have addressed this limitation with the release of software based interfaces for several platforms including Symbian OS. "The software interface provides versatility both for third party developers as well as for manufacturers and carriers," said Don Leick Product Marketing Manager for TestQuest. "It allows plug-and-play-style test connectivity, with only a small executable needing to be loaded on the device. Whereas the hardware interface, while being extremely functional and placing no software on the handset, requires the customer to send us the phone which we instrument to work with TestQuest Pro."

TestQuest Pro

TestQuest Pro consists of three main modules, Script Recorder, Test Information Manager and Test Execution Director, which together allow tests to be created, aggregated into test suites and executed.

Script Recorder

Script recorder is the heart of TestQuest Pro. It allows test scripts to be created by recording keystrokes (although tests can be created by editing test scripts directly) and defining verifications.

Setting up the TestQuest Pro software interface for use with a Series 60 device is simply a matter of loading and activating the TestQuest Pro agent on the phone. This involves the installation of 4 small utilities, which can be installed on the phone’s memory card and utilize about 40Kb of memory when running. The version of TestQuest Pro we had available (V5.0) provided for a connection over InfraRed and RS-232; the latest version can also utilize Bluetooth.

With the TQPro Agent installed and activated the Series 60 phone, in our case a Nokia 3650, is ready to connect to TestQuest Pro on a PC. TestQuest Pro sends keystroke commands to the agent which executes them and then sends back screen shots on demand. This means all the test control and validation work is done by TestQuest Pro on the PC.

Scripts are created within Script Recorder by either recording actions using a simulated 3650 keypad, and screen capture utility, or by editing a script directly. TestQuest Pro’s scripting language is C.

Scripts can be executed on the device while they are being recorded, or the keystrokes simply added to the script, it is also possible to drive the phone without recording the keystrokes. As the scripts are recorded the screen capture utility offers options to create verification which:

• Compare a bitmap with part or all of the current screen
• Test to find out whether a bit map appears in the current screen
• Test to determine whether a specific test string appears on the current screen, using Intelligent Text Recognition (ITR).

Our impression of Script Recorder is very positive, it hooked up and drove the Nokia 3650 easily and the recording and validation worked well, with simple bitmap and text validations being both easy to create and working accurately when tests were run.

For more advanced script development TestQuest Pro also provides a number of debugging features, similar to those you would expect in any Integrated Development Environment.

Test Information Manager

Testing usually involves the creation and execution of multiple scripts. The Test Information Manager (TIM) is a module of TestQuest Pro which allows testers to build up test suites out of individual scripts. Tests are organized into a hierarchical framework of projects, suites and then individual scripts which can then be executed as a batch.

Test Execution Director

Test Frameworks created in TIM are run by the Test Execution Driver (TED). TED also has capabilities for running individual scripts, batches of scripts or DLLs (compiled from a script).

Test results are recorded in log files and more structured reports on the outcomes of tests are produced by TIM.

One of TestQuest Pro’s strengths is its ability to work with multiple platforms. "Most commercial mobile developers are not just developing for a single OS," commented Don. "With TestQuest Pro they have a tool which allows them to test applications running on Windows Mobile, Symbian OS, and other platforms."

However while TestQuest Pro is able to run tests across multiple platforms using the standard Script Recorder developers would be required to create unique scripts for each platform. This is where Test Verb Technology (TVT) comes in. TVT is an architecture and methodology for the production and maintenance of scripts within TestQuest Pro. TVT provides a layer of abstraction between the low level script and "test verbs" (commands). A typical TVT test might read as follows:

TEST_CASE_START("Default System");

SET_POWER(ON);

NAVIGATE_TO("ADDRESS_BOOK");

ADD_CONTACT("Joe Smith")

……………..

Rather than the C script for a similar action which could read:

bt_KeypadKeysOf("SymbianTarget", "‹HangUp›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "‹Home›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "‹RightArrow›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "‹OK›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "‹LeftSelect›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "‹DownArrow›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "‹DownArrow›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "‹DownArrow›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "‹OK›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "5", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "666", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "33", 0, 0);

bt_Sleep( 500 );

bt_KeypadKeysOf("SymbianTarget", "‹DownArrow›", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "7777", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "6", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "444", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "8", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "44", 0, 0);

bt_KeypadKeysOf("SymbianTarget", "‹RightSelect›", 0, 0);

TVT has several advantages over vanilla scripts. They are device independent which means tests can be scripted once and run across multiple platforms. Even if the application’s implementation differs across multiple platforms it is possible for these differences to be abstracted in TVT. Tests are also easier to create and maintain as they are written in a way which is similar to natural language, which means they are easier to understand but also involve fewer lines of code which reduces the scope of any maintenance effort. Finally they also open script creation up to domain experts who may not be C programmers and therefore would struggle developing low level scripts. This final benefit is significant both because of the relative scarcity of good C developers but also due to the fact that domain experts are often much more effective at creating tests.

For Symbian OS application testing TestQuest has two main challengers, Digia’s Quality Kit and Mobile Innovation’s TRY. Both of these are Symbian OS specific test tools and run resident on the devices (or within the emulator). TestQuest Pro has several advantages over these tools. In addition to the ability to drive tests on multiple platforms, it has a low resource footprint on a device. "It’s really important to maintain the fidelity of tests on these devices," commented Don. "You want to ensure that you are using the minimum of memory and processor capacity, so that you do not change the behavior of the software you are testing. Executing off device is much better from a test fidelity standpoint." TQPro Agent certainly achieves this as we were able to use it to drive and capture screen shots of a Web application displaying complex graphics.

Our test of TestQuest Pro has been quite limited; it is a comprehensive testing product and we only scratched the surface of its capabilities. However we found it to be both effective and easy to use. Building scripts, creating test suites and analyzing the results was straightforward and the verifications, which relied on fuzzy technology, matching bit maps and text patterns, seemed to work well yielding no noticeable false results. TVT was not included in the software we had to test so we are unable to comment on how it works in practice but if it has inherited its quality from TestQuest Pro it should live up to its paper specification.

TestQuest Pro currently has software interfaces for Series 60, Pocket PC/Smartphone/WinCE and UTS (Qualcomm CDMA phones). A software interface for UIQ is in development and should be shipped early 2004. Support for Series 90 will also be added.

TestQuest Pro starts at US $12,500 for a license with the Symbian OS interface, with TVT starting at US $2,500.

Web:www.testquest.com