Richard: Penrith is not a place normally associated with Britain's high tech industries, so how did Penrillian start?

Charles: Penrillian has its roots back in 2000 when I wrote a book with James Noble called Small Memory Software.

I realized that the whole small memory thing was going to be big. Symbian were just starting and it was clear that there were going to be loads of small machines around. There was going to be a demand for this style of development, rather than it being a niche thing for us Acorn Micro hackers. The book resulted in an introduction to an Indian company, Saven, who wanted to get into the "next big thing". So we started Penrillian with the intention of training up Indian staff. It hasn't ended quite like that, but we are still going, happily.

Richard: One of the unique things about Penrillian is your open source testing product. Why have you developed it?

Charles: It all came about because of our use of the XP (Extreme Programming) methodology. I had tried out some of the practices over the 5 years I was a part-time consultant and technical lead at Symbian, and found that they worked really well. When we set up Penrillian we had a group of people who needed to learn a lot. I thought working in pairs, using some XP techniques, might be better than our staff working on their own. So we setup XP all the way though the company. We also brought in someone with XP experience to act as a coach, which we found invaluable.

XP requires continuous unit testing, so we sought a unit test framework. There wasn't one for Symbian OS, but there was one for C++. We took it and modified it for Symbian OS.

Richard: Did you release SymbianOSUnit because the original code was open source or out of some sense of alturism?

Charles: It was mainly because it was open source in the first place, and a feeling of giving something back to the community, as that's where we got it from. There are solid commercial reasons too. The release has meant that a lot of people have heard of us who otherwise might not have. So it is good publicity for our development work. Also there is always the hope that other people will provide feedback, or provide their own corrections to it.

Richard: Have there been many contributions?

Charles: Not yet. But there is always the hope that someone is going to come along with a wonderful modification, and that's another good reason for having released it. It's also good to have the discipline of an external version as well as an internal one.

Richard: Does the lack of feedback and involvement disappoint you?

Charles: Not particularly. We are not in a position to provide support on the open source version. If we did get contributions we would feel morally obliged to incorporate them into it. There is a cost to that. But if there were good improvements, it would be great.

Richard: Most of the commercial test tools concentrate in testing at the user interface. Why have you concentrated on unit testing?

Charles: A Symbian OS application is usually split into engine and UI. One of the benefits of this approach is that it encourages you to put as much as possible in the non-UI code. As a result it makes sense concentrate on testing the engine.

Richard: What are the advantages of the test first approach?

Charles: The principal is to first write a test for the functionality you are developing, then you develop the functionality. That has two benefits. It limits the amount of functionality that you write. You write only what you need to pass that test, although that limitation is to be taken with a pinch of salt. It's a matter of skill and judgment, making sure you don't waste time trying to deal with possible future functionality which you are not sure that you are going to need. So it is a discipline which says you only do what is necessary to satisfy the user requirements and no more.

The second advantage you get is that it forces you to think in terms of the interface rather than in terms of the implementation. Writing tests first means you have to think about the interface, and how to make the code testable. As a result I think you get a better design; it certainly seems to be the case in our work.

The advantage of the engine approach is you have a huge chunk of code that is portable between two different Symbian OS platform implementations. The test first approach helps increase the amount of portable code that you get because of that focus on the interface.

Richard: So how do do the UI testing?

Charles: Initially the developer does it. We also have a local tester here, who writes and runs test scripts as necessary. But remember that we are trying to keep the UI component as small as possible. Also it is not appropriate for us to do heavy testing at the user interface, as that's really acceptance testing.

Richard: Does this talk of minimal UI mean most of your work is with engines, not full blown applications?

Charles: No, not at all. It is just that the UI code is usually small and most projects we have worked on have a substantial amount of engine functionality. It's where all the interesting functionality is. So in theory the UI on top isn't going to have a huge number of possible defects and most of those it does have are usually obvious and straight forward.

We have experimented with some UI testing, capturing and replaying keystrokes or even mouse movements. We even talked about integrating that with SymbianOSUnit but it's a bit of an unhappy mix.

Richard: Why do you say its an unhappy mix?

Charles: If SymbianOSUnit is testing the engine then it is effectively the application UI. So it is difficult to combine that with another UI because the flow of the UI is going to be interrupted by the SymbianOSUnit UI suddenly appearing and doing its thing.

So an automated UI tester is a different proposition. But I have to say I'm dubious about the benefits. I recall a workshop write-up by Brian Marrick which concluded that these sort of automated tests weren't actually very good for anything except smoke testing. Fundamentally there wasn't an awful lot of point in automating a test that is going to be run maybe five times in it's life. And the majority of real testing, serious testing is probably not done more than five times before a release. In this case the cost of automating is more than the cost of doing them five times, and that's not allowing for the cost of maintaining the automated tests.

Richard: So most of the testing you do outside the unit testing is manual, and you don't see any need or reason to change that?

Charles: We find the combination a automated unit tests and manual non unit tests works for us. The reality is that almost anything that you could automate as a UI test we can probably automate as an engine test. Generally running though all the aspects of a UI doesn't take a tester very long. Its also unlikely to come up with anything terribly exciting.

Richard: So are you saying is that if you put, say, a listbox on a user interface most of the problems with it will be because the engine. If the content of the listbox is wrong it almost certainly the engine causing the error not the listbox?

Charles: Exactly. Another benefit of this approach to testing is we can use it as the basis for our porting as well. So we can start with tests for the original on the other platform, port the software and the test and we have a solid way of knowing what we've got is a portable engine.

Richard: So you can write a set of tests for a pre-existing engine on one platform and use those as verification that the ported engine is still doing what it, supposed to do?

Charles: Exactly, and that way, because of how we have written the tests, we have stripped out the UI dependencies. I think this makes it easier to take the software, turn it into an engine, and port it. Also this approach means porting the software between Symbian OS platforms come almost for free, because engines can be so generic.

Richard: So how much of the company's work is porting?

Charles: I think we would consider our specialization to be porting - not so much between Symbian OS platforms; more porting from other platforms.

Richard: What platforms are you being asked to port to Symbian OS from?

Charles: Windows CE or Pocket PC are the main ones. Most interest has been in porting Windows CE code to UIQ and Series 60. Interestingly there has been very little interest in porting Palm applications - and I have no idea why.

Richard: In terms the applications you are porting, how would you characterize them?

Charles: I would characterize most of them as infrastructure, but there are also lots of stand-alone applications: we did a dictionary recently. We have seen a few business oriented products for the end user. We completed a major one for T-Mobile Traffic and have done quite a lot of work with Extended Systems. These are the applications that individuals will pay money for at the moment, because it helps their business.

SymbianOSUnit can be downloaded from www.symbianosunit.co.uk. For more information on Penrillian visit its web site: www.penrillian.com.