Stratégie d'automatisation des tests: un filet de pêche ou une fusée ?

Jusqu'à récemment j'avais l'habitude d'utiliser la métaphore du filet de pêche pour illustrer la stratégie d'automatisation de tests que l'on applique le plus couramment sur les projets agiles. Cette stratégie est illustrée par la pyramide de Mike Cohn: une forte base de tests unitaires, un étage plus restreint de "spécifications exécutables", et un petit chapeau de tests "bout-en-bout". La métaphore du filet me vient d'un article que j'ai lu il y a quelques années *, qui parlait plutôt d'un filet de sécurité, et que j'ai tenté de m'approprier et remettre à la sauce pêcheur. Ça donne quelque chose comme ça:

"Une stratégie d'automatisation de tests est comme un filet de pêche, plus le filet est grand et ses mailles petites, plus la capacité à attraper des poissons est forte. NB: pour les esprits les moins vifs, les poissons représentent les anomalies. Le filet se doit d'être suffisamment solide pour permettre une utilisation pérenne, et suffisamment bien attaché au bateau pour ne pas perdre sa récolte.

Les tests unitaires représentent les petites mailles de votre filet à attraper des anomalies. Il en faut beaucoup pour couvrir une surface suffisante, et il faut qu'il soit bien unitaire pour que la maille de base soit suffisamment fine, sinon on laisse passer les petits poissons. Ce petit maillage est fait d'un matériel souple qui s'adapte à son environnement: les courants, les bancs de poissons, la résistance de l'eau. Le test unitaire l'est également: les outils et les pratiques (TDD) doivent permettre de le faire évoluer facilement dans son environnement (le code) . Tout comme les petites mailles d'un filet de pêche, un test unitaire peut "casser" sans remettre significativement en question votre capacité à attraper les poissons, il y a beaucoup d'autres mailles dans votre filet. De plus il est facile à réparer pour peu que le pêcheur consciencieux le fasse rapidement.

Ce filet de petites mailles nous paraîtrait presque suffisant. Mais voilà, va-t-il résister aux gros poissons ? Que se passe-t-il si un thon ou un requin blanc se pointe ? Et si l'on accroche le fond ? On risque de perdre tout simplement un gros morceau de son filet, ou d'en jeter une partie parce que "ça ne sert plus à rien". Il faut donc renforcer ce filet avec une structure plus rigide, fait d'un maillage plus large mais dans un matériel plus solide. Cette structure représente les spécifications exécutables. Les tests y sont d'ordre fonctionnel ou métier, ils sont moins fins au regard du code, mais plus solides car ils portent sur des vérifications plus stables dans le temps (le métier). Le code est lui soumis à un remaniement incessant et les vérifications des tests unitaires sont donc moins stables. Le métier, lui, ne change pas tous les jours, le cycle d'évolution est plus long. Néanmoins, le matériel utilisé reste souple pour ne pas subir négativement les contraintes de son environnement. Il faut également trouver le bon dosage de ce maillage afin de ne pas alourdir inutilement notre filet mais de le rendre suffisamment solide. De la même façon les tests des spécifications exécutables doivent représenter les exemples clés, qui facilitent la compréhension des exigences.

Alors, a-t-on maintenant un filet de pêche suffisant ? Non bien sûr. Mais que lui manque-t-il pour être utilisé ? Des fixations pardi! Des câbles, des cordes, des poulies, bref des attaches qui lui permette d'être relié au bateau et d'être manipulé dans son environnement cible, la mer! Ces attaches sont faites d'un matériel lourd et rigide et paraissent solides à première vue, mais elles sont aussi fragiles que les petites mailles car elles centralisent l'ensemble des contraintes de l'environnement. Ces attaches sont les tests de bout-en-bout automatisés. On teste ici l'application déployée dans son environnement (presque) réel. Ces tests ne sont pas nombreux mais indispensables. On teste ici ce qui ne peut pas l'être dans les 2 maillages précédents, comme l'interface graphique ou la performance par exemple."

Finalement je suis revenu à quelque chose de plus simple il y a quelques jours lors d'une soutenance: une métaphore basée sur le principe de la fusée à étage. C'est plus proche de la pyramide et cela s'explique plus rapidement que le filet de pêche. Chaque étage contribue à la mise en orbite du satellite, mais chacun a un rôle bien précis que les autres ne sauraient endosser. Plus on monte dans les étages, plus sa fonction se sophistique et se rapproche de l'objectif final. J'aime également le fait que les étages de la fusée servent directement un objectif métier, la mise en orbite du satellite, alors que le filet de pêche sert plutôt un objectif compensatoire à un système défaillant, la pêche aux anomalies. C'est plus proche de la vision moderne des tests que j'essaye de transmettre.

Et pour vous, l'automatisation des tests est-elle un filet de pêche ou une fusée à étage ?

*Brushing Up On Functional Test Effectiveness, Jennitta Andrea, Better Software Magazine Nov/Dec 2005

Outils de tests open-source

Les slides de ma présentation au dernier afterwork Valetch sont sur slideshare:

Outils de tests open-source

Qu'avez vous testé aujourdhui ?

C'était aux Valtech Days en Novembre dernier, vous pouvez retrouver la présentation en français et en vidéo sur Valtech TV. J'ai mis la présentation sur slideshare:

Qu'avez vous testé aujourdhui ?

What have you tested today ?

I will be presenting a lightning talk at the Valtech Days 2008 in Paris, the 21st of October.
It will be a "refactored" version of the lightning talk I made at the aa-ftt workshop the 4th of August just before the Agile 2008 conference. There is a low-fi shooting of this talk on google video, thanks to Elisabeth Hendrickson:



One of the attendees also wrote a blog post referring to the talk.

Since then, I went on thinking more and more on how to present these values and the rationale behind them. One of the consultant in Valtech also found a pretty good name for this talk: "What have you tested today ?". I think this question nicely wraps up the 3 values that I emphasize. There is the name of my lightning talk! Come the 21st of October to know more.

Test-Driven Requirements versus functional test automation

I am updating the case study of the TDR course that I recently developed for Valtech Training. During the first sessions of the course, it turned out that the trainees had a difficult time understanding the differences between Test-Driven Requirements and GUI Functional Tests. So I decided to improve the case study with real examples of automated GUI functional tests.

The case study is a simple webapp that is supposed to be an online banking system (it is of course greatly simplified). I am the product owner of this app, and one Valtech consultant developed it for me (because I can hardly code something now). I built the functional specifications with FitNesse, using test tables to interact with the developer and discuss the functional requirements. In turn, he used the fixtures in a TDD manner to drive his development. Below is an example of the functional requirement that describes an immediate money transfer :

As you can see, we mixed textual descriptions in a use-case style with an example to illustrate the description. The example is actually a test, specifying preconditions, actions, and verifications. We used a RowEntry fixture to specify the preconditions, i.e. to populate the database with test data, then we used an Action fixture to describe the actions, and a Row fixture to verify the results. The example is in french. If you cannot understand french, you just need to know that we spent some time to choose appropriate fixture names, so that the test tables do not feel like automated tests or freaky technical stuff inserted in plain text specifications. We really wanted that the test tables read like real examples, so we chose fixture names like "There exists the following accounts", or "Consult account details", which read more naturally than freaky names we often see with FIT/FitNesse examples "com.myapp.test.fixtures.ConsultAccount". We also put some effort to mature a domain specific testing language (DSTL), so that we could reuse the expressions for precondition or verification with different requirements. For example, we used a lot the fixture "There exists the following accounts" to set up initial test data.

Now let's move to the GUI testing. I made use of a Selenium-FitNesse bridge (the like of webtest) to integrate my automated GUI tests with my specifications. That bridge provides a testing mini-language to allow non-technical people like me writing automated test cases without knowing Selenium scripting. I created a couple of cases to test the webapp, below is an example of an automated test case that traces to the specification given above:



In this example, we see that the flow of actions and verifications is overwhelmed with interface details and strategic instructions (wait for page to reload, etc). The testing mini-language gives a better readability than the equivalent Selenium script would do (some test specialists like to call this "keyword-driven testing"), but still, the test case is not good at serving the specification side. Both types of tests are complementary and should co-exist. The first FIT-like type serves to drive and support the specification process, it also helps in building a regression safety net; the second Selenium-like type serves to test the application and make sure every piece of the software properly fits together. In the end, we should get something like this:

The screenshot above shows the organisation of my FitNesse wiki for the webapp. I created a topmost suite that contains two "sub-suites": one is the specification artefact, containing the FIT-like tests, one is the acceptance test artefact, containing the Selenium-like tests. I made use of links to trace tests from the second artefact to the first, so that I can have an idea of impacted GUI tests when I change the specifications. I can run seperately the specification tests and the acceptance tests, or within a single click. It should be clearer now that TDR and GUI fonctional testing serves a different purpose. I used FitNesse and Selenium because they are open-source, but the same kind of approach can be followed using test management proprietary tools like QualityCenter in combination with test robots and requirements management tools. It's not going to be the same price though.

Next generation testing tools

During an internal Open Space Technology in Valtech about one year ago (12th Feb 07) we had a discussion on software testing frameworks and test automation tools. I initiated this discussion by proposing a topic about how to make a framework that would hide the programming details of functional test automation and help focusing on writing test cases. We exchanged about existing commercial tools and their limits. We also discussed about frameworks, or wrappers, built on top of those tools to allow for better test creation. Valtech India has built such a wrapper, named LoFat, based on Excel spreadsheets and that can use QuickTestProfessional, Selenium, or SilkTest to execute the functional tests. My current client has also built such a wrapper on top of QuickTestProfessional. We finished the openspace discussion by sketching a simple and theorical architecture to support our approach. After this Open Space, we did not keep this discussion rolling.

Some time ago I came across an article from Jenitta Andrea titled Envisionning the Next Generation of Functional Testing Tools. That's a big piece of work that depicts our expectations towards test automation tools in a bright and comprehensive way. Following this article, Jenitta organised a workshop for the Agile Alliance in October 2007, gathering test automation tools experts to push the discussion further on. There are many reports of this workshop on participants' blogs, and a digest on InfoQ. A discussion group has been created on Yahoo to keep the discussion rolling. I think the consultants that participated in our open space discussion a year ago should subscribe to this group, because there are really interesting stuffs going on. As an example, an interesting web seminars is planned 19th of February, featuring Jenitta and Ward Cuningam, about next generation functional testing tools. Ward might present a new tool from Thoughtworks that supports the approach described by Jenitta. Well, by reading the description of the seminar, there is no question that Test-Driven Requirements and Functional Testing are now one single issue.

Continuous integration and functional tests

I just wrote an article for my current client, an investment bank, to explain to project managers what it takes to perform non-regression functional tests in a continuous integration process. When I say non-regression functional tests, I mean real functional tests that stimulates the GUI to perform tests. My main point is to argue that continuous integration is not exactly like continuous testing and that their respective objectives cannot be combined as easily as just integrating functional regression tests into the continuous integration process. This is actually what the managers ask for when we start automating the functional tests: "can we integrate these tests into the continuous integration process ?". Well, I could merely answer yes, for it is technically possible, but that answer would not be fair because they cannot integrate the functional tests into their specific continuous integration process. Let me explain this.

Usually the projects we are working with have a continuous integration process including the following steps: compile, build, code check (this step is usually referred to as "test" but I don't like this name), and (sometimes) package. So they are doing continuous integration and they are continuously checking the code, period. Problem is that they want to do continuous testing, I mean not only basic unit tests, they want to continuously perform functional non-regression tests on every build. And this is another story, because it usually requires a fully operational testing environment to test the application at runtime (which is needed for "real" functional tests). Between the time a build is ready for testing and the time it can actually be functionnaly tested, lots of operations are conducted manually, like deployment tasks, database and plateform setup, production data copy, startup of shared services, etc. Some managers don't know that they must fully automate the deployment in testing environment if they want to integrate fonctional regression tests in the continuous integration process. So first point was to stress on fully automating the deployment process.

But this is not the major issue to be addressed. Another point that I stressed was about not breaking the continuous integration benefits. Continous integration is based on quick feedback to the developers: the quicker the feedback, the better the fix. Anything that increases the delay between the time a developer checks his code in the source repository and the the time he gets notified of a failed build jeopardizes continuous integration success. Thing is that functional regression tests is time consuming, and comes after deployment (as noted above) which can be even more time consuming. So it does not look like a good option to automatically perform functional regression tests in the same workflow as continuous integration. Besides this, it does not make sense to functionally regression test every build

The solution that I advised is to create workflows that are triggered subsequently to provide some sort of increasing verification level while not breaking continuous integration. The target organization is to split the job into at least 2 workflows, but that can be more:

• the classical continuous integration workflow (compile, build) that is triggered as soon as a new piece of code is introduced in the repository. It is usually extended to check code (unit test, code rules, ...), but I advise to do so as long as it does not exceed 10-20 mins.
• It is better to add one more workflow to the process between continuous integration and regression workflow so that tasks like package, generate documentation, install, or even code verification could be removed from the continuous integration workflow and be included in this "intermediate" workflow. It should be triggered as soon as a successful continuous integration workflow has been performed (successful build).
• a regression workflow that includes deploying and regression testing (functionnaly) on the test plateform and is triggered at most once a day (say at night), most probably once a week depending on the development cycles and team size.

In short, my answer to the question was:
- fully automate the deployment process
- seperate functional regression tests in another workflow than the classical continuous integration workflow and trigger it at most once a
day

After releasing this article to my client, I started thinking that the name "continuous integration process" as used in the software programming field today is really a bad name. But this is another story.