During an internal Open Space Technology in Valtech about one year ago (12th Feb 07) we had a discussion on software testing frameworks and test automation tools. I initiated this discussion by proposing a topic about how to make a framework that would hide the programming details of functional test automation and help focusing on writing test cases. We exchanged about existing commercial tools and their limits. We also discussed about frameworks, or wrappers, built on top of those tools to allow for better test creation. Valtech India has built such a wrapper, named LoFat, based on Excel spreadsheets and that can use QuickTestProfessional, Selenium, or SilkTest to execute the functional tests. My current client has also built such a wrapper on top of QuickTestProfessional. We finished the openspace discussion by sketching a simple and theorical architecture to support our approach. After this Open Space, we did not keep this discussion rolling.
Some time ago I came across an article from Jenitta Andrea titled Envisionning the Next Generation of Functional Testing Tools. That's a big piece of work that depicts our expectations towards test automation tools in a bright and comprehensive way. Following this article, Jenitta organised a workshop for the Agile Alliance in October 2007, gathering test automation tools experts to push the discussion further on. There are many reports of this workshop on participants' blogs, and a digest on InfoQ. A discussion group has been created on Yahoo to keep the discussion rolling. I think the consultants that participated in our open space discussion a year ago should subscribe to this group, because there are really interesting stuffs going on. As an example, an interesting web seminars is planned 19th of February, featuring Jenitta and Ward Cuningam, about next generation functional testing tools. Ward might present a new tool from Thoughtworks that supports the approach described by Jenitta. Well, by reading the description of the seminar, there is no question that Test-Driven Requirements and Functional Testing are now one single issue.
February 11, 2008
January 3, 2008
The (Toyota) way to go
Toyota has become the biggest car producer in 2007 overtaking General Motors who had been holding this position for 80 years. This is quite a big news! Toyota has been paving the way to efficient production management for decades, inventing lean and other manufacturing management concepts. Lean can be applied everywere and be transposed to many other businesses. Lots of companies look in this field to improve their process throughput while keeping stable resources.
In software development field, the definitive reference for lean thinking is the work of the Poppendiecks, which is gathered on their website. Many agile development models and practices are either inspired by lean thinking or verify lean principles:
In software development field, the definitive reference for lean thinking is the work of the Poppendiecks, which is gathered on their website. Many agile development models and practices are either inspired by lean thinking or verify lean principles:
- TDD implements a perfect "pull" organisation of the code and unit tests tasks
- eXtreme Programming has many practices verifying lean principles (Jacques Couvreur wrote a good article on how XP relates to the Toyota way),
- Continuous Integration and software factories are a way to make jidoka happen in the software development process
- Test-Driven Requirements (TDR) is an extension of TDD, applying lean principles to the other steps of the software development process (I made a presentation, in French, on this topic during the Valtech Days 2007)
- and so on...
Although it seems Agile Values were not directly inspired by Lean Thinking by the time of the Agile Manifesto, there are so many connections today in the software development field that it is difficult to differentiate either one. However, I think that Lean covers a broader area of interest. Software development organisations that seek to be "agile" today will probably seek to be "lean" in the coming years.
December 12, 2007
Continuous integration and functional tests
Libellés :
Continuous integration,
EN,
Tests
I just wrote an article for my current client, an investment bank, to explain to project managers what it takes to perform non-regression functional tests in a continuous integration process. When I say non-regression functional tests, I mean real functional tests that stimulates the GUI to perform tests. My main point is to argue that continuous integration is not exactly like continuous testing and that their respective objectives cannot be combined as easily as just integrating functional regression tests into the continuous integration process. This is actually what the managers ask for when we start automating the functional tests: "can we integrate these tests into the continuous integration process ?". Well, I could merely answer yes, for it is technically possible, but that answer would not be fair because they cannot integrate the functional tests into their specific continuous integration process. Let me explain this.
Usually the projects we are working with have a continuous integration process including the following steps: compile, build, code check (this step is usually referred to as "test" but I don't like this name), and (sometimes) package. So they are doing continuous integration and they are continuously checking the code, period. Problem is that they want to do continuous testing, I mean not only basic unit tests, they want to continuously perform functional non-regression tests on every build. And this is another story, because it usually requires a fully operational testing environment to test the application at runtime (which is needed for "real" functional tests). Between the time a build is ready for testing and the time it can actually be functionnaly tested, lots of operations are conducted manually, like deployment tasks, database and plateform setup, production data copy, startup of shared services, etc. Some managers don't know that they must fully automate the deployment in testing environment if they want to integrate fonctional regression tests in the continuous integration process. So first point was to stress on fully automating the deployment process.
But this is not the major issue to be addressed. Another point that I stressed was about not breaking the continuous integration benefits. Continous integration is based on quick feedback to the developers: the quicker the feedback, the better the fix. Anything that increases the delay between the time a developer checks his code in the source repository and the the time he gets notified of a failed build jeopardizes continuous integration success. Thing is that functional regression tests is time consuming, and comes after deployment (as noted above) which can be even more time consuming. So it does not look like a good option to automatically perform functional regression tests in the same workflow as continuous integration. Besides this, it does not make sense to functionally regression test every build
The solution that I advised is to create workflows that are triggered subsequently to provide some sort of increasing verification level while not breaking continuous integration. The target organization is to split the job into at least 2 workflows, but that can be more:
- fully automate the deployment process
- seperate functional regression tests in another workflow than the classical continuous integration workflow and trigger it at most once a
day
Usually the projects we are working with have a continuous integration process including the following steps: compile, build, code check (this step is usually referred to as "test" but I don't like this name), and (sometimes) package. So they are doing continuous integration and they are continuously checking the code, period. Problem is that they want to do continuous testing, I mean not only basic unit tests, they want to continuously perform functional non-regression tests on every build. And this is another story, because it usually requires a fully operational testing environment to test the application at runtime (which is needed for "real" functional tests). Between the time a build is ready for testing and the time it can actually be functionnaly tested, lots of operations are conducted manually, like deployment tasks, database and plateform setup, production data copy, startup of shared services, etc. Some managers don't know that they must fully automate the deployment in testing environment if they want to integrate fonctional regression tests in the continuous integration process. So first point was to stress on fully automating the deployment process.
But this is not the major issue to be addressed. Another point that I stressed was about not breaking the continuous integration benefits. Continous integration is based on quick feedback to the developers: the quicker the feedback, the better the fix. Anything that increases the delay between the time a developer checks his code in the source repository and the the time he gets notified of a failed build jeopardizes continuous integration success. Thing is that functional regression tests is time consuming, and comes after deployment (as noted above) which can be even more time consuming. So it does not look like a good option to automatically perform functional regression tests in the same workflow as continuous integration. Besides this, it does not make sense to functionally regression test every build
The solution that I advised is to create workflows that are triggered subsequently to provide some sort of increasing verification level while not breaking continuous integration. The target organization is to split the job into at least 2 workflows, but that can be more:
- the classical continuous integration workflow (compile, build) that is triggered as soon as a new piece of code is introduced in the repository. It is usually extended to check code (unit test, code rules, ...), but I advise to do so as long as it does not exceed 10-20 mins.
- It is better to add one more workflow to the process between continuous integration and regression workflow so that tasks like package, generate documentation, install, or even code verification could be removed from the continuous integration workflow and be included in this "intermediate" workflow. It should be triggered as soon as a successful continuous integration workflow has been performed (successful build).
- a regression workflow that includes deploying and regression testing (functionnaly) on the test plateform and is triggered at most once a day (say at night), most probably once a week depending on the development cycles and team size.
- fully automate the deployment process
- seperate functional regression tests in another workflow than the classical continuous integration workflow and trigger it at most once a
day
After releasing this article to my client, I started thinking that the name "continuous integration process" as used in the software programming field today is really a bad name. But this is another story.
December 4, 2007
First post
Libellés :
EN
This is a simple, yet meaningful title for this post, so that no one will expect too much out of it.
I finally created this technical blog, at least more technical than my personal blog, as an answer to Eric's prayers. I intend to drop here my thoughts and readings on subjects that are of some interest to me (well, this is the point of a blog after all), actually everything that relates more or less closely to software testing, namely:
- functional test automation
- test-driven requirements
- quality management
- continuous testing
- ... and many more topics that I don't even know yet
I finally created this technical blog, at least more technical than my personal blog, as an answer to Eric's prayers. I intend to drop here my thoughts and readings on subjects that are of some interest to me (well, this is the point of a blog after all), actually everything that relates more or less closely to software testing, namely:
- functional test automation
- test-driven requirements
- quality management
- continuous testing
- ... and many more topics that I don't even know yet
Subscribe to:
Posts (Atom)
