Difference between revisions of "System Testing"

Latest revision as of 03:48, 18 October 2014

Overview[edit]

System testing is an essential part of a good Quality Control program. For a good general discussion of system testing, visit the Wikipedia article.

The Joomla System Testing Working Group[edit]

There is a group of volunteers focused in System testing. Find all updates at System Tests Working Group

System Testing in Joomla![edit]

System testing in Joomla! uses an open-source program called Selenium. Selenium allows you to create a test script that can be run automatically. The test script actually runs Joomla! in a browser and does the same keystrokes and mouse clicks that a user would do. It can also test for conditions such as the value of text or the presence of HTML elements and report when a test fails or when there is an error in the program.

The benefits of system testing are:

System tests help you test that the application is working correctly from the point of view of a user.
System tests help clearly specify how the application should behave.
System tests help you test that changes made to one part of the application haven't created a bug somewhere else.
System tests can be run automatically (for example, each night) so that testing is done as the application is being developed.

System Testing versus Unit Testing[edit]

System Testing and Unit Testing are complementary test strategies. In Joomla!, Unit Testing is mainly used to test the framework classes (for example, the classes in libraries/joomla). Unit tests test that an individual method (also known as function) in a class does what it is supposed to do. For example, a unit test is used to test that the methods in the JString class work as expected. Unit testing provides confidence that the framework classes work as expected and allows you to refactor these classes (improve the code without changing the functionality) and still have confidence that they still work correctly.

System tests test that the application works correctly from the user point of view. For example, a system test can test something simple, for example, that you can create a new menu item for a single article and show the menu item on the site. Or a system test can test something more detailed, for example that the parameters for a module all work as expected.

Designing and creating system tests requires that you know how to use the application. It does not require that you understand how the program is written. Application knowledge is more important than programming knowledge, so system tests can be designed and written by people with less technical knowledge of PHP or the Joomla! framework.

Writing System Tests with Selenium[edit]

For detailed documentation on writing system tests, please see the article Writing System Tests for Joomla! - Part 1.

System Testing Joomla extensions with Codeception[edit]

Codeception is a Framework for testing written in PHP. In Joomla we are creating and documenting a best practice using Codeception to test the component com_localise. Find out more at: Testing Joomla Extensions with Codeception

@@ Line 1: / Line 1: @@
-== News and Updates ==
+== Overview ==
-07 27 Document created.
+System testing is an essential part of a good Quality Control program.
+For a good general discussion of system testing, visit the [http://en.wikipedia.org/wiki/System_Testing Wikipedia article].
-== Functional Testing ==
+== The Joomla System Testing Working Group ==
-Functional (or system) testing is an essential part of a good Quality Control program.
+There is a group of volunteers focused in System testing. Find all updates at [[System Tests Working Group]]
-For a good general discussion of functional testing, visit the [http://en.wikipedia.org/wiki/Functional_Testing Wikipedia article].
-=== Functional Testing in Open Source ===
+== System Testing in Joomla! ==
-Web applications have been notoriously difficult to do functional testing with. It has traditionally required a great deal of time and effort. This can be a special challenge for open source projects because testing is often not very popular and designing tests that can be executed in a systematic way even less so. For a complicated application like Joomla!, this means that although we might do testing, we very seldom are able to perform extensive testing on each release that covers even a fraction of the elements in the CMS.
+System testing in Joomla! uses an open-source program called [http://www.seleniumhq.org Selenium]. Selenium allows you to create a test script that can be run automatically. The test script actually runs Joomla! in a browser and does the same keystrokes and mouse clicks that a user would do. It can also test for conditions such as the value of text or the presence of HTML elements and report when a test fails or when there is an error in the program.
+The benefits of system testing are:
+* System tests help you test that the application is working correctly from the point of view of a user.
+* System tests help clearly specify how the application should behave.
+* System tests help you test that changes made to one part of the application haven't created a bug somewhere else.
+* System tests can be run automatically (for example, each night) so that testing is done as the application is being developed.
-Relatively recently, a project called Selenium has arrived on the scene.  This project makes performing automated functional testing of web applications possible. This means that it is possible to devise a test and have it performed in an automated way without the requirement of somebody sitting in front of a browser and being a click monkey.
+== System Testing versus Unit Testing ==
+System Testing and Unit Testing are complementary test strategies. In Joomla!, Unit Testing is mainly used to test the framework classes (for example, the classes in libraries/joomla). Unit tests test that an individual method (also known as function) in a class does what it is supposed to do. For example, a unit test is used to test that the methods in the JString class work as expected. Unit testing provides confidence that the framework classes work as expected and allows you to refactor these classes (improve the code without changing the functionality) and still have confidence that they still work correctly.
-The benefits of functional testing are:
+System tests test that the application works correctly from the user point of view. For example, a system test can test something simple, for example, that you can create a new menu item for a single article and show the menu item on the site. Or a system test can test something more detailed, for example that the parameters for a module all work as expected.
-* Functional tests help highlight cases where changes in one element of the system might cause breakage in other, unexpected areas.
-* Functional tests help clearly specify how the application should behave.
-=== Functional Testing versus Unit Testing ===
+Designing and creating system tests requires that you know how to use the application. It does not require that you understand how the program is written. Application knowledge is more important than programming knowledge, so system tests can be designed and written by people with less technical knowledge of PHP or the Joomla! framework.
-In order to fully test an application, both functional and unit testing are essential. Each approach has their benefits and weaknesses. Unit testing is very good for isolating small chunks of code and ensuring that they are behaving as expected. Functional testing, on the other hand, is geared towards looking at the system as a whole and ensuring that the units are behaving together in a way that achieves the desired effect.
-The advantages of functional testing is that functional tests are easier to understand and conceptualize by people who are unfamiliar with the underlying code. With Selenium, tests can be designed by people who are not overly familiar with PHP - they only need to understand the application and what results should be expected.
+== Writing System Tests with Selenium ==
+For detailed documentation on writing system tests, please see the article [[Writing System Tests for Joomla! - Part 1]].
-The downside of functional testing is that it will only tell you that something broke. It isn't as obvious with functional testing exactly what went wrong or where the problem is.
+== System Testing Joomla extensions with Codeception ==
+Codeception is a Framework for testing written in PHP. In Joomla we are creating and documenting a best practice using Codeception to test the component com_localise. Find out more at: [[Testing Joomla Extensions with Codeception]]
-==== Test Objects ====
-If the purpose of unit tests is to isolate a module of code, then the purpose of functional tests is to evaluate the entire system. This is much easier for those unfamiliar with what happens under the hood of Joomla! because one is not required to understand what happens during a Joomla! page request. A functional test might evaluate, for example, if unpublishing an article works properly. Therefore, to design the test, it is only necessary to figure out how to change an article from Published to Unpublished, and how to verify that the article was actually unpublished.
-=== Functional Testing Testing in Joomla! ===
-I have recently been building a functional testing architecture that will make it possible to write tests once, and to execute those tests on a variety of platforms and clients. I currently have infrastructure in place that makes it possible to run one command and execute functional tests in one browser on one host system. I have designed the host system in such a way that it will run on 6 different PHP versions (stock Ubuntu PHP 5 and self-compiled PHP 4.3, 4.4, 5.0, 5.1 and 5.2) and 2 different web servers (Apache and Lighttpd).  These are all currently serving from the same database.
-The future steps will be to make it possible to run on multiple host systems (the requirement for this is to generalize the code that resets the database state after the tests are run) and to make it possible to use different clients (i.e. different systems with different web browsers). These steps should be fairly trivial, but I have not yet had time to focus on them.
-This will eventually make it possible to run the designed tests on every major browser and on multiple server platforms (various PHP versions, various configurations, various operating systems and various web servers).
-==== Writing Functional Tests ====
-Selenium IDE makes it fairly easy to write functional tests. It is a Firefox extension that operates as an Integrated Development Environment for creating functional tests. In essence, it records your clicks as you navigate in your browser and encodes them into a series of commands that can control the browser.
-==== Running Functional Tests ====
-== How to Get it Running ==
-== Troubleshooting ==
-== Frequently Asked Questions ==
-==See also==
 [[Category:Development]]
+[[Category:Testing]]
+[[Category:Bug Squad]]
+[[Category:Automated Testing]]