Code Metrics and Cyclomatic Complexity

Code Metrics are a tool for static testing it has a section about it in Section 3.3.2 in chapter 3 of the ISTQB book where they also briefly introduce the concept of Cyclomatic Complexity.  Since questions on Cyclomatic Complexity and code metrics are found in practice tests I thought it would be good to know.
Code metrics are the measurement scales and methods used in software testing, and Cyclomatic complexity is one example of a code metric used in static testing.  When performing static code analysis, code metrics can be used to calculate numerous attributes of the code such as comment frequency, depth of nesting, cyclomatic number and number of lines of code. Code metrics can not only compute these attributes but can also be done as the design and code are being created and as changes are made to a system, to see if the design or code is becoming bigger, more complex and more difficult to understand and maintain.
Cyclomatic complexity is a code metric used on static testing when the testers take all of the expected statements and decisions and compile them into a diagram or map of the system.  It involves an equation defined by the ISTQB book as L-N+2P. L is the number of edges or lines connecting nodes, N is the number of nodes or steps in the process flow reached when decisions are made, P is the number of connected components (usually the entire process is connected and therefore this is usually equal to 1) and example of this is below.



In this example there are 9 edges or lines, 8 nodes, and the entire thing is connected meaning P=1
the cyclomatic complexity equation would look like: 9-8+2*1=3

Designing Tests

For this post I have decided to write about how to design tests.  Testers design tests whether they are using Static or dynamic testing techniques.  To design a test you need 3 things according to ISTQB test conditions, test cases and test procedures (Scripts).  Each of these things are specified in their own documents created from information given by the developers and the tester's own interpretations of the provided info.  Testing is also designed and executed with varying degrees of formality depending on the customer's or testing team's preferences.


The designing process usually begins with the testers looking through something that they can derive test info from which is also known as a test basis.  A test basis can be many different things from an experienced user's knowledge, a business process, system requirements, specifications or the code.  Through test basis we find test conditions or the things we could test, because test conditions could come from anything they are usually vague.


Unlike test conditions, test cases are more specific.  Test cases will mean entering specific inputs into a system, and will also require some understanding of how the system works.  This mean knowing what the system should do and therefore the correct behavior of the system which is also referred to as a Test Oracle.  After choosing an input the tester needs to figure out what the expected result is and put it into the test case.  Expected results are what appears on the screen in response to the input values as well of changes to data or states and consequences.


The next and last step of the process is to put the test cases into a way that makes sense and the steps needed to run the test.  This is a test procedure or test script.  These procedures are then put into an execution schedule to determine which procedures are run first and whom runs them.


There are also many different categories of design techniques that I will get into later.

White Box vs. Black Box testing

Black box and White box testing are mentioned a lot in Chapter 4 of the ISTBQ book.  The text book definition of black box in the ISTQB book states  that it is "testing either functional or non-functional, without reference to the internal structure of the component or system".  Basically Black box testing is when a tester knows what the software does from having access to things like requirements, but has no knowledge as to how the software does these things, or anything about the branching logic.  Black box testing tends to involve a lot of exploratory testing as the tester goes through the system to understand how the system works.  Techniques of black box testing include but are not limited to equivalence partitioning, and boundary value analysis, the use of data tables, and state transition testing


Meanwhile White box testing is defined as "Testing based on an analysis of the internal structure of the component or system".  In other words this type of testing is when the tester knows all the details about not only what the software supposed to do, but how it works and does what it's supposed to do.  These kinds of tests allow tester to increase their test coverage by finding new tests that haven't been created.  It uses techniques like statement coverage, decision coverage, and branch coverage.


Black box testing is good when the tester is starting to run tests on a software they know little about, while white box testing is good for going more in depth into understanding the system.

Static Vs. Dynamic Testing

In the ISTQB book one of the first things mentioned in Chapter 3 is dynamic testing and static testing.


Dynamic testing involves executing a software with a set of given input values and then checking the outputs values and comparing them to the expected results.  Static testing involves using either a set of tools or manually examining software work products such as user guides, test cases, and scenarios but not actually executing them, reviews are actually an example of static testing at work.  Basically dynamic testing actually tests the software code while static testing looks through the documents.


Static and dynamic testing techniques are complimentary due to the fact that dynamic testing works on areas static testing does not and visa versa.  When they are used together they can greatly improve the quality of testing and help find a lot more defects.  Static testing methods are good for finding deviations from standards, missing requirements, design defects, non-maintainable code and inconsistent interface specifications.  Dynamic testing is good for finding defects in the code itself, as well as the quality of the software.  Static testing is good in the early stages of testing and in reviews while dynamic testing is good for the main executions.


Since a lot of the info in the ISTQB book is about the dynamic aspects of testing I'll go more in depth with static testing According to ISTQB the use of static testing, e.g. reviews, on software work products has various advantages:
Since static testing can start early in the life cycle, early feedback on quality issues can be established, e.g. an early validation of user requirements and not just late in the life cycle during acceptance testing.
By detecting defects at an early stage, rework costs are most often relatively low and thus a relatively cheap improvement of the quality of software products can be achieved.
Since rework effort is substantially reduced, development productivity figures are likely to increase.
The evaluation by a team has the additional advantage that there is an exchange of information between the participants.
Static tests contribute to an increased awareness of quality issues.

Formal Review Process

Since Formal reviews seem to be more elaborate and  complex than informal reviews I have decided to go more in depth.  While informal reviews are more common due to less structure and planning Formal reviews happen less often, usually every few weeks as opposed to informal review that can happen every few days, and have a lot more structure and planning involved.
 Typical formal reviews have 6 steps:
1. Planning
This is the beginning of the review process and starts when the author of the software requests a review from the moderator who's job is to lead the inspection.  The moderator then has to schedule the start and end dates of the review, and work with the author to come up with test documents including entry criteria.  The document can't be too long as most people can't retain a large number of pages of info.  during this phase the people who will be involved in the review will be decided on, and from these chosen reviewers the moderator hands out duties and tasks to them.


2. Kick-off
This step is stated by ISTQB to be optional but is highly recommended as it gives the review team members a degree of understanding about the document.  During this phase reviewers are given a short intro on the objective of the review and document.  The relationship between the review document and other documents used as sources are explained as well.


3. Preparation
This part of the review is where the individual reviewers gather up all of their data and materials they have compiled in testing to work on the document.  In this phase the reviewers identify defects, questions and comments and all the issues are recorded.  Using checklists in this phase is a way to make the testing more efficient.  The efficiency can also be checked by number of pages checked in an hour which varies greatly depending on the number of pages, and the complexity of the software.


4. Review Meeting
This is the actual review, where the review team gets together to talk about the defects that they have found as well as ask questions and collaborate with one another.  Bugs talked about in the review is recorded by scribe.  The severity of every defect is also recorded by the scribe and there are 3 levels of severity Critical, Major, and Minor


5. Rework
This is the phase where the reviewers try to retest and improve the documents step by step. It's up to the author to decided if the defects need to be fixed.  Then the author updates the software and the document.


6. Follow up
This is done by the moderator and is when they determine if the changes done by the author to the software and document are adequate, and the bugs have been removed from the affected areas.




The tasks of a formal review are divided up to include the author 9program writer), a moderator (inspection leader), a scribe (recorder), the reviewers (testers), and a manager (the boss).

Formal Vs Informal Review

Section 3.2 of the ISTQB book is about the test review process.  There are 2 types of review, informal review, and formal review.  While each one has it's differences they both share the goal of finding defects, and to help fellow testers gain a better understanding of the software, and test requirements, and cases.


Informal reviews are the more simple and commonly used of the 2 types. They can be done at anytime during the early stages of testing, it can only take 2 people to do an informal review, and they can be conducted with little planning, they don't required to be led by a trained moderator.  There are many informal testing types but one thing they all share in common is that they are undocumented meaning that no one needs to record a list of defects, and there is no compilation of test cases and defects given out to the review members.


Formal reviews on the other hand are a lot more complex.  They are usually done in the later stages of testing and because of that they require more people.  They have six main steps:
1. planning
2. kick-off
3. preparation
4. review meeting
5. rework
6. follow up
The formal meetings also have a number of roles that are allocated in the planning phase.  The roles are the Moderators, Scribes, Authors, Reviewers, and Managers.  Like informal reviews there are different types of formal testing such as technical reviews, and walkthroughs, and not every role is present in every review type, for instance, walkthroughs have the authors explaining the software and leading the review, while technical reviews have trained moderators leading the review instead and the authors are usually absent.  Scribes and reviewers however are present in most if not all formal review types.

The Ed Kit Principle

When I took a practice test for ISTQB certification I came across the question asking 'The quality and effectiveness of software testing are primarily determined by the quality of the test processes used is stated in' a) Bill Hetzel Principle b) Ed Kit principle c) IEEE 829 d) IEEE 8295 The correct answer was the Ed Kit Principle, and since I didn't know what that was I decided to do some research. Basically the Ed Kit principle states that "The quality and effectiveness of software testing are primarily determined by the quality of the test processes used.".  So in other words the way that a tester's testing methods are determines how good the tests they run are. Like the Bill Hetzel Principle the Ed Kit Principle isn't really mentioned by name in the ISTQB software testing book despite it appearing once in a while on online practice tests.  However that doesn't mean it isn't present in the book, and in fact has a significant presence.  If you have a copy of the book the principle is referenced a lot in at least in section 1.4 Fundamental Test Process.  Just remember if you're looking for the principle by name you will not be able to find it, just remember that the principle is about how quality of the test processes determine the quality of the tests.