Test Strategies

Test Strategies A.K.A Test Approaches are used during the planning phase of testing along with Test plans and Estimation techniques.  This factor of testing is under the control of testers and test leaders, and is one of the powerful factors in test effort and in the accuracy of the test plans and estimates.  (ISTQB, pg 142).  There are several major types of strategies some are more preventive some are more reactive:
Analytical: Test strategies that use some formal or informal analytical technique usually during the requirements and design stage of the project.


Model Based: These strategies tend to use the creation or selection of some formal or informal model for critical system behaviors, usually during the requirements and design stages of the project.


Methodical: These tend to rely on the adherence to a pre-planned, systematized approach that has been developed in-house, assembled from various concepts developed in- house and gathered from outside, or adapted significantly from outside ideas and may have an early or late point of involvement for testing.


Process: These strategies share a reliance upon an externally developed approach to testing, often with little or no customization and may have an early or late point of involvement for testing.


Dynamic: These strategies focus on finding as many defects as possible during test execution and adapting to the realities of the system under test as it is when delivered, and they typically emphasize the later stages of testing.




Consultative Or directed: These rely on a group of non-testers to guide or perform the testing effort and typically emphasize the later stages of testing simply due to the lack of recognition of the value of early testing.




Regression-Averse: These strategy types use a set of procedures (usually automated) that allow them to detect regression defects. A regression-averse strategy may involve automating functional tests prior to release of the function, in which case it requires early testing, but sometimes the testing is almost entirely focused on testing functions that already have been released, which is in some sense a form of post- release test involvement.

Estimation Techniques

Estimation Techniques along with Test Plans and Test Strategies are done throughout the Planning phase of testing.  First there are 2 techniques that ISTQB mentions, consulting people and analyzing metrics.  The consulting technique involves working with experienced staff members and testers drawing upon their collective wisdom to develop structure for testing the project. Then collaborate to understand, tasks, effort, duration, dependencies, and resource requirements that the tests will need to be successful.


Analyzing metrics can be as simple or sophisticated as the tester can make it. ISTQB states the easiest approach asks, 'How many testers do we typically have per developer on a project?'
A more reliable method classifies the project in terms of size (small, medium or large) and complexity (simple, moderate or complex) then sees the average length of projects of a particular size and complexity have taken in the past to test.
Another reliable yet easy approach  that ISTQB mentions is to look at the average effort per test case in similar past projects and to use the estimated number of test cases to estimate the total effort. The more sophisticated methods that ISTQB mentions involve building mathematical models in a spreadsheet that look at historical or industry averages for certain key parameters number of tests run by tester per day, number of defects found by tester per day, etc. and then plugging in those parameters to predict duration and effort for key tasks or activities on your project. The tester-to-developer ratio is an example of a top-down estimation technique, in that the entire estimate is derived at the project level, while the parametric technique is bottom-up, at least when it is used to estimate individual tasks or activities.
ISTQB testers prefer to start by drawing on the team's wisdom to create the work-breakdown structure and a detailed bottom-up estimate. We then apply models and rules of thumb to check and adjust the estimate bottom-up and top-down using past history. This approach tends to create an estimate that is both more accurate and more defensible than either technique by itself.
Even the best estimate must be negotiated with management however. Negotiating sessions exhibit amazing variety, depending on the people involved. However, there are some classic negotiating positions. It's not unusual for the test leader or manager to try to sell the management team on the value added by the testing or to alert management to the potential problems that would result from not testing enough. It's not unusual for management to look for smart ways to accelerate the schedule or to press for equivalent coverage in less time or with fewer resources. In between these positions, you and your colleagues can reach compromise, if the parties are willing. Our experience has been that successful negotiations about estimates are those where the focus is less on winning and losing and more about figuring out how best to balance competing pressures in the realms of quality, schedule, budget and features.

IEEE 829 template

As promised in the last post I will write about the IEEE 829 template which is the ISTQB standard template for test plans.


The book seemed kind of vague to me as to what it was about but below is what the book had on the test plan template and what should be in it.


IEEE 829 STANDARD TEST PLAN TEMPLATE


Test plan identifier                             Test deliverables
Introduction                                        Test tasks
Test items                                            Environmental needs
Features to be tested                            Responsibilities
Features not to be tested                      Staffing and training needs
Approach                                             Schedule
Item pass/fail criteria                           Risks and contingencies
Suspension and resumption criteria     Approvals


I might update this post when I figure out more about the IEEE 829 standard Test Plan Template.

Test Plans

Test plans Estimates and strategies are 3 testing topics that go together and are prepared concurrently during the planning phase of testing.  I will start by talking about test plans.  There are many different definitions for test plan but the ISTQB book states a test plan is mainly the project plan for the testing work to be done.  Test plans differ from Test Design Specification, Test Cases or a set of Test procedures in the fact that less description is used in Test Plans.


ISTQB states that there are 3 main reasons as to why testers write test plans:
1. Writing a test plan guides our thinking. ISTQB explained this reason's logic as 'if we can explain something in words, we understand it. If not, there's a good chance we don't.'
Writing a test plan forces us to confront the challenges that await us and focus our thinking on important topics.


2. The test planning process and the plan itself can be a good excuse for a tester to communicate with other members of the project team, testers, peers, managers and other stakeholders.  This is an important aspect since collaboration can help improve a tester's understanding of a system and their testing overall.


3. The test plan also helps us manage change. During early phases of the project, as we gather more information, we revise our plans. As the project evolves and situations change, we adapt our plans. Written test plans give us a baseline against which to measure such revisions and changes. Furthermore, updating the plan at major milestones helps keep testing aligned with project needs. As we run the tests, we make final adjustments to our plans based on the results. You might not have the time or the energy to update your test plans


ISTQB suggests to use a template when writing test plans to help remember important challenges.  ISTQB has it's own standard template called IEEE829 which I will write about in my next post.

Practice Quiz Links

Here are some websites that have practice quizzes for the ISTQB certification exam.
Most of these have full practice tests meant to simulate the number of tests on the real test.  There are supposed to be 40 questions on the real test and you need to get at least 65% to pass.


http://istqb.patshala.com/
This site has several full practice tests, as well as tests for the individual chapters for the ISTQB book although they are labeled as the main concepts that each chapter teaches as opposed to Chapter 1, 2, 3, 4, 5, 6. 
CH 1: Fundamentals of Testing
CH 2: Testing through the software life cycle
CH 3: Static Techniques
CH 4: Test design techniques
CH 5: Test Management
CH 6: Tool support for testing
These exams don't have a time limit so there is plenty of time to take notes or look for the answers if you're not sure what the right answer is.  However on the real test you won't be allowed to look for answers so you probably shouldn't do it too often for practice tests.


http://www.testingexcellence.com/istqb-quiz/istqb-foundation-practice-exam-1/
This has 2 full practice exams and a bunch of other related practice tests for stuff like testing fundamentals, management, and white box testing.  The practice exams aren't timed on this site so you can take your time.

Code Metrics and Cyclomatic Complexity

Code Metrics are a tool for static testing it has a section about it in Section 3.3.2 in chapter 3 of the ISTQB book where they also briefly introduce the concept of Cyclomatic Complexity.  Since questions on Cyclomatic Complexity and code metrics are found in practice tests I thought it would be good to know.
Code metrics are the measurement scales and methods used in software testing, and Cyclomatic complexity is one example of a code metric used in static testing.  When performing static code analysis, code metrics can be used to calculate numerous attributes of the code such as comment frequency, depth of nesting, cyclomatic number and number of lines of code. Code metrics can not only compute these attributes but can also be done as the design and code are being created and as changes are made to a system, to see if the design or code is becoming bigger, more complex and more difficult to understand and maintain.
Cyclomatic complexity is a code metric used on static testing when the testers take all of the expected statements and decisions and compile them into a diagram or map of the system.  It involves an equation defined by the ISTQB book as L-N+2P. L is the number of edges or lines connecting nodes, N is the number of nodes or steps in the process flow reached when decisions are made, P is the number of connected components (usually the entire process is connected and therefore this is usually equal to 1) and example of this is below.



In this example there are 9 edges or lines, 8 nodes, and the entire thing is connected meaning P=1
the cyclomatic complexity equation would look like: 9-8+2*1=3

Designing Tests

For this post I have decided to write about how to design tests.  Testers design tests whether they are using Static or dynamic testing techniques.  To design a test you need 3 things according to ISTQB test conditions, test cases and test procedures (Scripts).  Each of these things are specified in their own documents created from information given by the developers and the tester's own interpretations of the provided info.  Testing is also designed and executed with varying degrees of formality depending on the customer's or testing team's preferences.


The designing process usually begins with the testers looking through something that they can derive test info from which is also known as a test basis.  A test basis can be many different things from an experienced user's knowledge, a business process, system requirements, specifications or the code.  Through test basis we find test conditions or the things we could test, because test conditions could come from anything they are usually vague.


Unlike test conditions, test cases are more specific.  Test cases will mean entering specific inputs into a system, and will also require some understanding of how the system works.  This mean knowing what the system should do and therefore the correct behavior of the system which is also referred to as a Test Oracle.  After choosing an input the tester needs to figure out what the expected result is and put it into the test case.  Expected results are what appears on the screen in response to the input values as well of changes to data or states and consequences.


The next and last step of the process is to put the test cases into a way that makes sense and the steps needed to run the test.  This is a test procedure or test script.  These procedures are then put into an execution schedule to determine which procedures are run first and whom runs them.


There are also many different categories of design techniques that I will get into later.