role of tester

Mr.XYZ,Subject :Application forSoftware Quality Test Engineer (Quality Analyst)Respected Sir,I, undersigned, Miss.XYZwould like to apply for thepostofSoftwareQualityTester in your organization. I have completed myBachelor in Engineering degree with First Class from XYZ University.In confidence, I can excel myself under your esteemed guidance and I will abide the rulesand regulations of organization.Herewith, I am enclosing my resume as the first step in exploring my possibilities ofemployment with your organization. I would like to meet you after your kind consideration. I look forward to the pleasure of hearing from you.Classic Testing Mistakes22There are types of coverage that point more directly to design mistakes than statementcoverage does (branch coverage, for example).23 However, none - and not all of them puttogether - are so accurate that they can be used as test design techniques.One final note:Romances with coverage don’t seem to end with the former devoteewanting to be “just good friends”. When, at the end of a year’s use of coverage, it has notsolved the testing problem, I find testing groupsabandoning coverage entirely.That’s a shame.When I test, I spend somewhat less than 5% of my time looking atcoverage results, rethinking my test design, and writing some new tests to correct my[24Some Classic Testing MistakesThe role of testing• Thinking the testing team is responsible for assuring quality.• Thinking that the purpose of testing is to find bugs.• Not finding the important bugs.• Not reporting usability problems.• No focus on an estimate of quality (and on the quality of that estimate).• Reporting bug data without putting it into context.•Starting testing too late (bug detection, not bug reduction)Planning the complete testing effort• A testing effort biased toward functional testing.• Underemphasizing configuration testing.• Putting stress and load testing off to the last minute.• Not testing the documentation• Not testing installation procedures.• An overreliance on beta testing.• Finishing one testing task before moving on to the next.• Failing to correctly identify risky areas.• Sticking stubbornly to the test plan.Personnel issues• Using testing as a transitional job for new programmers.• Recruiting testers from the ranks of failed programmers.• Testers are not domain experts.• Not seeking candidates from the customer service staff or technical writing staff.• Insisting that testers be able to program.• A testing team that lacks diversity.• A physical separation between developers and testers.• Believing that programmers can’t test their own code.• Programmers are neither trained nor motivated to test.The tester at work• Paying more attention to running tests than to designing them.• Unreviewed test designs.• Being too specific about test inputs and procedures.• Not noticing and exploring “irrelevant” oddities.•Checking that the product does what it’s supposed to do, but not that it doesn’t dowhat it isn’t supposed to do.• Test suites that are understandable only by their owners.Brian MarickTesting Foundationsmarick@testing.comIt's easy to make mistakes when testing software or planning a testing effort.Somemistakes are made so often, so repeatedly, by so many different people, that they deservethe label Classic Mistake.Classic mistakes cluster usefully into five groups, which I’ve called “themes”:• The Role of Testing: who does the testing team serve, and how does it do that?• Planning the Testing Effort: how should the whole team’s work be organized?• Personnel Issues: who should test?• The Tester at Work: designing, writing, and maintaining individual tests.• Technology Rampant: quick technological fixes for hard problems.I have two goals for this paper.First, it should identify the mistakes, put them in context,describe why they’re mistakes, and suggest alternatives.Because the context of onemistake is usually prior mistakes, the paper is written in a narrative style rather than as alist that can be read in any order.Second, the paper should be a handy checklist ofmistakes.For that reason, the classic mistakes are printed in a larger bold font when theyappear in the text, and they’re also summarized at the end.Although many of these mistakes apply to all types of software projects, my specific focusis the testing of commercial software products, not custom software or software that issafety critical or mission critical.This paper is essentially a series of bug reports for the testing process.You may thinksome of them are features, not bugs.You may disagree with the severities I assign.Youmay want more information to help in debugging, or want to volunteer information ofyour own.Any decent bug reporting system will treat the original bug report as the firstpart of a conversation.So should it be with this paper.Therefore, seehttp://www.stlabs.com/marick/classic.htm for an ongoing discussion of this topic.Theme One: The Role of TestingA first major mistake people make is thinking thatthe testing team is responsiblefor assuring quality.This role, often assigned to the first testing team in anorganization, makes it the last defense, the barrier between the development team(accused of producing bad quality) and the customer (who must be protected from them).It’s characterized by a testing team (often called the “Quality Assurance Group”) that hasClassic Testing Mistakes2formal authority to prevent shipment of the product.That in itself is a disheartening task:the testing team can’t improve quality, only enforce a minimal level.Worse, that authorityis usually more apparent than real.Discovering that, together with the perverse incentivesof telling developers that quality is someone else’s job, leads to testing teams and testerswho are disillusioned, cynical, and view themselves as victims. We’ve learned fromDeming and others that products are better and cheaper to produce when everyone, atevery stage in development, is responsible for the quality of their work ([Deming86],[Ishikawa85]).In practice, whatever the formal role, most organizations believe thatthe purpose oftesting is to find bugs.This is a less pernicious definition than the previous one, butit’s missing a key word. When I talk to programmers and development managers abouttesters, one key sentence keeps coming up:“Testers aren’t finding the importantbugs.”Sometimes that’s just griping, sometimes it’s because the programmers have askewed sense of what’s important, but I regret to say that all too often it’s valid criticism.Too many bug reports from testers are minor or irrelevant, and too many important bugsare missed.What’s an important bug?Important to whom? To a first approximation, the answer mustbe “to customers”. Almost everyone will nod their head upon hearing this definition, butdo they mean it?Here’s a test of your organization’s maturity.Suppose your product is asystem that accepts email requests for service.As soon as a request is received, it sends areply that says “your request of 5/12/97 was accepted and its reference ID is NIC-051297-3”. A tester who sends in many requests per day finds she has difficulty keeping track ofwhich request goes with which ID.She wishes that the original request were appended tothe acknowledgement.Furthermore, she realizes that some customers will also generatemany requests per day, so would also appreciate this feature. Would she:1.file a bug report documenting a usability problem, with the expectation that it will beassigned a reasonably high priority (because the fix is clearly useful to everyone,important to some users, and easy to do)?2.file a bug report with the expectation that it will be assigned “enhancement request”priority and disappear forever into the bug database?3.file a bug report that yields a “works as designed” resolution code, perhaps with anemail “nastygram” from a programmer or the development manager?4.not bother with a bug report because it would end up in cases (2) or (3)?Ifusability problems are not considered valid bugs, your project defines thetesting task too narrowly.Testers are restricted to checking whether the product doeswhat was intended, not whether what was intended is useful.Customers do not careabout the distinction, and testers shouldn’t either.Testers are often the only people in the organization who use the system as heavily as anexpert.They notice usability problems that experts will see.(Formal usability testingalmost invariably concentrates on novice users.) Expert customers often don’t reportClassic Testing Mistakes3usability problems, because they’ve been trained to know it’s not worth their time.Instead, they wait (in vain, perhaps) for a more usable product and switch to it.Testerscan prevent that lost revenue.While defining the purpose of testing as “finding bugs important to customers” is a stepforward, it’s more restrictive than I like.It means that there is no focus on anestimate of quality (and on the quality of that estimate).Consider these twosituations for a product with five subsystems.1.100 bugs are found in subsystem 1 before release. (For simplicity, assume that all bugsare of the highest priority.)No bugs are found in the other subsystems.After release,no bugs are reported in subsystem 1, but 12 bugs are found in each of the othersubsystems.2.Before release, 50 bugs are found in subsystem 1.6 bugs are found in each of theother subsystems.After release, 50 bugs are found in subsystem 1 and 6 bugs in eachof the other subsystems.From the “find important bugs” standpoint, the first testing effort was superior.It found100 bugs before release, whereas the second found only 74.But I think you can make astrong case that the second effort is more useful in practical terms.Let me restate the twosituations in terms of what a test manager might say before release:1.“We have tested subsystem 1 very thoroughly, and we believe we’ve found almost allof the priority 1 bugs.Unfortunately, we don’t know anything about the bugginess ofthe remaining five subsystems.”2.“We’ve tested all subsystems moderately thoroughly.Subsystem 1 is still very buggy.The other subsystems are about 1/10th as buggy, though we’re sure bugs remain.”This is, admittedly, an extreme example, but it demonstrates an important point.Theproject manager has a tough decision:would it be better to hold on to the product formore work, or should it be shipped now? Many factors - all rough estimates of possiblefutures - have to be weighed:Will a competitor beat us to release and tie up the market?Will dropping an unfinished feature to make it into a particular magazine’s special “JavaDevelopment Environments” issue cause us to suffer in the review?Will critical customerX be more annoyed by a schedule slip or by a shaky product?Will the product be buggyenough that profits will be eaten up by support costs or, worse, a recall?1The testing team will serve the project manager better if it concentrates first on providingestimates of product bugginess (reducing uncertainty), then on finding more of the bugsthat are estimated to be there.That affects test planning, the topic of the next theme.It also affects status reporting. Test managers often err by reporting bug datawithout putting it into context.Without context, project management tends tofocus on one graph:1 Notice how none of the decisions depend solely on the product’s bugginess.That’s another reason why giving thetesting manager “stop ship” authority is a bad idea.He or she simply doesn’t have enough information to use thatauthority wisely.The project manager might not have enough either, but won’t have less.One factor is that waterfall development appears to make it easier to implement GUI-levelautomated tests because, in theory, the user interface is locked down relatively early. The fragilenature of much of the GUI-level automated test code results in high costs if the program undertest has significant user interface changes. However, user interfaces often change very late. Ifyou send code out for beta testing, for example, the user-testers will often ask for plenty ofchanges to the user interface. Many of those will be well-justified, will be made, and the GUItests will have to be redone.A different way to think about the life cycles is to consider the tradeoffs that project managershave to make. The project manager has to deliver a set of features, that work reasonably well, ontime and within budget.Under the waterfall, the features are designed up front, funding and time are spent on designingand developing all of the features, until finally, the code comes into system testing. If the code islate, or the product has a lot of bugs, we face the typical tradeoff between reliability and time tomarket. (The features are already coded, so even if we cut some, we don't save much time ormoney.) The testers want more time to test the product and get it fixed, while other stakeholdersare likely to argue that it is time to ship the product before it runs too far behind schedule.Iterative life cycles are designed to change this tradeoff. For example, in an evolutionary model(eXtreme Programming follows an evolutionary model, but evolution was in practice long beforeXP (Gilb, 1988)), you test each feature as you add it to the product, and bring it up to acceptablequality before adding the next feature.In the evolutionary case, the tradeoff is between features and time to market, rather than betweenreliability and time to market. Testers play a much less central role in the late-projectcontroversies because the issue is not whether the product works (it does) but instead is whetherthere's enough product.Testers should base test cases on documented characteristics of theprogram. If the software documentation is inadequate for this, testersshould assert a quality control function and demand betterspecification and requirements documentation?My experiences and disagreement with these two assertions played a key role in motivating meto start writing Testing Computer Software in 1983. Twenty years later, these assertions are stillin play. Not so long ago, I saw a leading consultant/author publicly tell a pair of newly-promotedtest managers to refuse to test software if it came without adequate specifications. The testmanagers asked whether they could really refuse, and the consultant explained that this wast hei rprofessional duty, because it is unprofessional to develop software without detailedspecifications.My first problem with guidance like this is that a tester whose test design and test planningprocess depends on a strong specification will be helpless in the face of a lightly specifiedproduct.•Many development groups choose not to write detailed specifications. Is this always bad?•Many product changes are made without updating the specification, because the spec isnot considered final authority in that company. Is this always bad?Testers should base their tests on the information they can get—and they can getpl ent y ofinformation from sources other than specifications.My second objection is that specifications describe how the program is supposed to work.Testers discover how the programdoesn't work. The specification isn't necessarily the bestsource of information on product risks.My third objection is that this approach to testing, in practice, narrows the range of the tester'sthinking. Testers often write only one test per specification item—because what we focus on iscovering every statement in the specification, rather than questioning all of the different ways inwhich the code associated with any given statement could go wrong.A specification is only one source of guidance, fallible guidance, about what to test and how theprogram should work. To the extent that a specification limits what you test, it is a source of risk.My final objection to this guidance is that a tester who refuses to proceed until the engineeringprocess is changed is hijacking the management of the project. If you want to manage theproject, become a project manager. But if you’re a service provider to a project, you are not theproject manager. Refusal to deliver critical services should be done only as a last resort underextreme circumstances. People who make a habit of it, in my experience, get fired. As, in myopinion, they should.Test Planning and DocumentationTesters should specify the expected result of every test, in advance?This is another guidance from Myers (1979) that has had lasting influence. See, for example,ISEB's current syllabus for test practitioner certification atwww1.bcs.org.uk/DocsRepository/00900/913/docs/practsyll.pdf.One fundamental problem with this idea is that it is misguided. Every test has many results. Noone specifies them all. An "expected result" points the tester at one or a few of these results, butaway from the others.For example, suppose we test a program that adds numbers. Give it 2+3 (the intended inputs) andget back 5 (the monitored output). This looks like a passing test, but suppose that the test took 6hours, or that there was a memory leak, or that it erased a file on the hard disk. If all we compareagainst is the expected result (5), we won't notice these other bugs.The problem of an observer (such as a tester) missing obvious failures is not just theoretical. I'veseen it at work, and psychologists have described it in research, naming it inattentional blindness(Mack & Rock, 2000). If you are focusing your attention on something else, then you have asignificant probability of completely missing a surprising event that happens right in front ofyou. Some striking demos are available at http://viscog.beckman.uiuc.edu/djs_lab/demos.html.Does this mean that it is a bad idea to develop expected results? No, of course not. It means thatthere are pluses and minuses in using, and relying on, expected results.Expected results are valuable when an answer is difficult or time-consuming to compute. Theyare valuable when testing is done by junior staff who don't know the product (but realize thatthese people are especially likely to miss other failures). They are valuable for the test plannerwho is thinking through how the product works while she design the tests—working the teststhrough to their predicted conclusions is an important tool for learning the product. They are ofcourse valuable for automated tests—but the ability to automaticallyd eri ve an expected result ismuch more valuable for automated testing (because it is so much faster) than the ability of ahuman to write the result down.My second issue with the requirement that all tests should be specified up to and including theirexpected results is that it bars many of the important types of high volume automated testing(Kaner, 2000; Kaner, Bond, & McGee, 2004; Nyman, 2000). TThe Role of TestersProjects differ. Organizations differ. Products and their clients and their risks differ. Rather thana specific answer good for all situations,the role of testers (and the testing they do)has todepend on the context in which they operate. (Kaner et al., 2001)The primary reason to test is to find bugs?This is widely accepted wisdom. Even Glen Myers said it, in the first book published aboutsoftware testing (Myers, 1979) and we promoted the same position (Kaner et al., 1999).Actually, though, we test for many different reasons. Here are some examples (Kaner, 2003b): Find defects. This is the classic objective of testing. A test is run in order to triggerfailures that expose defects. Generally, we look for defects in all interesting parts of theproduct.Maximize bug count. The distinction between this and “find defects” is that total numberof bugs is more important than coverage. We might focus narrowly, on only a few high-risk features, if this is the way to find the most bugs in the time available. Block premature product releases. This tester stops premature shipment by finding bugsso serious that no one would ship the product until they are fixed. For every release-decision meeting, the tester’s goal is to have new showstopper bugs.Help managers make ship / no-ship decisions. Managers are typically concerned withrisk in the field. They want to know about coverage (maybe not the simplistic codecoverage statistics, but some indicators of how much of the product has been addressedand how much is left), and how important the known problems are. Problems that appearsignificant on paper but will not lead to customer dissatisfaction are probably not relevantto the ship decision.Minimize technical support costs. Working in conjunction with a technical support orhelp desk group, the test team identifies the issues that lead to calls for support. These areoften peripherally related to the product under test--for example, getting the product towork with a specific printer or to import data successfully from a third party databasemight prevent more calls than a low-frequency, data-corrupting crash. Assess conformance to specification. Any claim made in the specification is checked.Program characteristics not addressed in the specification are not (as part ofthisobjective) checked.Conform to regulations. If a regulation specifies a certain type of coverage (such as, atleast one test for every claim made about the product), the test group creates theappropriate tests. If the regulation specifies a style for the specifications or otherdocumentation, the test group probably checks the style. In general, the test group isfocusing on anything covered by regulation and (in the context of this objective) nothingthat is not covered by regulation. Minimize safety-related lawsuit risk. Any error that could lead to an accident or injury isof primary interest. Errors that lead to loss of time or data or corrupt data, but that don’tcarry a risk of injury or damage to physical things are out of scope.Find safe scenarios for use of the product (find ways to get it to work, in spite of thebugs). Sometimes, all that you’re looking for is one way to do a task that will consistentlywork--one set of instructions that someone else can follow that will reliably deliver thebenefit they are supposed to lead to. In this case, the tester is not looking for bugs. He istrying out, empirically refining and documenting, a way to do a task. Assess quality. This is a tricky objective because quality is multi-dimensional. The natureof quality depends on the nature of the product. For example, a computer game that isrock solid but not entertaining is a lousy game. Toassess quality -- to measure and reportback on the level of quality -- you probably need a clear definition of the most importantquality criteria for this product, and then you need a theory that relates test results to thedefinition. For example,rel i abi l i t y is not just about the number of bugs in the product. Itis (or is often defined as being) about the number of reliability-related failures that can beexpected in a period of time or a period of use. (Reliability-related? In measuringreliability, an organization might not care, for example, about misspellings in errormessages.) To make this prediction, you need a mathematically and empirically soundmodel that links test results to reliability. Testing involves gathering the data needed bythe model. This might involve extensive work in areas of the product believed to bestable as well as some work in weaker areas. Imagine a reliability model based oncounting bugs found (perhaps weighted by some type of severity) per N lines of code orper K hours of testing. Finding the bugs is important. Eliminating duplicates is important.Troubleshooting to make the bug report easier to understand and more likely to fix is (inthe context of assessment) out of scope. Verify correctness of the product. It is impossible to do this by testing. You can provethat the product isnot correct or you can demonstrate that you didn’t find any errors in agiven period of time using a given testing strategy. However, you can’t test exhaustively,and the product might fail under conditions that you did not test. The best you can do (ifyou have a solid, credible model) is assessment--test-based estimation of the probabilityof errors. (See the discussion of reliability, above).The primary reason to test is to prove the program works correctly?Glen Myers vigorously attacked this view in The Art of Software Testing and so did I inT est i ngComputer Software.The problem from one viewpoint is that programmers are offended by testers who express a"negative mindset." I've heard this argument most recently from several programmers in the XPcommunity. Before then, for 15 years, I heard it from more traditional programmers and projectmanagers. A negative attitude causes poor morale. It causes testers to look at the wrong things.The problem from the other viewpoint is that people are more likely to find what they expect tofind and to miss what they aren't looking for (Rosenthal, 1966; Teasley, Leventhal, Mynatt, &Rohlman, 1994). The "positive viewpoint" (prove it works correctly) is an ineffective viewpointfor testers. I've pointed this out to programmers who object to testers' negative mindset and theirtypical answer is that they don't believe it, they don't care, it does't matter, and it's the wrong