The title of this blog entry is a part of one of the zillion wise and witty things that Douglas Adams wrote, this one comes from “The Salmon of Doubt”. It expands to “All opinions are not equal. Some are a very great deal more robust, sophisticated and well supported in logic and argument than others.”. Let me explain…
My great friend, peruser and co-beer-tester Ileana sent me this message: “Hi, I’ve just come across this article. What do you think about it?” with a link to the article https://artoftesting.com/exploratory-testing
I replied “Hello. My initial response, having read just a couple of pages, is “What a load of bollocks!” 😠😠
I’ll read more tomorrow”
I’ve read more, and haven’t changed my opinion. And here’s why:
The first misconception:
Exploratory testing is a type of testing where testers explore the system on the fly without any pre-prepared set of test documents.
In this type of testing, testers explore the system, perform test design and test execution simultaneously without any proper planning, which means there aren’t any formal or standard documents such as testing plans, test cases and so on. (1)
The first debunking:
- “Exploratory Testing has been around for several decades now. Every tester has been knowingly or unknowingly practicing it in their daily testing activities. There are various definitions and methodologies surrounding this testing approach. One of which is session based exploratory testing (SBET). Some confuse this testing approach with Ad-hoc testing without realizing it is way more powerful and structured. Here is a formal introduction to this testing approach and how to use it in your daily testing activities.” (2)
- “As I showed at European Testing Conference in my demo talk, exploratory testing is a skill that uses a great deal of techniques formed by testers while actually testing.” (3)
Author’s note: “On the fly” = “ad hoc” ≠ “exploratory testing”
Misconception # 2
The second misconception:
In the section headed ” Types of Exploratory Testing” we find:
Freestyle Exploratory Testing – In freestyle exploratory testing, as the name suggests, the tester has complete freedom of testing what s/he likes, there isn’t any set criteria or guidelines for testing. Testers test the application in an ad hoc way. Testers use this technique when they want to do a quick smoke testing. (1)
The second debunking:
- “Smoke Testing is not exhaustive testing but it is a group of tests that are executed to verify if the basic functionalities of that particular build are working fine as expected or not. This is and should always be the first test to be done on any ‘new’ build.” (4)
- “SMOKE TESTING, also known as “Build Verification Testing”, is a type of software testing that comprises of a non-exhaustive set of tests that aim at ensuring that the most important functions work.” (5)
Author’s note: Hence smoke testing consists of predetermined tests, and so is the exact opposite of freestyle. As such it’s a good candidate for automated checking, which is an activity diametrically opposed to Exploratory Testing.
Misconception # 3
The third misconception:
Also, in the section headed ” Types of Exploratory Testing” we’re told:
Strategy Based Exploratory Testing – In strategy based exploratory testing, testing strategies such as boundary value analysis, equivalence technique, cause-effect graphing, etc. are used to test the system. To perform this type of testing, the tester should be familiar with the functionalities and working of the system. (1)
The third debunking:
- “Because we (James Bach and Michael Bolton) now define all testing as exploratory. Our definition of testing is now this:
“Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.”” (6)
Author’s note: If we accept that James Bach and Michael Bolton are correct, and I see no reason not to, it’s exploratory testing that allows us to learn about a product/system, hence the assertion that “ To perform this type of testing, the tester should be familiar with the functionalities and working of the system.” is incompatible with the concept presented by the author(s) of (1)
Misconception # 4
Testing) takes less time for preparation”.
From there on it’s pretty much all downhill.
The fourth misconception:
As less or, no pre-planning is involved in exploratory testing, testers will start testing the system immediately without spending much time on the time-consuming planning processes. (1)
The fourth debunking:
- “How to do exploratory testing
Exploratory testing may sound ad hoc, but it is a disciplined approach and requires specific techniques. You should:
- State a goal for each testing session…”
- “How to Prepare for Exploratory Testing?
For preparing and executing exploratory tests, test charters are used with items like:
- What will be tested (scope)
- What will not be tested (out of scope)
- Why (questions to be answered)
- How (brainstorm)
- Expected problems
- Reference (7)”
Author’s note: I’ve given just two examples of web pages that talk about preparation for Exploratory Testing, both have the concept of preparing something like a test charter, before each ET session. This contradicts the (grammatically incorrect) claim that “ As less or, no pre-planning is involved in exploratory testing” assertion
I could go on, investigating and debunking the article until the wee small hours. But it’s Friday lunchtime and I’m heading up to the Catalan coast this weekend 😊.
Do I have a conclusion? In macro terms, yes: there’s a lot of bollocks written about testing. As dedicated, professional and committed testers, we should highlight and argue intelligently against this sort of thing.
Alex Schladebeck deserves the last word, because she’s cool, smart and English: “And finally, exploratory testing is where good testers really shine. It is a structured and systematic activity that involves working, thinking and evaluating at multiple levels.” (3)
1. Unknown. Exploratory Testing. ArtOfTesting. [Online] February 3, 2020. [Cited: February 20, 2020.] https://artoftesting.com/exploratory-testing.
2. “Testim”. The power of Session Based Exploratory Testing. testIM. [Online] March 29, 2019. [Cited: February 21, 2020.] https://www.testim.io/blog/exploratory-testing/.
3. Schladebeck, Alex. All Roads Lead to Exploratory Testing. Women Testers. [Online] March 9, 2018. [Cited: February 21, 2020.] https://www.womentesters.com/all_roads_lead_to_et/.
4. Unknown. Smoke Testing Vs Sanity Testing: Difference With Examples. Software Testing Help. [Online] November 10, 2019. [Cited: February 11, 2020.] https://www.softwaretestinghelp.com/smoke-testing-and-sanity-testing-difference/.
5. —. Smoke Testing. Software testing Fundamentals. [Online] [Cited: February 21, 2020.] http://softwaretestingfundamentals.com/smoke-testing/.
6. James Bach and Michael Bolton. Exploratory Testing 3.0. Satisfice- Software testing for Serious People. [Online] March 16, 2015. [Cited: February 21, 2020.] https://www.satisfice.com/blog/archives/1509.
7. Ghahrai, Amir. A Tester’s Quick Guide to Exploratory Testing. Testing Excellence. [Online] December 2 (my birthday!), 2018. [Cited: Februeary 21, 2020.] https://www.testingexcellence.com/testers-quick-guide-exploratory-testing/.
8. Exploratory testing. GOV.UK Service Manual. [Online] May 23, 2016. [Cited: February 21, 2020.] https://www.gov.uk/service-manual/technology/exploratory-testing.