Checking v/s Testing - part I

In this post I am writing what I am thinking. Please do not get offended. Thoughts rise in mind unbidden, sometime the offspring of bias hidden deep within and sometimes surprising the thinker with the mysterious profundity. What I mean to say is that I am writing out what I am thinking. Some of the thinking may be biased and I am not too worried about these. Ordinarily they would have been hidden in my mind but I decided I am going to be honest starting this week about my biases. I would try to expose them so that I can remove them. Sometimes though the thoughts are just random thoughts and I have can attribute them to nothing but randomness.


After browsing quickly through some posts on Testing vs. Checking on Michael's blog  I reach this page
I am thinking - Am I according too much importance to a at-best a fringe concept and giving it importance that is not due? Will I have to grudgingly say "Good Job, Michael?" I am NOT against Michael just that all "Good Job"'s come a bit reluctantly. I am being reminded of Hancock. It was very difficult to say Good Job.

This writing about my thoughts while I read is very distracting. What should I do? Read or Write? Decision is to read, try-to-remember and then write. Reading printed copy is so much easier. You can write some things quickly, draw diagrams or ...

OK. Comments start coming in

"Checking Is Confirmation" - The paragraph seems to say what I had thought. Checking is confirming existing belief. Checking is focused on making sure that the program doesn’t fail. POSITIVE TESTS.

I have a a few bones to pick with the paragraph.
1. "Checking is focused on making sure that the program doesn’t fail" - Unless you are tinkering with the code there is no way you can make sure that the program does not fail. Even the most minor of the inputs/actions can make the program fail. I would have written that checking is focused on "checking/verifying" that the program does not fail by providing appropriate conditions that are least likely to make the program fail. OK, I am verbose but am I not more clear? I have begun now to feel a little smug. I also have a thought that my smugness is going to be punctured soon. Is smugness a baloon that can be punctured?

2. "When we already believe something to be true, we verify our belief by checking." 
I don't think I already believe something to be true. I might hope it to be true but I do not believe it. Even if I think the sentence is correct I don't see the difference between checking and testing. At least YET! Maybe I will see it after I read about testing.

Reading on...

"Testing Is Exploration and Learning"
When we configure, operate, and observe a product with the intention of evaluating it, or with the intention of recognizing a problem that we hadn’t anticipated, we’re testing"
Hmmm, I knew that Michael is going to throw the word exploration into the mix.

As a tester do I anticipate that when I press Ctrl+F while using MS Word the find dialog would come up? Yes. Why? Because I expected it based on requirements, business knowledge, product knowledge and so on. So it is checking. Right? Positive tests.

If I pressed Alt button then removed my finger and then pressed Ctrl+F, will the find dialog come up? If I answer this as YES that means I am anticipating the behavior. This makes it chekcing.
However, if I answer it as NO or "I don't NO" err... sorry "I don't KNOW" then it is Testing because I have an INTENTION to RECOGNIZE a problem that I had not ANTICIPATED.

So far it seems that the defition of checking and testing depends on what I AM THINKING and not really on the test. That means one persons checking can be another person's testing. Does not sound very encouraing. Another important point is that the defition of TESTING also seems to match what I had anticiapted.

Time to read further...
"We’re testing when we’re trying to find out about the extents and limitations of the product and its design, and when we’re largely driven by questions that haven’t been answered or even asked before."

Looks like what is commonly understood as negative testing. Of course all the criticism that I have applied to Testing and Checking above are also applicable to the terms Positive and Negative testing as well. I don't have an answer or solution to that. At least NOT YET. ;-)

"Checks Are Machine-Decidable; Tests Require Sapience. A check provides a binary result—true or false, yes or no. 

A test has an open-ended result. Testing is about asking and answering the question “Is there a problem here?”

Sounds reasonable. Not different from my thought process if you take out the names check and test. However there are a few things that one has to understand. These are:


A. A binary true or false is not really the result of the test but the result of the knowledge of the person who is asking the question. To give you an example - Swastika is banned in Germany. Swastika is a religious symbol of Hindus, Buddhists, Jains and maybe a few more religions. It is a 4000+ year old symbol meaning peace and prosperity from all sides. What if I want to display Swastika as part of being a Hindu? Do I have freedom of my religion or my freedom is gone because of on evil person who distorted this symbol?

What is your answer? Is it yes or no? Oh! You are not sure so you would want to find out. Meaning you are exploring. So it is a test.
Once you find out from a lawyer or somebody that Hindu/Buddhist temples can have this Symbol, even in Germany, then this becomes a simple Yes/No answer from now onwards.

B. A tells us that a test can be a check depending on the knowledge of the person concerned. Thus what is a check for one person may be a test for another person. I am sure you can recall many incidents from your life where your knowledge of intricacies of a feature was a given, a yes or no and for somebody else it was exploration followed by discovery.

There is nothing wrong with this interpretaion of test and check except that it does not solve any problem for me in terms of communicating the same meaning. Check for one is Test for others. The only thing differentiating a check and test is the documentation which clearly spells out an EXPECTED RESULT. If I have an expected result then it is a check else it is a Test.

If it is that simple then what is all this fuss about? I am feeling definitely very happy for having so many pearls of wisdom at this later hour. I am sure most of them will turn out to be pearls made of ice that turn into water with time and heat (heat that I am going to get once this gets published!). I need to return back to reading rather than writing my thoughts out.

Turning out that I have similar arguments for other paragraphs as well. I will stop writing/recording my thoughts, do some serious reading and re-reading. Michael is not known to write things lightly and he usually has very good arguments so it is time I threw my ususal biased style dissecting the sentences, read the article/post as a whole and try to gain knowledge.

More later folks.

Breaking the Bias - Testing v/s checking

I will start with listing some of my biases and my views on the topic before I start reading the whole checking v/s testing stuff.
I suffer from an art of having an opinion on a book or a testing topic even before I have read anything about it. Just show me the cover page of a book and you will have a review (of course very biased) and show me title of an article (like testing v/s checking) and I will have an opinion about it.
Another bias I have is that I am anti-whatever. No matter what topic (well most of the new ones at least) I am anti-that-topic.
Both the biases have served me well so far, at least in testing. My wife gets upset when I critique some of her favorite books without reading them so I have learned to stay away from that. However, design, code, applications etc. I am suitable and sufficiently biased that it won't work. I attack everything with a zeal that no matter how good it looks, there has to be a subtle flaw which is going to be catastrophic. If I have to admit that something is good, I do that very grudgingly.
This checking v/s testing thing. I have not started reading about it but I can tell you one thing - it is not going to work. It is nothing but new words for the same old meaning. There is some phrase about wines and bottles, can't seem to remember that.
I also have an opinion on what is a check and what is a test. The words tell you a story, the names tell a story and the nomenclature tells a story. In my (biased) opinion (without reading anything on check and test) Check -  "test" the features that should work as defined. Or Checking is a procedure that is used to determine whether the feature works correctly when used in the predefined manner say, as written in the requirements.
Test - is more like trying to figure out if in a given undefined situation (not defined in the requirements, not (possibly) thought through by the developer and so on) or trying to see if application behaves or misbehaves in a given situation which is not the "normal" (whatever normal means). Some more bias - Testing ~ negative testing and checking ~ positive testing. My feeling says WHY CREATE A NEW TERMS when old one exists.
I might be wrong,  I might be right. I don't know. Only one way to find out - read about it. I will try to read about it. All this verbiage is just to tell you what I FEEL about the subject, in my heavily biased way. What I THINK about it will be dealt with once I am through with the reading. I am being honest about my biases here. You can fault me for my intellectual (!!!) understanding but I don't want you to fault me for my intellectual honesty.
Testing v/s checking articles here I come. Next post after reading those articles.

Phases in Blogger's Mental Life

This might seem like a condescnnding/sarcastic/hitting below the belt kind of post. It is NOT.


Boris Beizer wrote about 5 stages in testers mindset. See here.

For some of the blogging testers the phases look like the following:
1. (Lurker/Larval stage) I read blogs of A, B, C ...
2. (Admission-to-Bloggers-School stage) I blog
3. (Getting-noticed stage) A or B or C mentioned me/endorsed me
/advised me/wrote to me
4. (Metamorphosis/Finding-your-feet stage) I have things to say to A, B or C/I write to them
5. (Butterfly stage/Beauty for beautie's sake stage) I don't care who has anything to say about me or my blog. I write without fear and favor because I am a fearless tester

I told you it is not meant to be condescending. It is just how I see growth happening. I certainly have passed through some of these phases and I think I am still not at 5, in case you are wondering.

Happy Blogging

A Story and Another Story

Some more moral-less stories

Story 1 - Anticipating users

This story comes from a real-life experience with a user on a mailing list. His communication, his ability to guess the context of the question and provide the answer was brilliant. I reproduce it here without anybody's permission. Even if I wanted to seek permission, I would not know who to approach so I guess it has to be as it is here.

Seen on a mailing list of a product that allows users to display documents on handheld devices

User 1: I am trying to view a document on device running operating system XYZ using document viewer "ABC". I can only see boxes instead of actual text. Any reason why this could be happening?

User 2: (In response to User 1's question)

XYZ device supports following fonts "" natively. The operating system does not allow addition of fonts. ABC application can use only the native fonts for displaying the documents. Even if the fonts are embedded in the document, you can't view it on XYZ even when you can view it on Windows or Linux. For example, if you want to read a document with embedded Gurumukhi fonts, you will not be able to view it on XYZ.

User 1: Thanks for your response. It explained very clearly the answer to the questions I had. You explained Indeed I wanted to see a file with Gurumukhi fonts...

You might ask a question on how did User 2 know that User 1 wanted Gurumukhi ? How did he know the fonts were embedded? Answer to first question is - very smart deductive logic. Name of User 1 was say Gurpreet Singh. This is a typical Sikh name. Sikh's are people following a religion called Sikhhism. Gurumukhi is the script in which Sikh religious literature is written.

User 2 also knew that square boxes are seen when fonts are not available. However, User 2 anticipated the question of embedded fonts and answered that question even before it could be raised thus preventing protracted mail exchanges.

A fine example of anticipating what user wants and sending one reply, without any questions being asked.

Story 2 - Signboards

I do not remember where I read it. Very interesting.

There was a sweet shop selling all sorts of sweets. The shop owner wanted to have a nice shining board for his shop so he got one with words "Fresh sweets sold here."

After a few days a guy came to him and said why use the word "here" because everybody knows sweets are being sold HERE. Why have this redundancy? The shop owner thought about it and found it reasonable. The sign became "Fresh sweets sold"

Few days later another gentleman said "why user the word SOLD? Everybody knows sweets are SOLD here."

The sign soon became "Fresh Sweets" to which somebody said "Why use the word Fresh? Everybody knows sweets have to be Fresh" which made the sign "Sweets"

Very obvious - said somebody. Why use the words Sweets and this resulted in the board becoming blank.

The story would have ended here but for the well-wisher who asked about the blank board and suggested that he should have a board which says "Fresh Sweets Sold Here."

Software Testing, Sapient Software Testing, Scripted Testing, White-box testing, Testing Center of Excellence... So many boards and so many suggestions!

Don't look for morals. I am not preaching.

Do we understand the customer? Really?

Headlights to the project/Voice of the customer/User representative...

I have no idea who coined all these terms that I mentioned above but one thing is for sure that testers have to know users.

I was very smug in my knowledge that after 15 years in the industry, 3 years are Adobe managing Adobe Reader testing for multiple platforms, 7 years in other product company, 5 years of providing testing services to 50+ projects I know HOW TO KNOW what users want.

How do you know what users want? There are some well-trodden paths -

1. Requirements (if you are lucky to have)

2. Talking to Business People, domain experts

3. Talking to developers (I am keeping my options open here)

4. Talking to customers

5. User forums

6. Newsgroups

7. Articles, magazines etc...

8. Eat your own dog food - become the user yourself

9. Imagination (Why not!)

and so on...

I knew this worked. I had made this work myself, hadn't I?

I had, until I met this former manager of mine and had an eye opening discussion. It also brought home one point rather forcefully that as testers we find a large number of bugs which have no consequence, which need not be found (waiting for brickbats to land... Hold on please! I will explain later) and which need not be fixed.

Eye opener first-

Vipul: Ram (name changed to maintain anonymity of the person) looks like we have not learned our lessons still. We discussed X years ago, when I was working for you, about the need for doing performance engineering for our product. We spent quiet a lot of time doing that and performance was OK for next release but one more release and looks like everybody has forgotten their learnings. Your product has performance issues again!

Ram: Before I agree or disagree with this statement tell me one thing, how do you know that the developers have forgotten their lessons?

Vipul: It is obvious. Performance of the software has gone down.

Ram: Could it not be that we worked hard to keep the performance up to the mark but no more could be done at least given the fact that there is need for a lot of functionality which is required by a lot of people? We can't dump the features, we can't create multiple products each targeted towards a specific type of community and we can't make it complex by giving users all choices about which they may not know anything. Our products are used by all types of users.

Vipul: Well, that might be your problem but there are sure a lot of angry users there who do not want to see bad performance.

Ram: How do you know that?

Vipul: These tell me what I need to know.

Ram: Are you sure they represent all the users that we are talking about? Don't you think these could be a very small percentage of users who are more vocal than any other user group and they get more internet space/voice than the numbers they actually represent? When these users compare our product with other products do they compare only launch time performance or do they also compare the quality, user experience, all features (many of which) they use everyday?

Vipul: Hmmm! You have got a point there. However, to use your argument how do you know that they DO NOT represent the majority of users?

Ram: Business data. You did not and do not have access to this data but I do. Our corporate customers who form 90% of our users both in terms of numbers and revenue do not complaint. Many of them launch our software once every time they boot their machines/laptops and that is it. Launch is not a problem for them. We have talked to many of them and they see no problems. In fact performance is not a big problem for us. It is a problem, I agree but a far smaller problem than say security. It has far less priority than say, Usability.

I began thinking about what had been said. I understood what Ram was trying to say. Those who shout loudest do not necessarily represent the user. To understand what users want one needs data from lot of different sources. One also needs to understand the difference between what USERS want and what CUSTOMERS want. It did not matter whether Ram was right or wrong, what mattered was that I should not have been smug about my knowledge.

Don't be too sure. Meek shall inherit the earth.

What about “…as testers we find a large number of bugs which have no consequence...” Well, I have almost used up my time at Bangalore airport waiting for a delayed flight so let that be a topic for another post.

Last word - This discussion is part fiction, part paraphrased but the discussion did take place. Use at your own risk.

Why did I start blogging?

Tech-challenged I am but that does not explain being on twitter and not on blogs for a long-long time now. Why did I not blog earlier?

Writing-challenged as I may be, I am no stranger to writing. As part of speaking at various conferences and some times as Keynote (those poor souls who had to endure me!) I have some experience in writing. However, I had reasons not to write and I still have them. One reason to write has overridden all other reasons not to - A promise I made or I was made to make (under duress) by the following people:

• Parimala
• Santhosh Tuppad
• Ravisurya
• Pradeep Soundararajan
• Rahul Verma
• Vinay Baid
• Pravesh Pugalia

It happened one evening when I was discussing possibility of having a new testing conference in India. The motto being discussed was - "Idiots not allowed"

I know it is not a nice theme to have. Not politically correct. It has discordant notes. There is an unanswered question as well-who will decide who is an idiot? Well, that was not the idea. The ideas were

No bad presentation will be allowed. (every conference claims that)

It was to be a very low cost conference where testers meet and discuss (what is new in that? They do it all the time)

Invite testers who are interested in learning independen of Dogma. (Nothing new here mate!)

The idea was to NOT call big shots who manage companies or testing departments but TO CALL real testers. Who have something to show and teach. (That sounds new. New recipe for disaster, commercial failure, enmity with heads of testing of all companies and so on... Great new idea. Good luck buddy)

I was held to Ransom here. All these nice people (till then) showed their true colors and started a blackmail. They said that I keep telling testers in India to write whereas I do not write myself and this is not fair. I told them that there are good reasons for that but nobody would take that for an answer. The masterstroke that they had was - "We will do this conference IFF (if and only if - for the mathematically challenged people) you start blogging." I promised them, caught as I was in their vice like grip. I promised that I will start this activity within next 2 months and here I am burning midnight electricity writing this post.

Folks, will the conference happen now?

My experiments with ATM

"My experiments with Truth" is Mahatma Gandhi's autobiography. I have read only parts of it. That probably qualifies me to lift the title for this post. As you will see, there is a good reason to talk about "My experiments with ATM."

Automated teller machine (ATM) is probably the last machine one would want to play with, unless one has somebody else's card . Well, here is a story one such adventure...

To surprise my wife I decided to order something for her. When the delivery boy landed at my door with all the grocery (My wife has long given up on hoping to receive flowers, just in case you are wondering) I had ordered to surprise my wife, I found I had no cash. I decided to withdraw some cash from a cash machine.

ATM here I come

Insert the card, punch the PIN, punch the amount and behold! I get a message "Please enter your correct PIN."

I did enter correct PIN and the ATM accepted it, didn't it? I punched the PIN again and a message flashed "Only two attempts left. Please enter correct PIN"

I punched it again, very carefully, one button at a time, enough Think-Time included.

"Only one attempt left. Please enter correct PIN"

Mind you, I have not entered more than one incorrect PIN in my entire lifetime and here I am left with only one attempt and a guy waiting outside for the payment. Deeply ashamed, I called up the helpline. Other end of the connection spoke to me thus -

Helpline - Sir, please enter correct PIN and everything would be OK!

Vipul - Miss, I did that.

Helpline - OK Sir, please give me your customer identification number

Vipul - Sure. It is (You do not expect me to write it here, do you?)

Helpline - Sorry Sir. This is incorrect ID. If you can't give me correct customer ID how can I trust you when you say you entered correct PIN? Please try correct PIN again.

Speechless, I disconnected the call. I decided to give it one more shot and it turned out to be the last shot. The ATM said "Sorry, you have exceeded maximum number of retries. To get your card back please contact the bank during working hours"

Down but not out

I still have one more card with me. Same bank, different account, different card, same password. (I know, I know. It is not advisable to have same password on all your cards.)

The same story repeated itself with this card as well. One attempt gone, two more to go. I asked myself if I remembered the PIN correctly and the answer was - it was the same PIN I had been using since last 5 years! (I know, I know. It is not advisable to keep the same PIN.)

The guy outside was getting impatient. I told my-nervous-self that I need to isolate the problem. Maybe something was wrong with this particular machine. Should I try another machine?

I did that and now I was left with last attempt on this card as well. (Does anybody has any suggestion on what should I do? What would you do in such a situation?)

Now the problem that occupied my mind was "what is wrong here?"

I asked myself, what else can I do with this system that deals with PIN which may not count as an attempt to use an incorrect PIN. I have only one more chance. The idea that popped in my mind was about trying to change the PIN. I had no idea whether it will count as an attempt or not (even today I do not know that and I DO NOT want to experiment with it) however this was the only other thing that came to my mind.

I selected the "change password" option and in the old password field I entered my six digit PIN and entered new six digit PIN in new password and confirm new password field. The message that appeared was shocking, jolting, revealing and eye-opening. Light descended on me and realization dawned immediately.

The message said "PIN can't be more than 4 digits" and here I was who had configured a 6 digit PIN during pre-historic times and the machine somehow was suffering me gladly. Maybe one day a wise developer decided to deal severely with fools like me and made the PIN validation more strict. It is another issue that he/she did not apply the same logic to change password else my goose would be cooked and that is not a nice thing, I being vegetarian.

Rest of the story is relatively simple. I paid the guy, got myself a new debit card, I now keep two separate PINs for two different cards and I change my PINs often.

There is a side story though. Changes not applied to all the relevant places is an ailment that afflicted my use of AMT once again. I have to admit that it was mild, nothing like the nail biting suspense as the previous story, but does that matter? A bug is a bug is a bug, isn't it?

Say, there is a limit of 15K INR (Indian currency - Rupee) per withdrawal and a limit of 30K INR per day for every account that one holds. I wanted to withdraw more money than the supposed daily limit. I had heard that the banks might allow higher limits based on the directives of banking authority in India.

The ATM said clearly - 15K in one go. I took a risk and punched in 25K. I was expecting a rebuke from the machine. Surprise! Surprise! The ATM took some time in counting all the money but it did deliver me 25K!

I asked myself - What if I enter 30K as the amount to be withdrawn? Well, the machine did rebuke me this time and asked me to enter a valid amount of up to 15K. I was not about to be fooled by this limit of 15K. After all, the machine did give me 25K. I entered 25K and contrary to my expectation, the machine rebuked me again! 20K and the same result. 15K and it did give me 15K!

Looks like some developer, once again (no offence intended), did not do the change analysis completely. I am not complaining, this keeps secure my job as a tester.

Moral of the stories - No moral in these stories. I am not a Guru of any sorts. You want moral, go find it yourself. I am happy just telling stories not the morals.

For the curious testers - Should I have tried with 16K? 15.1K? Any other amount? Well, I did not have more money in my account. I am willing to try if you are willing to put enough money in my account for my experiments. Waiting for my banker to call me about unexplained source of millions in my account, all in the cause of software testing.