I recently attended a Project Managers meeting where PM specific, day to day problems were discussed.
Some were quite generic in nature and similar to a tester meeting, for example how do we make sure that people are properly trained, dealing with communication problems, etc. Others were more specific, for example talking about managing customer relationships, how a customer may perceive problems in a project from a completely different angle (for the better or the worse for the PM in question).
What occurred to me there is that we put a lot of emphasis in testing on adding “value for the customer”, however most testers that I know have no idea about customer management, and how our customers priorities might be different from the ones that the development team (programmers and testers) are thinking of. Managing that relationship sits squarely in the corner of the PM or Account Manager (where available). In a way it’s good that the Project Manager is shielding the development team from sometimes difficult customers, but to what degree can a tester really say that what they’re doing is actually adding value for the client?
That reminded me that what we think of as “adding value” is actually based on a model. To quote George Box “All models are wrong, some are useful”. Is our model built on validation and verification – in a broad way ensuring that the system works as intended and works without glitches? But what do we base that on since we all know that requirement documents are notoriously lacking? The argument I often hear is that testers bring their experience in adding what they perceive to be of value to other customers so it might be of value to the current project as well. With no knowledge of the customer or our relationship with them how can this be true?
At the next team meeting it might be an idea to invite the Account Manager as well or, if the PM is fulfilling that role as well, ask him about our relationship to the client. Is it perceived to be a difficult one? Have we worked with that client before? Is the project seen as a start of many others to follow? Do we know of particular things that “bug” the client that would help us direct our test effort?
I don’t know how other people address that but these are not questions that I have asked in the past but maybe I should. I attended a Project Management course and think that all testers should do that sooner or later, ideally followed with actually managing a project or two as well.
Many testers have an idea of how programmers think, many having a background in programming themselves. Maybe it’s time that we start to look through the eyes of a Project Manager as well and walk a mile in their shoes.
Thursday, 28 October 2010
Wednesday, 20 October 2010
The pictorial challenge answered
After a short discussion with Zeger (TestSideStory) on Twitter about the pictorial challenge it was time to do some work but I came back to the picture later. Have a look at it first before reading on.
I sat back and thought “How would I test this if I don’t know what’s important and don’t know to whom it will be important?”
I then collated some high level areas of approach
• Death by Goggle:
o Searching the net for the names on the signs to give me more avenues to consider
• Technical: It’s a .jpg of a certain size, resolution and colours used
o Is that on purpose, in which scenarios could it be a problem?
o Are there parts of the picture that look cut off, is there too much sky, etc.
o Are there hints that this is a collage of pictures or is it one photograph
• Emotional: How do I react to the picture when looking at it first.
o How does that change when picking out more details?
I started with the emotional part first because a) I can’t help it and b) to me testing a picture is more akin to criticizing a painting so I wanted see if the artist/photographer got an immediate emotional reaction from me. The signs look rusty and barely legible. Rust to me indicates age so I assume they’ve been there for a long time. The house looks like it has seen better days, my first reaction is that it’s been taken in a poor part of whatever town/country. The boy on the roof (I assume it’s a roof) would be unusual in a UK town, maybe it’s not where this picture is taken? Filed away for future consideration.
I then searched for “Zanchett” and “La Mejor Ropa”, the latter of which is Spanish for “The best clothes”. But maybe it's not "Zanchett" but "Zanchetti". That's where the font type could throw a spanner into the tester's work. You live and learn. Again, more pages but I didn't follow them up too closely as I'm more interested in the picture rather than where it was taken. Zanchett seems to be quite a common name, there are several facebook sites, a Portuguese Tourism page to name a few. I also found links to a breastfeeding site in Brazil, so maybe it’s not Spanish but Portuguese. The boy (man?) on the roof is wearing a t-shirt, short trousers and sandals which indicates a warm country. At a guess I’d say the photo was taken in Brasil or Argentina.
The breastfeeding site also made me look at the photo closer, maybe the blue plastic bowl hanging at the outside of the house is actually a baby bath. The bike on the left, caged birds on the right and the boy on the roof could mean it’s a nursery or orphanage (because of what I assume is a poor area, I could be wrong here). In my experience Mediterranean houses don’t give the impression that they’re looked after from the outside, that doesn’t mean that they’re not in very good condition from the inside. There’s an angle of cultural differences here and since I assume Southern America here as the location I’ll have to question my assumptions. The boy/man (I'm not able to clearly distinguish the features to say for sure) is wearing sports clothes which seem new or in good shape at least so maybe poor isn't actually correct here.
I didn’t go into the technical details other than noting that the size is 601 x 776 pixels – the 601 seems odd to me, maybe it was converted from a different size or type?
I’ll get in touch with Zeger to see what he thinks about it. :)
I sat back and thought “How would I test this if I don’t know what’s important and don’t know to whom it will be important?”
I then collated some high level areas of approach
• Death by Goggle:
o Searching the net for the names on the signs to give me more avenues to consider
• Technical: It’s a .jpg of a certain size, resolution and colours used
o Is that on purpose, in which scenarios could it be a problem?
o Are there parts of the picture that look cut off, is there too much sky, etc.
o Are there hints that this is a collage of pictures or is it one photograph
• Emotional: How do I react to the picture when looking at it first.
o How does that change when picking out more details?
I started with the emotional part first because a) I can’t help it and b) to me testing a picture is more akin to criticizing a painting so I wanted see if the artist/photographer got an immediate emotional reaction from me. The signs look rusty and barely legible. Rust to me indicates age so I assume they’ve been there for a long time. The house looks like it has seen better days, my first reaction is that it’s been taken in a poor part of whatever town/country. The boy on the roof (I assume it’s a roof) would be unusual in a UK town, maybe it’s not where this picture is taken? Filed away for future consideration.
I then searched for “Zanchett” and “La Mejor Ropa”, the latter of which is Spanish for “The best clothes”. But maybe it's not "Zanchett" but "Zanchetti". That's where the font type could throw a spanner into the tester's work. You live and learn. Again, more pages but I didn't follow them up too closely as I'm more interested in the picture rather than where it was taken. Zanchett seems to be quite a common name, there are several facebook sites, a Portuguese Tourism page to name a few. I also found links to a breastfeeding site in Brazil, so maybe it’s not Spanish but Portuguese. The boy (man?) on the roof is wearing a t-shirt, short trousers and sandals which indicates a warm country. At a guess I’d say the photo was taken in Brasil or Argentina.
The breastfeeding site also made me look at the photo closer, maybe the blue plastic bowl hanging at the outside of the house is actually a baby bath. The bike on the left, caged birds on the right and the boy on the roof could mean it’s a nursery or orphanage (because of what I assume is a poor area, I could be wrong here). In my experience Mediterranean houses don’t give the impression that they’re looked after from the outside, that doesn’t mean that they’re not in very good condition from the inside. There’s an angle of cultural differences here and since I assume Southern America here as the location I’ll have to question my assumptions. The boy/man (I'm not able to clearly distinguish the features to say for sure) is wearing sports clothes which seem new or in good shape at least so maybe poor isn't actually correct here.
I didn’t go into the technical details other than noting that the size is 601 x 776 pixels – the 601 seems odd to me, maybe it was converted from a different size or type?
I’ll get in touch with Zeger to see what he thinks about it. :)
Monday, 2 August 2010
The need for confirmation – why checking against requirements is not going away
I was thinking what prompts the need for traceability and ensuring that the final application actually satisfies the requirements that were captured at the beginning of the project. Who needs that traceability? I then thought about different companies. The type of company a tester works in has a huge impact on the approach of their testing. Broadly speaking (read: this is a gross generalisation) we have these types of companies:
1. Product companies selling shrink-wrap software. This is anything from operating systems, games, tax return helpers, computer helper apps (Adobe Reader), etc.
2. Consultancies developing a bespoke application for their clients to address a specific business problem
3. Hybrid between the first two, for example further developing/configuring a shrink-wrap application, for example SAP or other back-office systems.
4. Large company developing a bespoke application in-house for internal use only, for example banks or insurance companies.
Let’s have a look at each from different angles.
The first type of company producing products is somewhat unique, as there is not one, single client that determines the requirements. They need to rely on Product Managers or other staff who can guess or work out that there is a target audience who’s likely to use and/or buy their software. Anticipating what the market needs determines the requirements.
Many products, when selling well, will have several versions and releases. That’s good news for the automation people as products often need to be regression tested for a variety of platforms, i.e. different operating systems, hardware configurations, or when new functionality is added over time. The quality necessary for their products depends on the market they’re selling to.
The second type is very much focussed on the one client they’re selling to. What’s in the contract will get built (and tested if the contract specifies it). What’s not in the contract depends on the good will of the consultancy. If a client just spent £3M with you and you wave the contract instead of adding a 2 day RFC that’s probably the last big contract your company got.
Automated regression testing depends wholly on the client – have they specified that they want it, is the application large enough to warrant it for just the one version, are there likely to be more version of the software and contracts coming in from the client? Then it might make sense building something automated, otherwise forget it, time is money in this setup.
The hybrid company is closer to the Consultancy but with the added complication that now there is third party code to be considered for testing. More often than not that has already been tested but one has to be careful to exclude functions as they may be used differently in the context of the re-developed/configured new solution.
The in-house development test efforts very much depend on the company, however there usually are requirements that have been developed by internal groups. This type is usually similar to the Product companies in that software developed is there to stay for probably more than one version. That means more effort can be spent on testing it properly and with a long term view.
To come back to the need for confirmation – if you’re working in some sort of Consultancy, traceability is a no brainer. You’ll need to confirm that what your company is delivering is adhering to the requirements that were set out – and by extension that you fulfil the contract. From the Consultancy point of view satisfying the contract has to come first, otherwise the business would leave itself wide open to being sued or just not paid for breaching the contract. So once the confirmation is out of the way there might or might not be time to test the application in detail.
If you’re working in a product company or in-house development effort it’s a bit different. There are other factors to consider though. Are there any legal requirements to adhere to certain standards? Does the software need to be certified against particular standards? GLP or GCP comes to mind. Traceability is maybe not so important in this setup compared to the Consultancies, unless the PM or project owners insists on it, I’d argue out of the false hope that the system will then actually deliver what was expected when the requirements were written.
There was an argument on one of the LinkedIn groups if checking/testing against requirements is a hoax or if it adds value. As always, the answer is, it depends. If you have a 1:1 relationship of tests vs requirements I’d say it’s a hoax. All requirements need more than one simple test. So saying that a particular requirement has been tested against, then ticking the box, fine, that’s implemented is giving a wrong picture.
Imagine you have a requirements that says that you need to have several user roles in the system. Would you confirm that with one single check? You could, but would it actually give you the information and confidence in the application that the stakeholders ask us to provide? I don’t think so. Why artificially tick a box and say, yes, there are several user roles in the system if you’re telling a white lie if you haven’t tested the around that requirement?
I can see why checking against requirements is popular as it’s easy to put into contracts. The effectiveness of it from a quality point of view is almost non-existent, so it might help the sales and legal people (and not even that if the customer is set on suing a non-compliant company) but the PM should be looking elsewhere for confidence that the final product is actually what the customer wanted in the first place.
1. Product companies selling shrink-wrap software. This is anything from operating systems, games, tax return helpers, computer helper apps (Adobe Reader), etc.
2. Consultancies developing a bespoke application for their clients to address a specific business problem
3. Hybrid between the first two, for example further developing/configuring a shrink-wrap application, for example SAP or other back-office systems.
4. Large company developing a bespoke application in-house for internal use only, for example banks or insurance companies.
Let’s have a look at each from different angles.
The first type of company producing products is somewhat unique, as there is not one, single client that determines the requirements. They need to rely on Product Managers or other staff who can guess or work out that there is a target audience who’s likely to use and/or buy their software. Anticipating what the market needs determines the requirements.
Many products, when selling well, will have several versions and releases. That’s good news for the automation people as products often need to be regression tested for a variety of platforms, i.e. different operating systems, hardware configurations, or when new functionality is added over time. The quality necessary for their products depends on the market they’re selling to.
The second type is very much focussed on the one client they’re selling to. What’s in the contract will get built (and tested if the contract specifies it). What’s not in the contract depends on the good will of the consultancy. If a client just spent £3M with you and you wave the contract instead of adding a 2 day RFC that’s probably the last big contract your company got.
Automated regression testing depends wholly on the client – have they specified that they want it, is the application large enough to warrant it for just the one version, are there likely to be more version of the software and contracts coming in from the client? Then it might make sense building something automated, otherwise forget it, time is money in this setup.
The hybrid company is closer to the Consultancy but with the added complication that now there is third party code to be considered for testing. More often than not that has already been tested but one has to be careful to exclude functions as they may be used differently in the context of the re-developed/configured new solution.
The in-house development test efforts very much depend on the company, however there usually are requirements that have been developed by internal groups. This type is usually similar to the Product companies in that software developed is there to stay for probably more than one version. That means more effort can be spent on testing it properly and with a long term view.
To come back to the need for confirmation – if you’re working in some sort of Consultancy, traceability is a no brainer. You’ll need to confirm that what your company is delivering is adhering to the requirements that were set out – and by extension that you fulfil the contract. From the Consultancy point of view satisfying the contract has to come first, otherwise the business would leave itself wide open to being sued or just not paid for breaching the contract. So once the confirmation is out of the way there might or might not be time to test the application in detail.
If you’re working in a product company or in-house development effort it’s a bit different. There are other factors to consider though. Are there any legal requirements to adhere to certain standards? Does the software need to be certified against particular standards? GLP or GCP comes to mind. Traceability is maybe not so important in this setup compared to the Consultancies, unless the PM or project owners insists on it, I’d argue out of the false hope that the system will then actually deliver what was expected when the requirements were written.
There was an argument on one of the LinkedIn groups if checking/testing against requirements is a hoax or if it adds value. As always, the answer is, it depends. If you have a 1:1 relationship of tests vs requirements I’d say it’s a hoax. All requirements need more than one simple test. So saying that a particular requirement has been tested against, then ticking the box, fine, that’s implemented is giving a wrong picture.
Imagine you have a requirements that says that you need to have several user roles in the system. Would you confirm that with one single check? You could, but would it actually give you the information and confidence in the application that the stakeholders ask us to provide? I don’t think so. Why artificially tick a box and say, yes, there are several user roles in the system if you’re telling a white lie if you haven’t tested the around that requirement?
I can see why checking against requirements is popular as it’s easy to put into contracts. The effectiveness of it from a quality point of view is almost non-existent, so it might help the sales and legal people (and not even that if the customer is set on suing a non-compliant company) but the PM should be looking elsewhere for confidence that the final product is actually what the customer wanted in the first place.
Thursday, 17 June 2010
EWT22 - How to think about science/Something Fishy
At EWT22, Michael Bolton moderated a new type of session – listening to an audio recording “How To Think About Science” and finding parallels to software testing.
Background to the interview:
On July 3, 1992, the Canadian Fisheries Minister John Crosbie announced a moratorium on the fishing of northern cod. It was the largest single day lay-off in Canadian history: 30,000 people unemployed at a stroke. The ban was expected to last for two years, after which, it was hoped, the fishery could resume. But the cod have never recovered, and more than 15 years later the moratorium remains in effect. How could a fishery that had been for years under apparently careful scientific management just collapse? David Cayley talks to environmental philosopher Dean Bavington about the role of science in the rise and fall of the cod fishery."
The MISSION:
Listen to the recording. Take notes on what you’re hearing, with the goal of capturing lessons from the interview that might usefully influence how we test, and how we think about testing. Stories or patterns from your own testing experience that you can relate to something in the interview are especially valuable.
My experience tells me that it's good to take lots of notes, regardless if it's reviewing documents or acively testing. I fully expect to discard 90% of it but it can serve to confirm or discard ideas, theories and guesses.
So I listened to the interview and made notes about important events and their order, my raw data. I'd stop from time to time to listen again to passages to get a clearer understanding and to be able to keep up with note writing.
Michael clarified some terms that were used in Skype always shortly before I got to that passage. That was helpful on two fronts. One, I understood the term and context quicker. And two, I could tell that I was falling behind with my progress in listening as it took longer and longer between Michael explaining terms and me getting to the passage - a good way to measure progress, though fallible.
About 2/3rds in I changed the approach as I felt that I had enough data and because time was pressing. I started drawing parallels between the fishery events and software testing and noted those as well. Adding
to the raw data suffered as a result but that was fine as I was working towards the mission, drawing parallels and matching them to my experience. I concentrated on the similarities in process and process failings rather than thinking of 'fishing for bugs' or comparing the government quotas to test certification like some other people in the group did. Nothing wrong with either, I just took this particular approach.
Listening to the audio recording I noted several occasions of Hubris as reason for that monumental failure of preserving the cod population.
From Wiki: Hubris means extreme haughtiness or arrogance. Hubris often indicates being out of touch with reality and overestimating one's own competence or capabilities, especially for people in positions of power.
I reckon that is why the radio program was called "How to think about science", the idea that, while science has not all the answers, the ones it has can be taken as the truth.
That's only one part of it though, I don't want to make the same mistake and believe that one reason alone can explain the failures on many fronts.
Since I'm currently reading Jerry Weinberg's book "The Psychology of Computer Programming" (Thank you, Jerry, for giving me permission to use parts of this book) I saw some interesting similarities between his study of the psychology of computer programming and the events that lead to the disappearance of the Canadian cod. Here's Jerry's list from page 40 of his book.
1. Using introspection without validation
2. Using of observations on an overly narrow base
3. Observing the wrong variables, or failing to observe the right ones
4. Interfering with the observed phenomenon
5. Generating too much data and not enough information
6. Overly constraining experiments
7. Depending excessively on trainees as subjects
8. Failing to study group effects and group behaviour
9. Measuring what is simple to measure
10. Using unjustified precision
11. Transferring borrowed results to inapplicable situations
Matching my notes to this list I got the following results. We could argue with some of the matches, however the overall match of similarities should become clear:
1a. Key drivers were managing nature fluctuations in order to get back the money that was invested in the industry. Fluctuations turned out to be unmanageable – there was no driver to understand the model in detail
1b. Introducing averaging assumptions was one of the traps – the issue was not understood and two wildly different results had just been averaged with not thought of the consequences. One example of Hubris.
2. There was a simple model for the amount of cod in the sea which was far too simple and therefore wrong
3a Not all parameters of the model had been identified, for example parameter changes over time had been missed.
3b Some parameters had been misunderstood. These two points were the biggest mistakes made in my book. Science was not seen as fallible but a blind eye turned to the possibility that the current understanding does not fully cover the situation in question - see the definition for Hubris again.
4. There was constant interfering with the model as fishing continued throughout the years up to the point where the cod population didn't recover
5. Data from two sources was created, the scientific trawlers and the commercial fleet. They only seemed to look at total population, not age groups, etc
6. Local knowledge of inshore fishermen = “domain matter experts” has been ignored/dismissed as anecdotal – scientific methods were seen as more accurate and objective. Another example of Hubris
7. Not applicable in this scenario?
8. There were several examples of failing to recognise group behaviour, for example variation in location of spawning grounds and season, migration patterns, etc
9. Parameters found were dismissed as complicating the model too much (and making it unmanageable)
10. Not applicable in this scenario?
11. There were several examples of borrowing results from one are to another, for example assuming that inshore groups of cod behave the same as off shore groups
Some failings that I can't put into any of the categories but which are very important as well are:
A. The inshore fishermen’s protests have not been looked at in detail. They wanted to fish in the traditional way but reasons for that were not investigated. Scientists were supposed to speak truth. I'm wondering if, with a bit more questioning, some of the mistakes may have been prevented, for example damaging and destroying stock using the jigger (hook).
B. Investing in tools to make it cheaper became a driver to make more money which backfired. The parallel to automated software testing tools should be easy to see.
C. The fishing industry put pressure on the government to get higher allowable quotas. Available information was ignored and numbers changed against better knowledge to serve the industries need for money.
D. As the model got very complicated it didn’t help management as it didn't provide the control measures that were needed. In software terms this is the point where a project is either stopped or a drawn out death march begins
E. The target of management changed from figuring out how to achieve the goal to targeting people, the fisherman. Sounds oddly familiar, doesn't it?
I find it quite amazing how two, apparently unrelated areas like a book about the psychology of computer programming and discussing shortfalls of science in a fishery project are so similar at a higher level. I have a good number of other examples in mind, some of which I'll blog about shortly.
I'm in two minds about the message that science got it all wrong. I'd summarise it as scientists (bringing the people component in here) probably did their best but didn't look at their limitations. Results of their experiments were also disregarded and decisions taken based on politics and the industries needs/greed.
I come from a science background myself so may have a blind spot there. My overall (scientific approach/thinking) is to identify a problem, analyse it, put up some theories and experiments/tests that then either prove or disprove these theories. In other words, I'm building a close as possible picture or model of the issue in question. Any steps up to the results can be criticised, the result can not. The thinking behind that is that if you agree with the thinking behind all steps the result is independent. You don't get to argue with it if you don't like it. The result can be a pointer that something is wrong with your thinking, it's dangerous to build your thinking based on the results you want though. That's a different story though and far to involved to be discussed here.
I'd be happy to discuss these results in more detail so please feel free to agree or, even better, disagree with me.
I should note that these findings are based on a single source, the radio interview. What actually happened might or might not be exactly as portrayed, but I think it's useful to keep in mind that in testing as in life it's dangerous to believe only in one source without questioning it. Or would you believe a developer who tells you that a bug has been fixed and doesn't need re-tested?
PS: I would also like to point you towards an excellent blog of Jeroen Rosink who looks at this session from a slightly different angle.
Background to the interview:
On July 3, 1992, the Canadian Fisheries Minister John Crosbie announced a moratorium on the fishing of northern cod. It was the largest single day lay-off in Canadian history: 30,000 people unemployed at a stroke. The ban was expected to last for two years, after which, it was hoped, the fishery could resume. But the cod have never recovered, and more than 15 years later the moratorium remains in effect. How could a fishery that had been for years under apparently careful scientific management just collapse? David Cayley talks to environmental philosopher Dean Bavington about the role of science in the rise and fall of the cod fishery."
The MISSION:
Listen to the recording. Take notes on what you’re hearing, with the goal of capturing lessons from the interview that might usefully influence how we test, and how we think about testing. Stories or patterns from your own testing experience that you can relate to something in the interview are especially valuable.
My experience tells me that it's good to take lots of notes, regardless if it's reviewing documents or acively testing. I fully expect to discard 90% of it but it can serve to confirm or discard ideas, theories and guesses.
So I listened to the interview and made notes about important events and their order, my raw data. I'd stop from time to time to listen again to passages to get a clearer understanding and to be able to keep up with note writing.
Michael clarified some terms that were used in Skype always shortly before I got to that passage. That was helpful on two fronts. One, I understood the term and context quicker. And two, I could tell that I was falling behind with my progress in listening as it took longer and longer between Michael explaining terms and me getting to the passage - a good way to measure progress, though fallible.
About 2/3rds in I changed the approach as I felt that I had enough data and because time was pressing. I started drawing parallels between the fishery events and software testing and noted those as well. Adding
to the raw data suffered as a result but that was fine as I was working towards the mission, drawing parallels and matching them to my experience. I concentrated on the similarities in process and process failings rather than thinking of 'fishing for bugs' or comparing the government quotas to test certification like some other people in the group did. Nothing wrong with either, I just took this particular approach.
Listening to the audio recording I noted several occasions of Hubris as reason for that monumental failure of preserving the cod population.
From Wiki: Hubris means extreme haughtiness or arrogance. Hubris often indicates being out of touch with reality and overestimating one's own competence or capabilities, especially for people in positions of power.
I reckon that is why the radio program was called "How to think about science", the idea that, while science has not all the answers, the ones it has can be taken as the truth.
That's only one part of it though, I don't want to make the same mistake and believe that one reason alone can explain the failures on many fronts.
Since I'm currently reading Jerry Weinberg's book "The Psychology of Computer Programming" (Thank you, Jerry, for giving me permission to use parts of this book) I saw some interesting similarities between his study of the psychology of computer programming and the events that lead to the disappearance of the Canadian cod. Here's Jerry's list from page 40 of his book.
1. Using introspection without validation
2. Using of observations on an overly narrow base
3. Observing the wrong variables, or failing to observe the right ones
4. Interfering with the observed phenomenon
5. Generating too much data and not enough information
6. Overly constraining experiments
7. Depending excessively on trainees as subjects
8. Failing to study group effects and group behaviour
9. Measuring what is simple to measure
10. Using unjustified precision
11. Transferring borrowed results to inapplicable situations
Matching my notes to this list I got the following results. We could argue with some of the matches, however the overall match of similarities should become clear:
1a. Key drivers were managing nature fluctuations in order to get back the money that was invested in the industry. Fluctuations turned out to be unmanageable – there was no driver to understand the model in detail
1b. Introducing averaging assumptions was one of the traps – the issue was not understood and two wildly different results had just been averaged with not thought of the consequences. One example of Hubris.
2. There was a simple model for the amount of cod in the sea which was far too simple and therefore wrong
3a Not all parameters of the model had been identified, for example parameter changes over time had been missed.
3b Some parameters had been misunderstood. These two points were the biggest mistakes made in my book. Science was not seen as fallible but a blind eye turned to the possibility that the current understanding does not fully cover the situation in question - see the definition for Hubris again.
4. There was constant interfering with the model as fishing continued throughout the years up to the point where the cod population didn't recover
5. Data from two sources was created, the scientific trawlers and the commercial fleet. They only seemed to look at total population, not age groups, etc
6. Local knowledge of inshore fishermen = “domain matter experts” has been ignored/dismissed as anecdotal – scientific methods were seen as more accurate and objective. Another example of Hubris
7. Not applicable in this scenario?
8. There were several examples of failing to recognise group behaviour, for example variation in location of spawning grounds and season, migration patterns, etc
9. Parameters found were dismissed as complicating the model too much (and making it unmanageable)
10. Not applicable in this scenario?
11. There were several examples of borrowing results from one are to another, for example assuming that inshore groups of cod behave the same as off shore groups
Some failings that I can't put into any of the categories but which are very important as well are:
A. The inshore fishermen’s protests have not been looked at in detail. They wanted to fish in the traditional way but reasons for that were not investigated. Scientists were supposed to speak truth. I'm wondering if, with a bit more questioning, some of the mistakes may have been prevented, for example damaging and destroying stock using the jigger (hook).
B. Investing in tools to make it cheaper became a driver to make more money which backfired. The parallel to automated software testing tools should be easy to see.
C. The fishing industry put pressure on the government to get higher allowable quotas. Available information was ignored and numbers changed against better knowledge to serve the industries need for money.
D. As the model got very complicated it didn’t help management as it didn't provide the control measures that were needed. In software terms this is the point where a project is either stopped or a drawn out death march begins
E. The target of management changed from figuring out how to achieve the goal to targeting people, the fisherman. Sounds oddly familiar, doesn't it?
I find it quite amazing how two, apparently unrelated areas like a book about the psychology of computer programming and discussing shortfalls of science in a fishery project are so similar at a higher level. I have a good number of other examples in mind, some of which I'll blog about shortly.
I'm in two minds about the message that science got it all wrong. I'd summarise it as scientists (bringing the people component in here) probably did their best but didn't look at their limitations. Results of their experiments were also disregarded and decisions taken based on politics and the industries needs/greed.
I come from a science background myself so may have a blind spot there. My overall (scientific approach/thinking) is to identify a problem, analyse it, put up some theories and experiments/tests that then either prove or disprove these theories. In other words, I'm building a close as possible picture or model of the issue in question. Any steps up to the results can be criticised, the result can not. The thinking behind that is that if you agree with the thinking behind all steps the result is independent. You don't get to argue with it if you don't like it. The result can be a pointer that something is wrong with your thinking, it's dangerous to build your thinking based on the results you want though. That's a different story though and far to involved to be discussed here.
I'd be happy to discuss these results in more detail so please feel free to agree or, even better, disagree with me.
I should note that these findings are based on a single source, the radio interview. What actually happened might or might not be exactly as portrayed, but I think it's useful to keep in mind that in testing as in life it's dangerous to believe only in one source without questioning it. Or would you believe a developer who tells you that a bug has been fixed and doesn't need re-tested?
PS: I would also like to point you towards an excellent blog of Jeroen Rosink who looks at this session from a slightly different angle.
Monday, 10 May 2010
It’s NOT Rocket Science
This blog is to go into a bit more detail about the planning and execution of the European Weekend Testing mision 17.
If you don’t know what the mission was please have a look and read it.
In summary, the group should test an audio application, a software compressor plugin that needs its own host. The group was testing the plugin through the host which was an additional complication.
The mission for EWT17 was several weeks in the planning. I first came up with a rough outline, then bounced it off Anna and Markus who made it better without really knowing the application. We then pre-tested the whole mission with Anna and Markus on Thursday to make sure it was not too hard to do. I got interesting results to what they were looking at and we then decided that, yes, this mission is a go.
For me, the one of the main goals of the mission was to bring a bit more realism into it. So I introduced a bit of role playing in saying that all EWT testers are in one company in one group reporting to one test manager who has a problem he wants them to solve: Does this software compressor work and is stable enough for me to use tonight?
His problem is time limited and he wants an answer in 1.5 hours. I should rephrase that, he wants one answer. From his point of view it’s one problem, one question, one answer.
Of course in EWT people come from all over the world so this was always difficult to achieve but some people picked up on it as Jeroen also pointed out in his blog.
I wasn't actually expecting to get one but wasn't sure what people would do. Some testers refused an answer as per Michael Bolton's article.
Anna, Markus and me discussed if we should add a team lead role for this mission but decided against it as it wasn’t in line with the spirit of EWT to have a hierarchy. I was interested to see if one developed naturally or not, which didn’t happen – no problem there.
The other big difference in this mission compared to previous ones is that I picked a domain where not many people would have knowledge of – audio recording and engineering. There are actually a lot of parallels with software testing but that’s another story or blog.
When the mission was released people realised within a couple of minutes that they don’t know enough about the domain to be able to test the application. Some went into ET testing mode, some Googled and found the Wiki page that explains very well what it’s all about. No one thought about asking if there are domain experts in the group of which we had one, Zeger. He actually couldn’t get the software to start on his laptop so was available to answer questions. He picked up on some assumptions people made and threw in some very valuable comments. Some were picked up, others weren’t, for example the important one about asking if the test manager is using the same plug-in host or not as this could invalidate the test effort. Yes, he did and it would actually..
The Test Manager gave the group pretty much an impossible task. Find out if the application is stable. What does stable mean without context? Without knowing what hardware is used, operating system (as someone pointed out), other software installed stable is pretty much meaningless.
The other big area that I wanted to explore with this mission is testability. There are several possible meanings. One issue with testability is that the testers didn’t have enough domain knowledge to know how they would actually test the application. The Rocket reduces the dynamic range but how many testers know how to test that?
The other issue is about measuring if the dynamic range actually has been reduced and by the right amount. Since we’re talking about audio here we can’t rely on our senses as they aren’t up to the task. Sure we can hear if something gets quieter and louder but only if the difference is quite large. How large depends on the testers ears so it is really subjective.
There are some freeware tools that would help with this task, for example the freeware audio editor and recorder. A tester could then record the output of the plugin and compare waveforms or rely on the measure tools that come with the audio editor application. All in all a pretty difficult task. I only mention it here because the secondary objective was to find out if the attack of the plugin was fast enough. Something that could only be tested with additional tools and was largely ignored by the group, which was good. If you have your hands full with the main objective don’t get distracted with nice to have’s – thumbs up for the EWT testers.
The group dynamics where interesting to see and also the extend to which testers picked up on what information others typed into Skype. I tried to facilitate more group working which hasn’t worked so well. This is clearly something we need to think about again for future missions.
It was an intense session but one where I think everyone got something out of. To the new people who joined us for EWT17, it’s not always like that!
I’d like to everyone who participated who made this session a success and gave me lots to think about – and that’s a good thing.
Thomas
If you don’t know what the mission was please have a look and read it.
In summary, the group should test an audio application, a software compressor plugin that needs its own host. The group was testing the plugin through the host which was an additional complication.
The mission for EWT17 was several weeks in the planning. I first came up with a rough outline, then bounced it off Anna and Markus who made it better without really knowing the application. We then pre-tested the whole mission with Anna and Markus on Thursday to make sure it was not too hard to do. I got interesting results to what they were looking at and we then decided that, yes, this mission is a go.
For me, the one of the main goals of the mission was to bring a bit more realism into it. So I introduced a bit of role playing in saying that all EWT testers are in one company in one group reporting to one test manager who has a problem he wants them to solve: Does this software compressor work and is stable enough for me to use tonight?
His problem is time limited and he wants an answer in 1.5 hours. I should rephrase that, he wants one answer. From his point of view it’s one problem, one question, one answer.
Of course in EWT people come from all over the world so this was always difficult to achieve but some people picked up on it as Jeroen also pointed out in his blog.
I wasn't actually expecting to get one but wasn't sure what people would do. Some testers refused an answer as per Michael Bolton's article.
Anna, Markus and me discussed if we should add a team lead role for this mission but decided against it as it wasn’t in line with the spirit of EWT to have a hierarchy. I was interested to see if one developed naturally or not, which didn’t happen – no problem there.
The other big difference in this mission compared to previous ones is that I picked a domain where not many people would have knowledge of – audio recording and engineering. There are actually a lot of parallels with software testing but that’s another story or blog.
When the mission was released people realised within a couple of minutes that they don’t know enough about the domain to be able to test the application. Some went into ET testing mode, some Googled and found the Wiki page that explains very well what it’s all about. No one thought about asking if there are domain experts in the group of which we had one, Zeger. He actually couldn’t get the software to start on his laptop so was available to answer questions. He picked up on some assumptions people made and threw in some very valuable comments. Some were picked up, others weren’t, for example the important one about asking if the test manager is using the same plug-in host or not as this could invalidate the test effort. Yes, he did and it would actually..
The Test Manager gave the group pretty much an impossible task. Find out if the application is stable. What does stable mean without context? Without knowing what hardware is used, operating system (as someone pointed out), other software installed stable is pretty much meaningless.
The other big area that I wanted to explore with this mission is testability. There are several possible meanings. One issue with testability is that the testers didn’t have enough domain knowledge to know how they would actually test the application. The Rocket reduces the dynamic range but how many testers know how to test that?
The other issue is about measuring if the dynamic range actually has been reduced and by the right amount. Since we’re talking about audio here we can’t rely on our senses as they aren’t up to the task. Sure we can hear if something gets quieter and louder but only if the difference is quite large. How large depends on the testers ears so it is really subjective.
There are some freeware tools that would help with this task, for example the freeware audio editor and recorder. A tester could then record the output of the plugin and compare waveforms or rely on the measure tools that come with the audio editor application. All in all a pretty difficult task. I only mention it here because the secondary objective was to find out if the attack of the plugin was fast enough. Something that could only be tested with additional tools and was largely ignored by the group, which was good. If you have your hands full with the main objective don’t get distracted with nice to have’s – thumbs up for the EWT testers.
The group dynamics where interesting to see and also the extend to which testers picked up on what information others typed into Skype. I tried to facilitate more group working which hasn’t worked so well. This is clearly something we need to think about again for future missions.
It was an intense session but one where I think everyone got something out of. To the new people who joined us for EWT17, it’s not always like that!
I’d like to everyone who participated who made this session a success and gave me lots to think about – and that’s a good thing.
Thomas
Saturday, 17 April 2010
ACCU2010 writeup
Here’s a short (OK, I got carried away a bit) summary of the conference and the talks/presentations I attended. I only went down for one day - Friday 16th April due to work commitments.
Friday morning:
Keynote talk from Russell Winder “The Multicore Revolution - We’ve only just begun”. Originally Dan North was down to present “Simplicity – the way of the unusual architect” but couldn’t get to the UK due to the airspace being closed because of the Icelandic volcano eruption.
Russell is clearly very knowledgeable and presented a good talk about the architecture of computers the way different programming languages address those. Good presentation that was probably more interesting to the developer community than for me but interesting nonetheless.
After a coffee break which I used for frantic telephone calls and trying to book train tickets I attended Rachel Davies “Understanding User Stories”. Not having written any before this was an interesting presentation with just the right amount of practical exercises for the participants. She mentioned the book “Agile Coaching” which she wrote or co-wrote which I’ll have a look at.
She explained that she and her team used a user story template which is now being used by many people around the world. I got the impression that she didn’t expect that and was pleasantly surprised that such a small thing created to make life easier in her project team has such a big impact. She did not that this template doesn’t work in all cases – especially when there’s no discussion with the business or if technical implementations are the order of the day rather than user workflows.
The template followed a
As a...
I want...
So that...
format which goes a long way getting all the necessary information onto a small card. She noted the importance of this being a physical card rather than a spreadsheet or electronic document. To me this format is similar-ish to the way use cases are created as the actor (As a xyz..) and workflow (I want...) and goal description (so that..) are all present. That’s not bad and I don’t confuse the two but just note the similarities.
After a practical exercise of writing user stories for an imaginary tool of our choice we discussed that. What I found interesting is that breaking down the user stories to the right level and amount of detail is really a team decision – something many people struggle with when writing use cases.
The importance about the user stories should be “Do the user stories help us understand user context and business value.” That’s a harsh abbreviation and there’s more to it but to me these are the essentials (together with acceptance criteria).
At lunch I met James Bach and after listening in to another conversation we spoke about some senior managers and their viewpoints on testing, the TA model (parent/adult/child modes) and implementing rapid testing in companies. As I’m following both James and Michael B’s blogs there was nothing groundbreaking new for me but it rather reinforced my own views but also highlighted that the problems I’m facing are not with the understanding of the rapid testing approach but in other, more business related areas. I left contemplating and thinking about this (which is a good thing) but will need some time before I’ll find answer. Interesting.
After lunch I attended Paul Field’s “Change, change, change – the 5 year evolution of an agile team.” It was an interesting presentations about his experience about starting out with a 2 man agile team taking on a failing project in an IT unfriendly environment – the internal customer (another department) got burned in the past and was wary of IT in general.
He spoke about the pitfalls of building a team and scaling that from 2 to 7 after a while. Generally a good talk although it could’ve been a bit condensed from my point of view.
I liked his use of the Timeline Retrospective where he mapped colour coded issues as experienced by the team in sprint retrospectives over the whole project time. This can be a good way not to measure the way the project is going but the way the team is feeling about it which doesn’t necessarily needs to be the same. (And indeed was not, the team thought that they delivered badly at a sprint when the customer was delighted).
The retrospective starfish was also interesting consisting of 5 areas for
More of
Start doing
Stop doing
Less of
Keep doing
In the burnup chart he presented he mapped the story points “Done” and the story points as asked for from the business. This was not per sprint but over the whole project which showed that after 4 months (in a 6 month project) the team had created as many story points as the business originally wanted (great) but wouldn’t be able to finish by immovable 6 month deadline – a big problem. This burnup chart showed the problem in time and action was taken to get the project back on track.
When new team members where added he found that his former “flat” structure didn’t work anymore as existing team members were more senior in terms of knowledge so not everyone was equal. This needed to be addressed and after the new team member were brought up to speed through pair programming and other measures the team picked up some momentum.
After a coffee break I went to Astrid Byro’s “Just in time – last minute testing” presentation. She jumped in for another presenter who couldn’t come due to “Volcano awareness week” as James Bach coined it.
This was the weakest talk for me, the title didn’t have very much to do with the talk which was basically a war story about a 2 year RUP project with very high risks (accepted), lots of stress and a “command and control” approach should have died out 20 years ago. System migrators where used as sole testers which would’ve made me uncomfortable (maybe she was as well). I can see that from a PM point of view using people with domain knowledge is a good idea and using testing as a training exercise so you save handover further down the line is good for project timelines as well. But I can’t shake off the feeling that professional testers would’ve done some good – apparently that wasn’t possible (or thought necessary). At the end of the project performance testing was started (always a bad idea due to the high risks involved, again they were lucky and their application performed well). Their clients infrastructure didn’t but that wasn’t seen as a problem as the project in itself delivered, even if that didn’t actually help the customer. My view is more holistic than just pointing at the customer and saying “it’s their problem, our application performs well”, but maybe that’s just me.
That concluded the official presentations but I saw about a dozen lightning talks, the highlight for me was James Bach’s “And...also” presentation (I’m predictable in that regard, I know).His fiery talk had a lot of passion and humour to be able to appeal to even the most hardened developers and I believe he got some people thinking. It was about thinking not only what the expected positive outcome is but also what he also expects to be there (or not). For example NOT getting error messages, expecting the result to appear for more than half a second, etc – all things an automated test suite might fail to notice but a human being would notice – sometimes instinctively, sometimes with a bit of thinking.
I won’t mention the other lightning talks, there were too many to mention and I didn’t take notes of all so wouldn’t be able to give a complete picture anyway.
After a longer break the conference dinner started which was very interesting as well, after the starter we played musical chairs with only the speakers remaining seated so that people got more opportunity to speak to different speakers and other attendees. At my first table I sat next to Jeff Sutherland, creator of Scrum, without recognising him which I found extremely embarrassing. I promised to put in a good word with the conference organisers to get him back next year so that I could have a longer discussion with him – so organisers, if you’re reading this, do all of us a favour and invite Jeff for next year as well!
That’s it, thanks to Volcano Awareness Week I’ve been able to write down these notes while on my train journey rather than sitting at home already so let’s all say thanks to the volcano gods
:-)
Thomas
Friday morning:
Keynote talk from Russell Winder “The Multicore Revolution - We’ve only just begun”. Originally Dan North was down to present “Simplicity – the way of the unusual architect” but couldn’t get to the UK due to the airspace being closed because of the Icelandic volcano eruption.
Russell is clearly very knowledgeable and presented a good talk about the architecture of computers the way different programming languages address those. Good presentation that was probably more interesting to the developer community than for me but interesting nonetheless.
After a coffee break which I used for frantic telephone calls and trying to book train tickets I attended Rachel Davies “Understanding User Stories”. Not having written any before this was an interesting presentation with just the right amount of practical exercises for the participants. She mentioned the book “Agile Coaching” which she wrote or co-wrote which I’ll have a look at.
She explained that she and her team used a user story template which is now being used by many people around the world. I got the impression that she didn’t expect that and was pleasantly surprised that such a small thing created to make life easier in her project team has such a big impact. She did not that this template doesn’t work in all cases – especially when there’s no discussion with the business or if technical implementations are the order of the day rather than user workflows.
The template followed a
As a...
I want...
So that...
format which goes a long way getting all the necessary information onto a small card. She noted the importance of this being a physical card rather than a spreadsheet or electronic document. To me this format is similar-ish to the way use cases are created as the actor (As a xyz..) and workflow (I want...) and goal description (so that..) are all present. That’s not bad and I don’t confuse the two but just note the similarities.
After a practical exercise of writing user stories for an imaginary tool of our choice we discussed that. What I found interesting is that breaking down the user stories to the right level and amount of detail is really a team decision – something many people struggle with when writing use cases.
The importance about the user stories should be “Do the user stories help us understand user context and business value.” That’s a harsh abbreviation and there’s more to it but to me these are the essentials (together with acceptance criteria).
At lunch I met James Bach and after listening in to another conversation we spoke about some senior managers and their viewpoints on testing, the TA model (parent/adult/child modes) and implementing rapid testing in companies. As I’m following both James and Michael B’s blogs there was nothing groundbreaking new for me but it rather reinforced my own views but also highlighted that the problems I’m facing are not with the understanding of the rapid testing approach but in other, more business related areas. I left contemplating and thinking about this (which is a good thing) but will need some time before I’ll find answer. Interesting.
After lunch I attended Paul Field’s “Change, change, change – the 5 year evolution of an agile team.” It was an interesting presentations about his experience about starting out with a 2 man agile team taking on a failing project in an IT unfriendly environment – the internal customer (another department) got burned in the past and was wary of IT in general.
He spoke about the pitfalls of building a team and scaling that from 2 to 7 after a while. Generally a good talk although it could’ve been a bit condensed from my point of view.
I liked his use of the Timeline Retrospective where he mapped colour coded issues as experienced by the team in sprint retrospectives over the whole project time. This can be a good way not to measure the way the project is going but the way the team is feeling about it which doesn’t necessarily needs to be the same. (And indeed was not, the team thought that they delivered badly at a sprint when the customer was delighted).
The retrospective starfish was also interesting consisting of 5 areas for
More of
Start doing
Stop doing
Less of
Keep doing
In the burnup chart he presented he mapped the story points “Done” and the story points as asked for from the business. This was not per sprint but over the whole project which showed that after 4 months (in a 6 month project) the team had created as many story points as the business originally wanted (great) but wouldn’t be able to finish by immovable 6 month deadline – a big problem. This burnup chart showed the problem in time and action was taken to get the project back on track.
When new team members where added he found that his former “flat” structure didn’t work anymore as existing team members were more senior in terms of knowledge so not everyone was equal. This needed to be addressed and after the new team member were brought up to speed through pair programming and other measures the team picked up some momentum.
After a coffee break I went to Astrid Byro’s “Just in time – last minute testing” presentation. She jumped in for another presenter who couldn’t come due to “Volcano awareness week” as James Bach coined it.
This was the weakest talk for me, the title didn’t have very much to do with the talk which was basically a war story about a 2 year RUP project with very high risks (accepted), lots of stress and a “command and control” approach should have died out 20 years ago. System migrators where used as sole testers which would’ve made me uncomfortable (maybe she was as well). I can see that from a PM point of view using people with domain knowledge is a good idea and using testing as a training exercise so you save handover further down the line is good for project timelines as well. But I can’t shake off the feeling that professional testers would’ve done some good – apparently that wasn’t possible (or thought necessary). At the end of the project performance testing was started (always a bad idea due to the high risks involved, again they were lucky and their application performed well). Their clients infrastructure didn’t but that wasn’t seen as a problem as the project in itself delivered, even if that didn’t actually help the customer. My view is more holistic than just pointing at the customer and saying “it’s their problem, our application performs well”, but maybe that’s just me.
That concluded the official presentations but I saw about a dozen lightning talks, the highlight for me was James Bach’s “And...also” presentation (I’m predictable in that regard, I know).His fiery talk had a lot of passion and humour to be able to appeal to even the most hardened developers and I believe he got some people thinking. It was about thinking not only what the expected positive outcome is but also what he also expects to be there (or not). For example NOT getting error messages, expecting the result to appear for more than half a second, etc – all things an automated test suite might fail to notice but a human being would notice – sometimes instinctively, sometimes with a bit of thinking.
I won’t mention the other lightning talks, there were too many to mention and I didn’t take notes of all so wouldn’t be able to give a complete picture anyway.
After a longer break the conference dinner started which was very interesting as well, after the starter we played musical chairs with only the speakers remaining seated so that people got more opportunity to speak to different speakers and other attendees. At my first table I sat next to Jeff Sutherland, creator of Scrum, without recognising him which I found extremely embarrassing. I promised to put in a good word with the conference organisers to get him back next year so that I could have a longer discussion with him – so organisers, if you’re reading this, do all of us a favour and invite Jeff for next year as well!
That’s it, thanks to Volcano Awareness Week I’ve been able to write down these notes while on my train journey rather than sitting at home already so let’s all say thanks to the volcano gods
:-)
Thomas
Monday, 8 March 2010
First Scottish Software Testing Club event in Edinburgh
I’d like to announce the first Scottish Software Testing Club meeting on Wednesday, 10th March, 7pm in the Filmhouse Bar, Lothian Road in Edinburgh. Food is nice and they always have vegetarian and vegan meals on the menu. The beer and wine selection is good as well. Thanks to Rosie for letting us put the STC label on this meeting.
This is an informal event and is intended as a networking session and just saying hello to the other local testers. Everyone’s welcome, even if you are a developer :-). Glenn and I are thinking of maybe making this a monthly institution, 2nd Wednesday of each month or something along these lines – we’ll see what the turnout is going to be.
We'll meet in the main area, I'll try and put a picture of the STC logo up if I have the space.
Hope to see you there,
Thomas
This is an informal event and is intended as a networking session and just saying hello to the other local testers. Everyone’s welcome, even if you are a developer :-). Glenn and I are thinking of maybe making this a monthly institution, 2nd Wednesday of each month or something along these lines – we’ll see what the turnout is going to be.
We'll meet in the main area, I'll try and put a picture of the STC logo up if I have the space.
Hope to see you there,
Thomas
Friday, 5 February 2010
Process or training?
The raging battle between the process vs non-process people and Michael Bolton’s recent burning issue article got me thinking. If I assume that everyone in the testing field, be it tester, test lead, test manager and the wider software development arena wants to do a good job, why are their points of view so different?
OK, that question is too wide so let’s try again. Why does someone wants to put the process first and someone else the problem first?
In my experience that’s due to at least two major reasons, personality and experience. Let’s look at personality first.
Some people feel trapped by processes and limited in their freedom and abilities. They rely on their skills and abilities and trust whatever comes that they will find an answer even it they don’t know it, yet.
Some feel that with a process and documents in place there is something that they can fall back on to prove that what they’re doing is right. That they can give the process to someone else and that they can then also repeat it providing a base for a bigger group of people to learn from what they did before.
As a test manager I can’t influence people’s personality, that’s why there’s always an amount of touch and feel in any project decisions. Most decisions are emotionally impacted, I haven’t seen people looking at ROI and NOT decide based on their emotional reaction.
For the experience part, someone might have worked in a big, process driven corporation where there was a reason for processes. Or they worked in a small shop where anything goes and only a result counted, any result. I worked in the pharmaceutical industry and research scientist for a good number of years. GLP was a regulation we needed to comply with and if you ever heard of thalidomide you’ll know why that is a good idea. There’s an interesting article for further reading.
Processes were in place to satisfy these regulations. However having done a lot of the "work on the floor" most of the processes just didn’t make sense to me. After working for several years in the IT industry I now understand what they were trying to achieve, but no one explained it to me at the time. But I also now understand why the processes failed miserably. My point is that if you always worked in process driven companies you don’t know any other approach as very often there is a good reason why the processes are in place. That doesn’t mean they’re always working in practice though. I worked with a lot with red tape, ditched it all, and are now coming out the other end being somewhere in the middle.
Most companies want to be able to run a profitable project, have a satisfied customer and be able to repeat that over and over again. So why not putting your working habits into processes? Wrong, no two development projects are the same and a habit is just that. Wikipedia has this for Habit:
Habits are routines of behavior that are repeated regularly and tend to occur subconsciously, without directly thinking consciously about them.
The question is, do we actually want that? When you document a process and don’t want people to think about it but follow it, that’s fine. But when your next development project encounters an unforeseen problem don’t be surprised that your team can’t deal with it as you created a culture where people don’t think about what they’re doing.
I see the clash not between the process vs non-process approach but if you want to invest into training your people so that the whole team understands what’s expected from them and what’s important; or if you think that this route is too expensive and that you rather tell them what to do and have a few “in the know” who are creating the processes.
I think that training or lack thereof is at the root at the problem when we’re talking about processes as they’re often used to mask that people are not trained enough to do their jobs without them. If a developer doesn’t understand that throwing software over the wall to the testers impacts their work this is essentially a training issue. If a tester doesn’t understand what information is essential to the PM this is a training issue. If the PM doesn’t understand what to ask from the developer or tester, you guessed it, that’s a training issue as well.
I gave my 6 year old son a LEGO game, a Star Wars plane with some Androids to assemble. He was able to do that without help using the instructions. When asked if he could do that without he said that it would be quite hard. In other words, he would need to get out of his comfort zone. I prodded him a bit and he assembled the next one without the instructions. Of course that was harder and took longer. But what he learned there will stay with him, he’ll assemble the next one much easier. And the one after that. If he’s always following the instructions he learns exactly that, don’t think, follow the instructions.
In many ways it was easy for him to do something different as he’s a child and therefore constantly challenging his comfort zones and I’m encouraging that. But if that was never high on your parents agenda and your employers don’t expect that from you I can see why following a process is the easy option.
Comments welcome..
Thomas
OK, that question is too wide so let’s try again. Why does someone wants to put the process first and someone else the problem first?
In my experience that’s due to at least two major reasons, personality and experience. Let’s look at personality first.
Some people feel trapped by processes and limited in their freedom and abilities. They rely on their skills and abilities and trust whatever comes that they will find an answer even it they don’t know it, yet.
Some feel that with a process and documents in place there is something that they can fall back on to prove that what they’re doing is right. That they can give the process to someone else and that they can then also repeat it providing a base for a bigger group of people to learn from what they did before.
As a test manager I can’t influence people’s personality, that’s why there’s always an amount of touch and feel in any project decisions. Most decisions are emotionally impacted, I haven’t seen people looking at ROI and NOT decide based on their emotional reaction.
For the experience part, someone might have worked in a big, process driven corporation where there was a reason for processes. Or they worked in a small shop where anything goes and only a result counted, any result. I worked in the pharmaceutical industry and research scientist for a good number of years. GLP was a regulation we needed to comply with and if you ever heard of thalidomide you’ll know why that is a good idea. There’s an interesting article for further reading.
Processes were in place to satisfy these regulations. However having done a lot of the "work on the floor" most of the processes just didn’t make sense to me. After working for several years in the IT industry I now understand what they were trying to achieve, but no one explained it to me at the time. But I also now understand why the processes failed miserably. My point is that if you always worked in process driven companies you don’t know any other approach as very often there is a good reason why the processes are in place. That doesn’t mean they’re always working in practice though. I worked with a lot with red tape, ditched it all, and are now coming out the other end being somewhere in the middle.
Most companies want to be able to run a profitable project, have a satisfied customer and be able to repeat that over and over again. So why not putting your working habits into processes? Wrong, no two development projects are the same and a habit is just that. Wikipedia has this for Habit:
Habits are routines of behavior that are repeated regularly and tend to occur subconsciously, without directly thinking consciously about them.
The question is, do we actually want that? When you document a process and don’t want people to think about it but follow it, that’s fine. But when your next development project encounters an unforeseen problem don’t be surprised that your team can’t deal with it as you created a culture where people don’t think about what they’re doing.
I see the clash not between the process vs non-process approach but if you want to invest into training your people so that the whole team understands what’s expected from them and what’s important; or if you think that this route is too expensive and that you rather tell them what to do and have a few “in the know” who are creating the processes.
I think that training or lack thereof is at the root at the problem when we’re talking about processes as they’re often used to mask that people are not trained enough to do their jobs without them. If a developer doesn’t understand that throwing software over the wall to the testers impacts their work this is essentially a training issue. If a tester doesn’t understand what information is essential to the PM this is a training issue. If the PM doesn’t understand what to ask from the developer or tester, you guessed it, that’s a training issue as well.
I gave my 6 year old son a LEGO game, a Star Wars plane with some Androids to assemble. He was able to do that without help using the instructions. When asked if he could do that without he said that it would be quite hard. In other words, he would need to get out of his comfort zone. I prodded him a bit and he assembled the next one without the instructions. Of course that was harder and took longer. But what he learned there will stay with him, he’ll assemble the next one much easier. And the one after that. If he’s always following the instructions he learns exactly that, don’t think, follow the instructions.
In many ways it was easy for him to do something different as he’s a child and therefore constantly challenging his comfort zones and I’m encouraging that. But if that was never high on your parents agenda and your employers don’t expect that from you I can see why following a process is the easy option.
Comments welcome..
Thomas
Tuesday, 26 January 2010
Do I have worth?
When pondering the training plan for my team I remembered this question from Terry Pratchett’s book Unseen Academicals as the underlying driver for one of the main characters. In the book the character is constantly concerned that he doesn’t have “worth” and tries his best to prove, according to the guidelines he was brought up with, that the world is a better place with him in it.
In context with training a team of software tester I thought, who’s asking? Who does a software tester provide with something that is useful? And who decides what’s defined as useful? Depending on who’s doing the answering I’d get different responses. So I had a look at what job sites are looking for and compared that to what I’d be looking for in a tester in our company. Most job sites put an emphasis on the Skill and Domain section but leave out the business side. I had a picture in my mind how these three areas should overlap and it helps me to ponder the problem further, see below.
For clarification what I mean with Domain is knowledge as a subject matter expert (SME) – knowing about financial systems and accounting in banks, phone networks in telecommunication, etc.
Skill for me would be everything on the more technical side, ie writing test plans, scripts, test techniques, writing SQL statements, knowledge of particular tools, etc.
That leaves the Business side in this triangle with which I mean anything to do with working as part of a project team in a company. For example, knowing what to report to a Project Manager, when to report, knowing when to escalate matters , where to archive your test data, how to contact your system administrator, etc.
As a test manager, a lot of issues that I’m dealing with are NOT in the domain or skills section but the business area, still, no jobsite thinks it’s even worth mentioning. People not raising project risks at the time, telling PM’s about holidays they booked, etc is a common complaint I get, am I alone in that?
This diagram is a gross simplification and ignores things like experience and ability to communicate effectively. The latter I think is specific for each area as someone might be able to communicate rather well on a technical level but would be lost to explain the same thing to a senior manager. I rather see these unwritten parts like experience or communication as flowing around all three areas. In my opinion, if someone is very good in one area it makes up for weaknesses in others – to an extent.
I could extend it and use colour gradients or additional parts to signify experience, communication skill in that area but to me it’s not about making sure that I don’t miss anything or that I measure exactly the proficiency level. I don’t believe that this can be done reliably but it can be of help to remind me of the different areas that a tester should be competent in.
This diagram isn’t really restricted to testers, it’s more generic and could apply to any profession, really. I just started from the tester’s perspective as that is my background.
Now with the yearly appraisals around the corner (yes, I do work in a reasonably sized company), I will use this diagram and discuss how my team would like to expand their knowledge in each area.
I’ve got this nagging feeling that I’m overlooking something big as this diagram seems to be quite generic and wiser people than me must have done written something about it. If there are any links that you know of that go a bit deeper, please do let me know.
Thomas
In context with training a team of software tester I thought, who’s asking? Who does a software tester provide with something that is useful? And who decides what’s defined as useful? Depending on who’s doing the answering I’d get different responses. So I had a look at what job sites are looking for and compared that to what I’d be looking for in a tester in our company. Most job sites put an emphasis on the Skill and Domain section but leave out the business side. I had a picture in my mind how these three areas should overlap and it helps me to ponder the problem further, see below.
Skill for me would be everything on the more technical side, ie writing test plans, scripts, test techniques, writing SQL statements, knowledge of particular tools, etc.
That leaves the Business side in this triangle with which I mean anything to do with working as part of a project team in a company. For example, knowing what to report to a Project Manager, when to report, knowing when to escalate matters , where to archive your test data, how to contact your system administrator, etc.
As a test manager, a lot of issues that I’m dealing with are NOT in the domain or skills section but the business area, still, no jobsite thinks it’s even worth mentioning. People not raising project risks at the time, telling PM’s about holidays they booked, etc is a common complaint I get, am I alone in that?
This diagram is a gross simplification and ignores things like experience and ability to communicate effectively. The latter I think is specific for each area as someone might be able to communicate rather well on a technical level but would be lost to explain the same thing to a senior manager. I rather see these unwritten parts like experience or communication as flowing around all three areas. In my opinion, if someone is very good in one area it makes up for weaknesses in others – to an extent.
I could extend it and use colour gradients or additional parts to signify experience, communication skill in that area but to me it’s not about making sure that I don’t miss anything or that I measure exactly the proficiency level. I don’t believe that this can be done reliably but it can be of help to remind me of the different areas that a tester should be competent in.
This diagram isn’t really restricted to testers, it’s more generic and could apply to any profession, really. I just started from the tester’s perspective as that is my background.
Now with the yearly appraisals around the corner (yes, I do work in a reasonably sized company), I will use this diagram and discuss how my team would like to expand their knowledge in each area.
I’ve got this nagging feeling that I’m overlooking something big as this diagram seems to be quite generic and wiser people than me must have done written something about it. If there are any links that you know of that go a bit deeper, please do let me know.
Thomas
Wednesday, 20 January 2010
Testers - results orientation or integrity?
Hello,
this is my first blog post, I'd like to encourage you to leave criticism or feedback. I know I have a lot to learn so don't try and spare my feelings ;-) Here we go..
I started thinking about the importance of integrity for software testers after exchanging opinions on SoftwareTestingClub with Matt Heusser.
To me integrity has always been important but it was always a bit hazy and not very specific. I recently worked at job descriptions and put Integrity high on the list. It was later replaced with “Results Orientation” and “Planned and Organised” by the powers that be.
It could mean that these are a) more important or b) easier to understand by management. I argued for leaving integrity in but was overruled. In that case that wasn't a big issue for me even though I didn't like it.
If I say “Integrity”, what do I mean? I like the Wiki definition:
“Integrity is consistency of actions, values, methods, measures, principles, expectations and outcome.”
If a tester makes himself a name for having integrity what does that mean for their clients or employers? According to that definition it’s both clear and predictable what to expect from this person. By having a short discussion the values, methods, measures and principles can quickly be established. Or, in other words, integrity instils a sense of trust at the basic level. It still doesn’t mean that the tester in question is a good choice for a particular job, he might not have the right skillset, availability, etc.
In a software project context integrity is threatened in various ways – pressure from team members, Project Managers or people higher up in the food chain, customers, etc.
Project managers might ask testers to take shortcuts in the agreed test approach. When asked to mark tests as run even if they haven’t been executed or bugs not being reported as found; the tester will find themselves under a lot of pressure. Do you stand your ground and risk being shouted at, threatened or ridiculed? Or do you give in and take the shortcut to reduce the pressure? Under what conditions would you go against your values and principles?
Team members can put pressure on the tester to not log bugs as they might have run out of time to fix them before the deadline – anticipating pressure themselves from the PM.
In situations like that I found it important to remind myself that there is a two way relationship. One person asking to take the shortcut (maybe for a good reason) and the other person who can agree or disagree. Even though it doesn’t look like it at the time, one always has a choice.
For permanent staff and maybe even more so for contractors their integrity is an asset. If a contractor gets a name for not having integrity their career will quickly be at an end. Bowing out of the project might not be a good idea short term, long term that decision is invaluable.
Testers are not the gatekeepers of quality, even if it may sometimes feel that way, see the Enforcer in Rob Lambert’s light-hearted approach to tester types. Maybe there’s a reason why people are putting pressure on the tester and threaten their integrity.
Weinberg’s Rule of Three from his book “Secrets of Consulting” states "If you can't think of three things that might go wrong with your plans, then there's something wrong with your thinking." Finding out what the real goal is should be if not the first, but maybe the second reaction. Integrity might not be threatened at all.
The reaction to outside pressure is, of course, depending on the situation and personal values, methods, measures and principles. If the agreed test approach is no longer viable, maybe a new one should be agreed and documented.
If a project goes down that route there is a visible trace of why decisions have been taken and when. The tester protects themselves by getting agreement from the PM and other stakeholders which should be good engineering practice. The end result is the fear that comes with pressure – fear of not hitting the deadline resulting in loss of face, money or job – is removed by taking a realistic and professional stance.
Do I believe that integrity is important for a tester? Yes, of course. We’re some of the last cogs in the software development wheel who can highlight some of the things that might reduce the value of the product or solution our company is trying to sell. Should a tester be results orientated? Yes, knowing how to get the best results fast is important for the business. Let me ask though, when sitting in a plane waiting to take you on your holiday, would you rather have an airplane engineer with results orientation or integrity working on it?
Thomas
PS: Thanks to Rob Lambert for his help to get me started and kind words. I appreciate it.
this is my first blog post, I'd like to encourage you to leave criticism or feedback. I know I have a lot to learn so don't try and spare my feelings ;-) Here we go..
I started thinking about the importance of integrity for software testers after exchanging opinions on SoftwareTestingClub with Matt Heusser.
To me integrity has always been important but it was always a bit hazy and not very specific. I recently worked at job descriptions and put Integrity high on the list. It was later replaced with “Results Orientation” and “Planned and Organised” by the powers that be.
It could mean that these are a) more important or b) easier to understand by management. I argued for leaving integrity in but was overruled. In that case that wasn't a big issue for me even though I didn't like it.
If I say “Integrity”, what do I mean? I like the Wiki definition:
“Integrity is consistency of actions, values, methods, measures, principles, expectations and outcome.”
If a tester makes himself a name for having integrity what does that mean for their clients or employers? According to that definition it’s both clear and predictable what to expect from this person. By having a short discussion the values, methods, measures and principles can quickly be established. Or, in other words, integrity instils a sense of trust at the basic level. It still doesn’t mean that the tester in question is a good choice for a particular job, he might not have the right skillset, availability, etc.
In a software project context integrity is threatened in various ways – pressure from team members, Project Managers or people higher up in the food chain, customers, etc.
Project managers might ask testers to take shortcuts in the agreed test approach. When asked to mark tests as run even if they haven’t been executed or bugs not being reported as found; the tester will find themselves under a lot of pressure. Do you stand your ground and risk being shouted at, threatened or ridiculed? Or do you give in and take the shortcut to reduce the pressure? Under what conditions would you go against your values and principles?
Team members can put pressure on the tester to not log bugs as they might have run out of time to fix them before the deadline – anticipating pressure themselves from the PM.
In situations like that I found it important to remind myself that there is a two way relationship. One person asking to take the shortcut (maybe for a good reason) and the other person who can agree or disagree. Even though it doesn’t look like it at the time, one always has a choice.
For permanent staff and maybe even more so for contractors their integrity is an asset. If a contractor gets a name for not having integrity their career will quickly be at an end. Bowing out of the project might not be a good idea short term, long term that decision is invaluable.
Testers are not the gatekeepers of quality, even if it may sometimes feel that way, see the Enforcer in Rob Lambert’s light-hearted approach to tester types. Maybe there’s a reason why people are putting pressure on the tester and threaten their integrity.
Weinberg’s Rule of Three from his book “Secrets of Consulting” states "If you can't think of three things that might go wrong with your plans, then there's something wrong with your thinking." Finding out what the real goal is should be if not the first, but maybe the second reaction. Integrity might not be threatened at all.
The reaction to outside pressure is, of course, depending on the situation and personal values, methods, measures and principles. If the agreed test approach is no longer viable, maybe a new one should be agreed and documented.
If a project goes down that route there is a visible trace of why decisions have been taken and when. The tester protects themselves by getting agreement from the PM and other stakeholders which should be good engineering practice. The end result is the fear that comes with pressure – fear of not hitting the deadline resulting in loss of face, money or job – is removed by taking a realistic and professional stance.
Do I believe that integrity is important for a tester? Yes, of course. We’re some of the last cogs in the software development wheel who can highlight some of the things that might reduce the value of the product or solution our company is trying to sell. Should a tester be results orientated? Yes, knowing how to get the best results fast is important for the business. Let me ask though, when sitting in a plane waiting to take you on your holiday, would you rather have an airplane engineer with results orientation or integrity working on it?
Thomas
PS: Thanks to Rob Lambert for his help to get me started and kind words. I appreciate it.
Subscribe to:
Posts (Atom)