I recently attended a Project Managers meeting where PM specific, day to day problems were discussed.
Some were quite generic in nature and similar to a tester meeting, for example how do we make sure that people are properly trained, dealing with communication problems, etc. Others were more specific, for example talking about managing customer relationships, how a customer may perceive problems in a project from a completely different angle (for the better or the worse for the PM in question).
What occurred to me there is that we put a lot of emphasis in testing on adding “value for the customer”, however most testers that I know have no idea about customer management, and how our customers priorities might be different from the ones that the development team (programmers and testers) are thinking of. Managing that relationship sits squarely in the corner of the PM or Account Manager (where available). In a way it’s good that the Project Manager is shielding the development team from sometimes difficult customers, but to what degree can a tester really say that what they’re doing is actually adding value for the client?
That reminded me that what we think of as “adding value” is actually based on a model. To quote George Box “All models are wrong, some are useful”. Is our model built on validation and verification – in a broad way ensuring that the system works as intended and works without glitches? But what do we base that on since we all know that requirement documents are notoriously lacking? The argument I often hear is that testers bring their experience in adding what they perceive to be of value to other customers so it might be of value to the current project as well. With no knowledge of the customer or our relationship with them how can this be true?
At the next team meeting it might be an idea to invite the Account Manager as well or, if the PM is fulfilling that role as well, ask him about our relationship to the client. Is it perceived to be a difficult one? Have we worked with that client before? Is the project seen as a start of many others to follow? Do we know of particular things that “bug” the client that would help us direct our test effort?
I don’t know how other people address that but these are not questions that I have asked in the past but maybe I should. I attended a Project Management course and think that all testers should do that sooner or later, ideally followed with actually managing a project or two as well.
Many testers have an idea of how programmers think, many having a background in programming themselves. Maybe it’s time that we start to look through the eyes of a Project Manager as well and walk a mile in their shoes.
Thursday, 28 October 2010
Wednesday, 20 October 2010
The pictorial challenge answered
After a short discussion with Zeger (TestSideStory) on Twitter about the pictorial challenge it was time to do some work but I came back to the picture later. Have a look at it first before reading on.
I sat back and thought “How would I test this if I don’t know what’s important and don’t know to whom it will be important?”
I then collated some high level areas of approach
• Death by Goggle:
o Searching the net for the names on the signs to give me more avenues to consider
• Technical: It’s a .jpg of a certain size, resolution and colours used
o Is that on purpose, in which scenarios could it be a problem?
o Are there parts of the picture that look cut off, is there too much sky, etc.
o Are there hints that this is a collage of pictures or is it one photograph
• Emotional: How do I react to the picture when looking at it first.
o How does that change when picking out more details?
I started with the emotional part first because a) I can’t help it and b) to me testing a picture is more akin to criticizing a painting so I wanted see if the artist/photographer got an immediate emotional reaction from me. The signs look rusty and barely legible. Rust to me indicates age so I assume they’ve been there for a long time. The house looks like it has seen better days, my first reaction is that it’s been taken in a poor part of whatever town/country. The boy on the roof (I assume it’s a roof) would be unusual in a UK town, maybe it’s not where this picture is taken? Filed away for future consideration.
I then searched for “Zanchett” and “La Mejor Ropa”, the latter of which is Spanish for “The best clothes”. But maybe it's not "Zanchett" but "Zanchetti". That's where the font type could throw a spanner into the tester's work. You live and learn. Again, more pages but I didn't follow them up too closely as I'm more interested in the picture rather than where it was taken. Zanchett seems to be quite a common name, there are several facebook sites, a Portuguese Tourism page to name a few. I also found links to a breastfeeding site in Brazil, so maybe it’s not Spanish but Portuguese. The boy (man?) on the roof is wearing a t-shirt, short trousers and sandals which indicates a warm country. At a guess I’d say the photo was taken in Brasil or Argentina.
The breastfeeding site also made me look at the photo closer, maybe the blue plastic bowl hanging at the outside of the house is actually a baby bath. The bike on the left, caged birds on the right and the boy on the roof could mean it’s a nursery or orphanage (because of what I assume is a poor area, I could be wrong here). In my experience Mediterranean houses don’t give the impression that they’re looked after from the outside, that doesn’t mean that they’re not in very good condition from the inside. There’s an angle of cultural differences here and since I assume Southern America here as the location I’ll have to question my assumptions. The boy/man (I'm not able to clearly distinguish the features to say for sure) is wearing sports clothes which seem new or in good shape at least so maybe poor isn't actually correct here.
I didn’t go into the technical details other than noting that the size is 601 x 776 pixels – the 601 seems odd to me, maybe it was converted from a different size or type?
I’ll get in touch with Zeger to see what he thinks about it. :)
I sat back and thought “How would I test this if I don’t know what’s important and don’t know to whom it will be important?”
I then collated some high level areas of approach
• Death by Goggle:
o Searching the net for the names on the signs to give me more avenues to consider
• Technical: It’s a .jpg of a certain size, resolution and colours used
o Is that on purpose, in which scenarios could it be a problem?
o Are there parts of the picture that look cut off, is there too much sky, etc.
o Are there hints that this is a collage of pictures or is it one photograph
• Emotional: How do I react to the picture when looking at it first.
o How does that change when picking out more details?
I started with the emotional part first because a) I can’t help it and b) to me testing a picture is more akin to criticizing a painting so I wanted see if the artist/photographer got an immediate emotional reaction from me. The signs look rusty and barely legible. Rust to me indicates age so I assume they’ve been there for a long time. The house looks like it has seen better days, my first reaction is that it’s been taken in a poor part of whatever town/country. The boy on the roof (I assume it’s a roof) would be unusual in a UK town, maybe it’s not where this picture is taken? Filed away for future consideration.
I then searched for “Zanchett” and “La Mejor Ropa”, the latter of which is Spanish for “The best clothes”. But maybe it's not "Zanchett" but "Zanchetti". That's where the font type could throw a spanner into the tester's work. You live and learn. Again, more pages but I didn't follow them up too closely as I'm more interested in the picture rather than where it was taken. Zanchett seems to be quite a common name, there are several facebook sites, a Portuguese Tourism page to name a few. I also found links to a breastfeeding site in Brazil, so maybe it’s not Spanish but Portuguese. The boy (man?) on the roof is wearing a t-shirt, short trousers and sandals which indicates a warm country. At a guess I’d say the photo was taken in Brasil or Argentina.
The breastfeeding site also made me look at the photo closer, maybe the blue plastic bowl hanging at the outside of the house is actually a baby bath. The bike on the left, caged birds on the right and the boy on the roof could mean it’s a nursery or orphanage (because of what I assume is a poor area, I could be wrong here). In my experience Mediterranean houses don’t give the impression that they’re looked after from the outside, that doesn’t mean that they’re not in very good condition from the inside. There’s an angle of cultural differences here and since I assume Southern America here as the location I’ll have to question my assumptions. The boy/man (I'm not able to clearly distinguish the features to say for sure) is wearing sports clothes which seem new or in good shape at least so maybe poor isn't actually correct here.
I didn’t go into the technical details other than noting that the size is 601 x 776 pixels – the 601 seems odd to me, maybe it was converted from a different size or type?
I’ll get in touch with Zeger to see what he thinks about it. :)
Monday, 2 August 2010
The need for confirmation – why checking against requirements is not going away
I was thinking what prompts the need for traceability and ensuring that the final application actually satisfies the requirements that were captured at the beginning of the project. Who needs that traceability? I then thought about different companies. The type of company a tester works in has a huge impact on the approach of their testing. Broadly speaking (read: this is a gross generalisation) we have these types of companies:
1. Product companies selling shrink-wrap software. This is anything from operating systems, games, tax return helpers, computer helper apps (Adobe Reader), etc.
2. Consultancies developing a bespoke application for their clients to address a specific business problem
3. Hybrid between the first two, for example further developing/configuring a shrink-wrap application, for example SAP or other back-office systems.
4. Large company developing a bespoke application in-house for internal use only, for example banks or insurance companies.
Let’s have a look at each from different angles.
The first type of company producing products is somewhat unique, as there is not one, single client that determines the requirements. They need to rely on Product Managers or other staff who can guess or work out that there is a target audience who’s likely to use and/or buy their software. Anticipating what the market needs determines the requirements.
Many products, when selling well, will have several versions and releases. That’s good news for the automation people as products often need to be regression tested for a variety of platforms, i.e. different operating systems, hardware configurations, or when new functionality is added over time. The quality necessary for their products depends on the market they’re selling to.
The second type is very much focussed on the one client they’re selling to. What’s in the contract will get built (and tested if the contract specifies it). What’s not in the contract depends on the good will of the consultancy. If a client just spent £3M with you and you wave the contract instead of adding a 2 day RFC that’s probably the last big contract your company got.
Automated regression testing depends wholly on the client – have they specified that they want it, is the application large enough to warrant it for just the one version, are there likely to be more version of the software and contracts coming in from the client? Then it might make sense building something automated, otherwise forget it, time is money in this setup.
The hybrid company is closer to the Consultancy but with the added complication that now there is third party code to be considered for testing. More often than not that has already been tested but one has to be careful to exclude functions as they may be used differently in the context of the re-developed/configured new solution.
The in-house development test efforts very much depend on the company, however there usually are requirements that have been developed by internal groups. This type is usually similar to the Product companies in that software developed is there to stay for probably more than one version. That means more effort can be spent on testing it properly and with a long term view.
To come back to the need for confirmation – if you’re working in some sort of Consultancy, traceability is a no brainer. You’ll need to confirm that what your company is delivering is adhering to the requirements that were set out – and by extension that you fulfil the contract. From the Consultancy point of view satisfying the contract has to come first, otherwise the business would leave itself wide open to being sued or just not paid for breaching the contract. So once the confirmation is out of the way there might or might not be time to test the application in detail.
If you’re working in a product company or in-house development effort it’s a bit different. There are other factors to consider though. Are there any legal requirements to adhere to certain standards? Does the software need to be certified against particular standards? GLP or GCP comes to mind. Traceability is maybe not so important in this setup compared to the Consultancies, unless the PM or project owners insists on it, I’d argue out of the false hope that the system will then actually deliver what was expected when the requirements were written.
There was an argument on one of the LinkedIn groups if checking/testing against requirements is a hoax or if it adds value. As always, the answer is, it depends. If you have a 1:1 relationship of tests vs requirements I’d say it’s a hoax. All requirements need more than one simple test. So saying that a particular requirement has been tested against, then ticking the box, fine, that’s implemented is giving a wrong picture.
Imagine you have a requirements that says that you need to have several user roles in the system. Would you confirm that with one single check? You could, but would it actually give you the information and confidence in the application that the stakeholders ask us to provide? I don’t think so. Why artificially tick a box and say, yes, there are several user roles in the system if you’re telling a white lie if you haven’t tested the around that requirement?
I can see why checking against requirements is popular as it’s easy to put into contracts. The effectiveness of it from a quality point of view is almost non-existent, so it might help the sales and legal people (and not even that if the customer is set on suing a non-compliant company) but the PM should be looking elsewhere for confidence that the final product is actually what the customer wanted in the first place.
1. Product companies selling shrink-wrap software. This is anything from operating systems, games, tax return helpers, computer helper apps (Adobe Reader), etc.
2. Consultancies developing a bespoke application for their clients to address a specific business problem
3. Hybrid between the first two, for example further developing/configuring a shrink-wrap application, for example SAP or other back-office systems.
4. Large company developing a bespoke application in-house for internal use only, for example banks or insurance companies.
Let’s have a look at each from different angles.
The first type of company producing products is somewhat unique, as there is not one, single client that determines the requirements. They need to rely on Product Managers or other staff who can guess or work out that there is a target audience who’s likely to use and/or buy their software. Anticipating what the market needs determines the requirements.
Many products, when selling well, will have several versions and releases. That’s good news for the automation people as products often need to be regression tested for a variety of platforms, i.e. different operating systems, hardware configurations, or when new functionality is added over time. The quality necessary for their products depends on the market they’re selling to.
The second type is very much focussed on the one client they’re selling to. What’s in the contract will get built (and tested if the contract specifies it). What’s not in the contract depends on the good will of the consultancy. If a client just spent £3M with you and you wave the contract instead of adding a 2 day RFC that’s probably the last big contract your company got.
Automated regression testing depends wholly on the client – have they specified that they want it, is the application large enough to warrant it for just the one version, are there likely to be more version of the software and contracts coming in from the client? Then it might make sense building something automated, otherwise forget it, time is money in this setup.
The hybrid company is closer to the Consultancy but with the added complication that now there is third party code to be considered for testing. More often than not that has already been tested but one has to be careful to exclude functions as they may be used differently in the context of the re-developed/configured new solution.
The in-house development test efforts very much depend on the company, however there usually are requirements that have been developed by internal groups. This type is usually similar to the Product companies in that software developed is there to stay for probably more than one version. That means more effort can be spent on testing it properly and with a long term view.
To come back to the need for confirmation – if you’re working in some sort of Consultancy, traceability is a no brainer. You’ll need to confirm that what your company is delivering is adhering to the requirements that were set out – and by extension that you fulfil the contract. From the Consultancy point of view satisfying the contract has to come first, otherwise the business would leave itself wide open to being sued or just not paid for breaching the contract. So once the confirmation is out of the way there might or might not be time to test the application in detail.
If you’re working in a product company or in-house development effort it’s a bit different. There are other factors to consider though. Are there any legal requirements to adhere to certain standards? Does the software need to be certified against particular standards? GLP or GCP comes to mind. Traceability is maybe not so important in this setup compared to the Consultancies, unless the PM or project owners insists on it, I’d argue out of the false hope that the system will then actually deliver what was expected when the requirements were written.
There was an argument on one of the LinkedIn groups if checking/testing against requirements is a hoax or if it adds value. As always, the answer is, it depends. If you have a 1:1 relationship of tests vs requirements I’d say it’s a hoax. All requirements need more than one simple test. So saying that a particular requirement has been tested against, then ticking the box, fine, that’s implemented is giving a wrong picture.
Imagine you have a requirements that says that you need to have several user roles in the system. Would you confirm that with one single check? You could, but would it actually give you the information and confidence in the application that the stakeholders ask us to provide? I don’t think so. Why artificially tick a box and say, yes, there are several user roles in the system if you’re telling a white lie if you haven’t tested the around that requirement?
I can see why checking against requirements is popular as it’s easy to put into contracts. The effectiveness of it from a quality point of view is almost non-existent, so it might help the sales and legal people (and not even that if the customer is set on suing a non-compliant company) but the PM should be looking elsewhere for confidence that the final product is actually what the customer wanted in the first place.
Thursday, 17 June 2010
EWT22 - How to think about science/Something Fishy
At EWT22, Michael Bolton moderated a new type of session – listening to an audio recording “How To Think About Science” and finding parallels to software testing.
Background to the interview:
On July 3, 1992, the Canadian Fisheries Minister John Crosbie announced a moratorium on the fishing of northern cod. It was the largest single day lay-off in Canadian history: 30,000 people unemployed at a stroke. The ban was expected to last for two years, after which, it was hoped, the fishery could resume. But the cod have never recovered, and more than 15 years later the moratorium remains in effect. How could a fishery that had been for years under apparently careful scientific management just collapse? David Cayley talks to environmental philosopher Dean Bavington about the role of science in the rise and fall of the cod fishery."
The MISSION:
Listen to the recording. Take notes on what you’re hearing, with the goal of capturing lessons from the interview that might usefully influence how we test, and how we think about testing. Stories or patterns from your own testing experience that you can relate to something in the interview are especially valuable.
My experience tells me that it's good to take lots of notes, regardless if it's reviewing documents or acively testing. I fully expect to discard 90% of it but it can serve to confirm or discard ideas, theories and guesses.
So I listened to the interview and made notes about important events and their order, my raw data. I'd stop from time to time to listen again to passages to get a clearer understanding and to be able to keep up with note writing.
Michael clarified some terms that were used in Skype always shortly before I got to that passage. That was helpful on two fronts. One, I understood the term and context quicker. And two, I could tell that I was falling behind with my progress in listening as it took longer and longer between Michael explaining terms and me getting to the passage - a good way to measure progress, though fallible.
About 2/3rds in I changed the approach as I felt that I had enough data and because time was pressing. I started drawing parallels between the fishery events and software testing and noted those as well. Adding
to the raw data suffered as a result but that was fine as I was working towards the mission, drawing parallels and matching them to my experience. I concentrated on the similarities in process and process failings rather than thinking of 'fishing for bugs' or comparing the government quotas to test certification like some other people in the group did. Nothing wrong with either, I just took this particular approach.
Listening to the audio recording I noted several occasions of Hubris as reason for that monumental failure of preserving the cod population.
From Wiki: Hubris means extreme haughtiness or arrogance. Hubris often indicates being out of touch with reality and overestimating one's own competence or capabilities, especially for people in positions of power.
I reckon that is why the radio program was called "How to think about science", the idea that, while science has not all the answers, the ones it has can be taken as the truth.
That's only one part of it though, I don't want to make the same mistake and believe that one reason alone can explain the failures on many fronts.
Since I'm currently reading Jerry Weinberg's book "The Psychology of Computer Programming" (Thank you, Jerry, for giving me permission to use parts of this book) I saw some interesting similarities between his study of the psychology of computer programming and the events that lead to the disappearance of the Canadian cod. Here's Jerry's list from page 40 of his book.
1. Using introspection without validation
2. Using of observations on an overly narrow base
3. Observing the wrong variables, or failing to observe the right ones
4. Interfering with the observed phenomenon
5. Generating too much data and not enough information
6. Overly constraining experiments
7. Depending excessively on trainees as subjects
8. Failing to study group effects and group behaviour
9. Measuring what is simple to measure
10. Using unjustified precision
11. Transferring borrowed results to inapplicable situations
Matching my notes to this list I got the following results. We could argue with some of the matches, however the overall match of similarities should become clear:
1a. Key drivers were managing nature fluctuations in order to get back the money that was invested in the industry. Fluctuations turned out to be unmanageable – there was no driver to understand the model in detail
1b. Introducing averaging assumptions was one of the traps – the issue was not understood and two wildly different results had just been averaged with not thought of the consequences. One example of Hubris.
2. There was a simple model for the amount of cod in the sea which was far too simple and therefore wrong
3a Not all parameters of the model had been identified, for example parameter changes over time had been missed.
3b Some parameters had been misunderstood. These two points were the biggest mistakes made in my book. Science was not seen as fallible but a blind eye turned to the possibility that the current understanding does not fully cover the situation in question - see the definition for Hubris again.
4. There was constant interfering with the model as fishing continued throughout the years up to the point where the cod population didn't recover
5. Data from two sources was created, the scientific trawlers and the commercial fleet. They only seemed to look at total population, not age groups, etc
6. Local knowledge of inshore fishermen = “domain matter experts” has been ignored/dismissed as anecdotal – scientific methods were seen as more accurate and objective. Another example of Hubris
7. Not applicable in this scenario?
8. There were several examples of failing to recognise group behaviour, for example variation in location of spawning grounds and season, migration patterns, etc
9. Parameters found were dismissed as complicating the model too much (and making it unmanageable)
10. Not applicable in this scenario?
11. There were several examples of borrowing results from one are to another, for example assuming that inshore groups of cod behave the same as off shore groups
Some failings that I can't put into any of the categories but which are very important as well are:
A. The inshore fishermen’s protests have not been looked at in detail. They wanted to fish in the traditional way but reasons for that were not investigated. Scientists were supposed to speak truth. I'm wondering if, with a bit more questioning, some of the mistakes may have been prevented, for example damaging and destroying stock using the jigger (hook).
B. Investing in tools to make it cheaper became a driver to make more money which backfired. The parallel to automated software testing tools should be easy to see.
C. The fishing industry put pressure on the government to get higher allowable quotas. Available information was ignored and numbers changed against better knowledge to serve the industries need for money.
D. As the model got very complicated it didn’t help management as it didn't provide the control measures that were needed. In software terms this is the point where a project is either stopped or a drawn out death march begins
E. The target of management changed from figuring out how to achieve the goal to targeting people, the fisherman. Sounds oddly familiar, doesn't it?
I find it quite amazing how two, apparently unrelated areas like a book about the psychology of computer programming and discussing shortfalls of science in a fishery project are so similar at a higher level. I have a good number of other examples in mind, some of which I'll blog about shortly.
I'm in two minds about the message that science got it all wrong. I'd summarise it as scientists (bringing the people component in here) probably did their best but didn't look at their limitations. Results of their experiments were also disregarded and decisions taken based on politics and the industries needs/greed.
I come from a science background myself so may have a blind spot there. My overall (scientific approach/thinking) is to identify a problem, analyse it, put up some theories and experiments/tests that then either prove or disprove these theories. In other words, I'm building a close as possible picture or model of the issue in question. Any steps up to the results can be criticised, the result can not. The thinking behind that is that if you agree with the thinking behind all steps the result is independent. You don't get to argue with it if you don't like it. The result can be a pointer that something is wrong with your thinking, it's dangerous to build your thinking based on the results you want though. That's a different story though and far to involved to be discussed here.
I'd be happy to discuss these results in more detail so please feel free to agree or, even better, disagree with me.
I should note that these findings are based on a single source, the radio interview. What actually happened might or might not be exactly as portrayed, but I think it's useful to keep in mind that in testing as in life it's dangerous to believe only in one source without questioning it. Or would you believe a developer who tells you that a bug has been fixed and doesn't need re-tested?
PS: I would also like to point you towards an excellent blog of Jeroen Rosink who looks at this session from a slightly different angle.
Background to the interview:
On July 3, 1992, the Canadian Fisheries Minister John Crosbie announced a moratorium on the fishing of northern cod. It was the largest single day lay-off in Canadian history: 30,000 people unemployed at a stroke. The ban was expected to last for two years, after which, it was hoped, the fishery could resume. But the cod have never recovered, and more than 15 years later the moratorium remains in effect. How could a fishery that had been for years under apparently careful scientific management just collapse? David Cayley talks to environmental philosopher Dean Bavington about the role of science in the rise and fall of the cod fishery."
The MISSION:
Listen to the recording. Take notes on what you’re hearing, with the goal of capturing lessons from the interview that might usefully influence how we test, and how we think about testing. Stories or patterns from your own testing experience that you can relate to something in the interview are especially valuable.
My experience tells me that it's good to take lots of notes, regardless if it's reviewing documents or acively testing. I fully expect to discard 90% of it but it can serve to confirm or discard ideas, theories and guesses.
So I listened to the interview and made notes about important events and their order, my raw data. I'd stop from time to time to listen again to passages to get a clearer understanding and to be able to keep up with note writing.
Michael clarified some terms that were used in Skype always shortly before I got to that passage. That was helpful on two fronts. One, I understood the term and context quicker. And two, I could tell that I was falling behind with my progress in listening as it took longer and longer between Michael explaining terms and me getting to the passage - a good way to measure progress, though fallible.
About 2/3rds in I changed the approach as I felt that I had enough data and because time was pressing. I started drawing parallels between the fishery events and software testing and noted those as well. Adding
to the raw data suffered as a result but that was fine as I was working towards the mission, drawing parallels and matching them to my experience. I concentrated on the similarities in process and process failings rather than thinking of 'fishing for bugs' or comparing the government quotas to test certification like some other people in the group did. Nothing wrong with either, I just took this particular approach.
Listening to the audio recording I noted several occasions of Hubris as reason for that monumental failure of preserving the cod population.
From Wiki: Hubris means extreme haughtiness or arrogance. Hubris often indicates being out of touch with reality and overestimating one's own competence or capabilities, especially for people in positions of power.
I reckon that is why the radio program was called "How to think about science", the idea that, while science has not all the answers, the ones it has can be taken as the truth.
That's only one part of it though, I don't want to make the same mistake and believe that one reason alone can explain the failures on many fronts.
Since I'm currently reading Jerry Weinberg's book "The Psychology of Computer Programming" (Thank you, Jerry, for giving me permission to use parts of this book) I saw some interesting similarities between his study of the psychology of computer programming and the events that lead to the disappearance of the Canadian cod. Here's Jerry's list from page 40 of his book.
1. Using introspection without validation
2. Using of observations on an overly narrow base
3. Observing the wrong variables, or failing to observe the right ones
4. Interfering with the observed phenomenon
5. Generating too much data and not enough information
6. Overly constraining experiments
7. Depending excessively on trainees as subjects
8. Failing to study group effects and group behaviour
9. Measuring what is simple to measure
10. Using unjustified precision
11. Transferring borrowed results to inapplicable situations
Matching my notes to this list I got the following results. We could argue with some of the matches, however the overall match of similarities should become clear:
1a. Key drivers were managing nature fluctuations in order to get back the money that was invested in the industry. Fluctuations turned out to be unmanageable – there was no driver to understand the model in detail
1b. Introducing averaging assumptions was one of the traps – the issue was not understood and two wildly different results had just been averaged with not thought of the consequences. One example of Hubris.
2. There was a simple model for the amount of cod in the sea which was far too simple and therefore wrong
3a Not all parameters of the model had been identified, for example parameter changes over time had been missed.
3b Some parameters had been misunderstood. These two points were the biggest mistakes made in my book. Science was not seen as fallible but a blind eye turned to the possibility that the current understanding does not fully cover the situation in question - see the definition for Hubris again.
4. There was constant interfering with the model as fishing continued throughout the years up to the point where the cod population didn't recover
5. Data from two sources was created, the scientific trawlers and the commercial fleet. They only seemed to look at total population, not age groups, etc
6. Local knowledge of inshore fishermen = “domain matter experts” has been ignored/dismissed as anecdotal – scientific methods were seen as more accurate and objective. Another example of Hubris
7. Not applicable in this scenario?
8. There were several examples of failing to recognise group behaviour, for example variation in location of spawning grounds and season, migration patterns, etc
9. Parameters found were dismissed as complicating the model too much (and making it unmanageable)
10. Not applicable in this scenario?
11. There were several examples of borrowing results from one are to another, for example assuming that inshore groups of cod behave the same as off shore groups
Some failings that I can't put into any of the categories but which are very important as well are:
A. The inshore fishermen’s protests have not been looked at in detail. They wanted to fish in the traditional way but reasons for that were not investigated. Scientists were supposed to speak truth. I'm wondering if, with a bit more questioning, some of the mistakes may have been prevented, for example damaging and destroying stock using the jigger (hook).
B. Investing in tools to make it cheaper became a driver to make more money which backfired. The parallel to automated software testing tools should be easy to see.
C. The fishing industry put pressure on the government to get higher allowable quotas. Available information was ignored and numbers changed against better knowledge to serve the industries need for money.
D. As the model got very complicated it didn’t help management as it didn't provide the control measures that were needed. In software terms this is the point where a project is either stopped or a drawn out death march begins
E. The target of management changed from figuring out how to achieve the goal to targeting people, the fisherman. Sounds oddly familiar, doesn't it?
I find it quite amazing how two, apparently unrelated areas like a book about the psychology of computer programming and discussing shortfalls of science in a fishery project are so similar at a higher level. I have a good number of other examples in mind, some of which I'll blog about shortly.
I'm in two minds about the message that science got it all wrong. I'd summarise it as scientists (bringing the people component in here) probably did their best but didn't look at their limitations. Results of their experiments were also disregarded and decisions taken based on politics and the industries needs/greed.
I come from a science background myself so may have a blind spot there. My overall (scientific approach/thinking) is to identify a problem, analyse it, put up some theories and experiments/tests that then either prove or disprove these theories. In other words, I'm building a close as possible picture or model of the issue in question. Any steps up to the results can be criticised, the result can not. The thinking behind that is that if you agree with the thinking behind all steps the result is independent. You don't get to argue with it if you don't like it. The result can be a pointer that something is wrong with your thinking, it's dangerous to build your thinking based on the results you want though. That's a different story though and far to involved to be discussed here.
I'd be happy to discuss these results in more detail so please feel free to agree or, even better, disagree with me.
I should note that these findings are based on a single source, the radio interview. What actually happened might or might not be exactly as portrayed, but I think it's useful to keep in mind that in testing as in life it's dangerous to believe only in one source without questioning it. Or would you believe a developer who tells you that a bug has been fixed and doesn't need re-tested?
PS: I would also like to point you towards an excellent blog of Jeroen Rosink who looks at this session from a slightly different angle.
Monday, 10 May 2010
It’s NOT Rocket Science
This blog is to go into a bit more detail about the planning and execution of the European Weekend Testing mision 17.
If you don’t know what the mission was please have a look and read it.
In summary, the group should test an audio application, a software compressor plugin that needs its own host. The group was testing the plugin through the host which was an additional complication.
The mission for EWT17 was several weeks in the planning. I first came up with a rough outline, then bounced it off Anna and Markus who made it better without really knowing the application. We then pre-tested the whole mission with Anna and Markus on Thursday to make sure it was not too hard to do. I got interesting results to what they were looking at and we then decided that, yes, this mission is a go.
For me, the one of the main goals of the mission was to bring a bit more realism into it. So I introduced a bit of role playing in saying that all EWT testers are in one company in one group reporting to one test manager who has a problem he wants them to solve: Does this software compressor work and is stable enough for me to use tonight?
His problem is time limited and he wants an answer in 1.5 hours. I should rephrase that, he wants one answer. From his point of view it’s one problem, one question, one answer.
Of course in EWT people come from all over the world so this was always difficult to achieve but some people picked up on it as Jeroen also pointed out in his blog.
I wasn't actually expecting to get one but wasn't sure what people would do. Some testers refused an answer as per Michael Bolton's article.
Anna, Markus and me discussed if we should add a team lead role for this mission but decided against it as it wasn’t in line with the spirit of EWT to have a hierarchy. I was interested to see if one developed naturally or not, which didn’t happen – no problem there.
The other big difference in this mission compared to previous ones is that I picked a domain where not many people would have knowledge of – audio recording and engineering. There are actually a lot of parallels with software testing but that’s another story or blog.
When the mission was released people realised within a couple of minutes that they don’t know enough about the domain to be able to test the application. Some went into ET testing mode, some Googled and found the Wiki page that explains very well what it’s all about. No one thought about asking if there are domain experts in the group of which we had one, Zeger. He actually couldn’t get the software to start on his laptop so was available to answer questions. He picked up on some assumptions people made and threw in some very valuable comments. Some were picked up, others weren’t, for example the important one about asking if the test manager is using the same plug-in host or not as this could invalidate the test effort. Yes, he did and it would actually..
The Test Manager gave the group pretty much an impossible task. Find out if the application is stable. What does stable mean without context? Without knowing what hardware is used, operating system (as someone pointed out), other software installed stable is pretty much meaningless.
The other big area that I wanted to explore with this mission is testability. There are several possible meanings. One issue with testability is that the testers didn’t have enough domain knowledge to know how they would actually test the application. The Rocket reduces the dynamic range but how many testers know how to test that?
The other issue is about measuring if the dynamic range actually has been reduced and by the right amount. Since we’re talking about audio here we can’t rely on our senses as they aren’t up to the task. Sure we can hear if something gets quieter and louder but only if the difference is quite large. How large depends on the testers ears so it is really subjective.
There are some freeware tools that would help with this task, for example the freeware audio editor and recorder. A tester could then record the output of the plugin and compare waveforms or rely on the measure tools that come with the audio editor application. All in all a pretty difficult task. I only mention it here because the secondary objective was to find out if the attack of the plugin was fast enough. Something that could only be tested with additional tools and was largely ignored by the group, which was good. If you have your hands full with the main objective don’t get distracted with nice to have’s – thumbs up for the EWT testers.
The group dynamics where interesting to see and also the extend to which testers picked up on what information others typed into Skype. I tried to facilitate more group working which hasn’t worked so well. This is clearly something we need to think about again for future missions.
It was an intense session but one where I think everyone got something out of. To the new people who joined us for EWT17, it’s not always like that!
I’d like to everyone who participated who made this session a success and gave me lots to think about – and that’s a good thing.
Thomas
If you don’t know what the mission was please have a look and read it.
In summary, the group should test an audio application, a software compressor plugin that needs its own host. The group was testing the plugin through the host which was an additional complication.
The mission for EWT17 was several weeks in the planning. I first came up with a rough outline, then bounced it off Anna and Markus who made it better without really knowing the application. We then pre-tested the whole mission with Anna and Markus on Thursday to make sure it was not too hard to do. I got interesting results to what they were looking at and we then decided that, yes, this mission is a go.
For me, the one of the main goals of the mission was to bring a bit more realism into it. So I introduced a bit of role playing in saying that all EWT testers are in one company in one group reporting to one test manager who has a problem he wants them to solve: Does this software compressor work and is stable enough for me to use tonight?
His problem is time limited and he wants an answer in 1.5 hours. I should rephrase that, he wants one answer. From his point of view it’s one problem, one question, one answer.
Of course in EWT people come from all over the world so this was always difficult to achieve but some people picked up on it as Jeroen also pointed out in his blog.
I wasn't actually expecting to get one but wasn't sure what people would do. Some testers refused an answer as per Michael Bolton's article.
Anna, Markus and me discussed if we should add a team lead role for this mission but decided against it as it wasn’t in line with the spirit of EWT to have a hierarchy. I was interested to see if one developed naturally or not, which didn’t happen – no problem there.
The other big difference in this mission compared to previous ones is that I picked a domain where not many people would have knowledge of – audio recording and engineering. There are actually a lot of parallels with software testing but that’s another story or blog.
When the mission was released people realised within a couple of minutes that they don’t know enough about the domain to be able to test the application. Some went into ET testing mode, some Googled and found the Wiki page that explains very well what it’s all about. No one thought about asking if there are domain experts in the group of which we had one, Zeger. He actually couldn’t get the software to start on his laptop so was available to answer questions. He picked up on some assumptions people made and threw in some very valuable comments. Some were picked up, others weren’t, for example the important one about asking if the test manager is using the same plug-in host or not as this could invalidate the test effort. Yes, he did and it would actually..
The Test Manager gave the group pretty much an impossible task. Find out if the application is stable. What does stable mean without context? Without knowing what hardware is used, operating system (as someone pointed out), other software installed stable is pretty much meaningless.
The other big area that I wanted to explore with this mission is testability. There are several possible meanings. One issue with testability is that the testers didn’t have enough domain knowledge to know how they would actually test the application. The Rocket reduces the dynamic range but how many testers know how to test that?
The other issue is about measuring if the dynamic range actually has been reduced and by the right amount. Since we’re talking about audio here we can’t rely on our senses as they aren’t up to the task. Sure we can hear if something gets quieter and louder but only if the difference is quite large. How large depends on the testers ears so it is really subjective.
There are some freeware tools that would help with this task, for example the freeware audio editor and recorder. A tester could then record the output of the plugin and compare waveforms or rely on the measure tools that come with the audio editor application. All in all a pretty difficult task. I only mention it here because the secondary objective was to find out if the attack of the plugin was fast enough. Something that could only be tested with additional tools and was largely ignored by the group, which was good. If you have your hands full with the main objective don’t get distracted with nice to have’s – thumbs up for the EWT testers.
The group dynamics where interesting to see and also the extend to which testers picked up on what information others typed into Skype. I tried to facilitate more group working which hasn’t worked so well. This is clearly something we need to think about again for future missions.
It was an intense session but one where I think everyone got something out of. To the new people who joined us for EWT17, it’s not always like that!
I’d like to everyone who participated who made this session a success and gave me lots to think about – and that’s a good thing.
Thomas
Saturday, 17 April 2010
ACCU2010 writeup
Here’s a short (OK, I got carried away a bit) summary of the conference and the talks/presentations I attended. I only went down for one day - Friday 16th April due to work commitments.
Friday morning:
Keynote talk from Russell Winder “The Multicore Revolution - We’ve only just begun”. Originally Dan North was down to present “Simplicity – the way of the unusual architect” but couldn’t get to the UK due to the airspace being closed because of the Icelandic volcano eruption.
Russell is clearly very knowledgeable and presented a good talk about the architecture of computers the way different programming languages address those. Good presentation that was probably more interesting to the developer community than for me but interesting nonetheless.
After a coffee break which I used for frantic telephone calls and trying to book train tickets I attended Rachel Davies “Understanding User Stories”. Not having written any before this was an interesting presentation with just the right amount of practical exercises for the participants. She mentioned the book “Agile Coaching” which she wrote or co-wrote which I’ll have a look at.
She explained that she and her team used a user story template which is now being used by many people around the world. I got the impression that she didn’t expect that and was pleasantly surprised that such a small thing created to make life easier in her project team has such a big impact. She did not that this template doesn’t work in all cases – especially when there’s no discussion with the business or if technical implementations are the order of the day rather than user workflows.
The template followed a
As a...
I want...
So that...
format which goes a long way getting all the necessary information onto a small card. She noted the importance of this being a physical card rather than a spreadsheet or electronic document. To me this format is similar-ish to the way use cases are created as the actor (As a xyz..) and workflow (I want...) and goal description (so that..) are all present. That’s not bad and I don’t confuse the two but just note the similarities.
After a practical exercise of writing user stories for an imaginary tool of our choice we discussed that. What I found interesting is that breaking down the user stories to the right level and amount of detail is really a team decision – something many people struggle with when writing use cases.
The importance about the user stories should be “Do the user stories help us understand user context and business value.” That’s a harsh abbreviation and there’s more to it but to me these are the essentials (together with acceptance criteria).
At lunch I met James Bach and after listening in to another conversation we spoke about some senior managers and their viewpoints on testing, the TA model (parent/adult/child modes) and implementing rapid testing in companies. As I’m following both James and Michael B’s blogs there was nothing groundbreaking new for me but it rather reinforced my own views but also highlighted that the problems I’m facing are not with the understanding of the rapid testing approach but in other, more business related areas. I left contemplating and thinking about this (which is a good thing) but will need some time before I’ll find answer. Interesting.
After lunch I attended Paul Field’s “Change, change, change – the 5 year evolution of an agile team.” It was an interesting presentations about his experience about starting out with a 2 man agile team taking on a failing project in an IT unfriendly environment – the internal customer (another department) got burned in the past and was wary of IT in general.
He spoke about the pitfalls of building a team and scaling that from 2 to 7 after a while. Generally a good talk although it could’ve been a bit condensed from my point of view.
I liked his use of the Timeline Retrospective where he mapped colour coded issues as experienced by the team in sprint retrospectives over the whole project time. This can be a good way not to measure the way the project is going but the way the team is feeling about it which doesn’t necessarily needs to be the same. (And indeed was not, the team thought that they delivered badly at a sprint when the customer was delighted).
The retrospective starfish was also interesting consisting of 5 areas for
More of
Start doing
Stop doing
Less of
Keep doing
In the burnup chart he presented he mapped the story points “Done” and the story points as asked for from the business. This was not per sprint but over the whole project which showed that after 4 months (in a 6 month project) the team had created as many story points as the business originally wanted (great) but wouldn’t be able to finish by immovable 6 month deadline – a big problem. This burnup chart showed the problem in time and action was taken to get the project back on track.
When new team members where added he found that his former “flat” structure didn’t work anymore as existing team members were more senior in terms of knowledge so not everyone was equal. This needed to be addressed and after the new team member were brought up to speed through pair programming and other measures the team picked up some momentum.
After a coffee break I went to Astrid Byro’s “Just in time – last minute testing” presentation. She jumped in for another presenter who couldn’t come due to “Volcano awareness week” as James Bach coined it.
This was the weakest talk for me, the title didn’t have very much to do with the talk which was basically a war story about a 2 year RUP project with very high risks (accepted), lots of stress and a “command and control” approach should have died out 20 years ago. System migrators where used as sole testers which would’ve made me uncomfortable (maybe she was as well). I can see that from a PM point of view using people with domain knowledge is a good idea and using testing as a training exercise so you save handover further down the line is good for project timelines as well. But I can’t shake off the feeling that professional testers would’ve done some good – apparently that wasn’t possible (or thought necessary). At the end of the project performance testing was started (always a bad idea due to the high risks involved, again they were lucky and their application performed well). Their clients infrastructure didn’t but that wasn’t seen as a problem as the project in itself delivered, even if that didn’t actually help the customer. My view is more holistic than just pointing at the customer and saying “it’s their problem, our application performs well”, but maybe that’s just me.
That concluded the official presentations but I saw about a dozen lightning talks, the highlight for me was James Bach’s “And...also” presentation (I’m predictable in that regard, I know).His fiery talk had a lot of passion and humour to be able to appeal to even the most hardened developers and I believe he got some people thinking. It was about thinking not only what the expected positive outcome is but also what he also expects to be there (or not). For example NOT getting error messages, expecting the result to appear for more than half a second, etc – all things an automated test suite might fail to notice but a human being would notice – sometimes instinctively, sometimes with a bit of thinking.
I won’t mention the other lightning talks, there were too many to mention and I didn’t take notes of all so wouldn’t be able to give a complete picture anyway.
After a longer break the conference dinner started which was very interesting as well, after the starter we played musical chairs with only the speakers remaining seated so that people got more opportunity to speak to different speakers and other attendees. At my first table I sat next to Jeff Sutherland, creator of Scrum, without recognising him which I found extremely embarrassing. I promised to put in a good word with the conference organisers to get him back next year so that I could have a longer discussion with him – so organisers, if you’re reading this, do all of us a favour and invite Jeff for next year as well!
That’s it, thanks to Volcano Awareness Week I’ve been able to write down these notes while on my train journey rather than sitting at home already so let’s all say thanks to the volcano gods
:-)
Thomas
Friday morning:
Keynote talk from Russell Winder “The Multicore Revolution - We’ve only just begun”. Originally Dan North was down to present “Simplicity – the way of the unusual architect” but couldn’t get to the UK due to the airspace being closed because of the Icelandic volcano eruption.
Russell is clearly very knowledgeable and presented a good talk about the architecture of computers the way different programming languages address those. Good presentation that was probably more interesting to the developer community than for me but interesting nonetheless.
After a coffee break which I used for frantic telephone calls and trying to book train tickets I attended Rachel Davies “Understanding User Stories”. Not having written any before this was an interesting presentation with just the right amount of practical exercises for the participants. She mentioned the book “Agile Coaching” which she wrote or co-wrote which I’ll have a look at.
She explained that she and her team used a user story template which is now being used by many people around the world. I got the impression that she didn’t expect that and was pleasantly surprised that such a small thing created to make life easier in her project team has such a big impact. She did not that this template doesn’t work in all cases – especially when there’s no discussion with the business or if technical implementations are the order of the day rather than user workflows.
The template followed a
As a...
I want...
So that...
format which goes a long way getting all the necessary information onto a small card. She noted the importance of this being a physical card rather than a spreadsheet or electronic document. To me this format is similar-ish to the way use cases are created as the actor (As a xyz..) and workflow (I want...) and goal description (so that..) are all present. That’s not bad and I don’t confuse the two but just note the similarities.
After a practical exercise of writing user stories for an imaginary tool of our choice we discussed that. What I found interesting is that breaking down the user stories to the right level and amount of detail is really a team decision – something many people struggle with when writing use cases.
The importance about the user stories should be “Do the user stories help us understand user context and business value.” That’s a harsh abbreviation and there’s more to it but to me these are the essentials (together with acceptance criteria).
At lunch I met James Bach and after listening in to another conversation we spoke about some senior managers and their viewpoints on testing, the TA model (parent/adult/child modes) and implementing rapid testing in companies. As I’m following both James and Michael B’s blogs there was nothing groundbreaking new for me but it rather reinforced my own views but also highlighted that the problems I’m facing are not with the understanding of the rapid testing approach but in other, more business related areas. I left contemplating and thinking about this (which is a good thing) but will need some time before I’ll find answer. Interesting.
After lunch I attended Paul Field’s “Change, change, change – the 5 year evolution of an agile team.” It was an interesting presentations about his experience about starting out with a 2 man agile team taking on a failing project in an IT unfriendly environment – the internal customer (another department) got burned in the past and was wary of IT in general.
He spoke about the pitfalls of building a team and scaling that from 2 to 7 after a while. Generally a good talk although it could’ve been a bit condensed from my point of view.
I liked his use of the Timeline Retrospective where he mapped colour coded issues as experienced by the team in sprint retrospectives over the whole project time. This can be a good way not to measure the way the project is going but the way the team is feeling about it which doesn’t necessarily needs to be the same. (And indeed was not, the team thought that they delivered badly at a sprint when the customer was delighted).
The retrospective starfish was also interesting consisting of 5 areas for
More of
Start doing
Stop doing
Less of
Keep doing
In the burnup chart he presented he mapped the story points “Done” and the story points as asked for from the business. This was not per sprint but over the whole project which showed that after 4 months (in a 6 month project) the team had created as many story points as the business originally wanted (great) but wouldn’t be able to finish by immovable 6 month deadline – a big problem. This burnup chart showed the problem in time and action was taken to get the project back on track.
When new team members where added he found that his former “flat” structure didn’t work anymore as existing team members were more senior in terms of knowledge so not everyone was equal. This needed to be addressed and after the new team member were brought up to speed through pair programming and other measures the team picked up some momentum.
After a coffee break I went to Astrid Byro’s “Just in time – last minute testing” presentation. She jumped in for another presenter who couldn’t come due to “Volcano awareness week” as James Bach coined it.
This was the weakest talk for me, the title didn’t have very much to do with the talk which was basically a war story about a 2 year RUP project with very high risks (accepted), lots of stress and a “command and control” approach should have died out 20 years ago. System migrators where used as sole testers which would’ve made me uncomfortable (maybe she was as well). I can see that from a PM point of view using people with domain knowledge is a good idea and using testing as a training exercise so you save handover further down the line is good for project timelines as well. But I can’t shake off the feeling that professional testers would’ve done some good – apparently that wasn’t possible (or thought necessary). At the end of the project performance testing was started (always a bad idea due to the high risks involved, again they were lucky and their application performed well). Their clients infrastructure didn’t but that wasn’t seen as a problem as the project in itself delivered, even if that didn’t actually help the customer. My view is more holistic than just pointing at the customer and saying “it’s their problem, our application performs well”, but maybe that’s just me.
That concluded the official presentations but I saw about a dozen lightning talks, the highlight for me was James Bach’s “And...also” presentation (I’m predictable in that regard, I know).His fiery talk had a lot of passion and humour to be able to appeal to even the most hardened developers and I believe he got some people thinking. It was about thinking not only what the expected positive outcome is but also what he also expects to be there (or not). For example NOT getting error messages, expecting the result to appear for more than half a second, etc – all things an automated test suite might fail to notice but a human being would notice – sometimes instinctively, sometimes with a bit of thinking.
I won’t mention the other lightning talks, there were too many to mention and I didn’t take notes of all so wouldn’t be able to give a complete picture anyway.
After a longer break the conference dinner started which was very interesting as well, after the starter we played musical chairs with only the speakers remaining seated so that people got more opportunity to speak to different speakers and other attendees. At my first table I sat next to Jeff Sutherland, creator of Scrum, without recognising him which I found extremely embarrassing. I promised to put in a good word with the conference organisers to get him back next year so that I could have a longer discussion with him – so organisers, if you’re reading this, do all of us a favour and invite Jeff for next year as well!
That’s it, thanks to Volcano Awareness Week I’ve been able to write down these notes while on my train journey rather than sitting at home already so let’s all say thanks to the volcano gods
:-)
Thomas
Monday, 8 March 2010
First Scottish Software Testing Club event in Edinburgh
I’d like to announce the first Scottish Software Testing Club meeting on Wednesday, 10th March, 7pm in the Filmhouse Bar, Lothian Road in Edinburgh. Food is nice and they always have vegetarian and vegan meals on the menu. The beer and wine selection is good as well. Thanks to Rosie for letting us put the STC label on this meeting.
This is an informal event and is intended as a networking session and just saying hello to the other local testers. Everyone’s welcome, even if you are a developer :-). Glenn and I are thinking of maybe making this a monthly institution, 2nd Wednesday of each month or something along these lines – we’ll see what the turnout is going to be.
We'll meet in the main area, I'll try and put a picture of the STC logo up if I have the space.
Hope to see you there,
Thomas
This is an informal event and is intended as a networking session and just saying hello to the other local testers. Everyone’s welcome, even if you are a developer :-). Glenn and I are thinking of maybe making this a monthly institution, 2nd Wednesday of each month or something along these lines – we’ll see what the turnout is going to be.
We'll meet in the main area, I'll try and put a picture of the STC logo up if I have the space.
Hope to see you there,
Thomas
Subscribe to:
Posts (Atom)