I left a comment in a LinkedIn group that people seemed to like so I'm putting it in a blog now.
The discussion was about testing time getting squeezed by the sponsor and PM as extension of that in many (non-agile) projects and what the Test Managers responsibilities are.
If you ask the team to prepare a cake for twelve but only buy ingredients for six you only have a few options. Either you reduce the number of people. Or everyone get's a slimmer piece. Or someone goes out and buys more ingredients. I'll let you work out the infamous Project Management Triangle for yourself..
And yes, there are more options like extending the dough with other materials, fake more cake by making it shallower but with a bigger diameter, etc which would be options that a clever and motivated team may come up with.
My point though is that it's not the Test Manager who takes on sole responsibility when the customer doesn't like the cake. If you shorten preparation time you'll find some lumps at the end, no question about it.
Quality of the software is a team effort with the sponsor / owner and by extension the PM setting the boundaries. It is up to the Test / Dev Manager to then speak up and describe what the practical effects of these boundaries are but that's where it ends.
With a good team a lot can be done to find out what it is that is really required and how to get there. And hey, at the end there may be cake for everyone involved.
Wednesday, 28 November 2012
Tuesday, 27 November 2012
The discussion and learning about "it" is more important than the definition of "it".
On one of the LinkedIn Groups, Steve Green, who I value a lot both for his experience in exploratory testing as well as a person argued against the use of exit criteria as they are almost meaningless in his experience.
I agree but meaningless doesn't mean useless. Let's say that:
The discussion and learning about "it" is more important than the definition of "it".
While in many cases exit criteria are sacrificed to get a project out of the door I think that it would be a mistake to dismiss them. By trying to write down exit criteria and discussion within the team a lot can be learned about the way software is developed within a given company. When one struggles to come up with an exit criteria that may point to one or several problems.
Are stakeholders not clear? Are decisions made ad hoc? Is planning or estimating a problem?
A discussion about what exit criteria may be realistic and which ones aren't including the reasoning behind it will unearth a lot of important information for the project.
You may end up without exit criteria after all and that's OK. But the important bit is that information was gained and learning happened.
Of course "it" could be anything. Exit Criteria; a bug; a process; a workflow; an approach; a standard. As long as there's learning we have gained something that cannot be taken away.
I agree but meaningless doesn't mean useless. Let's say that:
The discussion and learning about "it" is more important than the definition of "it".
While in many cases exit criteria are sacrificed to get a project out of the door I think that it would be a mistake to dismiss them. By trying to write down exit criteria and discussion within the team a lot can be learned about the way software is developed within a given company. When one struggles to come up with an exit criteria that may point to one or several problems.
Are stakeholders not clear? Are decisions made ad hoc? Is planning or estimating a problem?
A discussion about what exit criteria may be realistic and which ones aren't including the reasoning behind it will unearth a lot of important information for the project.
You may end up without exit criteria after all and that's OK. But the important bit is that information was gained and learning happened.
Of course "it" could be anything. Exit Criteria; a bug; a process; a workflow; an approach; a standard. As long as there's learning we have gained something that cannot be taken away.
Wednesday, 22 August 2012
Trials and Tribulations of a Test Manager (Part IV)
The stuff they didn’t tell you about when you got the job
Some of the most challenging and tragic but also some absolutely hilarious things happened in this category. Some were sort of harmless where I found out some time after I started that the Test Manager job meant that I’m also expected to take the role as the Environment and Configuration Manager for the test environments at least.
Some examples of the more serious situations were that I had people come to me telling me they’ve been diagnosed to be bipolar; partners with failing kidneys; alcoholics; parents in hospital with live threatening illnesses; current divorce proceedings and the list goes on. At times like these the people approaching me were usually quite vulnerable. So while knowing that the next project will surely be impacted by sometimes grave news what they want to hear is that I’m taking care of the business side and say “Don’t you worry, I’ll take care of the work side, how can I help you – do you need time off, flexible working, etc?”
I always gave people leeway in situations like these mostly because it’s the right thing to do and I would like to be greeted by understanding from my line manager myself. But at the end of the day there isn’t really an alternative. Under pressure people will put their personal interests above the company anyway. Sometimes without knowing, sometimes with full understanding they don’t really ask for permission for time off / flexible working hours / understanding/ etc. They are only letting me know that their work will be impacted for some time. They do the right thing and expect the same in return.
Once they get a positive response it usually makes their lives a bit easier with one thing less to worry about. I had several instances where people then put in double the amount of effort and working hours once the situation was resolved, far in excess of what they took in the first place. Sometimes HR needs to be involved, sometimes it was fine to deal with it myself depending on the situation.
I worked in multicultural teams and what one perceives to be a joke can be an insult to the next. Careful explaining and setting of boundaries of what’s allowed and where the no man’s land begins becomes very important here. Sexual harassment, invoking grievance policies and offering a trip to the nearest dark alley are all part of this area which I have been lucky enough to diffuse before anything too serious happened.
Something related which tended to be between hilarious and so far out there that it stretched my years of experience to the extreme are what I call the “Rammstein” jokers. (After the band – insult and make fun of everybody, no taboos but expect to get some feedback). How do you react if someone sends an email round to the team saying something along the lines of “Peter is OK but he smells a bit this morning, probably borrowed his wife’s perfume again. I worry more about John who got out his thing under the table again, talks to it a bit in a Gollum voice and then looks at ‘hoff pictures.” (Names and wording changed but that email happened...) In these cases it’s either a “Thank you for working for us, don’t bother logging off. Let me escort you out of the building.” Or seeing how the rest of the team handles it, have a quiet word with the author and then take it from there.
I hope you enjoyed the musings of my last few years as a Test Manager. It was an interesting experience and I like to manage teams. There’s always something new, exciting and sometimes unexpected to learn and I hope I gave something in return.
Thanks for reading.
Tuesday, 21 August 2012
Trials and Tribulations of a Test Manager (Part III)
You may want to read Trials and Tribulations of a Test Manager (Part II) if you haven't done so already.
The Test Manager as a company approach
The Test Manager as a company approach
A Test Manager (or Development Manager or any other type of Manager with line management responsibilities and a certain degree of freedom and responsibility) usually has roles that reflect the whole company. This allegory works best for people who lead a team or department.
She would be the CEO, expected to prepare a vision for her team and provide guidance; the CTO to provide the technical vision and advise on which technology path is suitable to follow and which isn’t; the Finance section for keeping track of budgets; HR for hiring and firing people and providing guidance about policies in the company and so on.
So in very many ways a Manager is a company acting in a Micro-cosmos in their own little world. Of course the good ones make sure it fits with the Macro-cosmos of the outside world.
Not every Manager can be good at everything so what are you going to do about it? Delegate or get train yourself in the weak areas? This is very much a personal and context question which I can’t really answer here, it’s only there to remind us that the day to day tasks not necessarily encompass everything a Test Manager is responsible for and that some time and thought may need to go in these areas as we tend to forget about the things that we’re not so comfortable with.
One role that I haven’t mentioned above is that of a trainer. If I want to train, coach and mentor I should know a thing or two about the area in question. So everyday learning is a must in my opinion. This does not mean that I’m reading a testing book or internet articles every day although I try – often live interferes. Some days I learn a lot just by talking to colleagues, how they approach a particular problem and how their experiences play a part in that. Sometimes I learn how to do something, sometimes how not to. But it’s all useful, one way or another.
One thing that should be pretty clear is that you can’t manage what you don’t know. If you rely on reports about how the team is doing a lot is lost in translation. If the Test Manager and the Test team are located in the same building there shouldn’t be any excuse for the Test Manager NOT to sit with the team. That private office may look fancier and more comfortable but you could just as well sitting in another country. All the day to day problems, friendly banter and everything that makes up that team will be lost to the manager. In short, co-location with the team is a must, otherwise the Manager will not properly understand the problems that the team is facing. If the Manager doesn’t know where the rocks on the road to success are he can’t remove them.
It works the other way round as well. The team sees the problems and challenges and work that the Test Manager does. Overhearing discussions I had with dev managers, sys admins, etc has helped my team understand what’s expected from their manager so they could work toward helping towards these goals. When sitting with the team any small problems can be nipped in the bud before they become serious problems. Morale issues can be spotted early before they become serious problems. If there are talks that need to be had without the team that’s what meetings rooms are for. Letting people overhear all discussions also fosters an atmosphere of openness. It's saying “see I have nothing to hide” and we're all heading in the same direction.
Read on in the final chapter of Trials and Tribulations of a Test Manager (Part IV) coming soon.
Monday, 20 August 2012
Trials and Tribulations of a Test Manager (Part II)
You may want to read Trials and Tribulations of a Test Manager (Part I) if you haven't done so already.
The Shield, the Sword and the Horn of Met
Read on in Trials and Tribulations of a Test Manager (Part III).
The Shield, the Sword and the Horn of Met
In my experience one of the most important functions of any Team Leader is that they can shield their team and make sure all the rubbish and rotten tomatoes coming flying don’t hit them. The importance of that can’t be overemphasized from my point of view. If the Team Leader is seen to put his money where his mouth is in protecting the team from external distractions they will rally when called upon.
As an ex-Manager of mine once said “Get them to work for you, not for the company. Sooner or later the business will do something that will disappoint them. But if they work for you they will do their best because they don’t want to let you down, even if they’re unhappy with the company.”
I’ve been in both camps in my working live and can say it’s a lot easier to work at a place where this Shield is in place. It breeds mutual respect and is in very many ways the return to the tribal society.
If any threats to the team are swiftly intercepted and struck down it also sends a message to people outside the team– don’t mess with these guys. They’re a professional bunch and we don’t tolerate substandard behavior. Saying “no” for matters concerning the team sets boundaries for the behavior that the manager sees as acceptable. Precedence is created.
Something else I learned from an ex-Manager is that the team manager is not there to do the hands on work although it doesn’t hurt if they can jump in when necessary. The main goal for the team manager is to remove the rocks on the way and make sure that there is nothing impeding or slowing them down. Only if everyone in the team can work unobstructed should the team manager look at their own work tasks.
I can’t count how many times I got absolutely invaluable information about problems in the team or company in the pub. A week or two after joining I went out with the team for a drink and learned more there in two hours than in the weeks before. Things that people genuinely cared about or that upset them for one reason or other came to light in a more relaxed atmosphere than the workplace. Having some distance between the manager and the team is fine. When it comes to the crunch the manager sticks their head out and take a decision. But being seen as part of the team is a must in my book. If people can’t see the real me behind the manager facade, how are they going to trust that I stand up for their interests?
Leading by example is not only old news it was ancient when dinosaurs roamed the earth. What it does for the team is manifold. It sets the boundaries of what is acceptable and what isn’t. If the Test Manager comes late and leaves early that will soon be copied. Is a two line bug report acceptable? Depends on the company, maybe a bit more information would be helpful. So logging bug reports as you’d like to see them done is a good way to lead by example. These reports can become the standard that everyone should be working towards. Same with many other small things, emails, written and verbal reporting, etc. Setting precedencies and explaining what the reasoning behind it is and why it’s important will help form a professional team that works as the Test Manager expects.
I’m not promoting the lone hero approach, far from it. In order to get a team into shape you need people to help you. Someone who has seen the light and believes your approach is the right thing to do. The video Leadership Lessons from the dancing guy makes it clearer than I could do. It's been around for quite a while and is worth a look.
If you can get some momentum going with the help of a believer from within the group it will go a long way to building a successful team.
Read on in Trials and Tribulations of a Test Manager (Part III).
Friday, 17 August 2012
Trials and Tribulations of a Test Manager (Part I)
It’s been a
while since I last wrote here but I was busy moving country and jobs again. I
left Gibraltar behind and am now back in Germany in my home town but that’s a
different story..
Let’s
share the experience
I’ve been thinking about Test Management and Management in general a lot in
the last couple of months having observed both the good, the bad and the
downright ugly approaches. Stuart Reid in the Foreword to Essential Test Design from Torbjörn Ryber says
“… I would argue that the majority of software
testing problems stem from poor test management …”. I’m sure that context
dictates what the majority would be and that it depends on how it’s measured
but I agree with the sentiment – that test approach, process, technology and
technique all have a role to play but that good or bad management will dictate
for many a project if it succeeds or fails (This is also true for work done
outside a project).
A couple of weeks ago I had chat with a
colleague from another department about my test team. He told me that he and others wished they
could be part of that team – not because they wanted to test but because they
wanted a part of that team spirit and ambience that comes with it. I was quite
flustered – it was one of the nicest compliments I ever got as I put that team
together. If people want to get in not because they like the work or pay but
because it’s cool I must have done something right. I’ll try to put into words
what I believe that is over the next few entries.
It’s
all about connections man...
A Test Manager like everyone else in a
company structure is a link in the chain of business responsibilities. There’s
a link leading to your team and there’s a link towards Senior Management; and
possibly several more to your peers (Having no identifiable peers is a sign
something went wrong but that’s another story). If you consider yourself as a
key ring to which many chains are attached that picture works best for what I’m
trying to convey.
Each direction is a relationship with
slightly different expectations and requirements at the end of it. My approach
for each relationship in every direction in the past has been the same – read:
show the same courtesy and listen to everyone about what their requirements are,
no matter what their position. Each
person at the end of the relationship chain has their own needs and
requirements that I may be involved with. Some will be the same across all
levels, others differ wildly from person to person.
Some want help with their career, others
want to be left alone as there’s enough going on in their lives already. Some
want gentle pushing some object violently. Some can be rewarded with money,
others with flexible work hours, visits to conferences. A pat on the back when something
was well done has never hurt and is the strongest reward that people will not
ask for.
It helps to have worked in similar jobs
comparable to that of the person at the end of each chain. In other words, if
you worked in support before it’s easier to understand the needs and
requirements of a support role. Imagine walking a mile in their shoes. That’s a bit harder if you look up at the food
chain but as the requirements of the CTO or other Senior Managers become
clearer building a team that can satisfy their high level goals becomes a lot easier.
If you haven’t worked in a comparable
job before sit down with the person in question and ask them to tell you about
what the tasks you just gave them means for them. Is it a 5 minute job? Is it a
long nightmare tasks because of complications that you haven’t foreseen? Talk
to them and say that you genuinely want to know. Of course it has to be
heartfelt or that one backfires... A few months ago I asked for something that
I considered to be a half hour job, restructuring some of our virtual machines and
renaming some URLs. Turned out it would’ve been a week or more with serious
impacts on another team. I sat down with the person that I requested it from
and subsequently now know more about networks and VMs. These kind of sit downs
were great learning experiences for me. Next time I won’t make this particular
unreasonable request as my knowledge has increased.
Sunday, 29 April 2012
Performance Testing on a shoestring
The setting for this particular story is the run-up to the biggest yearly Horseracing event in the UK, the
Grand National in Aintree.
Even people usually not betting have a flutter heading to the nearest betting shop or hitting the net. Good news for an online gambling company, mixed news when it comes to the servers that need to cope with a lot more traffic than any other time of the year. Quite a few online gaming companies had problems on the day.
Our servers held and we only load tested one aspect of our system. Here's the story.
About two weeks before the race it was decided that we want our payment section tested after all. Right, plenty of time, better get started. We looked into JMeter, however there's limited knowledge in the team. So we asked some developers for help. After a lot of headscratching it was decided, yes, it can be done within about 3 weeks and 2 devs. So the idea of using JMeter was out of the window. A commercial solution was not feasable due to timescales and some other factors.
We decided, since it was to be something that had to be quick and was to be a one off (we'll worry about next year later), to do an in-house solution. While I was running around, gathering required information, investigating possible pitfalls, etc our automation expert (don't tell him, he will ask for more money) used AutoIT to code the load tool.
Yes, you read right. That's about the most unlikely tool for a performance test that I can think of but I employed most of my team members, I have trust in their abilities so time to put my money where my mouth is.
We knew that we wanted to get 8+ transactions a second to test that our system and our third party payment provider could take the heat. And we wanted that sustained over a period of time, say half an hour before the race is busiest so let's just run it for that time. That's the detailed requirements gathering out of the way...
I won't go into the details of the system for obvious reasons, rather explain how we went about it. We ran a transaction manually and found that it took between 10-25 seconds to complete, including filling out the form again for a second try. We also found that we could fit 8 browser windows on a monitor and show all necessary fields and buttons when setting the window size to 35%.
The idea was then to click "Submit" on all 8 browsers at the same time (or a millisecond apart). That worked fine. So, we could hit the system with 8 transactions. Once.
Ok, no problem, since we needed about 25 seconds for a round trip that was maintainable, we added another 5 seconds in case the system slowed down and then went to hunt down 30 PCs in the office (an experience in itself) to start them a second apart.
After having identified the machines we set all 30 to the same screen resolution (I didn't know how many different monitors we had until that day), deployed our performance tool to each machine and set up the machines. That involved setting the IE homepage to the desired URL, running the setup tool, clicking the necessary buttons and fields so that the correct X/Y coordinates were captured. The AutoIT application would then calculate the relative difference for the other 7 windows.
That was a lot of manual setup but we haven't had the time to code for more. Each machine got it's own ID (set up in the application and post it on the monitor) from 1 to 30 and we coded an editable start time into it. For example, machine with ID 1 would start at 08:30:01, ID 2 would start at 08:30:02, etc and then loop round starting again at 08:30:31.
Quite a bit of thinking and discussion has gone into this bit because we could have let each window just start again after each transaction was finished. Depending on the system response time transaction would have drifted apart though resulting in potentially a lot higher transaction number for some seconds and none at others. I was more concerned about the former as it could have been a theoretical 30*8=240 transactions at the same time which would very likely have created serious problems.
So we had our 8 transactions running for 30 seconds, looping around indefinetely until we stopped them. Setup was completed in the evening after a 13 hour+ day. We also did a small test run just to make sure it would work.
Before starting the next morning we had sys admins and several other people in place to monitor the various system components during the test and in case something went seriously wrong.
Execution threw up some problems, not least some network issues that weren't identified before. Also, some pages threw errors that the application couldn't recover from. Time was running out and people came into the office wanting their PCs to work on, so work was stopped. Two days later, after resolving the issues and having two more days to put some resilience into the AutoIT scripts so that recovery after errors was better we tried again.
This time it worked really well. Some machines had to be started again but overall we put the desired load on the system.
Lessons learnt:
Thanks for reading.
Grand National in Aintree.
Even people usually not betting have a flutter heading to the nearest betting shop or hitting the net. Good news for an online gambling company, mixed news when it comes to the servers that need to cope with a lot more traffic than any other time of the year. Quite a few online gaming companies had problems on the day.
Our servers held and we only load tested one aspect of our system. Here's the story.
About two weeks before the race it was decided that we want our payment section tested after all. Right, plenty of time, better get started. We looked into JMeter, however there's limited knowledge in the team. So we asked some developers for help. After a lot of headscratching it was decided, yes, it can be done within about 3 weeks and 2 devs. So the idea of using JMeter was out of the window. A commercial solution was not feasable due to timescales and some other factors.
We decided, since it was to be something that had to be quick and was to be a one off (we'll worry about next year later), to do an in-house solution. While I was running around, gathering required information, investigating possible pitfalls, etc our automation expert (don't tell him, he will ask for more money) used AutoIT to code the load tool.
Yes, you read right. That's about the most unlikely tool for a performance test that I can think of but I employed most of my team members, I have trust in their abilities so time to put my money where my mouth is.
We knew that we wanted to get 8+ transactions a second to test that our system and our third party payment provider could take the heat. And we wanted that sustained over a period of time, say half an hour before the race is busiest so let's just run it for that time. That's the detailed requirements gathering out of the way...
I won't go into the details of the system for obvious reasons, rather explain how we went about it. We ran a transaction manually and found that it took between 10-25 seconds to complete, including filling out the form again for a second try. We also found that we could fit 8 browser windows on a monitor and show all necessary fields and buttons when setting the window size to 35%.
The idea was then to click "Submit" on all 8 browsers at the same time (or a millisecond apart). That worked fine. So, we could hit the system with 8 transactions. Once.
Ok, no problem, since we needed about 25 seconds for a round trip that was maintainable, we added another 5 seconds in case the system slowed down and then went to hunt down 30 PCs in the office (an experience in itself) to start them a second apart.
After having identified the machines we set all 30 to the same screen resolution (I didn't know how many different monitors we had until that day), deployed our performance tool to each machine and set up the machines. That involved setting the IE homepage to the desired URL, running the setup tool, clicking the necessary buttons and fields so that the correct X/Y coordinates were captured. The AutoIT application would then calculate the relative difference for the other 7 windows.
That was a lot of manual setup but we haven't had the time to code for more. Each machine got it's own ID (set up in the application and post it on the monitor) from 1 to 30 and we coded an editable start time into it. For example, machine with ID 1 would start at 08:30:01, ID 2 would start at 08:30:02, etc and then loop round starting again at 08:30:31.
Quite a bit of thinking and discussion has gone into this bit because we could have let each window just start again after each transaction was finished. Depending on the system response time transaction would have drifted apart though resulting in potentially a lot higher transaction number for some seconds and none at others. I was more concerned about the former as it could have been a theoretical 30*8=240 transactions at the same time which would very likely have created serious problems.
So we had our 8 transactions running for 30 seconds, looping around indefinetely until we stopped them. Setup was completed in the evening after a 13 hour+ day. We also did a small test run just to make sure it would work.
Before starting the next morning we had sys admins and several other people in place to monitor the various system components during the test and in case something went seriously wrong.
Execution threw up some problems, not least some network issues that weren't identified before. Also, some pages threw errors that the application couldn't recover from. Time was running out and people came into the office wanting their PCs to work on, so work was stopped. Two days later, after resolving the issues and having two more days to put some resilience into the AutoIT scripts so that recovery after errors was better we tried again.
This time it worked really well. Some machines had to be started again but overall we put the desired load on the system.
Lessons learnt:
- Regardless how much you ask people for information, there's always something that someone has forgot to mention that will ruin your day. Do a test run and plan for round 2.
- Don't use IE if you don't have a standard company build on all PCs. Use a browser that you install yourself and control so you don't run into configuration issues when you least need it.
- Ask people for help. Running it yourself will only stress you out. I found that once we explained the somewhat mad plan people were happy to assist
- Tell people if you're highjacking their PCs, you may not get a chance to revert all changes. That was done for most but the ones we forgot were understandably miffed that we changed their settings without warning.
- Get a demonstration of what will be monitored and what can be saved for analysis later. Assuming it will all be there can lead to disappointment and repeats of the test.
- Don't ask others to do the boring jobs. Being hands on helps the rest of the team to see that you're serious about making this work.
- It's possible to get the job done, regardless if you're prepared, have the tools or the time. Determination, the belief that the job can be done and trust in others will get you a long way.
Thanks for reading.
Saturday, 28 April 2012
Do I want to write this?
I usually keep private and work net presence clearly separated. This one is different but I decided to post it as I couldnt' get the testing mindset out of my private life and it may help. It also explains why I haven't blogged or been active in the testing scene in the last couple of months. Not that I think that needs explaining.
Germany/Wuppertal, December 2012, Intensive Care Unit. I browse through the notes the nurses left for the last two days, noting the structure of it to be easily recognisable for the next ICU nurse. It's easy to see at a glance what drugs and treatments the patient got in the last 24 hours. It has to be, any failure here could be fatal.
Looking at the syringes next to my fathers bed I wonder what they all are and take note of the names. He's been in a coma since before Christmas. Reading the drug names comes easy, I worked as a pharmaceutical research scientist for over a decade. Memories of that come back. At home I find that Ketamine is for disassociating the body from pain and that the street price has fallen over the last couple of years. For some reason that stuck with me. I read up on resuscitation, survival rates and the side effects like personality changes and brain disorders. I read a lot and educated myself but sometimes I learn things that I really don't want to know.
Beds, syringes, forms, power, gas supplies, room layout, etc are all standardized so that any nurse or doctor can take over where the last one left off. In an ICU that's of vital importance. Each shift has a 1 hour handover/scrum to brief the next shift of what happened. I wonder what would happen if we were to do 1 hour handovers each day. In my current line of work it's not that important to know exactly where the last person left of. It's useful but no one dies if you miss a piece of information.
Seeing that everyone working in the ICU absolutely has to know everything about each patient/project was an eye opener. Of course that comes at a cost. But in that context that cost is worth paying.
So how much is it worth in our projects that everyone knows everything about the project? How many people in the project have no idea what their colleagues are working on? Is that OK or is that acceptable? How is the risk covered that information goes missing?
I learned a lot more in these weeks. How to recognise if people make mistakes and where the system fails; who puts in more effort than the rest; for some nurses the relatives play a bigger part, for others the patient is the only important thing. Most are somewhere in between. I reckon that's the Manager in me making these observations.
The people who I think of as "best" without defining what I mean exactly all have a passion for what they're doing. They're not only knowledgeable but are emotionally involved. I can say the same about the testing scene or probably any other craft that people are working in.
Of course the "learning" during this time wasn't purely to do with this mindset. Most of it was on the emotional side as can be expected. I learned quite a bit about what my approach to thinking and learning is compared to my parents and what is self-learned. But watching myself making these observations was a convincing sign that I'm working in the right job.
My whole family spent Christmas in Germany (unplanned and at very short notice; 6 hours from getting the call to leaving for the airport with my wife, 8 year old son and all Christmas presents) while my father was in coma all the time. He woke up in January and I flew back to Germany to spend some time with him. He died in February.
You won't be forgotten.
Germany/Wuppertal, December 2012, Intensive Care Unit. I browse through the notes the nurses left for the last two days, noting the structure of it to be easily recognisable for the next ICU nurse. It's easy to see at a glance what drugs and treatments the patient got in the last 24 hours. It has to be, any failure here could be fatal.
Looking at the syringes next to my fathers bed I wonder what they all are and take note of the names. He's been in a coma since before Christmas. Reading the drug names comes easy, I worked as a pharmaceutical research scientist for over a decade. Memories of that come back. At home I find that Ketamine is for disassociating the body from pain and that the street price has fallen over the last couple of years. For some reason that stuck with me. I read up on resuscitation, survival rates and the side effects like personality changes and brain disorders. I read a lot and educated myself but sometimes I learn things that I really don't want to know.
Beds, syringes, forms, power, gas supplies, room layout, etc are all standardized so that any nurse or doctor can take over where the last one left off. In an ICU that's of vital importance. Each shift has a 1 hour handover/scrum to brief the next shift of what happened. I wonder what would happen if we were to do 1 hour handovers each day. In my current line of work it's not that important to know exactly where the last person left of. It's useful but no one dies if you miss a piece of information.
Seeing that everyone working in the ICU absolutely has to know everything about each patient/project was an eye opener. Of course that comes at a cost. But in that context that cost is worth paying.
So how much is it worth in our projects that everyone knows everything about the project? How many people in the project have no idea what their colleagues are working on? Is that OK or is that acceptable? How is the risk covered that information goes missing?
I learned a lot more in these weeks. How to recognise if people make mistakes and where the system fails; who puts in more effort than the rest; for some nurses the relatives play a bigger part, for others the patient is the only important thing. Most are somewhere in between. I reckon that's the Manager in me making these observations.
The people who I think of as "best" without defining what I mean exactly all have a passion for what they're doing. They're not only knowledgeable but are emotionally involved. I can say the same about the testing scene or probably any other craft that people are working in.
Of course the "learning" during this time wasn't purely to do with this mindset. Most of it was on the emotional side as can be expected. I learned quite a bit about what my approach to thinking and learning is compared to my parents and what is self-learned. But watching myself making these observations was a convincing sign that I'm working in the right job.
My whole family spent Christmas in Germany (unplanned and at very short notice; 6 hours from getting the call to leaving for the airport with my wife, 8 year old son and all Christmas presents) while my father was in coma all the time. He woke up in January and I flew back to Germany to spend some time with him. He died in February.
You won't be forgotten.
Wednesday, 25 January 2012
Why using custom pictures in XMind is useful
I read this Blog from Albert Gareev, thought that's great but there's another way where you can use not only marker packs but also any pictures which is useful as described in my Mindmaps for managing test projects blog.
For example, I wrote the names of my team members in Word, Arial, 14pt, bold and took a picture of the name and now use them as markers. That's very useful when assigning projects to people so I can see at a glance who is working on which projects and where they may be stretched a bit thin.
In XMind this can be done by
navigating to your installation folder, for example
C:\Program Files\XMind\plugins\org.xmind.ui.resources_3.2.1.201011212218\markers
Drop your *.png or even *.jpg files in there.
Open "markerSheet.xml" in the text editor of your choice and add XML entries using the existing entries as a guideline. It's pretty straight forward.
Save and re-open XMind if you had it open before.
Now right click on an object and insert a Marker - voila, your new pictures as markers are now available. You are no longer restricted to pre-designed packs but can cook your own.
For example, I wrote the names of my team members in Word, Arial, 14pt, bold and took a picture of the name and now use them as markers. That's very useful when assigning projects to people so I can see at a glance who is working on which projects and where they may be stretched a bit thin.
In XMind this can be done by
navigating to your installation folder, for example
C:\Program Files\XMind\plugins\org.xmind.ui.resources_3.2.1.201011212218\markers
Drop your *.png or even *.jpg files in there.
Open "markerSheet.xml" in the text editor of your choice and add XML entries using the existing entries as a guideline. It's pretty straight forward.
Save and re-open XMind if you had it open before.
Now right click on an object and insert a Marker - voila, your new pictures as markers are now available. You are no longer restricted to pre-designed packs but can cook your own.
Sunday, 8 January 2012
Bringing Information Back Into IT
I wrote a bit for the EuroStar blog roll some time ago called "Bringing Information Back Into IT".
Any comments please leave at this post as the EuroStar one is out of date now.
Any comments please leave at this post as the EuroStar one is out of date now.
Wednesday, 17 August 2011
MindMaps for managing test projects
I work in an operational company where the test effort is geared towards in-house projects (see "The need for confirmation" blog for company types).
From a test management view this means that the majority of our time and effort is spent on small version increases of existing software, for example bug fixes, small pieces of new functionality or adding temporary items like pictures and web pages for specific events. Some larger projects, read a few months worth of effort, exist as well and run alongside the smaller ones.
We usually batch together changes so that we get a decent amount of test time together, unless the projects are decent sized already. I then check what system modules the bug fixes or new functions affect as we always have several of these small projects on the go, although usually less than a dozen. Once the interaction between these small mini-projects is known I can then decide in which test environment they'll go.
In short, this approach helps with resource management, where the resources managed are testers and test environments. I consciously excluded time as a resource (although that could be tracked as well) and only track the priority of a given project.
In order to track what the status is for each project or change I use a Mindmap which is divided into three main parts:
I experimented a bit and found that colour coding the projects and allocated environments works very well to highlight where a particular project will go to and where potentially clashes are when a project in that environment gets delayed. Putting the names of testers on the top level of a project also helps to see how much projects every tester in the team has going on at any moment in time.
Finally, putting numbers in for priority of projects helps to see which projects should go into one of the test environments next.
Here's an example of one of the MindMaps I use, for obvious reasons I made the project and tester names quite generic.
I use XMind in which Flags can be added to an entry which works in most MindMap applications. The flags are all pictures so I created one with the name of each tester, then updated the XMind configuration which is basically an XML file to add the additional folder and tester names. This makes for quick labelling of who's working on which project.
Here's a link to the generic project in case you want to download it (XMind file).
If you have any questions, please do not hesitate to get in touch.
From a test management view this means that the majority of our time and effort is spent on small version increases of existing software, for example bug fixes, small pieces of new functionality or adding temporary items like pictures and web pages for specific events. Some larger projects, read a few months worth of effort, exist as well and run alongside the smaller ones.
We usually batch together changes so that we get a decent amount of test time together, unless the projects are decent sized already. I then check what system modules the bug fixes or new functions affect as we always have several of these small projects on the go, although usually less than a dozen. Once the interaction between these small mini-projects is known I can then decide in which test environment they'll go.
In short, this approach helps with resource management, where the resources managed are testers and test environments. I consciously excluded time as a resource (although that could be tracked as well) and only track the priority of a given project.
In order to track what the status is for each project or change I use a Mindmap which is divided into three main parts:
- Projects that I know about and that are either in analysis or still in development
- Projects that are nearly development complete or already queuing to get tested
- Projects that are in one of our test environments and have a tester assigned
I experimented a bit and found that colour coding the projects and allocated environments works very well to highlight where a particular project will go to and where potentially clashes are when a project in that environment gets delayed. Putting the names of testers on the top level of a project also helps to see how much projects every tester in the team has going on at any moment in time.
Finally, putting numbers in for priority of projects helps to see which projects should go into one of the test environments next.
Here's an example of one of the MindMaps I use, for obvious reasons I made the project and tester names quite generic.
I use XMind in which Flags can be added to an entry which works in most MindMap applications. The flags are all pictures so I created one with the name of each tester, then updated the XMind configuration which is basically an XML file to add the additional folder and tester names. This makes for quick labelling of who's working on which project.
Here's a link to the generic project in case you want to download it (XMind file).
If you have any questions, please do not hesitate to get in touch.
Wednesday, 16 February 2011
Writing a job description for a Tester Part II
In the first part I described how I started out with the job desciption for a Test Analyst and why the first sentence was so important for me. In this part I'll go through the body of the job description.
This part is both describing what the applicant is expected to do in the job and what working in our company is like.
A new job is always a two way relationship. The employer is looking for the skills, personality, etc to fill a gap in their team. The potential candidate is looking for an interesting and rewarding place to work in so I try and address both.
Again this section is written tongue in cheek to give the potential applicant both an idea of what's required of them and to lure the right people into applying. In that first part I'm describing the part of personality I'm looking for, I mention some skills and hint at what level the candidate should be.
For someone experienced in ET the sentence ..."know your oracles from your heuristics" should be clear. I didn't want to put "Should have exploratory testing experience" as they keyword driven people would just put that in their CV and I'm none the wiser.
Asking the question "If you're not scared away by the above you may want to apply." should convey that yes, I'm asking for a lot but that I trust the applicant to be a confident and good tester and will welcome them with open arms.
The Principal Accountabilities section more closely resembles a traditional job spec. I nearly left it out as an experienced software tester would be able to do these tasks anyway. In the end I left it in as the tasks in there convey that I'm not only looking for a tester but also a facilitator that can make things happen with other teams and people in the business. Many testers do that but not all so I felt that it would explain to the potential candidate that this is a "get out of your chair and make it happen" job rather than one where the test manager or lead tell you what to do.
The Qualification section contains some standard stuff but also some controversial and opposing if not mutually exclusive statements, that Ajay has picked up on. The idea here is that the candidate asks for clarifications or state their opinion about it so that we can then have a good discussion about it during which we both get to know the others skill and opinion better than with standard interview questions. The idea for this came from the book Lessons Learnt in Software Testing which has opposing titles in chapter 6 about whether to use test document templates.
The bonus section for Qualifications is tongue in cheek again. I start that having no ISEB/ISTQB qualification is a good thing for me. The idea here is that
a) it's unusual and people take note - no skimming over this job description, I want your attention here
b) I don't think much of these certification programmes
c) If the tester doesn't have certificates, how do they educate themselves? Another opportunity for a good discussion
d) I want to see if people change their CV when applying for the job or if they send a generic one. Custom build CVs are more interesting to me - I'd like to see if they're interested enough in the job to put some effort into the CV or if they can't be bothered. Since I removed the chance for just putting in the keywords from the job spec by hardly mentioning any the custom CV should more closely reflect their real experiences.
The other items on the list are mostly nice to have's and cover a wide area so that almost everyone can tick one if not several of these boxes.
If the result is worth the effort you ask? No idea, yet, I'll find out soon. From the applications I've got so far I could easily see who read the job spec and who just skim read and send their generic CV. Makes my job easier as I can just bin the latter.
This part is both describing what the applicant is expected to do in the job and what working in our company is like.
A new job is always a two way relationship. The employer is looking for the skills, personality, etc to fill a gap in their team. The potential candidate is looking for an interesting and rewarding place to work in so I try and address both.
Again this section is written tongue in cheek to give the potential applicant both an idea of what's required of them and to lure the right people into applying. In that first part I'm describing the part of personality I'm looking for, I mention some skills and hint at what level the candidate should be.
For someone experienced in ET the sentence ..."know your oracles from your heuristics" should be clear. I didn't want to put "Should have exploratory testing experience" as they keyword driven people would just put that in their CV and I'm none the wiser.
Asking the question "If you're not scared away by the above you may want to apply." should convey that yes, I'm asking for a lot but that I trust the applicant to be a confident and good tester and will welcome them with open arms.
The Principal Accountabilities section more closely resembles a traditional job spec. I nearly left it out as an experienced software tester would be able to do these tasks anyway. In the end I left it in as the tasks in there convey that I'm not only looking for a tester but also a facilitator that can make things happen with other teams and people in the business. Many testers do that but not all so I felt that it would explain to the potential candidate that this is a "get out of your chair and make it happen" job rather than one where the test manager or lead tell you what to do.
The Qualification section contains some standard stuff but also some controversial and opposing if not mutually exclusive statements, that Ajay has picked up on. The idea here is that the candidate asks for clarifications or state their opinion about it so that we can then have a good discussion about it during which we both get to know the others skill and opinion better than with standard interview questions. The idea for this came from the book Lessons Learnt in Software Testing which has opposing titles in chapter 6 about whether to use test document templates.
The bonus section for Qualifications is tongue in cheek again. I start that having no ISEB/ISTQB qualification is a good thing for me. The idea here is that
a) it's unusual and people take note - no skimming over this job description, I want your attention here
b) I don't think much of these certification programmes
c) If the tester doesn't have certificates, how do they educate themselves? Another opportunity for a good discussion
d) I want to see if people change their CV when applying for the job or if they send a generic one. Custom build CVs are more interesting to me - I'd like to see if they're interested enough in the job to put some effort into the CV or if they can't be bothered. Since I removed the chance for just putting in the keywords from the job spec by hardly mentioning any the custom CV should more closely reflect their real experiences.
The other items on the list are mostly nice to have's and cover a wide area so that almost everyone can tick one if not several of these boxes.
If the result is worth the effort you ask? No idea, yet, I'll find out soon. From the applications I've got so far I could easily see who read the job spec and who just skim read and send their generic CV. Makes my job easier as I can just bin the latter.
Monday, 14 February 2011
Writing a job description for a Tester Part I
This two part post is about my recent job description for a Test Analyst.
I recently started a new job taking over a team of four Test Analysts only to find that one moved to another position within the company after two weeks. Relieved that it wasn't me but that the move had been planned before I even arrived I quickly had to think about replacing the 7 years+ domain knowledge that just walked out of the door and the nice guy owning that knowledge. Nice start.
Since I didn't know yet the strengths and weaknesses of the team I decided to get the skill matrix out that I used in my last company. In short, it's a table that lists all sorts of skills and knowledge areas against a column for each tester and values ranging from 0 to 3. Zero meaning no knowledge, three meaning mastered that skill. It's a quick and dirty way to get a feeling how people see themselves and if there are areas that no one in the team really has knowledge of. This is worth a blog in itself but I don't want to overburden this one.
After discussions with the team I then found that structured exploratory testing was new to them so I decided that I wanted to get someone who's got some experience and interest in that area, maybe someone who took part in weekend testing sessions. Automation was another "no knowledge" area but there are other problems that we need to solve before I'm prepared to tackle that one.
Armed with that knowledge I then trawled the internet for job descriptions of Test Analyst, QA Engineer and whatever other job titles people gave to software testers. The majority I found were generic, non-descriptive documents that were interchangeable or they were looking for programmers in disguise.
Most used keywords and things like "experience with Quality Centre". I never understand that one as any tester worth their salt should be able to learn that within a couple of days. One of the requirements for the job is that we learn quickly how to use a new piece of software so why make it a requirement for a job?
I can understand it in specific cases if you're using a specific tool and your expert just left, you may want someone with that knowledge to pick up where the old guy left off.Or you need help implementing a specific tool or need certain skills in your team. Generic statements don't help in this area though IMHO.
I then had a think of what it is that I really wanted, what I didn't want to compromise on, what's my most important requirement. I then put that as the first sentence of the job desciption for a Test Analyst.
"We need someone who knows how to test software first and foremost."
This sentence is very simple but has several functions, similar to the "Pricing has many functions, only one of which is the exchange of money" which was coined by Jerry Weinberg. I heard that first at the Rapid Software Testing course with Michael Bolton. One is that it's a non-standard way to start a job description. As any successful novel writer will know, you need a strong first sentence that captures the readers attention and creates a feeling of wanting to know more. I believe this one qualifies.
It also makes very clear what's required of the applicant - people blindly ticking of scripts need not apply.
It also conveys that the person writing this knows what he wants and that this description hasn't been written by someone in HR who doesn't know about testing.
Of course the sentence is cheeky with a touch of humour as stating the bloody obvious is done in many CV's - need to be able to write test scripts, test plans, work with developers, etc, but not in such an extreme way. That software testers should actually be able to test doesn't appear in any CV's I've seen so I thought I'd put it there.
Testers who take pride in their work and want to learn more about their craft should get the feeling that there is a Test Manager who cares about these things rather than asking "How many percent of the test scripts have you written/executed" or "How many bugs have you found today?".
I want these testers to apply.
More to follow tomorrow in part II
I recently started a new job taking over a team of four Test Analysts only to find that one moved to another position within the company after two weeks. Relieved that it wasn't me but that the move had been planned before I even arrived I quickly had to think about replacing the 7 years+ domain knowledge that just walked out of the door and the nice guy owning that knowledge. Nice start.
Since I didn't know yet the strengths and weaknesses of the team I decided to get the skill matrix out that I used in my last company. In short, it's a table that lists all sorts of skills and knowledge areas against a column for each tester and values ranging from 0 to 3. Zero meaning no knowledge, three meaning mastered that skill. It's a quick and dirty way to get a feeling how people see themselves and if there are areas that no one in the team really has knowledge of. This is worth a blog in itself but I don't want to overburden this one.
After discussions with the team I then found that structured exploratory testing was new to them so I decided that I wanted to get someone who's got some experience and interest in that area, maybe someone who took part in weekend testing sessions. Automation was another "no knowledge" area but there are other problems that we need to solve before I'm prepared to tackle that one.
Armed with that knowledge I then trawled the internet for job descriptions of Test Analyst, QA Engineer and whatever other job titles people gave to software testers. The majority I found were generic, non-descriptive documents that were interchangeable or they were looking for programmers in disguise.
Most used keywords and things like "experience with Quality Centre". I never understand that one as any tester worth their salt should be able to learn that within a couple of days. One of the requirements for the job is that we learn quickly how to use a new piece of software so why make it a requirement for a job?
I can understand it in specific cases if you're using a specific tool and your expert just left, you may want someone with that knowledge to pick up where the old guy left off.Or you need help implementing a specific tool or need certain skills in your team. Generic statements don't help in this area though IMHO.
I then had a think of what it is that I really wanted, what I didn't want to compromise on, what's my most important requirement. I then put that as the first sentence of the job desciption for a Test Analyst.
"We need someone who knows how to test software first and foremost."
This sentence is very simple but has several functions, similar to the "Pricing has many functions, only one of which is the exchange of money" which was coined by Jerry Weinberg. I heard that first at the Rapid Software Testing course with Michael Bolton. One is that it's a non-standard way to start a job description. As any successful novel writer will know, you need a strong first sentence that captures the readers attention and creates a feeling of wanting to know more. I believe this one qualifies.
It also makes very clear what's required of the applicant - people blindly ticking of scripts need not apply.
It also conveys that the person writing this knows what he wants and that this description hasn't been written by someone in HR who doesn't know about testing.
Of course the sentence is cheeky with a touch of humour as stating the bloody obvious is done in many CV's - need to be able to write test scripts, test plans, work with developers, etc, but not in such an extreme way. That software testers should actually be able to test doesn't appear in any CV's I've seen so I thought I'd put it there.
Testers who take pride in their work and want to learn more about their craft should get the feeling that there is a Test Manager who cares about these things rather than asking "How many percent of the test scripts have you written/executed" or "How many bugs have you found today?".
I want these testers to apply.
More to follow tomorrow in part II
Thursday, 28 October 2010
I'll try walking in your shoes
I recently attended a Project Managers meeting where PM specific, day to day problems were discussed.
Some were quite generic in nature and similar to a tester meeting, for example how do we make sure that people are properly trained, dealing with communication problems, etc. Others were more specific, for example talking about managing customer relationships, how a customer may perceive problems in a project from a completely different angle (for the better or the worse for the PM in question).
What occurred to me there is that we put a lot of emphasis in testing on adding “value for the customer”, however most testers that I know have no idea about customer management, and how our customers priorities might be different from the ones that the development team (programmers and testers) are thinking of. Managing that relationship sits squarely in the corner of the PM or Account Manager (where available). In a way it’s good that the Project Manager is shielding the development team from sometimes difficult customers, but to what degree can a tester really say that what they’re doing is actually adding value for the client?
That reminded me that what we think of as “adding value” is actually based on a model. To quote George Box “All models are wrong, some are useful”. Is our model built on validation and verification – in a broad way ensuring that the system works as intended and works without glitches? But what do we base that on since we all know that requirement documents are notoriously lacking? The argument I often hear is that testers bring their experience in adding what they perceive to be of value to other customers so it might be of value to the current project as well. With no knowledge of the customer or our relationship with them how can this be true?
At the next team meeting it might be an idea to invite the Account Manager as well or, if the PM is fulfilling that role as well, ask him about our relationship to the client. Is it perceived to be a difficult one? Have we worked with that client before? Is the project seen as a start of many others to follow? Do we know of particular things that “bug” the client that would help us direct our test effort?
I don’t know how other people address that but these are not questions that I have asked in the past but maybe I should. I attended a Project Management course and think that all testers should do that sooner or later, ideally followed with actually managing a project or two as well.
Many testers have an idea of how programmers think, many having a background in programming themselves. Maybe it’s time that we start to look through the eyes of a Project Manager as well and walk a mile in their shoes.
Some were quite generic in nature and similar to a tester meeting, for example how do we make sure that people are properly trained, dealing with communication problems, etc. Others were more specific, for example talking about managing customer relationships, how a customer may perceive problems in a project from a completely different angle (for the better or the worse for the PM in question).
What occurred to me there is that we put a lot of emphasis in testing on adding “value for the customer”, however most testers that I know have no idea about customer management, and how our customers priorities might be different from the ones that the development team (programmers and testers) are thinking of. Managing that relationship sits squarely in the corner of the PM or Account Manager (where available). In a way it’s good that the Project Manager is shielding the development team from sometimes difficult customers, but to what degree can a tester really say that what they’re doing is actually adding value for the client?
That reminded me that what we think of as “adding value” is actually based on a model. To quote George Box “All models are wrong, some are useful”. Is our model built on validation and verification – in a broad way ensuring that the system works as intended and works without glitches? But what do we base that on since we all know that requirement documents are notoriously lacking? The argument I often hear is that testers bring their experience in adding what they perceive to be of value to other customers so it might be of value to the current project as well. With no knowledge of the customer or our relationship with them how can this be true?
At the next team meeting it might be an idea to invite the Account Manager as well or, if the PM is fulfilling that role as well, ask him about our relationship to the client. Is it perceived to be a difficult one? Have we worked with that client before? Is the project seen as a start of many others to follow? Do we know of particular things that “bug” the client that would help us direct our test effort?
I don’t know how other people address that but these are not questions that I have asked in the past but maybe I should. I attended a Project Management course and think that all testers should do that sooner or later, ideally followed with actually managing a project or two as well.
Many testers have an idea of how programmers think, many having a background in programming themselves. Maybe it’s time that we start to look through the eyes of a Project Manager as well and walk a mile in their shoes.
Wednesday, 20 October 2010
The pictorial challenge answered
After a short discussion with Zeger (TestSideStory) on Twitter about the pictorial challenge it was time to do some work but I came back to the picture later. Have a look at it first before reading on.
I sat back and thought “How would I test this if I don’t know what’s important and don’t know to whom it will be important?”
I then collated some high level areas of approach
• Death by Goggle:
o Searching the net for the names on the signs to give me more avenues to consider
• Technical: It’s a .jpg of a certain size, resolution and colours used
o Is that on purpose, in which scenarios could it be a problem?
o Are there parts of the picture that look cut off, is there too much sky, etc.
o Are there hints that this is a collage of pictures or is it one photograph
• Emotional: How do I react to the picture when looking at it first.
o How does that change when picking out more details?
I started with the emotional part first because a) I can’t help it and b) to me testing a picture is more akin to criticizing a painting so I wanted see if the artist/photographer got an immediate emotional reaction from me. The signs look rusty and barely legible. Rust to me indicates age so I assume they’ve been there for a long time. The house looks like it has seen better days, my first reaction is that it’s been taken in a poor part of whatever town/country. The boy on the roof (I assume it’s a roof) would be unusual in a UK town, maybe it’s not where this picture is taken? Filed away for future consideration.
I then searched for “Zanchett” and “La Mejor Ropa”, the latter of which is Spanish for “The best clothes”. But maybe it's not "Zanchett" but "Zanchetti". That's where the font type could throw a spanner into the tester's work. You live and learn. Again, more pages but I didn't follow them up too closely as I'm more interested in the picture rather than where it was taken. Zanchett seems to be quite a common name, there are several facebook sites, a Portuguese Tourism page to name a few. I also found links to a breastfeeding site in Brazil, so maybe it’s not Spanish but Portuguese. The boy (man?) on the roof is wearing a t-shirt, short trousers and sandals which indicates a warm country. At a guess I’d say the photo was taken in Brasil or Argentina.
The breastfeeding site also made me look at the photo closer, maybe the blue plastic bowl hanging at the outside of the house is actually a baby bath. The bike on the left, caged birds on the right and the boy on the roof could mean it’s a nursery or orphanage (because of what I assume is a poor area, I could be wrong here). In my experience Mediterranean houses don’t give the impression that they’re looked after from the outside, that doesn’t mean that they’re not in very good condition from the inside. There’s an angle of cultural differences here and since I assume Southern America here as the location I’ll have to question my assumptions. The boy/man (I'm not able to clearly distinguish the features to say for sure) is wearing sports clothes which seem new or in good shape at least so maybe poor isn't actually correct here.
I didn’t go into the technical details other than noting that the size is 601 x 776 pixels – the 601 seems odd to me, maybe it was converted from a different size or type?
I’ll get in touch with Zeger to see what he thinks about it. :)
I sat back and thought “How would I test this if I don’t know what’s important and don’t know to whom it will be important?”
I then collated some high level areas of approach
• Death by Goggle:
o Searching the net for the names on the signs to give me more avenues to consider
• Technical: It’s a .jpg of a certain size, resolution and colours used
o Is that on purpose, in which scenarios could it be a problem?
o Are there parts of the picture that look cut off, is there too much sky, etc.
o Are there hints that this is a collage of pictures or is it one photograph
• Emotional: How do I react to the picture when looking at it first.
o How does that change when picking out more details?
I started with the emotional part first because a) I can’t help it and b) to me testing a picture is more akin to criticizing a painting so I wanted see if the artist/photographer got an immediate emotional reaction from me. The signs look rusty and barely legible. Rust to me indicates age so I assume they’ve been there for a long time. The house looks like it has seen better days, my first reaction is that it’s been taken in a poor part of whatever town/country. The boy on the roof (I assume it’s a roof) would be unusual in a UK town, maybe it’s not where this picture is taken? Filed away for future consideration.
I then searched for “Zanchett” and “La Mejor Ropa”, the latter of which is Spanish for “The best clothes”. But maybe it's not "Zanchett" but "Zanchetti". That's where the font type could throw a spanner into the tester's work. You live and learn. Again, more pages but I didn't follow them up too closely as I'm more interested in the picture rather than where it was taken. Zanchett seems to be quite a common name, there are several facebook sites, a Portuguese Tourism page to name a few. I also found links to a breastfeeding site in Brazil, so maybe it’s not Spanish but Portuguese. The boy (man?) on the roof is wearing a t-shirt, short trousers and sandals which indicates a warm country. At a guess I’d say the photo was taken in Brasil or Argentina.
The breastfeeding site also made me look at the photo closer, maybe the blue plastic bowl hanging at the outside of the house is actually a baby bath. The bike on the left, caged birds on the right and the boy on the roof could mean it’s a nursery or orphanage (because of what I assume is a poor area, I could be wrong here). In my experience Mediterranean houses don’t give the impression that they’re looked after from the outside, that doesn’t mean that they’re not in very good condition from the inside. There’s an angle of cultural differences here and since I assume Southern America here as the location I’ll have to question my assumptions. The boy/man (I'm not able to clearly distinguish the features to say for sure) is wearing sports clothes which seem new or in good shape at least so maybe poor isn't actually correct here.
I didn’t go into the technical details other than noting that the size is 601 x 776 pixels – the 601 seems odd to me, maybe it was converted from a different size or type?
I’ll get in touch with Zeger to see what he thinks about it. :)
Monday, 2 August 2010
The need for confirmation – why checking against requirements is not going away
I was thinking what prompts the need for traceability and ensuring that the final application actually satisfies the requirements that were captured at the beginning of the project. Who needs that traceability? I then thought about different companies. The type of company a tester works in has a huge impact on the approach of their testing. Broadly speaking (read: this is a gross generalisation) we have these types of companies:
1. Product companies selling shrink-wrap software. This is anything from operating systems, games, tax return helpers, computer helper apps (Adobe Reader), etc.
2. Consultancies developing a bespoke application for their clients to address a specific business problem
3. Hybrid between the first two, for example further developing/configuring a shrink-wrap application, for example SAP or other back-office systems.
4. Large company developing a bespoke application in-house for internal use only, for example banks or insurance companies.
Let’s have a look at each from different angles.
The first type of company producing products is somewhat unique, as there is not one, single client that determines the requirements. They need to rely on Product Managers or other staff who can guess or work out that there is a target audience who’s likely to use and/or buy their software. Anticipating what the market needs determines the requirements.
Many products, when selling well, will have several versions and releases. That’s good news for the automation people as products often need to be regression tested for a variety of platforms, i.e. different operating systems, hardware configurations, or when new functionality is added over time. The quality necessary for their products depends on the market they’re selling to.
The second type is very much focussed on the one client they’re selling to. What’s in the contract will get built (and tested if the contract specifies it). What’s not in the contract depends on the good will of the consultancy. If a client just spent £3M with you and you wave the contract instead of adding a 2 day RFC that’s probably the last big contract your company got.
Automated regression testing depends wholly on the client – have they specified that they want it, is the application large enough to warrant it for just the one version, are there likely to be more version of the software and contracts coming in from the client? Then it might make sense building something automated, otherwise forget it, time is money in this setup.
The hybrid company is closer to the Consultancy but with the added complication that now there is third party code to be considered for testing. More often than not that has already been tested but one has to be careful to exclude functions as they may be used differently in the context of the re-developed/configured new solution.
The in-house development test efforts very much depend on the company, however there usually are requirements that have been developed by internal groups. This type is usually similar to the Product companies in that software developed is there to stay for probably more than one version. That means more effort can be spent on testing it properly and with a long term view.
To come back to the need for confirmation – if you’re working in some sort of Consultancy, traceability is a no brainer. You’ll need to confirm that what your company is delivering is adhering to the requirements that were set out – and by extension that you fulfil the contract. From the Consultancy point of view satisfying the contract has to come first, otherwise the business would leave itself wide open to being sued or just not paid for breaching the contract. So once the confirmation is out of the way there might or might not be time to test the application in detail.
If you’re working in a product company or in-house development effort it’s a bit different. There are other factors to consider though. Are there any legal requirements to adhere to certain standards? Does the software need to be certified against particular standards? GLP or GCP comes to mind. Traceability is maybe not so important in this setup compared to the Consultancies, unless the PM or project owners insists on it, I’d argue out of the false hope that the system will then actually deliver what was expected when the requirements were written.
There was an argument on one of the LinkedIn groups if checking/testing against requirements is a hoax or if it adds value. As always, the answer is, it depends. If you have a 1:1 relationship of tests vs requirements I’d say it’s a hoax. All requirements need more than one simple test. So saying that a particular requirement has been tested against, then ticking the box, fine, that’s implemented is giving a wrong picture.
Imagine you have a requirements that says that you need to have several user roles in the system. Would you confirm that with one single check? You could, but would it actually give you the information and confidence in the application that the stakeholders ask us to provide? I don’t think so. Why artificially tick a box and say, yes, there are several user roles in the system if you’re telling a white lie if you haven’t tested the around that requirement?
I can see why checking against requirements is popular as it’s easy to put into contracts. The effectiveness of it from a quality point of view is almost non-existent, so it might help the sales and legal people (and not even that if the customer is set on suing a non-compliant company) but the PM should be looking elsewhere for confidence that the final product is actually what the customer wanted in the first place.
1. Product companies selling shrink-wrap software. This is anything from operating systems, games, tax return helpers, computer helper apps (Adobe Reader), etc.
2. Consultancies developing a bespoke application for their clients to address a specific business problem
3. Hybrid between the first two, for example further developing/configuring a shrink-wrap application, for example SAP or other back-office systems.
4. Large company developing a bespoke application in-house for internal use only, for example banks or insurance companies.
Let’s have a look at each from different angles.
The first type of company producing products is somewhat unique, as there is not one, single client that determines the requirements. They need to rely on Product Managers or other staff who can guess or work out that there is a target audience who’s likely to use and/or buy their software. Anticipating what the market needs determines the requirements.
Many products, when selling well, will have several versions and releases. That’s good news for the automation people as products often need to be regression tested for a variety of platforms, i.e. different operating systems, hardware configurations, or when new functionality is added over time. The quality necessary for their products depends on the market they’re selling to.
The second type is very much focussed on the one client they’re selling to. What’s in the contract will get built (and tested if the contract specifies it). What’s not in the contract depends on the good will of the consultancy. If a client just spent £3M with you and you wave the contract instead of adding a 2 day RFC that’s probably the last big contract your company got.
Automated regression testing depends wholly on the client – have they specified that they want it, is the application large enough to warrant it for just the one version, are there likely to be more version of the software and contracts coming in from the client? Then it might make sense building something automated, otherwise forget it, time is money in this setup.
The hybrid company is closer to the Consultancy but with the added complication that now there is third party code to be considered for testing. More often than not that has already been tested but one has to be careful to exclude functions as they may be used differently in the context of the re-developed/configured new solution.
The in-house development test efforts very much depend on the company, however there usually are requirements that have been developed by internal groups. This type is usually similar to the Product companies in that software developed is there to stay for probably more than one version. That means more effort can be spent on testing it properly and with a long term view.
To come back to the need for confirmation – if you’re working in some sort of Consultancy, traceability is a no brainer. You’ll need to confirm that what your company is delivering is adhering to the requirements that were set out – and by extension that you fulfil the contract. From the Consultancy point of view satisfying the contract has to come first, otherwise the business would leave itself wide open to being sued or just not paid for breaching the contract. So once the confirmation is out of the way there might or might not be time to test the application in detail.
If you’re working in a product company or in-house development effort it’s a bit different. There are other factors to consider though. Are there any legal requirements to adhere to certain standards? Does the software need to be certified against particular standards? GLP or GCP comes to mind. Traceability is maybe not so important in this setup compared to the Consultancies, unless the PM or project owners insists on it, I’d argue out of the false hope that the system will then actually deliver what was expected when the requirements were written.
There was an argument on one of the LinkedIn groups if checking/testing against requirements is a hoax or if it adds value. As always, the answer is, it depends. If you have a 1:1 relationship of tests vs requirements I’d say it’s a hoax. All requirements need more than one simple test. So saying that a particular requirement has been tested against, then ticking the box, fine, that’s implemented is giving a wrong picture.
Imagine you have a requirements that says that you need to have several user roles in the system. Would you confirm that with one single check? You could, but would it actually give you the information and confidence in the application that the stakeholders ask us to provide? I don’t think so. Why artificially tick a box and say, yes, there are several user roles in the system if you’re telling a white lie if you haven’t tested the around that requirement?
I can see why checking against requirements is popular as it’s easy to put into contracts. The effectiveness of it from a quality point of view is almost non-existent, so it might help the sales and legal people (and not even that if the customer is set on suing a non-compliant company) but the PM should be looking elsewhere for confidence that the final product is actually what the customer wanted in the first place.
Thursday, 17 June 2010
EWT22 - How to think about science/Something Fishy
At EWT22, Michael Bolton moderated a new type of session – listening to an audio recording “How To Think About Science” and finding parallels to software testing.
Background to the interview:
On July 3, 1992, the Canadian Fisheries Minister John Crosbie announced a moratorium on the fishing of northern cod. It was the largest single day lay-off in Canadian history: 30,000 people unemployed at a stroke. The ban was expected to last for two years, after which, it was hoped, the fishery could resume. But the cod have never recovered, and more than 15 years later the moratorium remains in effect. How could a fishery that had been for years under apparently careful scientific management just collapse? David Cayley talks to environmental philosopher Dean Bavington about the role of science in the rise and fall of the cod fishery."
The MISSION:
Listen to the recording. Take notes on what you’re hearing, with the goal of capturing lessons from the interview that might usefully influence how we test, and how we think about testing. Stories or patterns from your own testing experience that you can relate to something in the interview are especially valuable.
My experience tells me that it's good to take lots of notes, regardless if it's reviewing documents or acively testing. I fully expect to discard 90% of it but it can serve to confirm or discard ideas, theories and guesses.
So I listened to the interview and made notes about important events and their order, my raw data. I'd stop from time to time to listen again to passages to get a clearer understanding and to be able to keep up with note writing.
Michael clarified some terms that were used in Skype always shortly before I got to that passage. That was helpful on two fronts. One, I understood the term and context quicker. And two, I could tell that I was falling behind with my progress in listening as it took longer and longer between Michael explaining terms and me getting to the passage - a good way to measure progress, though fallible.
About 2/3rds in I changed the approach as I felt that I had enough data and because time was pressing. I started drawing parallels between the fishery events and software testing and noted those as well. Adding
to the raw data suffered as a result but that was fine as I was working towards the mission, drawing parallels and matching them to my experience. I concentrated on the similarities in process and process failings rather than thinking of 'fishing for bugs' or comparing the government quotas to test certification like some other people in the group did. Nothing wrong with either, I just took this particular approach.
Listening to the audio recording I noted several occasions of Hubris as reason for that monumental failure of preserving the cod population.
From Wiki: Hubris means extreme haughtiness or arrogance. Hubris often indicates being out of touch with reality and overestimating one's own competence or capabilities, especially for people in positions of power.
I reckon that is why the radio program was called "How to think about science", the idea that, while science has not all the answers, the ones it has can be taken as the truth.
That's only one part of it though, I don't want to make the same mistake and believe that one reason alone can explain the failures on many fronts.
Since I'm currently reading Jerry Weinberg's book "The Psychology of Computer Programming" (Thank you, Jerry, for giving me permission to use parts of this book) I saw some interesting similarities between his study of the psychology of computer programming and the events that lead to the disappearance of the Canadian cod. Here's Jerry's list from page 40 of his book.
1. Using introspection without validation
2. Using of observations on an overly narrow base
3. Observing the wrong variables, or failing to observe the right ones
4. Interfering with the observed phenomenon
5. Generating too much data and not enough information
6. Overly constraining experiments
7. Depending excessively on trainees as subjects
8. Failing to study group effects and group behaviour
9. Measuring what is simple to measure
10. Using unjustified precision
11. Transferring borrowed results to inapplicable situations
Matching my notes to this list I got the following results. We could argue with some of the matches, however the overall match of similarities should become clear:
1a. Key drivers were managing nature fluctuations in order to get back the money that was invested in the industry. Fluctuations turned out to be unmanageable – there was no driver to understand the model in detail
1b. Introducing averaging assumptions was one of the traps – the issue was not understood and two wildly different results had just been averaged with not thought of the consequences. One example of Hubris.
2. There was a simple model for the amount of cod in the sea which was far too simple and therefore wrong
3a Not all parameters of the model had been identified, for example parameter changes over time had been missed.
3b Some parameters had been misunderstood. These two points were the biggest mistakes made in my book. Science was not seen as fallible but a blind eye turned to the possibility that the current understanding does not fully cover the situation in question - see the definition for Hubris again.
4. There was constant interfering with the model as fishing continued throughout the years up to the point where the cod population didn't recover
5. Data from two sources was created, the scientific trawlers and the commercial fleet. They only seemed to look at total population, not age groups, etc
6. Local knowledge of inshore fishermen = “domain matter experts” has been ignored/dismissed as anecdotal – scientific methods were seen as more accurate and objective. Another example of Hubris
7. Not applicable in this scenario?
8. There were several examples of failing to recognise group behaviour, for example variation in location of spawning grounds and season, migration patterns, etc
9. Parameters found were dismissed as complicating the model too much (and making it unmanageable)
10. Not applicable in this scenario?
11. There were several examples of borrowing results from one are to another, for example assuming that inshore groups of cod behave the same as off shore groups
Some failings that I can't put into any of the categories but which are very important as well are:
A. The inshore fishermen’s protests have not been looked at in detail. They wanted to fish in the traditional way but reasons for that were not investigated. Scientists were supposed to speak truth. I'm wondering if, with a bit more questioning, some of the mistakes may have been prevented, for example damaging and destroying stock using the jigger (hook).
B. Investing in tools to make it cheaper became a driver to make more money which backfired. The parallel to automated software testing tools should be easy to see.
C. The fishing industry put pressure on the government to get higher allowable quotas. Available information was ignored and numbers changed against better knowledge to serve the industries need for money.
D. As the model got very complicated it didn’t help management as it didn't provide the control measures that were needed. In software terms this is the point where a project is either stopped or a drawn out death march begins
E. The target of management changed from figuring out how to achieve the goal to targeting people, the fisherman. Sounds oddly familiar, doesn't it?
I find it quite amazing how two, apparently unrelated areas like a book about the psychology of computer programming and discussing shortfalls of science in a fishery project are so similar at a higher level. I have a good number of other examples in mind, some of which I'll blog about shortly.
I'm in two minds about the message that science got it all wrong. I'd summarise it as scientists (bringing the people component in here) probably did their best but didn't look at their limitations. Results of their experiments were also disregarded and decisions taken based on politics and the industries needs/greed.
I come from a science background myself so may have a blind spot there. My overall (scientific approach/thinking) is to identify a problem, analyse it, put up some theories and experiments/tests that then either prove or disprove these theories. In other words, I'm building a close as possible picture or model of the issue in question. Any steps up to the results can be criticised, the result can not. The thinking behind that is that if you agree with the thinking behind all steps the result is independent. You don't get to argue with it if you don't like it. The result can be a pointer that something is wrong with your thinking, it's dangerous to build your thinking based on the results you want though. That's a different story though and far to involved to be discussed here.
I'd be happy to discuss these results in more detail so please feel free to agree or, even better, disagree with me.
I should note that these findings are based on a single source, the radio interview. What actually happened might or might not be exactly as portrayed, but I think it's useful to keep in mind that in testing as in life it's dangerous to believe only in one source without questioning it. Or would you believe a developer who tells you that a bug has been fixed and doesn't need re-tested?
PS: I would also like to point you towards an excellent blog of Jeroen Rosink who looks at this session from a slightly different angle.
Background to the interview:
On July 3, 1992, the Canadian Fisheries Minister John Crosbie announced a moratorium on the fishing of northern cod. It was the largest single day lay-off in Canadian history: 30,000 people unemployed at a stroke. The ban was expected to last for two years, after which, it was hoped, the fishery could resume. But the cod have never recovered, and more than 15 years later the moratorium remains in effect. How could a fishery that had been for years under apparently careful scientific management just collapse? David Cayley talks to environmental philosopher Dean Bavington about the role of science in the rise and fall of the cod fishery."
The MISSION:
Listen to the recording. Take notes on what you’re hearing, with the goal of capturing lessons from the interview that might usefully influence how we test, and how we think about testing. Stories or patterns from your own testing experience that you can relate to something in the interview are especially valuable.
My experience tells me that it's good to take lots of notes, regardless if it's reviewing documents or acively testing. I fully expect to discard 90% of it but it can serve to confirm or discard ideas, theories and guesses.
So I listened to the interview and made notes about important events and their order, my raw data. I'd stop from time to time to listen again to passages to get a clearer understanding and to be able to keep up with note writing.
Michael clarified some terms that were used in Skype always shortly before I got to that passage. That was helpful on two fronts. One, I understood the term and context quicker. And two, I could tell that I was falling behind with my progress in listening as it took longer and longer between Michael explaining terms and me getting to the passage - a good way to measure progress, though fallible.
About 2/3rds in I changed the approach as I felt that I had enough data and because time was pressing. I started drawing parallels between the fishery events and software testing and noted those as well. Adding
to the raw data suffered as a result but that was fine as I was working towards the mission, drawing parallels and matching them to my experience. I concentrated on the similarities in process and process failings rather than thinking of 'fishing for bugs' or comparing the government quotas to test certification like some other people in the group did. Nothing wrong with either, I just took this particular approach.
Listening to the audio recording I noted several occasions of Hubris as reason for that monumental failure of preserving the cod population.
From Wiki: Hubris means extreme haughtiness or arrogance. Hubris often indicates being out of touch with reality and overestimating one's own competence or capabilities, especially for people in positions of power.
I reckon that is why the radio program was called "How to think about science", the idea that, while science has not all the answers, the ones it has can be taken as the truth.
That's only one part of it though, I don't want to make the same mistake and believe that one reason alone can explain the failures on many fronts.
Since I'm currently reading Jerry Weinberg's book "The Psychology of Computer Programming" (Thank you, Jerry, for giving me permission to use parts of this book) I saw some interesting similarities between his study of the psychology of computer programming and the events that lead to the disappearance of the Canadian cod. Here's Jerry's list from page 40 of his book.
1. Using introspection without validation
2. Using of observations on an overly narrow base
3. Observing the wrong variables, or failing to observe the right ones
4. Interfering with the observed phenomenon
5. Generating too much data and not enough information
6. Overly constraining experiments
7. Depending excessively on trainees as subjects
8. Failing to study group effects and group behaviour
9. Measuring what is simple to measure
10. Using unjustified precision
11. Transferring borrowed results to inapplicable situations
Matching my notes to this list I got the following results. We could argue with some of the matches, however the overall match of similarities should become clear:
1a. Key drivers were managing nature fluctuations in order to get back the money that was invested in the industry. Fluctuations turned out to be unmanageable – there was no driver to understand the model in detail
1b. Introducing averaging assumptions was one of the traps – the issue was not understood and two wildly different results had just been averaged with not thought of the consequences. One example of Hubris.
2. There was a simple model for the amount of cod in the sea which was far too simple and therefore wrong
3a Not all parameters of the model had been identified, for example parameter changes over time had been missed.
3b Some parameters had been misunderstood. These two points were the biggest mistakes made in my book. Science was not seen as fallible but a blind eye turned to the possibility that the current understanding does not fully cover the situation in question - see the definition for Hubris again.
4. There was constant interfering with the model as fishing continued throughout the years up to the point where the cod population didn't recover
5. Data from two sources was created, the scientific trawlers and the commercial fleet. They only seemed to look at total population, not age groups, etc
6. Local knowledge of inshore fishermen = “domain matter experts” has been ignored/dismissed as anecdotal – scientific methods were seen as more accurate and objective. Another example of Hubris
7. Not applicable in this scenario?
8. There were several examples of failing to recognise group behaviour, for example variation in location of spawning grounds and season, migration patterns, etc
9. Parameters found were dismissed as complicating the model too much (and making it unmanageable)
10. Not applicable in this scenario?
11. There were several examples of borrowing results from one are to another, for example assuming that inshore groups of cod behave the same as off shore groups
Some failings that I can't put into any of the categories but which are very important as well are:
A. The inshore fishermen’s protests have not been looked at in detail. They wanted to fish in the traditional way but reasons for that were not investigated. Scientists were supposed to speak truth. I'm wondering if, with a bit more questioning, some of the mistakes may have been prevented, for example damaging and destroying stock using the jigger (hook).
B. Investing in tools to make it cheaper became a driver to make more money which backfired. The parallel to automated software testing tools should be easy to see.
C. The fishing industry put pressure on the government to get higher allowable quotas. Available information was ignored and numbers changed against better knowledge to serve the industries need for money.
D. As the model got very complicated it didn’t help management as it didn't provide the control measures that were needed. In software terms this is the point where a project is either stopped or a drawn out death march begins
E. The target of management changed from figuring out how to achieve the goal to targeting people, the fisherman. Sounds oddly familiar, doesn't it?
I find it quite amazing how two, apparently unrelated areas like a book about the psychology of computer programming and discussing shortfalls of science in a fishery project are so similar at a higher level. I have a good number of other examples in mind, some of which I'll blog about shortly.
I'm in two minds about the message that science got it all wrong. I'd summarise it as scientists (bringing the people component in here) probably did their best but didn't look at their limitations. Results of their experiments were also disregarded and decisions taken based on politics and the industries needs/greed.
I come from a science background myself so may have a blind spot there. My overall (scientific approach/thinking) is to identify a problem, analyse it, put up some theories and experiments/tests that then either prove or disprove these theories. In other words, I'm building a close as possible picture or model of the issue in question. Any steps up to the results can be criticised, the result can not. The thinking behind that is that if you agree with the thinking behind all steps the result is independent. You don't get to argue with it if you don't like it. The result can be a pointer that something is wrong with your thinking, it's dangerous to build your thinking based on the results you want though. That's a different story though and far to involved to be discussed here.
I'd be happy to discuss these results in more detail so please feel free to agree or, even better, disagree with me.
I should note that these findings are based on a single source, the radio interview. What actually happened might or might not be exactly as portrayed, but I think it's useful to keep in mind that in testing as in life it's dangerous to believe only in one source without questioning it. Or would you believe a developer who tells you that a bug has been fixed and doesn't need re-tested?
PS: I would also like to point you towards an excellent blog of Jeroen Rosink who looks at this session from a slightly different angle.
Monday, 10 May 2010
It’s NOT Rocket Science
This blog is to go into a bit more detail about the planning and execution of the European Weekend Testing mision 17.
If you don’t know what the mission was please have a look and read it.
In summary, the group should test an audio application, a software compressor plugin that needs its own host. The group was testing the plugin through the host which was an additional complication.
The mission for EWT17 was several weeks in the planning. I first came up with a rough outline, then bounced it off Anna and Markus who made it better without really knowing the application. We then pre-tested the whole mission with Anna and Markus on Thursday to make sure it was not too hard to do. I got interesting results to what they were looking at and we then decided that, yes, this mission is a go.
For me, the one of the main goals of the mission was to bring a bit more realism into it. So I introduced a bit of role playing in saying that all EWT testers are in one company in one group reporting to one test manager who has a problem he wants them to solve: Does this software compressor work and is stable enough for me to use tonight?
His problem is time limited and he wants an answer in 1.5 hours. I should rephrase that, he wants one answer. From his point of view it’s one problem, one question, one answer.
Of course in EWT people come from all over the world so this was always difficult to achieve but some people picked up on it as Jeroen also pointed out in his blog.
I wasn't actually expecting to get one but wasn't sure what people would do. Some testers refused an answer as per Michael Bolton's article.
Anna, Markus and me discussed if we should add a team lead role for this mission but decided against it as it wasn’t in line with the spirit of EWT to have a hierarchy. I was interested to see if one developed naturally or not, which didn’t happen – no problem there.
The other big difference in this mission compared to previous ones is that I picked a domain where not many people would have knowledge of – audio recording and engineering. There are actually a lot of parallels with software testing but that’s another story or blog.
When the mission was released people realised within a couple of minutes that they don’t know enough about the domain to be able to test the application. Some went into ET testing mode, some Googled and found the Wiki page that explains very well what it’s all about. No one thought about asking if there are domain experts in the group of which we had one, Zeger. He actually couldn’t get the software to start on his laptop so was available to answer questions. He picked up on some assumptions people made and threw in some very valuable comments. Some were picked up, others weren’t, for example the important one about asking if the test manager is using the same plug-in host or not as this could invalidate the test effort. Yes, he did and it would actually..
The Test Manager gave the group pretty much an impossible task. Find out if the application is stable. What does stable mean without context? Without knowing what hardware is used, operating system (as someone pointed out), other software installed stable is pretty much meaningless.
The other big area that I wanted to explore with this mission is testability. There are several possible meanings. One issue with testability is that the testers didn’t have enough domain knowledge to know how they would actually test the application. The Rocket reduces the dynamic range but how many testers know how to test that?
The other issue is about measuring if the dynamic range actually has been reduced and by the right amount. Since we’re talking about audio here we can’t rely on our senses as they aren’t up to the task. Sure we can hear if something gets quieter and louder but only if the difference is quite large. How large depends on the testers ears so it is really subjective.
There are some freeware tools that would help with this task, for example the freeware audio editor and recorder. A tester could then record the output of the plugin and compare waveforms or rely on the measure tools that come with the audio editor application. All in all a pretty difficult task. I only mention it here because the secondary objective was to find out if the attack of the plugin was fast enough. Something that could only be tested with additional tools and was largely ignored by the group, which was good. If you have your hands full with the main objective don’t get distracted with nice to have’s – thumbs up for the EWT testers.
The group dynamics where interesting to see and also the extend to which testers picked up on what information others typed into Skype. I tried to facilitate more group working which hasn’t worked so well. This is clearly something we need to think about again for future missions.
It was an intense session but one where I think everyone got something out of. To the new people who joined us for EWT17, it’s not always like that!
I’d like to everyone who participated who made this session a success and gave me lots to think about – and that’s a good thing.
Thomas
If you don’t know what the mission was please have a look and read it.
In summary, the group should test an audio application, a software compressor plugin that needs its own host. The group was testing the plugin through the host which was an additional complication.
The mission for EWT17 was several weeks in the planning. I first came up with a rough outline, then bounced it off Anna and Markus who made it better without really knowing the application. We then pre-tested the whole mission with Anna and Markus on Thursday to make sure it was not too hard to do. I got interesting results to what they were looking at and we then decided that, yes, this mission is a go.
For me, the one of the main goals of the mission was to bring a bit more realism into it. So I introduced a bit of role playing in saying that all EWT testers are in one company in one group reporting to one test manager who has a problem he wants them to solve: Does this software compressor work and is stable enough for me to use tonight?
His problem is time limited and he wants an answer in 1.5 hours. I should rephrase that, he wants one answer. From his point of view it’s one problem, one question, one answer.
Of course in EWT people come from all over the world so this was always difficult to achieve but some people picked up on it as Jeroen also pointed out in his blog.
I wasn't actually expecting to get one but wasn't sure what people would do. Some testers refused an answer as per Michael Bolton's article.
Anna, Markus and me discussed if we should add a team lead role for this mission but decided against it as it wasn’t in line with the spirit of EWT to have a hierarchy. I was interested to see if one developed naturally or not, which didn’t happen – no problem there.
The other big difference in this mission compared to previous ones is that I picked a domain where not many people would have knowledge of – audio recording and engineering. There are actually a lot of parallels with software testing but that’s another story or blog.
When the mission was released people realised within a couple of minutes that they don’t know enough about the domain to be able to test the application. Some went into ET testing mode, some Googled and found the Wiki page that explains very well what it’s all about. No one thought about asking if there are domain experts in the group of which we had one, Zeger. He actually couldn’t get the software to start on his laptop so was available to answer questions. He picked up on some assumptions people made and threw in some very valuable comments. Some were picked up, others weren’t, for example the important one about asking if the test manager is using the same plug-in host or not as this could invalidate the test effort. Yes, he did and it would actually..
The Test Manager gave the group pretty much an impossible task. Find out if the application is stable. What does stable mean without context? Without knowing what hardware is used, operating system (as someone pointed out), other software installed stable is pretty much meaningless.
The other big area that I wanted to explore with this mission is testability. There are several possible meanings. One issue with testability is that the testers didn’t have enough domain knowledge to know how they would actually test the application. The Rocket reduces the dynamic range but how many testers know how to test that?
The other issue is about measuring if the dynamic range actually has been reduced and by the right amount. Since we’re talking about audio here we can’t rely on our senses as they aren’t up to the task. Sure we can hear if something gets quieter and louder but only if the difference is quite large. How large depends on the testers ears so it is really subjective.
There are some freeware tools that would help with this task, for example the freeware audio editor and recorder. A tester could then record the output of the plugin and compare waveforms or rely on the measure tools that come with the audio editor application. All in all a pretty difficult task. I only mention it here because the secondary objective was to find out if the attack of the plugin was fast enough. Something that could only be tested with additional tools and was largely ignored by the group, which was good. If you have your hands full with the main objective don’t get distracted with nice to have’s – thumbs up for the EWT testers.
The group dynamics where interesting to see and also the extend to which testers picked up on what information others typed into Skype. I tried to facilitate more group working which hasn’t worked so well. This is clearly something we need to think about again for future missions.
It was an intense session but one where I think everyone got something out of. To the new people who joined us for EWT17, it’s not always like that!
I’d like to everyone who participated who made this session a success and gave me lots to think about – and that’s a good thing.
Thomas
Saturday, 17 April 2010
ACCU2010 writeup
Here’s a short (OK, I got carried away a bit) summary of the conference and the talks/presentations I attended. I only went down for one day - Friday 16th April due to work commitments.
Friday morning:
Keynote talk from Russell Winder “The Multicore Revolution - We’ve only just begun”. Originally Dan North was down to present “Simplicity – the way of the unusual architect” but couldn’t get to the UK due to the airspace being closed because of the Icelandic volcano eruption.
Russell is clearly very knowledgeable and presented a good talk about the architecture of computers the way different programming languages address those. Good presentation that was probably more interesting to the developer community than for me but interesting nonetheless.
After a coffee break which I used for frantic telephone calls and trying to book train tickets I attended Rachel Davies “Understanding User Stories”. Not having written any before this was an interesting presentation with just the right amount of practical exercises for the participants. She mentioned the book “Agile Coaching” which she wrote or co-wrote which I’ll have a look at.
She explained that she and her team used a user story template which is now being used by many people around the world. I got the impression that she didn’t expect that and was pleasantly surprised that such a small thing created to make life easier in her project team has such a big impact. She did not that this template doesn’t work in all cases – especially when there’s no discussion with the business or if technical implementations are the order of the day rather than user workflows.
The template followed a
As a...
I want...
So that...
format which goes a long way getting all the necessary information onto a small card. She noted the importance of this being a physical card rather than a spreadsheet or electronic document. To me this format is similar-ish to the way use cases are created as the actor (As a xyz..) and workflow (I want...) and goal description (so that..) are all present. That’s not bad and I don’t confuse the two but just note the similarities.
After a practical exercise of writing user stories for an imaginary tool of our choice we discussed that. What I found interesting is that breaking down the user stories to the right level and amount of detail is really a team decision – something many people struggle with when writing use cases.
The importance about the user stories should be “Do the user stories help us understand user context and business value.” That’s a harsh abbreviation and there’s more to it but to me these are the essentials (together with acceptance criteria).
At lunch I met James Bach and after listening in to another conversation we spoke about some senior managers and their viewpoints on testing, the TA model (parent/adult/child modes) and implementing rapid testing in companies. As I’m following both James and Michael B’s blogs there was nothing groundbreaking new for me but it rather reinforced my own views but also highlighted that the problems I’m facing are not with the understanding of the rapid testing approach but in other, more business related areas. I left contemplating and thinking about this (which is a good thing) but will need some time before I’ll find answer. Interesting.
After lunch I attended Paul Field’s “Change, change, change – the 5 year evolution of an agile team.” It was an interesting presentations about his experience about starting out with a 2 man agile team taking on a failing project in an IT unfriendly environment – the internal customer (another department) got burned in the past and was wary of IT in general.
He spoke about the pitfalls of building a team and scaling that from 2 to 7 after a while. Generally a good talk although it could’ve been a bit condensed from my point of view.
I liked his use of the Timeline Retrospective where he mapped colour coded issues as experienced by the team in sprint retrospectives over the whole project time. This can be a good way not to measure the way the project is going but the way the team is feeling about it which doesn’t necessarily needs to be the same. (And indeed was not, the team thought that they delivered badly at a sprint when the customer was delighted).
The retrospective starfish was also interesting consisting of 5 areas for
More of
Start doing
Stop doing
Less of
Keep doing
In the burnup chart he presented he mapped the story points “Done” and the story points as asked for from the business. This was not per sprint but over the whole project which showed that after 4 months (in a 6 month project) the team had created as many story points as the business originally wanted (great) but wouldn’t be able to finish by immovable 6 month deadline – a big problem. This burnup chart showed the problem in time and action was taken to get the project back on track.
When new team members where added he found that his former “flat” structure didn’t work anymore as existing team members were more senior in terms of knowledge so not everyone was equal. This needed to be addressed and after the new team member were brought up to speed through pair programming and other measures the team picked up some momentum.
After a coffee break I went to Astrid Byro’s “Just in time – last minute testing” presentation. She jumped in for another presenter who couldn’t come due to “Volcano awareness week” as James Bach coined it.
This was the weakest talk for me, the title didn’t have very much to do with the talk which was basically a war story about a 2 year RUP project with very high risks (accepted), lots of stress and a “command and control” approach should have died out 20 years ago. System migrators where used as sole testers which would’ve made me uncomfortable (maybe she was as well). I can see that from a PM point of view using people with domain knowledge is a good idea and using testing as a training exercise so you save handover further down the line is good for project timelines as well. But I can’t shake off the feeling that professional testers would’ve done some good – apparently that wasn’t possible (or thought necessary). At the end of the project performance testing was started (always a bad idea due to the high risks involved, again they were lucky and their application performed well). Their clients infrastructure didn’t but that wasn’t seen as a problem as the project in itself delivered, even if that didn’t actually help the customer. My view is more holistic than just pointing at the customer and saying “it’s their problem, our application performs well”, but maybe that’s just me.
That concluded the official presentations but I saw about a dozen lightning talks, the highlight for me was James Bach’s “And...also” presentation (I’m predictable in that regard, I know).His fiery talk had a lot of passion and humour to be able to appeal to even the most hardened developers and I believe he got some people thinking. It was about thinking not only what the expected positive outcome is but also what he also expects to be there (or not). For example NOT getting error messages, expecting the result to appear for more than half a second, etc – all things an automated test suite might fail to notice but a human being would notice – sometimes instinctively, sometimes with a bit of thinking.
I won’t mention the other lightning talks, there were too many to mention and I didn’t take notes of all so wouldn’t be able to give a complete picture anyway.
After a longer break the conference dinner started which was very interesting as well, after the starter we played musical chairs with only the speakers remaining seated so that people got more opportunity to speak to different speakers and other attendees. At my first table I sat next to Jeff Sutherland, creator of Scrum, without recognising him which I found extremely embarrassing. I promised to put in a good word with the conference organisers to get him back next year so that I could have a longer discussion with him – so organisers, if you’re reading this, do all of us a favour and invite Jeff for next year as well!
That’s it, thanks to Volcano Awareness Week I’ve been able to write down these notes while on my train journey rather than sitting at home already so let’s all say thanks to the volcano gods
:-)
Thomas
Friday morning:
Keynote talk from Russell Winder “The Multicore Revolution - We’ve only just begun”. Originally Dan North was down to present “Simplicity – the way of the unusual architect” but couldn’t get to the UK due to the airspace being closed because of the Icelandic volcano eruption.
Russell is clearly very knowledgeable and presented a good talk about the architecture of computers the way different programming languages address those. Good presentation that was probably more interesting to the developer community than for me but interesting nonetheless.
After a coffee break which I used for frantic telephone calls and trying to book train tickets I attended Rachel Davies “Understanding User Stories”. Not having written any before this was an interesting presentation with just the right amount of practical exercises for the participants. She mentioned the book “Agile Coaching” which she wrote or co-wrote which I’ll have a look at.
She explained that she and her team used a user story template which is now being used by many people around the world. I got the impression that she didn’t expect that and was pleasantly surprised that such a small thing created to make life easier in her project team has such a big impact. She did not that this template doesn’t work in all cases – especially when there’s no discussion with the business or if technical implementations are the order of the day rather than user workflows.
The template followed a
As a...
I want...
So that...
format which goes a long way getting all the necessary information onto a small card. She noted the importance of this being a physical card rather than a spreadsheet or electronic document. To me this format is similar-ish to the way use cases are created as the actor (As a xyz..) and workflow (I want...) and goal description (so that..) are all present. That’s not bad and I don’t confuse the two but just note the similarities.
After a practical exercise of writing user stories for an imaginary tool of our choice we discussed that. What I found interesting is that breaking down the user stories to the right level and amount of detail is really a team decision – something many people struggle with when writing use cases.
The importance about the user stories should be “Do the user stories help us understand user context and business value.” That’s a harsh abbreviation and there’s more to it but to me these are the essentials (together with acceptance criteria).
At lunch I met James Bach and after listening in to another conversation we spoke about some senior managers and their viewpoints on testing, the TA model (parent/adult/child modes) and implementing rapid testing in companies. As I’m following both James and Michael B’s blogs there was nothing groundbreaking new for me but it rather reinforced my own views but also highlighted that the problems I’m facing are not with the understanding of the rapid testing approach but in other, more business related areas. I left contemplating and thinking about this (which is a good thing) but will need some time before I’ll find answer. Interesting.
After lunch I attended Paul Field’s “Change, change, change – the 5 year evolution of an agile team.” It was an interesting presentations about his experience about starting out with a 2 man agile team taking on a failing project in an IT unfriendly environment – the internal customer (another department) got burned in the past and was wary of IT in general.
He spoke about the pitfalls of building a team and scaling that from 2 to 7 after a while. Generally a good talk although it could’ve been a bit condensed from my point of view.
I liked his use of the Timeline Retrospective where he mapped colour coded issues as experienced by the team in sprint retrospectives over the whole project time. This can be a good way not to measure the way the project is going but the way the team is feeling about it which doesn’t necessarily needs to be the same. (And indeed was not, the team thought that they delivered badly at a sprint when the customer was delighted).
The retrospective starfish was also interesting consisting of 5 areas for
More of
Start doing
Stop doing
Less of
Keep doing
In the burnup chart he presented he mapped the story points “Done” and the story points as asked for from the business. This was not per sprint but over the whole project which showed that after 4 months (in a 6 month project) the team had created as many story points as the business originally wanted (great) but wouldn’t be able to finish by immovable 6 month deadline – a big problem. This burnup chart showed the problem in time and action was taken to get the project back on track.
When new team members where added he found that his former “flat” structure didn’t work anymore as existing team members were more senior in terms of knowledge so not everyone was equal. This needed to be addressed and after the new team member were brought up to speed through pair programming and other measures the team picked up some momentum.
After a coffee break I went to Astrid Byro’s “Just in time – last minute testing” presentation. She jumped in for another presenter who couldn’t come due to “Volcano awareness week” as James Bach coined it.
This was the weakest talk for me, the title didn’t have very much to do with the talk which was basically a war story about a 2 year RUP project with very high risks (accepted), lots of stress and a “command and control” approach should have died out 20 years ago. System migrators where used as sole testers which would’ve made me uncomfortable (maybe she was as well). I can see that from a PM point of view using people with domain knowledge is a good idea and using testing as a training exercise so you save handover further down the line is good for project timelines as well. But I can’t shake off the feeling that professional testers would’ve done some good – apparently that wasn’t possible (or thought necessary). At the end of the project performance testing was started (always a bad idea due to the high risks involved, again they were lucky and their application performed well). Their clients infrastructure didn’t but that wasn’t seen as a problem as the project in itself delivered, even if that didn’t actually help the customer. My view is more holistic than just pointing at the customer and saying “it’s their problem, our application performs well”, but maybe that’s just me.
That concluded the official presentations but I saw about a dozen lightning talks, the highlight for me was James Bach’s “And...also” presentation (I’m predictable in that regard, I know).His fiery talk had a lot of passion and humour to be able to appeal to even the most hardened developers and I believe he got some people thinking. It was about thinking not only what the expected positive outcome is but also what he also expects to be there (or not). For example NOT getting error messages, expecting the result to appear for more than half a second, etc – all things an automated test suite might fail to notice but a human being would notice – sometimes instinctively, sometimes with a bit of thinking.
I won’t mention the other lightning talks, there were too many to mention and I didn’t take notes of all so wouldn’t be able to give a complete picture anyway.
After a longer break the conference dinner started which was very interesting as well, after the starter we played musical chairs with only the speakers remaining seated so that people got more opportunity to speak to different speakers and other attendees. At my first table I sat next to Jeff Sutherland, creator of Scrum, without recognising him which I found extremely embarrassing. I promised to put in a good word with the conference organisers to get him back next year so that I could have a longer discussion with him – so organisers, if you’re reading this, do all of us a favour and invite Jeff for next year as well!
That’s it, thanks to Volcano Awareness Week I’ve been able to write down these notes while on my train journey rather than sitting at home already so let’s all say thanks to the volcano gods
:-)
Thomas
Monday, 8 March 2010
First Scottish Software Testing Club event in Edinburgh
I’d like to announce the first Scottish Software Testing Club meeting on Wednesday, 10th March, 7pm in the Filmhouse Bar, Lothian Road in Edinburgh. Food is nice and they always have vegetarian and vegan meals on the menu. The beer and wine selection is good as well. Thanks to Rosie for letting us put the STC label on this meeting.
This is an informal event and is intended as a networking session and just saying hello to the other local testers. Everyone’s welcome, even if you are a developer :-). Glenn and I are thinking of maybe making this a monthly institution, 2nd Wednesday of each month or something along these lines – we’ll see what the turnout is going to be.
We'll meet in the main area, I'll try and put a picture of the STC logo up if I have the space.
Hope to see you there,
Thomas
This is an informal event and is intended as a networking session and just saying hello to the other local testers. Everyone’s welcome, even if you are a developer :-). Glenn and I are thinking of maybe making this a monthly institution, 2nd Wednesday of each month or something along these lines – we’ll see what the turnout is going to be.
We'll meet in the main area, I'll try and put a picture of the STC logo up if I have the space.
Hope to see you there,
Thomas
Subscribe to:
Posts (Atom)