The methods of analysis are quite simple. Cross-tabulations and correlations are interesting. What you often see is plain frequencies or averages which are usually not interesting. Here is an example
Did you use a consultant in your ITIL implementation project?
- The project was executed by outside consultants
- The project was supported by outside consultants
- The project was executed by own staff
Has ITIL Fulfilled your expectations
- Do not know
Did the project budget and timetable hold?
The answers were not very interesting, roughly 60% used a consultant to support, 60 % were satisfied and 60 % had missed the timetable or budget. What would have been interesting was the cross-tabulation of the use of consultants to results but the team doing the survey used a free tool that only reported the plain frequencies.
Reporting averages or frequencies is not analysis. Proper analysis means that you need to cross tabulate or analyze correlations. It will be much easier if you have only three questions (and some background info) than if you have 30 questions and 7 background variables.
4.1 Weeding the data
There is a good rule in data analysis: GARBAGE IN – GARBAGE OUT.
Data analysis does not refine data, it just compresses it. And it does not produce diamonds out of rubbish.
It is important to weed the data. If you get surprising results the most likely reason is a data error. People misunderstand questions and give impossible answers. Human errors happen. 3,4 becomes 34. A representative of a six person internal IT department might answer the she represents a 700 person service company and they have 1.5 million customers as their parent indeed is a company in service business but it is not IT service that they are doing.
4.1 Background info
Blind surveys are generally not good. You should know some basic facts about your respondents. My surveys are usually aimed at IT service business and there are always two things I want to know.
- Is it an internal or external service unit?
- Is it a small or large unit?
Sometimes it is quite important to know what the role of the respondent is related to the survey issue. Here is an example of the importance of the role.
The survey tried to find out what method IT users employ when they are trying to solve their problems. One of the choices was: Using Service Desk. I noticed that Service Desk people were giving a high number on this question. What I did was that sent a new questions to all who had responded and asked what was their role in support, were they providing the service, using the service or administering the service. Incidentally this is a benefit of using the e-mail survey. It is easy to contact the respondents.
The answers confirmed my suspicions. The Service Desk people esitimated that their share of support was 58%, the users estimated it as 38 %.
It is quite common that amateurs designing surveys want to add many background questions. The reason for this is that they have seen similar questions in professional surveys and the want to look like pro’s. The problem is that they have no clue to why pro’s have these questions. I will not waste time by explaining why. My advice is: DO NOT ASK UNNECESSARY QUESTIONS.
4.2 Moving targets
In many cases the number you are trying to measure is not stable. The next graph shows the value of Customer satisfaction over a period of 20 weeks.
It is easy to see that the time of measurement is quite important. In this case there was a problem on week 7 which was fixed temproraily on week 8. When it reocurred on week 9, it caused a drop in staisfaction which laster for five weeks.
4.2 What do numbers tell?
Let us continue with application example and assume that the fact was that nearly everybody had problems in the beginning but after a couple of months the number of problems would stabilize to a level of one problem per two months. In fact we already know this as we have been able to follow the number of incidents that users have reported to the service desk.
Let us also assume that we know the number of weeks the person has been using the application. Now we can calculate how many times a person has a problem based on how many weeks they have been using it.
Again we see that we have a moving target and the numbers do not have a constant value. It can be useful to know this relationship. What about if a person who has been using the system for 8 weeks, reports that she still has one problem per week. Is there something wrong with this person, did she miss the training? We see that the survey has been given a new meaning.
As we see, the number survey can be useful but we are still missing an important point. How important were these problems? Did it hurt business? The number of incidents does not tell us that.
I have seen several surveys where they ask which ITIL processes the respondent have implemented. Typically the results indicate that a large number of respondent have implemented the operational processes like Problem Management. At the same time I have been trying to get some basic numbers on processes. It seems quite typical that a unit may report that they have implemented Problem Management but are unable to report the number of problems they open per month.
A general rule is that SURVEY NUMBERS DON’T MEAN MUCH
4.2 Example of analysis
In this case I did a 3 question survey on internal and external cooperation and ITIL implementation
All questions used a 1-5 scale between agree – disagree.
To get this final result I calculated the difference in internal and external cooperation and compared that to the ITIL implementation situation. It shows that internal cooperation is better when you have implemented ITIL and external cooperation is better when you have not implemented ITIL. Quite significant result.
4.2 Analyzing free text
The free text survey is usually the most powerful tool. Analyzing the free questions takes a little effort but is not usually difficult. Here is a simple technique.
Copy/enter the answers in an Excel table. Split all separate statements in a cell. Use one table for each question. Add one column for classifying the data.
Here is an example with three answers:
What was best in the course?
- The exercises.
- The atmosphere was relaxed and the trainer really knows this stuff
- I liked best the opportunity do discuss and hear other people’s experiences in this area. It was good that the trainer allowed people to discuss and the exercises were also quite helpful.
What could be improved?
- The material. Course could be shorter.
- The room was hot. There should be more explanations in the material.
- The room was too cold. There should be more examples in the material. Too many acronyms in the slides. The course was too short.
The processed answers look like this
|good||What was best in the course?|
|atmosphere||The atmosphere was relaxed|
|atmosphere||It was good that the trainer allowed people to discuss|
|discussion||I liked best the opportunity do discuss and hear other people’s experiences in this area.|
|exercises||the exercises were also quite helpful.|
|knowledge||the trainer really knows this stuff|
|improve||What could be improved?|
|length||Course could be shorter.|
|length||The course was too short|
|material||There should be more explanations in the material.|
|material||There should be more examples in the material.|
|material||Too many acronyms in the slides.|
|room||The room was hot.|
|room||The room was too cold.|
We can see that the material needs to be improved as there 4 improvement comments on that.
The idea in the analysis is finding common issues from the open answers. If there is a problem or a special strength, many people will say it using a little different words. The classification will help you to find it. You can do the classification a few times to get it right. Sometimes it useful to combine a few similar classes and sometimes you need to split a class in two different classes.
It is possible to get a numerical report out of these open questions. I usually show the frequencies in a graph. (Here we assume there were more answers).
You can even calculate a satisfaction index from positive (POS) and negative comments (NEG)
On this index a positive number like +15% is a good result.