When we discuss gamification in general, the typical question occurs, does it work? In my personal experience I have seen it work. Quantifiable proof however is somewhat harder to get your hands on. In our case the data are often protected by non-disclosure agreements, because they may give away data that are company sensitive, which organisations prefer not to share.
I continuously keep up-to-date and my assistant has a regular task to look for data and specific proof of gamification in HR and Learning, where we work the most. He found me an interesting piece of work, an article published this year in the International Journal of Educational Technology in Higher Education by Dichev and Dicheva. The researchers examined empirical studies carried out about the effectiveness of gamification in education between 2014 and 2015. Anything that didn’t focus solely on gamification and included game based learning was excluded.
Interestingly enough research methods varied, which lead to the majority of studies to be deemed inconclusive. I am not sure if this a a sign of an industry desperately trying to prove it’s value where providers then make all sorts of crazy claims and find a basic study to confirm why you should buy from them. I have seen some mad claims being mad, which often sound like finger in the air make up a number style research. My usual yardstick is if it has 3 figures before the decimal point and a % sign then really look for proof, because in most cases this is not going to happen and in any case rarely sustainable. It also challenges me statistically, how can you have 200% of something when you already had a starting point before then and usually the maximum is 100%? Maybe I followed different statistics classes to those magicians.
The challenge pointed out by the researchers also is that multiple game mechanics are examined at once, hence not allowing to judge which of them enhanced learning if at all. Setting up a similar series of game mechanics may not have resulted in replicating the outcome of the research. I know I am not painting the most rosy picture here, it is at the same time important to realise that for one gamification is quite complex and applied to another complex field such as learning a lot of factors may come into play. I would love to have the opportunity to be part of a proper research study around gamification of learning to test what works best, across age, gender, socio-economic backgrounds, etc.
The overall result of the comparative study was not all doom in gloom, even though the majority of studies did not give conclusive evidence. The remaining studies were more positive than negative. The more research comes out and judging by the weekly requests from students around the world, it won’t be long hopefully before we can also address the inconclusive factor and have meaningful data to work with.
Distribution of behavioural studies by degree of evidence:
Positive 26%
Negative 10%
Inconclusive 64%
As most of my clients know, I don’t want to paint on overly rosy picture when presenting to them. I, maybe wrongly, tend to err on the side of caution. I know that we have achieved improvements on the original goals of our clients, typically in double figures. The average is probably 20%. If you think this is small, look at it this way, if as a result of 20% increase in retention, you sell more, your people can do more, then for a business the impact is incremental. Proving what exactly made things hit the sweet spot, I think is often also claiming to know everything about everyone and personally I don’t find that realistic. But that is just my opinion.
If you have solid research and great data to share, by all means let me know.
Leave a Reply