Backing up your claims

genie out of the bottle for blog article on

At a recent event, a speaker was making claims about how what they were building was totally new to the market and nobody else had done it. They also went on to say that they had been specifically selected to build this new thing for a major software house. As a listener to the talk and with in-depth knowledge of the target market this new thing was aimed at, I knew of actual examples of companies already doing what the speaker spoke about and also of instances where it was being used in business. I didn’t address it publicly but went to speak to the person later privately and no matter how many examples I had, all my information was overruled and dismissed.

It made me wonder, how much research the speaker had done to make the claims they made. First mover advantage and those claims tend to be for first movers for real, not for followers of a trend. At least that is my opinion.

I find it hard to make big claims without evidence for everything we do, where possible I look for research to confirm our thinking or dismiss it for a different approach. It is why I advise any company we work with to do their research and test extensively what works or doesn’t.

If we can’t find any evidence from other sources, we tend to A/B test and explore options to find out what is the most effective way forward for all users and everyone involved.

I cringe when I hear we have achieved 300% improvement in an initiative. I always wonder, compared to what really? Just like I cringed, when the speaker made claims I knew were actually untrue. Maybe it’s modesty, maybe it’s being overly harsh, but I personally believe it is also what I see as wrong in business. False claims becoming make belief.

I think current technology allows us to test and build data analysis to verify our designs, hunches and substantiate our claims. Ideally, they work in our favour, but when they don’t we need to listen and analyse what would make it better, more inclusive, more engaging, more pervasive, etc.

Here is what I would suggest doing when you are making claims about your product or service, look for research that either confirms or rejects what you are aiming to do. This gives you an educated view and can handle objectives. Equally, check whether you have competition in your space or someone has created something very similar. Having competition and being aware of what they do, is a good thing, it means there tends to be a market for your service.

If you then want to go a bit further and back up your product claims, then partner with a university or independent research body to have your product tested for the claims you want to make. Their independence is important. Having your clients doing the talking about the claims you make is even better also. 3rd party endorsement has been powerful for years.


Researching benefits of gamification

Together with a client we are embarking on research to verify if our gamified learning solution is actually more effective than the non-gamified learning resources. In my quest to prepare for this research I reached out to a number of academics in this space in the US, which surprisingly lead to some rather direct variations of ‘please get lost’. Only not expressed in such words.

My list of connections hasn’t been exhausted yet and some of the UK based academia have been much more friendly and at least willing to engage in conversation. It is one thing for academics to criticise that our field hasn’t been researched enough but then in the same sentence refuse to work with practitioners who actually are willing to engage in research, I find dumbfounding.

We will still go ahead with our project regardless of academic input. It is both in the interest of our work and that of our client too. What we are looking to get input on, is research methodology for short term research and what variables to test for.

The obvious items we want to discover is whether the learner indeed retains more information with gamification. We would like to find out which of the game mechanics we used actually stood out and made a difference to the learner as expressed by the learner.

Due to time constraints the sample may be small for now and very targeted at our clients target audience. It also means we may need to keep the testing relatively limited.

So a question to all my readers, what would you test and want to find out from research on the effectiveness of a gamified learning solution?

Is gamification in learning really working?

When we discuss gamification in general, the typical question occurs, does it work? In my personal experience I have seen it work. Quantifiable proof however is somewhat harder to get your hands on. In our case the data are often protected by non-disclosure agreements, because they may give away data that are company sensitive, which organisations prefer not to share.

I continuously keep up-to-date and my assistant has a regular task to look for data and specific proof of gamification in HR and Learning, where we work the most. He found me an interesting piece of work, an article published this year in the International Journal of Educational Technology in Higher Education by Dichev and Dicheva. The researchers examined empirical studies carried out about the effectiveness of gamification in education between 2014 and 2015. Anything that didn’t focus solely on gamification and included game based learning was excluded.

Interestingly enough research methods varied, which lead to the majority of studies to be deemed inconclusive. I am not sure if this a a sign of an industry desperately trying to prove it’s value where providers then make all sorts of crazy claims and find a basic study to confirm why you should buy from them. I have seen some mad claims being mad, which often sound like finger in the air make up a number style research. My usual yardstick is if it has 3 figures before the decimal point and a % sign then really look for proof, because in most cases this is not going to happen and in any case rarely sustainable. It also challenges me statistically, how can you have 200% of something when you already had a starting point before then and usually the maximum is 100%? Maybe I followed different statistics classes to those magicians.

The challenge pointed out by the researchers also is that multiple game mechanics are examined at once, hence not allowing to judge which of them enhanced learning if at all. Setting up a similar series of game mechanics may not have resulted in replicating the outcome of the research. I know I am not painting the most rosy picture here, it is at the same time important to realise that for one gamification is quite complex and applied to another complex field such as learning a lot of factors may come into play. I would love to have the opportunity to be part of a proper research study around gamification of learning to test what works best, across age, gender, socio-economic backgrounds, etc.

The overall result of the comparative study was not all doom in gloom, even though the majority of studies did not give conclusive evidence. The remaining studies were more positive than negative. The more research comes out and judging by the weekly requests from students around the world, it won’t be long hopefully before we can also address the inconclusive factor and have meaningful data to work with.

Distribution of behavioural studies by degree of evidence:

Positive 26%

Negative 10%

Inconclusive 64%

As most of my clients know, I don’t want to paint on overly rosy picture when presenting to them. I, maybe wrongly, tend to err on the side of caution. I know that we have achieved improvements on the original goals of our clients, typically in double figures. The average is probably 20%. If you think this is small, look at it this way, if as a result of 20% increase in retention, you sell more, your people can do more, then for a business the impact is incremental. Proving what exactly made things hit the sweet spot, I think is often also claiming to know everything about everyone and personally I don’t find that realistic. But that is just my opinion.

If you have solid research and great data to share, by all means let me know.

Average gamer in the UK: 43 year old female

Recent new published research by Nesta on the average gamer in the UK found that they are probably a highly educated female aged 43. I have to admit I was even slightly stunned with this number. Although most of my peer group does play. Games ranged from computer to video games played on consoles, desktops and mobile devices.

What is further interesting is that the study found that most gamers, were also active consumer of other cultural activities such as read, going to events, theatre, museums, etc. So the interest was much more varied than one medium. Financial background or wealth was not a statistically significant indicator to differentiate.

Playing video games has come of age as a leisure activity, and the generation that grew up on video games is now reaching mid-life. The study wanted to examine the socio-economic characteristics of games players and their wider cultural behaviours. The results, first of all, do not reveal any obviously negative associations with socio- economic variables. If anything, those who play games are typically better educated, no less wealthy and are greater consumers of culture. The presented correlations are
not proof of causal relationships according to the researchers; nonetheless, the results challenge the stereotypical perception that playing video games has harmful effects on the individual.

Neither do the results lend any support to the idea that playing video games has longer term detrimental impacts on an individual’s socio-economic circumstances. That is, in general, the study did not observe any socio-economic differences between those who played when growing up and those who did not. Interestingly, it found that those who played when growing up also participated in other forms of culture, in particular, they were more likely to read, paint, attend performing arts and visit heritage sites or libraries. Not very surprising to hear that there is a path-dependency in playing video games: those who played when growing up are also more likely to do so nowadays.

Although the study confirmed that video games playing is more popular among younger individuals, with the gamer’s average age of 43.2 years, the common perception that the medium is ‘immature’ is far from being supported. Interestingly, the average gamer is more likely to be female, however among those who play, females do so less often than males.

An important question was whether playing video games makes people more creative. Although the study wasn’t conclusive in that regard, they did observe, however, that those who play video games have a stronger tendency to engage in more active forms of cultural activity which is consistent at least with greater creativity. However, it remains an open question whether this finding is a reflection of a creativity- enhancing effect of playing video games, or is rather driven by self-selection of a certain type of person towards both activities – video games playing and active forms of cultural participation.

I find it fascinating to read studies like these and to hear the positive side of games. I made games as a child, read a lot of books and as a family we definitely did a lot of cultural activities. I find games help me to re-foucs my thoughts and sometimes they also help me be more creative. I still to this day read a lot and enjoy cultural adventures. So I guess I fit the profile 🙂

Men in STEM reject the gender bias findings

Feminine gamification viewpoint: Men in STEM reject gender bias research

Recently I was quoted to be feminist as a result of pointing out that in behavioural change and gamification design, we need to be mindful of the subtle differences between feminine and masculine decision and behavioural styles. What I have always said in each presentation on the topic that both men and women have the ability to behave more masculine or feminine in any given situation. Still what some people take away is that women need to be given empathy and that’s it. In my view that’s missing the point a bit.

One of the reasons I pointed out the differences and started this blog theme, is because I kept hearing that there is no difference between men and women for the purpose of gamification design. Intuitively this felt wrong in my view and I went looking for examples as well as research to find out if this was true or untrue. Study upon study I found actually confirmed the differences, mainly because of how we are brought up in society and very culturally influenced gender roles. In designing for behaviour change, in my mind it makes sense to be mindful of your audience and their way of working in order to have your change program accepted first of all and then utilised for the purpose of what it was designed for. I would also encourage taking into account age, racial difference, company and local culture, etc.

Then yesterday I came across an article on a research study that indicated that men in STEM in fact reject the findings of gender bias in STEM much more than other men in non-STEM roles. It was a university study carried out by Princeton researchers. They presented finding of a study to 200 people in a non-university setting and then to university professors, what they found that the male professors in STEM faculties disbelieved the research to a much larger extent than any of the other faculties. When they took out the STEM faculty results, no difference in acceptance was found. Women didn’t give it more or less value than their counterparts in other departments.

When I spoke on the topic at the gamification world congress a lot of my male friends in gamification admitted being a bit rattled by my presentation all the way to completely challenging it. Equally interesting was the response from app and product developers with mainly male teams, who admitted struggling to engage the female audience they were designing for. In cultures where male and female divides are higher, interestingly the suggestion of having women hold workshops with women or finding out one-to-one what you need to know did hit a positive vibe. In my view good design starts with good base understanding of your audience and that implies being mindful of differences however subtle they may be.

I may never convince some designers, but I seriously hope that in gamification design, we don’t fall in the same bracket as the research with male STEM professors.

How do you account for differences in gender, age, race, culture in your gamification design?

Pink brain, blue brain

Feminine gamification viewpoint: Pink brain, blue brain

In this blog theme I often discuss the differences between male and female motivations, because in gamification we are often asked to change behaviour and it doesn’t take a rocket scientist to realise how men and women react is different. Science and neuroscience is divided and research studies can come up with conflicting views. Lisa Eliot did further research in the differences and similarities of the brain, she found she couldn’t explain differences based on the hardware, yet behaviours were still different, which she is assigning to experience.

I found her talk very refreshing and like the fact she also looked at refuting some the common beliefs about the differences. Have a look for yourself at Lisa Elliot talking about Pink brain, blue brain.

She realised in her book research that hormones and synaptic connections are probably more key in driving difference between the sexes. When we are born the differences are very slight, but then thanks to societal conditioning it changes over time. She assigns most of the differences to nurturing certain behaviours, which happens as we grow up through positive and negative reinforcement in our environment.

She gave the example of spatial awareness, where there is a slight difference at birth with boys having a little more ability, but the types of games we play as children such as lego, building and design type games, Tetris etc can diminish this difference.

In terms of gamification design, I would recommend to be mindful of the differences, because a lot of them are learned behaviours because of society, we also have an opportunity to debunk some of the common expected behaviour patterns and develop new ones. Again as with everything in our design, the ethical perspective is important.

How do you handle differences in behaviour in your gamification design?

Feminine gamification viewpoint: Awkward Moment

Feminine gamification viewpoint: Awkward Moment

Researchers at Darthmouth  in the USA, used games and embedded game design techniques to address gender bias in a recent study. They designed two card games one is called Awkward Moment, which challenges players to react to funny, embarrassing and stressful situations; and Buffalo: the Name Dropping Game, which asks players to name real or fictional examples who fit the game’s unexpected combination of attributes.

Neither game will mention the purpose or intent of trying to change gender bias at all, it is just embedded in the game design through intermixing, sometimes suggestions are ‘on topic’, sometimes completely ‘off topic’. They used this technique to counteract the inherent resistance most people have when the intent is shared, but rather they chose to focus on enjoyment and growth in their game. When designing games and gamification with the purpose of being persuasive then this is something to take into serious consideration.

Awkward Moment mixes in situations that involve a bias against girls in STEM or a lack of gender equity as well as a range of cards that do not address these situations at all. Researchers Kaufman and Flanagan found that after playing Awkward Moment, young players had more inspiring and assertive responses to multiple forms of social bias in relation to women and science. To verify if the game had changed perceptions, students were asked after playing 1 round of the game to match pictures of men and women with various job roles: 58% of participants placed a woman in the scientist role, which was 33% more than a control group who didn’t play the game and 40% more than a group who played a neutral game that had no references to gender bias.

The study confirms my thinking when it comes to gamification design that although the techniques don’t need to be obvious nor shared,  your gamification design is better when it addresses some of the real things that live in an organisation even if they are no longer desired such as gender bias for example. Most companies want to be seen as inclusive of all groups, yet reality is often different because of long held biases, so research like this will help us as game and gamification designer to create more transformational experiences for clients.

How will you embed gamification techniques to decrease gender bias?

Feminine gamification viewpoint: Fitocracy research

Feminine  gamification viewpoint: Fitocracy research

Survey based research carried out by J. Koivisto and J. Hamari confirmed and highlighted some interesting findings regarding the use of the gamified fitness app Fitocracy and the impacts on gender. From their literature review it seems a lot of the findings are confirming what is already known from other studies.

Namely women tend to be more engaged when a social setting is provided, equally they gain more from the recognition of others on social networks in comparison to males. One underlying requirement that is important for genders is that the community is also supportive of the goal one is trying to achieve by using the gamified system, in this case fitness. It confirms what we have been discussing in previous posts that women tend to rely on their network of friends to pick them up as well as feel recognised and we are not afraid to do this on a social networking platform. Men may still enjoy social elements, but they gain less positive benefits from it and are more interested in the app doing what it is supposed to do.

Ease of use was another key factor for women, but not as important for men. Again I would suggest for all app and systems designers get a group of women to test your system for ease of use, they will give you more feedback even if it may mean more work. An interesting observation was that when women do engage with a game they tend to play it for longer especially if it is social and immersive.

Instant feedback loops were important to both males and females. When you are trying to engage an older demographic also consider stimulating both mental and physical activity, social connectedness and support and feedback that endorses their levels of self-efficacy which builds the confidence to use a system more.

What are you using in your gamification design to attract and engage the ladies?

Feminine Gamification Viewpoint: Gender design preferences

Feminine Gamification Viewpoint: Gender design preferences

When women and men design they will naturally go for a number of different elements:

Type of objectsSelf-propelling objects (ships, cars, planes, rockets)Static objects (plants, flowers, furniture, landscape)
MessagePrinted wordNot printed word
TypographyStandard typographyDecorated typography
Main actorMalesFemales
Type of imageCaricaturePhotograph
Human formProfileFrontal and smiling
BackgroundSkyscrapers and towersHouses, windows, and roomsn
ThemesViolence / TechnologyDaily life

The reason for this can be found in neuro-scientific research, where test of being shown ads resulted in subtle differences in perception between the genders. Men recognise fewer claims and prefer simple advertisements with no more than one or two features. This is because they have item-specific processing skills, which lead them to extract the core of the ad and neglect the rest.

Women, on the the other hand, are comprehensive processors who attempt to assimilate all available information before rendering judgment. In this way, they prefer complex ads containing rich and detailed information on multiple features. Women also have a greater affinity for purely verbal information, whereas men benefit from visual reinforcement.

A greater proportion of men than women are colourblind, which means they only have two cones for perceiving pigments in their eyes as opposed to three which is the norm. A larger proportion of women have four cones, which allows them to perceive hundreds of millions of color pigments, a hundred times more than most men can perceive.

Biology can also explain preferences for certain colours over others. Men and women have different cortical responses to the stimulation of blue and red light wavelengths: women are more sensitive than men to the long-wave spectrum of light that detects red. This means that women were found to have a significant memory advantage for the purple-pink range of colours. Interesting that my preference for pink and purple may well find it’s origins in my brain’s ability to see colours 😉

The most robust and persistent biological difference between sexes, however, lies in visio-spatial abilities. A meta-analysis of 300 studies on visio-spatial differences concluded that sex differences in spatial abilities favoring males are highly significant; on average, males outperform females in mental rotation and spatial perception.

So the fact that men like more high speed moving objects in games can well be linked to their better ability to deal with them and that women may well be attracted to less movement and more static type games such as puzzles and also the aseptically colourful varieties such as Candy Crush where the colours are indeed very appealing and often pink.

So from a gamification design perspective it is important to keep these differences in mind when we are designing tools that appeal to both genders.

I am curious: What is your favourite game and why? What do you find most important in a game is it movement or colour?