The core research questions we sought to answer when assessing both games were (1) how effective have they been in engaging children (2) which elements/stimuli of the game were most and least engaging and why? (3) to what extent are game interfaces and mechanics simple enough for children to use with minimal adult supervision?

Usability Study Findings and Areas for Improvement

The usability study determined that both games were age appropriate, highly engaging for children, and had the right pacing for mechanics difficulty. Youth maintained high levels of engagement playing both games and the learning content within — and reported they would choose to continue to play the games at home, even though over 50% had other mobile gaming options. Youth also suggested that what would make the games better would be to expand their offerings (add more levels or different game modes). The Evaluation Executive Overview can be found here, the Antura and the Letters report can be found here, and the Feed the Monster report can be found here.

In addition to the initial positive findings, the usability study uncovered several areas for improvement. The key design decisions which had an impact on the usability of the games rested primarily on the (1) narratives (2) mechanics and (3) features for personalization. These include:

• Conducting significant work on the narratives of both games to successfully convey the overall storyline and create a greater investment in the game environment

• Designing additional instructions or communication so that children fully understand how the rewards systems of the games work

• Improving the user experience of the mechanics of the games so that they don’t have a negative impact on the learning goals

• Providing children with sufficient feedback such as why their wrong answers are incorrect

A specific example related to the areas of improvement above was an observed difficulty for 50% of children with a drag mechanic in the Feed the Monster game. Although there were low levels of frustration, extra efforts required to overcome challenges of mechanics can have a negative impact on learning goals by diverting unnecessary attention to mastering a mechanics issue. In the case of both games (and all educational games for that matter), the initial user experience should be as simple as possible to create a low threshold for engaging learning.

Both games succeeded in providing low-hanging fruit for acquiring early-grade Arabic literacy, but this design-based research approach produced valuable insights into how both games can be enhanced for better outcomes. These findings will inform future iterations of both games to create a better user experience, immersive and clear narratives and increase the adaptability of the games with more scaffolding.

According to Liv Nordhaug, administrator of the competition, the player progression of both games has been improved in the current iteration. “We saw that the average improvements in oral reading fluency masked wide variations. Some children were really learning a lot, whereas others actually became confused because the learning progression was too steep. We have taken this very seriously in the follow-up with the game developers. We want to make sure that the games remain broadly engaging and appealing to most children, but that they also address diverse learning patterns and do not leave learners behind. This is complex to implement within our cost and time constraints, but I am very optimistic about the updated games. Both games have made important changes to make sure children understand the gameplay and rules and goals, including deleting too complex mini-games and introducing new tutorial features. Both games also enable the player to easily repeat previous tasks, and there are also some more interactive responses to diverse learning needs; like the opportunity to use a freeze icon to make the gameplay sloweror a match icon to remove distractors. Finally, both of the games now have great reward systems that are easy to personalize, which can motivate children to stay engaged over time to achieve their own goals.”  

More findings from the report will be explored on “The Power of Digital Gaming for Literacy” webinar on Tuesday, March 20th at 10AM (EDT).

Scaling Feed the Monster

The UNICEF Innovation Fund has made further investments in the Feed the Monster game through the development of an open source localization kit in partnership with Curious Learning. The localization kit will be developed through piloting a localization process of “Feed the Monster”, and this experience will be leveraged as a case study to create a practical action kit for content and software developers which outlines key design decisions that result in failures and successes of the localization process. The kit will support localisation efforts by start-ups and UNICEF Country Offices developing open source content and software, allowing products to reach greater scale more efficiently. It will be published and disseminated in June 2018.