A couple of weeks back my wife found details of a hackathon event, organized with a local school who work with children with Autism. Luckily I was able to take part – despite the language barrier (I’m slowly learning…) – and it was great fun and a very rewarding experience. In the end my team finished runner-up, which we were very proud of – we had the smallest team, of just 2 of us, and we were the only team without either an artist or a games developer. If you want to check it out, the event page is here, and all of the games are now available for download. A big thanks from me to the organizers for making the event happen, and for translating for me, and to my team-mate Rafal!

Friday evening

The first evening started with a presentation on Autism, explaining some of the challenges, and how games can help therapists in their work. My prior knowledge of Autism is based mainly on watching Rainman, but it is of course a very complex condition that we still know relatively little about. As I understand it, the main difference between a “normal” person and one who has Autism is in their perception of their surroundings. A “normal” person is able to generalize, and able to filter information, where a person with Autism may be over-stimulated by some things and under-stimulated by others. Researchers have found that interacting with screens can be less stressful for children with autism, and that working with a therapist and a screen is more enjoyable with better results than working with only a therapist. The aim of the hackathon was to develop concepts for games to target activities that are fun for the child, and that can help in their learning. Typical examples of this might be things that a child with autism might struggle with; identifying things, identifying emotions, generalizing, finding patterns, special awareness, identifying dangerous situations, and so on. There are apparently relatively few games available that are suitable – which seems a real shame. The twist of course is that conventional game mechanics may not be appropriate for a child with Autism, and may even cause distress, so it’s important to bear this in mind. We had access to a team of therapists at all times, so that we could ask them for ideas and feedback, and I found it really interesting to learn about this different perspective on things I take for granted.

Our first thought in brainstorming ideas for a game was that this is a really complex thing, with lots of subtleties, especially as the support that every child needs can be very different. We wanted from the outset to design a game in such a way that the therapist was in control; we wanted to make everything as configurable as possible, so that the therapist could tailor the experience to the individual needs of the child they were playing with. Some of the examples of games that therapists play to help children include “find the odd one out”, “what is this object”, “which face is smiling”, and similar. We decided that we could make randomly generate these games based on configurable data provided by a therapist.

A challenge in events like this is that you don’t know who’ll be on your team, what skills will be available to you, and so on – in our case, we quickly established that we were both engineer types – my team-mate’s day job is building high-performance phone networks, and writing low-level drivers against chips for communications devices, and I specialize in services architectures. We would need to bear this in mind when deciding what to try to make; clearly graphics-rich 3D games simply wouldn’t happen! We’d also need to find a common ground in terms of tech stack and languages; we opted for Electron.js, essentially a browser shell that can be used to make cross-platform desktop applications with HTML and JavaScript, because it provided a relatively accessible thing to work with that would meet the needs of our ideas.

Of course the other challenge of a hackathon event is time – there’s not a lot of it to spare! The approach we took was to ruthlessly prioritize, work in really short cycles. Our strategy was to think about the 3 things that most needed our attention, and choose the one of these we could meaningfully work on (together) next, with a simple but concrete objective. We’d then focus on that one thing, until the objective was met (and no further). If we felt we’d got stuck, and made no progress for say 20 minutes, we’d park the item and go back to the list. At the end of the task, we’d re-consider what the next 3 things would be – and this would often change based on what we’d uncovered while working on the previous thing – and go again. This way we’d avoid over-polishing one thing at the expense of other areas, and we’d be able to constantly adjust our focus as needed.

Still on the first evening, we had an idea of what we’d do – we’d take an input set of items, and randomly generate a “find the odd one out” game. At this stage, we planned to make the game web-based, so we started by spiking out a really simple API to serve game data, that we could hit from a browser and just see the raw results. To keep things as lean as possible, everything would be hard-coded or stubbed – no databases, files, etc, just an in-memory array of data. One tiny iteration at a time we added things like a game id, so that the server could support more than one player(!), and after an hour or so we had a basic but working implementation that was serving JSON data to the browser. Time to call it a night, and sleep on the ideas to see where to go next! The final thing we did for the evening – while it was fresh in our minds – was to make a “next 3 things” list for the morning; the clear number 1 was to make a simple user interface.

Saturday

The next morning we started nice and early, and kicked things off by starting on the UI. This was something neither of us specialized in, so there was a definite incentive to keep things lean and not over-complicate! We started off with sketches of the basic details on paper, before mocking this up in HTML. We opted to use KnockoutJS, a nice lightweight MVVM library with very little ceremony but more than enough power for our needs. Before long we had a basic UI that rendered the JSON from the server, and posted back the chosen option – a minimalist but playable game.

Next up on our list was to think about a reward when the child guessed right, and what to do next, as well as feedback for incorrect guesses. Once again, our thought was that we should yield control as much as possible to the therapist – they might want to start a new game straight away, but they might want to take a couple of minutes to dwell on the current game, discussing something before moving on. We opted for a nice colorful “Well done” message, with a sound effect, and a “Play Again” button that the therapist could tap when it suited them. This was also a good time to start improving the aesthetics; not something either of us would ever claim to be experts in! We googled and fumbled with CSS and layouts, eventually getting something “good enough” (for this iteration). If we could have chosen 1 person to draft in as an extra to our team, it would definitely be someone with a talent for design!

At this stage one of our main impediments was the client/server structure; we’d prototyped the server in .NET, and were having trouble porting it to run on Rafal’s Linux machine – something we definitely didn’t have spare time to figure out. One option of course was to simply pair and work on the Windows machine, which is what we’d stuck to so far. It was time to think about a next thing to work on though, and it’d be nice to do something about the hard-coded data. We brainstormed this one together, reluctant to build a full-fat admin tool, and Rafal’s excellent idea was that we could simply use folders/files as categories/items. This was a massive win for 2 reasons – the first is that it saved us building admin tools; Windows Explorer was all a therapist would need; the second is that folders and files are really portable, such that a therapist could simply have a USB stick for each child in the class, and could share materials with colleagues really easily as simple email attachments or via their sharing app of choice. We had been thinking of building a database, and accounts, and data entry forms… which would have taken us a lot more effort, and given a much less flexible result. It seemed like we were onto something!

To use files/folders, we needed to change course from the client/server model we’d spiked so far. Together we ported our simple server to JavaScript, and moved our client into an Electron.js shell. This took a bit of googling; things like modules didn’t work the same way, and we had a little spell of trying to find answers. A really nice thing though is that staying in JavaScript and Electron meant we were fully cross-platform straight away, and that we could iterate with a simple refresh – no compilation or reloading. The other big win is the dev tools available in Electron, which gave us debugging as we worked.

Before long we had a fully working ported version of the game running in Electron.js. We went back to our “top 3 things”, and decided to add more game modes next. There were a bunch of ideas we could do, of which quite a few used very similar concepts with small variations, so these were what we chose to work on. From “odd one out”, we added “find the item”, “find the category”, and “what’s in the picture” with quite little effort. We were still struggling with modules in Electron, so resorted to copy/paste, but it worked well enough for now! By the end of the first full day we felt really good – we had a working game, with configurable data, and 4 game modes, with a simple menu to choose game. Again, before clocking off for the night we prepared our “next 3” list for the morning. At this stage, styling was definitely our weak point.

Sunday

When I got to the school in the morning some of the teams had done an all-nighter! They were a young team, all students, who did quite a few hackathons together. It was interesting to chat with them and see what they were working on; every other team had built a game in Unity. Seeing their beautifully rendered artwork made us painfully aware of our biggest weakness! We decided to divide and conquer for a bit; Rafal worked on some animations and graphics, while I focused on layout and tidying up. After a couple of hours things were looking… better… and we felt like moving on to other areas for a break!

One thing we wanted to do was to add some more types of interaction with the game; one of the things that we discovered was actually surprisingly easy to do was text-to-speech. We started with having the game saying “well done” (which actually sounded a bit creepy, at least with the default voice!), which we decided against; but for things like incorrect guesses this was great. We put in an array of random messages, from which one would be selected and read out. The messages were all positive and encouraging rather than harsh or corrective, and we figured this would be the kind of thing we could make configurable if we had spare time. We also made the game read the question when clicked on.

We’d been doing little demos to the therapists throughout and one of the pieces of feedback was that some children respond differently to bright colors – and so as a next feature we decided to add a choice of color scheme. We had a “bright” theme, the default, but would add a more subdued theme that the therapist could choose. This actually turned out to be much harder; I could get Knockout to selectively render or hide the tag to import the other CSS theme, but the browser would load it before applying the binding… the solution I eventually came up with was to wait until the bindings were applied, and append or delete the tag the old fashioned way, which did the job. Definitely not an elegant solution, but we had our feature!

We got to a stage around lunch time, with judging and presentations at 4pm, where we started to weigh up what to do next. We decided by now that actually the most valuable thing we could work on was not the product itself, but on preparing a compelling demo. We had made an engine that would show whatever content was provided, but with no content it would look a bit rubbish! We made a few sample data sets from images of things like animals, faces, vehicles and plants, to show some of the things. While we were doing this, Rafal had another great idea (there’s a theme…). The idea this time was that the code we had, with no modifications, would show whatever label was presented, which was normally just the name of the item… what we could do with very little effort would be to give the item a longer tag, which became a fact about the picture – which gave therapists another way to prepare games. This was another simple adjustment; we just changed the layout to fit longer pieces of text, and everything else worked the same. Five game modes!

In the end, we were the first to be ready to present, thanks to our constant prioritization and iteration. The therapists were really keen on our game because of the range of options and configurability it gave, and the main minus point against us was that it wasn’t an “out of the box” experience for a child, unlike the others, without some configuration first. I think what we would probably do to get round that would be to have some sample games available, and a slicker way of loading them – we were simply showing the standard file picker, which isn’t the most visually appealing for a children’s game!

All the teams demonstrated their games – which I was really impressed by! There was a lot of variety, and I was surprised at how far people had got in the time. I’ll have to check Unity out some time! In the final results we were awarded 2^nd, which we were very pleased with. We didn’t win – mostly because the game’s text and speech were all in English and needed to be in Polish – but we had an idea that we felt could go further…

What next?

About a month before the event, we were walking our puppy in the park when we met a young boy with autism and his father. The boy wanted to play with our dog, and she didn’t seem to mind, so they spent a couple of hours playing. It was through the boy’s father that we’d found out about the event. He’s got a lot of energy, and is devoted to finding as many new experiences as possible for his young son. When he heard we’d joined in, he was really interested!

One of the limitations we found in Electron.js is that unlike in a regular browser, the Electron.js text-to-speech only reads in the default voice – English. I’m currently in the process of porting the game into Xamarin, a .NET framework that gives access to multi-language while remaining cross-platform. The game will now work with any installable language pack on Windows and Android! I’ve also expanded out the other text in the game to come from files rather than hard-coded, again to enable multi-language. There’s a bit more work to do, and then we’ll get into play testing, with a community of children.

Beyond that, I will release the game into app stores for Android, Windows and possibly iOS. It will of course be free of charge, because it’s aimed at people who really don’t need any more barriers in life, and I’ll release the code as open source. Watch this space…