Moral cheating, part 2: GUI automation on Khan Academy

GUI automa­tion means writ­ing a pro­gram that pre­tends to be a human using a mouse and key­board. Assum­ing your tar­get doesn’t look for bots, you have enough to exploit the sys­tem.

In seri­ous games, bots must become indis­tin­guish­able from legit­i­mate play­ers. Naïve bots give them­selves away imme­di­ate­ly; they click the exact same pix­el on a frame to per­form the same action, they chain actions togeth­er with remark­ably per­sis­tent tim­ing, and they don’t respond when addressed in con­ver­sa­tion. Humans fid­get in antic­i­pa­tion of their next task, wig­gle the game cam­era, click to inspect enti­ties, chat with friends, and self-cor­rect. The best cheat­ing bot needs to exhib­it these flaws to the degree and style of the human they replace, which involves train­ing. This of course makes the bots woe­ful­ly slow to con­fig­ure and run, but if you are aim­ing to get past bot-detec­tion, these are nec­es­sary evils.

We will start with a sim­ple bot tar­get­ing Khan Acad­e­my, the lat­ter being an edu­ca­tion ser­vice that rewards users with ener­gy points and badges akin to gold stars. Here I write a bot that could earn all ener­gy points pos­si­ble for videos; This breaks the com­pul­sion loop for users retained by KA rewards.

Of course I am not going to vio­late KA’s ToS. I want to pay spe­cial atten­tion to sec­tion 8.14, mean­ing I will not run my bot to its extreme con­clu­sion and risk harm­ing “any user’s enjoy­ment of [the Web­site],” what­ev­er that means to legal demons. I think call­ing the badges/points “valu­able” is debat­able, but I won’t risk cheap­en­ing the val­ue oth­er users find in earn­ing badges and ener­gy points. We will only emu­late a human user’s activ­i­ty that would put no more load on their servers than nor­mal, and I will per­form a small demo for edu­ca­tion­al pur­pos­es. If you try using this bot or some­thing like it for your­self, then I am not respon­si­ble for your deci­sions.

The job entails open­ing a video in a brows­er, watch­ing it, and then mov­ing to the next one. KA’s embed­ded play­er moves auto­mat­i­cal­ly to the next relat­ed bit of con­tent, but it might redi­rect the user to some­thing oth­er than a video. There­fore we need to con­trol the nav­i­ga­tion our­selves.

GUI automa­tion requires many assump­tions to work, so just reuse the con­cepts if KA changes and breaks the code here.

If you want to max out your account, you could use curl and pup to scrape togeth­er a text file with every video link. For CYA rea­sons it would not be a good idea to dis­trib­ute the playlist should you build one. I am only show­ing you what you could do with a playlist if you had it for the sake of this dis­cus­sion on com­pul­sion loops, and you can test this just by copy­ing some links from your address bar man­u­al­ly in a text file.

Let’s look at a Python con­troller for Sikuli imple­ment­ing the fol­low­ing behav­ior giv­en a playlist text file:

1. Find the next video to play.
2. Open the video link, assum­ing a KA ses­sion is active.
3. Start Sikuli, block­ing until it ends.
4. Go to step 1.
5. (On ter­mi­na­tion) Mark the line of the cur­rent video being watched.

Sikuli allowed me to pro­gram the bot using images, so I didn’t have to fuss with hard-cod­ed coor­di­nates. Since KA videos cur­rent­ly auto­play on load, the Sikuli script only needs to wait until the video ends and then close the brows­er.

You can study the source code for the above bot on GitHub.

What was the point of this GUI automation talk?

We just cov­ered an exam­ple of GUI automa­tion to break a sim­ple com­pul­sion loop in a real-world sys­tem. The ques­tion I want peo­ple to ask is how many users would KA have if every­one used this bot? If the bot killed reten­tion, then sites like KA can’t get by on con­tent alone. I think for marketing’s sake, that’s impor­tant to know. I hypoth­e­size is that few would return to KA if their rewards were use­less. If so, do we have free will to improve our­selves with­out a com­pul­sion loop to pro­gram us?

The bot shown here rais­es every red flag on cheat-detec­tion soft­ware. Bots cheat­ing com­pet­i­tive games must bet­ter emu­late human behav­ior.

But bet­ter bots to cheat at games with detec­tion soft­ware is not the direc­tion we are going to go, because as I stat­ed in part 1, the moral good from cheat­ing ends in com­pet­i­tive envi­ron­ments. Instead, we will next look at break­ing com­pul­sion loops by replac­ing them with bet­ter ones. Those who wish to reassess their pri­or­i­ties and reclaim con­trol of their health and atten­tion would rather sub­sti­tute, for exam­ple, gam­i­fied exer­cise in place of their World of War­craft account.