Mind map freeze-dried web searches with Eyedact

I seri­ous­ly love mindmaps. If you’ve nev­er used them before, a mind map dia­grams a knowl­edge domain. They’re awe­some because you can make one in no time at all, and if they are well-made you can infer what you have to under­stand about any top­ic.

Mind maps are easy to write on paper, and there are many apps out there for mak­ing them fan­cy for pre­sen­ta­tions. Yet, if you must use mindmap­ping soft­ware for per­son­al use, you only need a text edi­tor and Google. You can write a mindmap as an indent­ed text file. Since I typ­i­cal­ly draw mind maps by hand, writ­ing a text file is the most I would do when mak­ing one on a com­put­er.

American Football
    Rules
        Downs
        Penalties
            Leverage
            Clipping
        Positions
            Linebacker
            Defense
            Offense
            Quarterback
    Teams
        Patriots
        Giants
        Cowboys
        Packers

This exam­ple seems like it is offen­sive­ly incom­plete, but it works because we still see a knowl­edge domain’s shape. For most pur­pos­es this is enough. You prob­a­bly don’t need mind map apps that allow detailed notes or file attach­ments due to their pro­pri­etary file for­mats and lock-in. Mind maps are not for hold­ing small details any­way; they are for note-tak­ing. Fur­ther­more, we already can search the web any­thing, so mind maps should facil­i­tate access to exist­ing knowl­edge.

Using a mind map to freeze-dry web searches

Eye­dact helps mind map­pers search the web. Much like with Aloe, it’s com­plete­ly non-inva­sive. Unin­stall it, and you can still do what it does, just a lit­tle slow­er.

Obvi­ous­ly you could skip Eye­dact and Google what you want to know direct­ly, but there’s rea­son to think about doing bet­ter: Those new to a sub­ject might deal with vague tech­ni­cal terms. How many def­i­n­i­tions of “com­po­nent”, “remove” and “link” do you think are out there? What if you need to search for that word and you don’t remem­ber where it was rel­e­vant to you?  Eye­dact lets you write search­es that are too vague for Google and still get spe­cif­ic results using pre­served con­text. Hence “freeze-dried” web search­es.

Mind maps shape upcom­ing research for note-tak­ers, and pro­vide implic­it doc­u­men­ta­tion for main­tain­ers. Let’s talk more about that bold­ed point. If I’m writ­ing a frame­work with a big stack, I could write a para­graph explain­ing what we use and why, or I can just splat con­tent in front of peo­ple and tell them to read. Assume I have an NPM pack­age with the fol­low­ing con­tent in what-you-should-know.

GraphQL
    best practices
    caching
    tutorial
Docker
    best practices
    build
    container
    image
    tutorial
Webpack
    plugins
    tutorial
Restify
    caching
    plugins
    tutorial
Redux
    actions
    async
    tutorial
    best practices
React
    component
    async
    tutorial
    best practices

In package.json, I can do some­thing like this:

{
    ...
    "scripts": {
        "doc": "eyedact what-you-should-know,"
        "doc:beginner": "npm run doc tutorial -al",
        "doc:journeyman": "npm run doc practices -al",
        "doc:advanced": "npm run doc caching async plugins -a",
        "doc:local": "less README.md",
    }
}

in this hypo­thet­i­cal, if you fin­ish review­ing the con­tent all the way down the list, you are ready to work on the project.

Recap

We briefly cov­ered the text rep­re­sen­ta­tion a of mind map and how you can use one to give con­text to many web search­es. Doing this guides peo­ple in search­ing with­in a new knowl­edge domain, and auto­mat­i­cal­ly gives peo­ple direc­tion in learn­ing lots of rel­e­vant mate­r­i­al from a sin­gle com­mand.

Moral cheating, part 2: GUI automation on Khan Academy

GUI automa­tion means writ­ing a pro­gram that pre­tends to be a human using a mouse and key­board. Assum­ing your tar­get doesn’t look for bots, you have enough to exploit the sys­tem.

In seri­ous games, bots must become indis­tin­guish­able from legit­i­mate play­ers. Naïve bots give them­selves away imme­di­ate­ly; they click the exact same pix­el on a frame to per­form the same action, they chain actions togeth­er with remark­ably per­sis­tent tim­ing, and they don’t respond when addressed in con­ver­sa­tion. Humans fid­get in antic­i­pa­tion of their next task, wig­gle the game cam­era, click to inspect enti­ties, chat with friends, and self-cor­rect. The best cheat­ing bot needs to exhib­it these flaws to the degree and style of the human they replace, which involves train­ing. This of course makes the bots woe­ful­ly slow to con­fig­ure and run, but if you are aim­ing to get past bot-detec­tion, these are nec­es­sary evils.

We will start with a sim­ple bot tar­get­ing Khan Acad­e­my, the lat­ter being an edu­ca­tion ser­vice that rewards users with ener­gy points and badges akin to gold stars. Here I write a bot that could earn all ener­gy points pos­si­ble for videos; This breaks the com­pul­sion loop for users retained by KA rewards.

Of course I am not going to vio­late KA’s ToS. I want to pay spe­cial atten­tion to sec­tion 8.14, mean­ing I will not run my bot to its extreme con­clu­sion and risk harm­ing “any user’s enjoy­ment of [the Web­site],” what­ev­er that means to legal demons. I think call­ing the badges/points “valu­able” is debat­able, but I won’t risk cheap­en­ing the val­ue oth­er users find in earn­ing badges and ener­gy points. We will only emu­late a human user’s activ­i­ty that would put no more load on their servers than nor­mal, and I will per­form a small demo for edu­ca­tion­al pur­pos­es. If you try using this bot or some­thing like it for your­self, then I am not respon­si­ble for your deci­sions.

The job entails open­ing a video in a brows­er, watch­ing it, and then mov­ing to the next one. KA’s embed­ded play­er moves auto­mat­i­cal­ly to the next relat­ed bit of con­tent, but it might redi­rect the user to some­thing oth­er than a video. There­fore we need to con­trol the nav­i­ga­tion our­selves.

GUI automa­tion requires many assump­tions to work, so just reuse the con­cepts if KA changes and breaks the code here.

If you want to max out your account, you could use curl and pup to scrape togeth­er a text file with every video link. For CYA rea­sons it would not be a good idea to dis­trib­ute the playlist should you build one. I am only show­ing you what you could do with a playlist if you had it for the sake of this dis­cus­sion on com­pul­sion loops, and you can test this just by copy­ing some links from your address bar man­u­al­ly in a text file.

Let’s look at a Python con­troller for Sikuli imple­ment­ing the fol­low­ing behav­ior giv­en a playlist text file:

  1. Find the next video to play.
  2. Open the video link, assum­ing a KA ses­sion is active.
  3. Start Sikuli, block­ing until it ends.
  4. Go to step 1.
  5. (On ter­mi­na­tion) Mark the line of the cur­rent video being watched.

Sikuli allowed me to pro­gram the bot using images, so I didn’t have to fuss with hard-cod­ed coor­di­nates. Since KA videos cur­rent­ly auto­play on load, the Sikuli script only needs to wait until the video ends and then close the brows­er.

You can study the source code for the above bot on GitHub.

What was the point of this GUI automation talk?

We just cov­ered an exam­ple of GUI automa­tion to break a sim­ple com­pul­sion loop in a real-world sys­tem. The ques­tion I want peo­ple to ask is how many users would KA have if every­one used this bot? If the bot killed reten­tion, then sites like KA can’t get by on con­tent alone. I think for marketing’s sake, that’s impor­tant to know. I hypoth­e­size is that few would return to KA if their rewards were use­less. If so, do we have free will to improve our­selves with­out a com­pul­sion loop to pro­gram us?

The bot shown here rais­es every red flag on cheat-detec­tion soft­ware. Bots cheat­ing com­pet­i­tive games must bet­ter emu­late human behav­ior.

But bet­ter bots to cheat at games with detec­tion soft­ware is not the direc­tion we are going to go, because as I stat­ed in part 1, the moral good from cheat­ing ends in com­pet­i­tive envi­ron­ments. Instead, we will next look at break­ing com­pul­sion loops by replac­ing them with bet­ter ones. Those who wish to reassess their pri­or­i­ties and reclaim con­trol of their health and atten­tion would rather sub­sti­tute, for exam­ple, gam­i­fied exer­cise in place of their World of War­craft account.

Terminal recording for GIF screencasts using toughtty

Ter­mi­nal record­ing soft­ware isn’t too friend­ly to con­tent pro­duc­ers, I’ve found. But there’s a Node.js imple­men­ta­tion of a ter­mi­nal recorder called ttys­tu­dio that showed promise even if it had not been touched for two years. I didn’t have to deal with weird sys­tem depen­den­cies and con­fig­u­ra­tion depend­ing on my dis­tro. I just ran npm i and went to town.

ttystudio was real­ly cool but I wasn’t real­ly agree­ing with the whole idea of record­ing the ter­mi­nal in real-time. I want­ed to have time to think about what I was typ­ing, and then have the out­put GIF not show any disk-space-hog­ging paus­es relat­ed to my doubt, inde­ci­sion or inter­mit­tent typ­ing accu­ra­cy.

After fork­ing and tak­ing sev­er­al shame­less­ly sub­jec­tive cre­ative lib­er­ties I end­ed up with this com­par­i­son. The first GIF is by ttystudio, and the sec­ond is by my ver­sion, toughtty*

ttystudio’s GIF weighs in at 84 frames and 3805 bytes, against toughtty’s 38 frames and 2980 bytes for the same ses­sion. Also, if I sat and spent 15 min­utes typ­ing that echo com­mand, the met­rics for toughtty’s GIF would not change, where­as ttystudio’s GIF would have grown beyond a rea­son­able size for use on the web. The only dif­fer­ence is that ttystudio cap­tures frames over time, where­as toughtty cap­tures frames in response to changes.

Throw­ing out record­ing in real-time does come with a caveat; You can’t add a pause in your GIF by just sit­ting back and wait­ing a moment. Want paus­es added auto­mat­i­cal­ly with­out real-time record­ing? Well, toughtty. I added a --padding option that lets the user add padding frames one, two or ten at a time to allow the view­er to review key con­tent.

I eat my own dog­food, so you can see GIFs pro­duced by toughtty on my Aloe arti­cle. If you want to record your ter­mi­nal ses­sions and save them as GIFs, check out the README in the toughtty source for instal­la­tion and usage instruc­tions.

*(I can’t believe that name wasn’t tak­en!)

Aloe enables rapid scaffolding for SCSS projects

aloe is a Sass-author­ing tool writ­ten in Python 2.7 that I use to write styles. Like oth­er style-cen­tric tools it is opin­ion­at­ed and does not do any­thing spe­cial. But, it is friend­ly to ter­mi­nal junkies. Run pip install aloe-pattern and fol­low along. I think you’ll like it.

Structuring projects with an aloe plant analogy

To enable a few fea­tures I apply a niche fla­vor of the Com­pos­ite pat­tern:

Any stylesheet direc­to­ry can be con­vert­ed to a new project at will, or vise-ver­sa.

This assumes fea­tures do not cut across direc­to­ries. Like with many oth­er pat­terns, the ben­e­fits to aloe’s pat­tern come from con­ve­nient­ly-avail­able con­text.

Sass, LESS or CSS on a laptop screen

Aloe works with a few goals in mind:

  1. Code shar­ing. One change may affect as many projects as I choose with­out involv­ing a remote resource.
  2. Rapid scal­able scaf­fold­ing. We won’t type @import  any­more, we will gen­er­ate con­tex­tu­al­ized rule­sets and we will have a cus­tom SMACSS project with BEM-ready wid­gets and themes for Christ­mas and Hal­loween ready yes­ter­day, with­out using tem­plates!
  3. Non-inva­sive­ness. I typ­i­cal­ly dis­like soft­ware that makes me feel like I need it. Aloe has no con­fig­u­ra­tion file and does not do any­thing weird to your code. You can use aloe and then imme­di­ate­ly unin­stall it with­out los­ing any of its ben­e­fits.

Tech Demo

Managing local dependencies

It’s eas­i­er to show than tell, so let’s start that SMACSS exam­ple I was talk­ing about. I’ll run aloe + base/normalize layout/{grid,sticky-footer} module/clock state/{form,notifications} theme/{xmas,halloween} to spin up a filesys­tem.

Using aloe to spin up a custom SMACSS project

This builds out of the box. The aloe plant anal­o­gy comes in with the __.scss par­tials. I call them rinds as part of the anal­o­gy, and all they do is import oth­er par­tials for me and pro­vide the frac­tal struc­ture for Sass com­pi­la­tions. The fol­low­ing rules apply:

  1. Every direc­to­ry con­tains exact­ly one rind named __.scss.
  2. Rinds con­tain only @import direc­tives, and are also the only files to con­tain @import direc­tives. They import par­tials inside their own direc­to­ries before oth­er rinds.
  3. Rinds may only import oth­er rinds in imme­di­ate sub­di­rec­to­ries. So ~/scss/__.scss may import ~/scss/foo/__.scss, but not ~/scss/bar/baz/_.scss.

These rules con­sol­i­date depen­den­cy man­age­ment to one file at a time, and you can use rinds as a sin­gle point to include or exclude entire swaths of func­tion­al­i­ty from a com­pi­la­tion, as we will see lat­er.

Get­ting back to our SMACSS exam­ple, if I need to reorder or remove any depen­den­cies, it’s just as easy. If I run aloe - state for exam­ple, state styles will be exclud­ed if I build any styles from the root of the project.

To elab­o­rate, let’s look at a fresh exam­ple. Here I run aloe + a b c d e. You will see that my root rind imports the depen­den­cies in the order I declare. If I clear out this file (>__.scss) and then link up the par­tials in reverse order (aloe + e d c b a), the effect is as expect­ed.

Writing partial contents

This seems like its all well and good, but what about set­ting the con­tents for these par­tials? One thing I do as a next step for scaf­fold­ing is use the aloe plot com­mand that lets me write a sum­ma­ry of rule­sets. If I run some­thing like aloe plot "clock(hourhand,minhand,sechand)", I get SCSS rule­sets. There’s a switch I throw to enable BEM-style writ­ing à la Stu­art Rob­son, so aloe plot -b "button(--big,__text(--upper,--lower))" gives me a BEM-y par­tial.

Don’t wor­ry about leav­ing these rule­sets emp­ty, because Sass will not include any emp­ty rule­sets in com­piled CSS.

On that note, let’s build some CSS. Here I use aloe build to out­put CSS using rinds. I can spec­i­fy mul­ti­ple tar­gets to build and con­cate­nate in order so I can deploy tar­get styles depend­ing on what I need. This nice­ly mesh­es with per-request style deploy­ment.

aloe fol­lows sym­links, so I can main­tain a link to any (S)CSS stylesheet to ben­e­fit from its fea­tures as if it were a par­tial.

In this post we cov­ered a non-inva­sive Sass author­ing tool used for rapid scaf­fold­ing and cus­tom deploy­ment of styles using a frac­tal orga­ni­za­tion pat­tern.

Moral cheating, part 1: Breaking the Compulsion Loop

Devel­op­ers of video games or gam­i­fied sys­tems use com­pul­sion loops to oblige con­tin­ued use of their prod­uct. A com­pul­sion loop is a iter­a­tive process that instills in you a new habit. Con­ve­nient­ly, your new habit nor­mal­ly makes mon­ey for some­one else.

Com­pul­sion loops cre­ate the illu­sion of val­ue, as South Park stat­ed bril­liant­ly in S18E6Freemi­um isn’t Free.

If you must exploit human psy­chol­o­gy to keep users, then your prod­uct lacks intrin­sic val­ue. If it weren’t for com­pul­sion loops, some­one should still find rea­son to stay with you.

But we live with com­pul­sion loops, and those prone to addic­tion arguably suf­fer the most. Addicts end up stuck in com­pul­sion loops on a down­ward spi­ral for the next dopamine hit. Take Runescape, a MMORPG by Jagex Ltd. designed to keep you play­ing regard­less of your mood or health. I don’t even need to men­tion World of War­craft. I’ve played both games and had fun, but not because the games were fun in them­selves. Because the games had no intrin­sic val­ue, I had to rely on friends and the antic­i­pa­tion com­pul­sion loops offer to tol­er­ate the grind.

This is not to say com­pul­sion loops are bad, just that they are pow­er­ful. We can only hope to get stuck to com­pul­sion loops that make us want to diet, exer­cise and keep the house clean. Unfor­tu­nate­ly busi­ness­es are excel­lent at pro­gram­ming your habits to suit them before you set your­self up for suc­cess. Neale Mar­tin, author of Habit: The 95% of Behav­ior Mar­keters Ignore, helps busi­ness­es shape their own cus­tomers in this way.

I am a staunch oppo­nent of bad com­pul­sion loops. This is not to be tak­en as an active and hyp­o­crit­i­cal push to make peo­ple behave dif­fer­ent­ly. Rather, I would hope to guide peo­ple out of com­pul­sion loops them­selves if they want out but are strug­gling to leave.

If you know­ing­ly choose to stay in a loop and main­tain the cost, more pow­er to you. My favorite self-pro­claimed alco­holic come­di­an Doug Stan­hope has more to say on that sub­ject.

Stan­hope says (like­ly in jest) that addic­tion doesn’t exist. Even if that were true, the pur­pose of this arti­cle series is to break an addict’s spi­ral and com­pul­sion loop so they can find more con­trol and enjoy­ment that ben­e­fits them more.

For con­ve­nience I will refer to both gam­i­fied sys­tems and video games as “activ­i­ties,” because we should talk about when it’s okay to cheat them both.

But why cheat? It’s not the only way to fight addic­tion. You have cold turkey and habit sub­sti­tu­tion, so why pick this more con­tro­ver­sial approach?

We need the rewards from cheat­ing to cheap­en the val­ue of our own work. If you were banned from a game for cheat­ing or sim­ply went cold turkey, you still have antic­i­pa­tion for reward, which is one of the dri­vers keep­ing you in a com­pul­sion loop.

In my expe­ri­ence hav­ing spent more time and mon­ey than I want to admit addict­ed to games, cheat­ing breaks a com­pul­sion loop so emphat­i­cal­ly that you feel no antic­i­pa­tion, pur­pose or dri­ve to con­tin­ue. When you run iddqd in DOOM or max out all of your stats after run­ning a bot, you start your last joyride. With noth­ing left to antic­i­pate, cheat­ing means ruin­ing the game for your­self. You won’t play it again for a long time after the nov­el­ty of god-teir game­play runs off.

The Compulsion Loop has its own darkness

This is why cheat­ing ruins games for the play­er on an indi­vid­ual lev­el. Now, if pulling the lever in the Skin­ner box for a food pel­let takes more away from you than the pel­let can ever give back, then cheat­ing is an under­rat­ed detox pro­gram that breaks a down­ward spi­ral. Cheat­ing becomes the right thing to do. You WANT to ruin the activ­i­ty for your­self before the activ­i­ty ruins you.

Even if you enjoy some addict­ing activ­i­ties and wish to keep doing them, you should learn how to cheat as a way to con­trol your habits and the influ­ence that com­pul­sion loops have on you. If some­thing changes in your life and the habit starts to take its toll, you need that out.

I can’t cure addic­tion, but when pos­si­ble I can teach addicts to cheat as a means to regain lost con­trol. Next in the series we will dis­cuss GUI automa­tion with Sikuli to cheat our way to fake points on Khan Acad­e­my, a gamei­fied edu­ca­tion plat­form. Our goal will be to break the com­pul­sion loop in Khan Acad­e­my so that only the peo­ple who val­ue edu­ca­tion for it’s own sake will con­sume its con­tent.

Addendum: What about the other players?

Cheat­ing in games that involve mon­ey and com­pe­ti­tion rais­es one sim­ple ques­tion: Is it ever right to cheat if one play­er can gain an advan­tage over oth­ers that play legit­i­mate­ly?

It’s nev­er right. That’s an easy answer because this ques­tion assumes a cheater is com­pet­ing. There’s a dif­fer­ence between using a mem­o­ry edi­tor to win addic­tive sin­gle-play­er games and using a bot to win a Rock­et League sea­son.

Cheat­ing in com­pet­i­tive envi­ron­ments must stop imme­di­ate­ly because the reward for the obvi­ous cheat­ing harms oth­ers. Oth­er­wise, cheat­ing sab­o­tages addic­tive activ­i­ties, which is a good thing.

Morals become grey for me when cheat­ing hurts only the activity’s own­ing busi­ness. Mobile freemi­um games or MMOs might cre­ate a per­fect storm where busi­ness­es wins at the expense of addicts.

Design­ers should allow play­ers to cheat so long as the cheats don’t affect com­pe­ti­tion. Revert­ing a char­ac­ter to a point before they start­ed cheat­ing is a good approach. This gives seri­ous play­ers an incen­tive to grind and train legit­i­mate­ly, and gives addicts an out.

Games like Runescape won’t allow this because real-world cur­ren­cy is tied an in-game mar­ket. Any cheat­ing would vapor­ize their busi­ness and reduce the game to blind rage and lia­bil­i­ty issues. If Jagex could cap­i­tal­ize only on the com­pet­i­tive nature of the game, maybe they wouldn’t be in this mess.

Despite the harm it would do to Bliz­zard, Jagex, etc., cheat­ing their games frees addicts. Addicts can­not make lucid or informed con­sent. Busi­ness­es should not take mon­ey from addicts because those trans­ac­tions aren’t legit­i­mate.

Giv­ing addicts an out so that they can invest more in them­selves is the moti­va­tion behind this series. That, and encour­ag­ing for intrin­sic val­ue in prod­ucts once again.

Creative writing with git-flow and the Snowflake Method

git-flow’s val­ue depends on the nature of a project. Take cre­ative writ­ing: Randy Ingermanson’s Snowflake Method makes you start from a crude—yet complete—one-sentence sto­ry and iter­ate until you are left with a good sto­ry. Require­ments imposed by The Snowflake Method are anal­o­gous to git-flow’s role for the master branch. Giv­en a LaTeX project man­aged with a com­bi­na­tion of git-flow and the Snowflake Method (“Snowflow”), we get some inter­est­ing prop­er­ties.

Assume this file sys­tem:

build.sh          # Compile PDF(s) in dist using story/
dist/             # .pdf files
concepts/         # whatever
story/            # .tex files
    aggregate.tex # \document{book} 

At min­i­mum build.sh runs some­thing like pdflatex -halt-on-error -output-directory ./dist ./story/aggregate.tex to make a PDF of your sto­ry. The concepts/ direc­to­ry con­tains assets describ­ing char­ac­ters, set­tings, con­flicts and plot deci­sions. One rule for this project is that the concepts/ direc­to­ry be checked in, but nev­er be processed by code. This allows free-form cre­ativ­i­ty in asset pro­duc­tion that a pre­cise process would oth­er­wise cur­tail.

A woman writing on a Mac, hopefully with help from the Snowflake Method

Snowflow branch­es behave anal­o­gous­ly to their git-flow coun­ter­parts, with some added expec­ta­tions.

  • The master branch holds a project that com­piles to PDF and tells a com­plete sto­ry.
  • An elaborate (develop) branch adds detail to the sto­ry.
  • concept (feature) branch intro­duces at least one unique, self-con­tained con­cept in concept/ lat­er used to elaborate.
  • review (release) branch holds a com­plete sto­ry for review. If the sto­ry is of suf­fi­cient edi­to­r­i­al qual­i­ty it may merge to master.
  • reconcile (hotfix) branch fix­es press­ing log­i­cal errors such as plot­holes.
  • seed (support) branch address­es cir­cum­stan­tial con­cerns such as dif­fer­ences in sto­ry arcs or fic­tion­al uni­vers­es.

To enable the Snowflake Method, master should a com­plete sto­ry, but that sto­ry does not have to be pub­lish­able. On that note an ear­ly iter­a­tion for Therac-25’s firmware shouldn’t have seen the light of day. It may seem insen­si­tive to com­pare death by radi­a­tion over­dose to bad writ­ing, but only if you’ve nev­er read any­thing by Dan Sacharow.

A Snowflow project will face a “soft end” on a com­mit to master rep­re­sent­ing a pub­lish­able sto­ry. Unless you come up with dif­fer­ent uni­vers­es, sto­ry arcs or deriv­a­tive prod­ucts there may be no need to mea­sure pro­gres­sion or com­pat­i­bil­i­ty with ver­sion num­bers or tags.

In exper­i­ment­ing with this work­flow my favorite dis­cov­ery is that concepts/ even­tu­al­ly takes the shape of “behind the scenes” con­tent for read­ers, which may be sep­a­rate­ly pack­aged and sold. So long as com­mits are dis­ci­plined, the com­mit his­to­ry helps you build sev­er­al prod­ucts at once, where the main sto­ry you intend to pub­lish implic­it­ly pro­motes con­tent you could pro­duce using infor­ma­tion in concepts/.

The concepts/ direc­to­ry also serves as a sand­box for inspired writ­ing ses­sions. Writ­ing is pret­ty moody. Some­times I feel dis­ci­plined and can see how to pack­age my com­mits, and oth­er times I just want to write with no dis­trac­tions. So if you want to ham­mer out a few thou­sand words in a text file in concepts/, go nuts. You can always wor­ry about struc­tur­ing the con­tent with Snowflow when you are ready.

Elaboration process

I must elab­o­rate on the elaborate branch. elaborate may either expand on the sto­ry or per­form tech­ni­cal tasks using LaTeX. In the for­mer sce­nario, I use foot­notes con­tain­ing at least one ques­tion or instruc­tion to iden­ti­fy oppor­tu­ni­ties to build on the sto­ry.

You don’t have to use foot­notes, but keep in mind that some­one who reviews your work should not have to be a devel­op­er. I like hav­ing some­thing vis­i­ble in the prod­uct for an author and edi­tor to dis­cuss.

For exam­ple:

Bob jumped over Alice. \footnote{But WHY did Bob jump over Alice?}

To make the elab­o­ra­tion process sen­si­ble, you should write con­tent that address­es a foot­note either in the vicin­i­ty of where the foot­note appeared, or in a loca­tion that bet­ter estab­lish­es con­text. When you resolve the mat­ter raised by a foot­note, remove that foot­note.

When you com­mit to an elaborate branch you may add at least zero foot­notes, but you must remove at least one of the foot­notes found when you start­ed. By the time you start a review branch there should not exist any foot­notes.

Review process

  1. Elab­o­rate on all rel­e­vant foot­notes.
  2. git flow release start ...
  3. Com­pile a PDF to release to trust­ed read­ers for feed­back.
  4. From the feed­back, insert a list of foot­notes (or oth­er anno­ta­tions) where applic­a­ble accord­ing to your own best judge­ment.
  5. Address all foot­notes.
  6. Repeat steps 3–6 until there exist no foot­notes.
  7. git flow release finish

Guidelines

Writ­ing can’t be con­strained by too many rules, but I did note these guide­lines and obser­va­tions down as I worked.

  • Do not adjust the story/ to the concepts/, adjust the concepts/ to the story/.
  • Do not mod­i­fy story/ when on a concept branch.
  • Your com­fort and focus on writ­ing comes before the process. Don’t be afraid of relax­ing with pen and paper to decide what to do. Lay down on the floor and sketch on a bunch of Post-Its next to a cup of tea before typ­ing any­thing.

Licensing Snowflake Method content

If you decide to write using this process, stay mind­ful of where you pub­lish your work­ing code. If your prod­uct is a book, license it like a book. But more than any­thing, con­sult some­one qual­i­fied to talk about licens­ing. Of course some books like You Don’t Know JS are open source, but if you are con­cerned about dis­tri­b­u­tion, do your research and choose a license.

Generate tailored résumés with Spin

Recruiters might ask me to change my résumé, and by “might” I mean “always.” Some­times it’s reorder­ing sec­tions. Oth­er times it’s giv­ing them a Word doc­u­ment instead of a PDF. This hap­pened enough times to jus­ti­fy at least some lev­el of automa­tion, so I wrote Spin, a CLI that gen­er­ates résumés tai­lored to dif­fer­ent audi­ences. Spin con­verts a super­set of Mark­down Extra to HTML5. So if I made fur­ni­ture on the side, I can gen­er­ate two résumés with $ spin ./woodworker ./programmer.

The woodworker and programmer pro­files are writ­ten in Mark­down Extra mixed with @ direc­tives. The direc­tives define sec­tions and expe­ri­ences with infor­ma­tion like start and end dates. Check out the exam­ples in Spin’s source.

Mak­ing résumés com­piled prod­ucts yield some ben­e­fits:

  • Code reuse. Two pro­files may share let­ter­heads, con­tact info and sec­tions. Corol­lary to this, chang­ing a résumé to entice Agile Cube Farm LLC doesn’t have to change the résumé I lat­er send to Hour­Long Standup Co.
  • Con­sol­i­da­tion. I can write every­thing I ever did in my life in one place. I may now jour­nal my qual­i­fi­ca­tions freely in world’s most obses­sive­ly struc­tured diary.
  • It’s free. Spin is not friend­ly to peo­ple who aren’t used to cod­ing. But all of the résumé man­age­ment tools I’ve seen so far cost mon­ey, are down, don’t sup­port pro­files, or require more labor than I want to put in. I got every­thing I want with Spin for free after the ini­tial time invest­ment.

Implementation notes

I picked PHP to churn this out quick­ly and gen­er­ate doc­u­ments in HTML5 to allow an inter­me­di­ary step for styling and script­ing for those who care enough. I didn’t want to use PHP­Word because my use of it would be too opin­ion­at­ed and would force con­sumers to fuss with PHP exten­sions. Pan­doc works just fine thank youverymuch.

One can use Spin to main­tain a few pro­files and con­vert every out­put doc­u­ment to DOCX and PDF using Pan­doc. My qual­i­fi­ca­tions serve as a real-world exam­ple of this. The base HTML doc­u­ments them­selves may also have extra fea­tures that make them more pleas­ant for view­ing on a brows­er.

If Spin helps you, please con­sid­er buy­ing me a beer!

Iterating Git commit history using Qi

In the Code­less Code, a cre­ative blog on soft­ware engi­neer­ing, there’s this scribe named Qi and he is a total badass. Qi is source con­trol per­son­i­fied; a prag­mat­ic sto­ry­teller that rea­sons about sys­tems as the sum total of all that came before.

I wrote the qi CLI tool (GitHub, PyPi) as a hat-tip to the author—and with his gra­cious per­mis­sion for use of the name. The idea is to bring devel­op­ers lever­age in dis­cus­sions with man­age­ment through a notice­ably less vio­lent vari­ant of Case 89qi is only a Python wrap­per to git that prints revi­sion hash­es at the end of fixed time inter­vals. This enables run­ning an analy­sis at, say, the end of every week over the past two quar­ters to check if a team deci­sion in Jan­u­ary is hav­ing an intend­ed effect in June.

This helps devel­op­ers and man­agers accept the same facts under pres­sure. I once worked in a high­ly active JS shop with over 30 devel­op­ers and over one hun­dred com­mits every day. We had Bam­boo run­ning builds and auto­mat­ed tests every so often to catch prob­lem­at­ic changes, but it was hard to know if our changes this sprint fol­lowed a spe­cif­ic “direc­tion” for sys­tem qual­i­ty, what­ev­er that meant.  After I wrote and ran qi, I dis­cov­ered that the team was, on aver­age, cre­at­ing sev­en pro­duc­tion code files for every one test suite. To make mat­ters worse, rough­ly 20% of our com­bined JS and CSS went com­plete­ly unused.

We all knew that these things hap­pened, but once I had the num­bers to show that mat­ters were get­ting worse due to man­age­ment pres­sure, we re-pri­or­i­tized to allow more time to increase code cov­er­age for crit­i­cal tests. Qi knew that com­mit his­to­ry could not just be read, but lever­aged.

qi is use­ful when one of the fol­low­ing con­di­tions apply to you:

  • You want to gath­er facts and cor­re­la­tions to study the code’s his­to­ry, review recent changes, or to prove a point.
  • You have no way to mon­i­tor the state of a code­base in terms of met­rics or cor­re­la­tions that are impor­tant to you or man­age­ment.

qi is not use­ful under these con­di­tions:

  • Your project(s) are small or neglect­ed because there is rarely a big pic­ture to derive, let alone a desire to find one.
  • Your team has a strong grasp on the sys­tem being main­tained, or already per­forms suf­fi­cient analy­sis.

What’s happening?

Con­sid­er the com­mand qi --every month --past "3 years".

The com­mand does what it says. It will print revi­sion hash­es that mark the end of every month for the past three years, with some caveats:

  • qi starts from HEAD. The query against his­to­ry will search 3 years pri­or to the com­mit date from HEAD, not from the time you ran the com­mand.
  • If HEAD is younger than 3 years, then the out­put will end at the com­mit mark­ing the end of the first month of devel­op­ment.
  • If HEAD is not even a month old, there will be no out­put.

Print­ing hash­es is one thing, but we want to do some­thing with this infor­ma­tion. Since qi is basi­cal­ly a time iter­a­tor, we can tell it to run a com­mand at each com­mit. Note that qi’s short options use the same let­ter that starts their long options.

qi -e month -p "3 years" -c "./analyze"

When you spec­i­fy a com­mand for qi to run, qi will check out each sig­nif­i­cant com­mit (detach­ing HEAD) and run the com­mand against the code­base at that time.
If your analy­sis fin­ish­es or crash­es, qi will attempt to put HEAD back where it start­ed. But if qi crash­es, HEAD will be left detached and you will need to run git checkout to put it back your­self.

qi -c lets you mod­el infor­ma­tion­al met­rics over time, such as the pro­por­tion of dead code in the project and the cur­rent code cov­er­age by tests. You decide what met­rics mat­ter, because qi is not a report­ing tool, it is one kind of iter­a­tor a report­ing tool might use.

Let’s look at a cou­ple of fun exam­ples.

For triv­ia, if you run qi -e year -p "12 years" -c "du -sh ." on Torvald’s Lin­ux source, you will see the disk usage growth of the project over it’s time on GitHub.

[sage linux]$ qi -e year -p "12 years" -c "du -sh ."
2.8G . # 2017 (to date)
2.7G . # 2016
2.6G . # 2015
2.6G . # 2014
2.5G . # 2013
2.5G . # 2012
2.4G . # 2011
2.4G . # 2010
2.4G . # 2009
2.3G . # 2008
2.2G . # 2007
2.2G . # 2006
2.2G . # 2005

Anoth­er exam­ple is if you want React’s com­plex­i­ty report at the end of every year of its devel­op­ment. In this case you could use this set­up:

$ git clone https://github.com/facebook/react.git && cd react
$ qi --every year --past "3 years" --command "./analyze.sh"

analyze.sh

#!/bin/bash
TIME=$(git show -s --format=%ct HEAD)

# Using https://www.npmjs.com/package/complexity-report
cr -e -f json ./src > $TIME.json

 

Hiding annoying diffs on BitBucket pull request pages

Some shops man­date check­ing in long, auto­gen­er­at­ed files into source con­trol that clut­ter up Bit­Buck­et pull request pages with large diffs, leav­ing you to scroll epic dis­tances or depend on the top nav­i­ga­tion to vis­it files key to a code review.

Despite at least three dif­fer­ent tick­ets dat­ing back to 2014 ask­ing Atlass­ian to sup­port exclud­ing spe­cif­ic file diffs from pull request pages, a John Gar­cia explained why they would not imple­ment this fea­ture:

This response is rea­son­able for the ref­er­enced case, yet unhelp­ful to users as a whole. Even if the peo­ple ref­er­enced in the con­ver­sa­tion were using source con­trol inap­pro­pri­ate­ly by track­ing files that they should not track, oth­ers review­ing code might not be in a posi­tion to change that. Then there are big (as in, often has diffs that take up at least one screen) auto­gen­er­at­ed files that fre­quent­ly change, aren’t nor­mal­ly worth read­ing and should be in source con­trol, such as yarn.lock. Cit­ing a Stack­Over­flow answer address­ing one case with 11 votes at the time of this writ­ing does not make the fea­ture request unjus­ti­fied.

I had the same prob­lem as the peo­ple sup­port­ing the tick­et (with many yarn.lock files, no less) so I wrote a user­script cre­ative­ly named “Hide Annoy­ing Bit­Buck­et Diffs (HABD).” It’s set to exe­cute when you view a pull-request page pow­ered by Bit­Buck­et. The script does not sup­port bina­ry files. Instal­la­tion instruc­tions are at the bot­tom of this arti­cle.

In order for this script to func­tion, you will need to con­fig­ure a cou­ple of things. Thank­ful­ly it’s quick!

You can con­trol what diffs to remove by ref­er­enc­ing a plain text resource list­ing one reg­u­lar expres­sion per line (Point­ing to a raw Paste­bin helps here). A diff is removed from the pull request page if the file­name cor­re­spond­ing to that diff match­es ANY of the expres­sions in the resource. 

For exam­ple, this resource hides file diff blocks for package.json and yarn.lock, both of which are com­mon in Node projects and are checked into source con­trol.

package.json$
yarn.lock$

If we’re clear up to this point and you want to use the script, let’s get you set up. If you encounter any prob­lems, please report an issue.

Script installation for TamperMonkey

  1. While using Chrome, add Tam­per­Mon­key.
  2. Go to HABD’s page on OpenUserJs.org and click “Install.” You will see the source code and a con­fir­ma­tion. Con­firm the instal­la­tion.
  3. Click the Tam­per­mon­key icon in Chrome and click “Dash­board.” You will see a list of installed scripts.
  4. At the far right of the row where you see HABD, you will see a lit­tle notepad icon. Click that to edit the script.
  5. Change the line start­ing with “@resource” to point to a raw Paste­Bin or some oth­er clear text resource with one reg­u­lar expres­sion per line. The link already in the script points to an exam­ple. Remem­ber that the reg­u­lar expres­sions match against file­names. If ANY of the expres­sions match, the diff is removed from the page.
  6. (Option­al) Change the line start­ing with “@match” so that it match­es only the pages in which you review Bit­Buck­et pull requests.
  7. (Option­al) If you want to hide diffs on both pull request and com­mit pages, use the @include rule to add anoth­er URL match­ing pat­tern.

Script installation for GreaseMonkey

  1. While using Fire­fox (or some oth­er sup­port­ed brows­er), add Grease­Mon­key.
  2. Go to HABD’s page on OpenUserJs.org and click “Install.” You will see the source code and a con­fir­ma­tion. Con­firm the instal­la­tion.
  3. Click the arrow next to the Grease­Mon­key icon in Fire­fox and click “Man­age User Scripts.” You will see a list of installed scripts.
  4. On the row for “Hide Annoy­ing Bit­buck­et Diffs”, click “Pref­er­ences,” then “Edit This User Script.”
  5. Change the line start­ing with “@resource” to point to a raw Paste­Bin or some oth­er clear text resource with one reg­u­lar expres­sion per line. The link already in the script points to an exam­ple. Remem­ber that the reg­u­lar expres­sions match against file­names. If ANY of the expres­sions match, the diff is removed from the page.
  6. (Option­al) Change the line start­ing with “@match” so that it match­es only the pages in which you review Bit­Buck­et pull requests.
  7. (Option­al) If you want to hide diffs on both pull request and com­mit pages, use the @include rule to add anoth­er URL match­ing pat­tern.
Do NOT follow this link or you will be banned from the site!