Ashton Kemerling

AK

Introducing Butler.el

I’m a big fan of reducing barriers to increase adoption and compliance. One of the largest examples for me in the past few months was when I tried out Eclipse for Python Development. The most important plugins for me (other than Python and Git) were the Jenkins and Jira plugins. Quick access to my tasks and instantaneous in-editor feedback about job status increased the likelihood that I would respond to failed builds or fill out tickets correctly.

I eventually decided that Eclipse was overkill for Python development, and way too heavy for a Macbook Air. But in my switch back to Emacs I particularly missed my Jira and Jenkins plugins. This is why I’m writing to announce the 0.1.0 release of Butler. Butler allows for you to view the jobs on your Jenkins servers, and trigger new jobs without switching out of Emacs.

At this point Butler’s feature set is modest: view, refresh, and trigger. These are just the groundwork functions for a larger feature set in the coming weeks, including console output, visual indication of running jobs, watching jobs, build queue, etc. I have high hopes that in a few months this plugin will be as mature as its eclipse counterpart.

If you have any bugs or feature requests, please add them to the Github issue tracker.

A New Addiction

Well, I have officially started collecting watches. My first decent purchase was a few months ago, but I’m counting it as a collection now because I have more pieces than is functionally required, which arguably is 1 in the cell phone era.

Why collect watches? Well, first of all they’re modern Mens jewelry. Mens fashion has changed remarkably little in the century, especially compared to Womens’. So with suits that may stay “in style” for 2-6 years depending on the cut, men only have a few places to go to express their style: shoes, watches, ties, and cuff links.

Further still, very few men under the age of 25 (my group at the moment) wear suits. I usually wear a button up or t-shirt to work, and rarely wear any pants more fancy than jeans. This leaves me shoes and watches. This of course also limits my shoe choices a bit, as does the weather in Chicago and my commuting method of choice (bike).

So, that leaves the lonely watch as my main fashion item. What I particularly love about watches is their intricacy, especially mechanical ones. What I love even more about them is how functional yet counter-functional they can be. Watches do one thing quite well: deal with the passing of time. But at the same time, they’re hardly required in the era of LED signs, car clocks, and cell phones. Furthermore, many have far more complications (features beyond basic date & time) than most people will use. How many people actually time things with their chronographs? How many divers watches rated to 200m go futher underwater than 8 feet? Not many.

At this point, I have two pieces in my collection which I will photograph and post later:

  • A quartz Fossil chronograph, with leather band.
  • An Invicta Pro Diver Automatic, in stainless steel.

I enjoy both quite a bit. The Fossil has a surprising amount of sentimental value for me already, as it was my Valentine’s day gift this year. But I’m also growing to love the Invicta, it feels more serious with its weight, and the hands glow to a degree I’d never seen before.

Next step? Time to start hitting up estate sales and thrift shops in look of a hidden treasure.

Binding vs. Assignment

A coworker of mine was recently running into problems with the following snippet of Python code, and turned to another functionally oriented developer and myself for help.

Odd Functions
1
2
3
4
5
6
7
funs = []
for i in ["a" "b"]:
    def internal_function():
        return i
    funs.append(internal_function)
print funs[0]()
print funs[1]()

At first glance, one would expect this to print “a” then “b”. But much to my surprise (and my coworkers frustration), it was returning “b” both times.

My first instinct was that Python was closing properly, but was messing up the namespacing. So I asked him to use lambdas instead, creating this:

Lambdas
1
2
3
4
5
funs = []
for i in ["a", "b"]:
    funs.append(lambda : i)
print funs[0]()
print funs[1]()

Same result! So by this point we know that its not namespacing that is causing the problems, as you can’t accidentally assign two lambdas the same name. So I began to suspect that the use of “i” was the issue. Perhaps we’re closing around i, which is a mutable variable, and Python can’t anticipate that we want a copy instead of the actual reference. Since i is mutable, both closures are having what i means changed from underneath them after closure creation time. (Coincidentally, Python 2.7.3 keeps i around even after the loop is completed, which I think is odd).

So the solution here would be to use a temporary variable. Any variable first created inside a loop should theoretically only have a lifespan of one iteration through the loop. So let’s try the following:

Temporary Variables
1
2
3
4
5
6
funs = []
for i in ["a", "b"]:
    temp = i
    funs.append(lambda : temp)
print funs[0]()
print funs[1]()

Same result, again! So clearly python is mishandling variables by a large amount. So what’s going on here? My fellow functional coworker was finally able to explain: the real problem here is that Python conflates binding and assignment. Binding is used to create a new variable within the current context, while assignment can only change the value of a given variable within the narrowest bound scope.

Why is this an issue? Well, Python uses the same symbol “=”, to specify binding and assignment, while most other languages use two syntactic rules. For example Go apparently uses “:=” to bind, and “=” to assign. Java uses type declarations to denote bindings, and Clojure/Haskell more or less only have bindings underneath. Unfortunately there’s an upper limit to how intelligent an imperative compiler can get, especially in interpreted languages where the compiler must be fairly quick. So Python doesn’t do the leg work to realize that temp is only used inside the loop, and binds it only once outside the loop. The side effect of this is that anyone who closes over any local variable inside the loop will be surprised when it is mutated by further iterations, and said variables will be left laying around after the loop is done.

So how do we get out of this? One obvious option is to pick a language that understands the difference. Short of that, use a separate function to force Python’s hand.

Ugly Fix
1
2
3
4
5
6
7
8
9
def create_lambda(x):
    return lambda : x

funs = []
for i in ["a", "b"]:
    funs.append(create_lambda(i))

funs[0]()
funs[1]()

Which unfortunately is pretty ugly. Other languages will get you out of this bind though. Clojure has no concept of assigning variables, only binding. So the only way to create mutable state in Clojure is through successive binds, or via the special Software Transactional Memory functions, which are specially designed for concurrency.

Nice and Clean
1
2
3
4
5
6
7
(def funs
     (for [i ["a" "b"]]
         (fn []
             i)))

(println ((first funs)))
(println ((second funs))

And according to my Go enthused coworker, this also works:

1
2
3
4
5
6
7
8
9
10
11
12
13
func main() {
    for i:=0; i<4; i++ {
        tmp := i

        funs[i] = func () int {
            return tmp

            }
    }
    fmt.Println(funs[0]())
    fmt.Println(funs[1]())
    fmt.Println(funs[2]())
}

If you are currently working on a Python project, or are a Python enthusiast, the only hope is to be careful, and hope that Python 3’s compiler might be more intelligent.

Size Matters

About 6 months ago I started to look for a new language for hobby projects. I wanted something practical, yet fun. I managed to limit the criteria down to:

  • Functional
  • Good community
  • Growing acceptance

This automatically eliminates a lot common languages, like Python, Ruby, and Java. After my Common Lisp job, I knew that a sane compilation system/target were an absolute must, so I looked at Scala first. On the surface Scala seemed a real winner: growing fan base, increasing commercial adoption, functional, and targeting the JVM. But while trying out Scala I realized that I had missed a huge criteria: language size.

There’s only so much information that a programmer can keep in their mind at a time. Every programmer needs to split their internal memory in at least 4 different ways, if not more. There are language details to remember, library or API interfaces to recall, domain knowledge to retain, and possibly coordination or social issues to be concerned about. Once this internal memory is exceeded details end up being forgotten, and recalling forgotten information involves an expensive round trip to documentation, code, or the internet.

There’s no escaping the need to recall language details, at least until someone invents a syntax free programming languages; similarly it’s near impossible to write an application of any size without relying on libraries or APIs. Domain knowledge is often expensive to acquire and beneficial to have. And social issues cannot be resolved by programming language choice alone.

So the real question is, what are you going to skimp on? Are you going to pay the price of memory “cache misses”? Are you going to forget portions of your language, or certain libraries? Or are you going to segregate domain knowledge among the team so that everyone knows their bit only?

The real world answer is probably a bit of each. But I contend that reduction of language features in memory is the least damaging of all the outcomes. I doubt it’s controversial to say that domain knowledge should be shared widely among developers, and should be as much in memory as possible; programmers who understand what they’re programming for tend to program better on average. Similarly it’s hard to eliminate a decently written library: that functionality has to come from somewhere, and it’ll either be an external library, an internal one, or duplicated code.

But language reduction is an interesting idea. Programmers long ago got away with previous few features, yet managed some amazing things. Don’t believe me? Consider how small the Scheme spec is compared to say, Java. It reduces the need for fancy and hard to learn IDEs, allows the programmer to get more done between looking up function or class definitions, and increases the speed with which an engineer can fully understand a piece of code.

It was actually this feature that finally drove me away from Scala to Clojure. After a few hours of Scala work I was turned off by the size of the language, and got tired of saying “Why did it do that?” Clojure on the other hand is relatively small, only a dozen “special forms” which have to be hard coded into the language, and then a 3-4 dozen core functions. This means that I can both keep the entire language in memory, and retain a large amount of library and domain knowledge to get work done.

Octopress

I’m currently in the process of changing my website over from Wordpress.com to Octopress on Github pages. There are two things driving me from Wordpress.com: control and price.

Wordpress is fine if you are either a big PHP afficianado, or are afraid of web programming in general. It’s quite easy to install themes and plugins without any knowledge of what this means. However, this only is true if you’re self-hosting Wordpress.org (the project) and not using Wordpress.com (the commercial website using Wordpress.org). On Wordpress.org you’re stuck using a limited number of (mostly for pay) themes and plugins, and are restricted from inserting HTML, JS, or CSS into your website.

So if you want Google Analytics, you’ll need to use a third party CDN who can inject the Google Analytics JS into the DOM. No thanks. If you want any social networking integration, you better hope that Wordpress.com supports it (sorry, app.net users!). And if you want pictures or other media, you’ll have to manually control the links to a third party system or hope that you don’t exceed the cheap data storage tier on wordpress.com.

So what’s so great about Octopress?

  1. Free or cheap hosting wherever.
  2. Easy dev preview.
  3. Version controlling.

Octopress uses Jekyll, which makes it really easy to separate content from presentation, and produces purely static webpages that can be tossed onto any webhost. It also has great integration for thid party systems, and adding your own is only a hop, skip, and a bit of JS away. Local development is super straightforward with the rake generate and rake preview commands, which produce static HTML and a testing webserver, respectively. And everything is stored in Git, so upgrading is only a matter of merging (!), and you can host on Github by pushing to the correct repo (or use rake deploy). And since your side is code, you don’t have to count on flimsy undo support, merely revert commits as needed to go back in history.

So far the only hiccup is moving all of the comments over from Worpress to Disqus. All of the docs on this I’ve seen are out of date, I’ll report if I get this working.

Java Doesn’t Deserve Half the Shit We Give It

It’s very popular among programmers to diss Java on Reddit and Hacker News. But Java really isn’t that bad of a language, and I really think there are other reasons why people hate it.

While we like to pretend that programming language choice is purely about practicality, there is a very large social aspect to it. Nobody wants to admit to liking Cobol or VBA, and being at the forefront of a new, exciting language can bring a type of prestige in programming circles. Thus to some degree, our choice of programming language reflects on how we wish to see ourselves. Haskell programmers think of themselves as highly skilled mathematicians, python users think of themselves as massively productive, etc.

Personally, I think the easiest way to sum up the design decisions behind Java is thusly: You have a large number of average programmers, how can you get them to design large and complicated systems reliably. This is not the most exciting idea in the world, but it’s thoroughly realistic. Your average programmer is, well, average. While this may be reason enough to not bother with Java in your hobby projects, I don’t really see this as a reason to heap scorn upon it. It may not be exciting, but Java really isn’t that bad.

Right vs. Useful

I spend a fairly decent amount of my time thinking about how my mindset and world view affects my decisions and actions. In the past few months I’ve found that how I evaluate these mindsets has begun to change; I used to be mostly concerned with whether or not a mindset was correct, but now I’ve started to think about whether or not it’s useful.

Determining whether or not a mindset is correct is rather tricky philosophical ground, as it requires some sort of platonic “reality” to compare it against. But for the sake of argument, let us say that a “correct” mindset matches the most unbiased observations of the world we can make, or logically follows from a set of axioms that are relatively well accepted. Good examples of correct mindsets are “Not lying on my taxes reduces the chance of getting in trouble”, and “driving according to traffic laws means I won’t get a ticket”.

Useful mindsets are even easier. A useful mindset is one that produces more beneficial outcomes for the user, where beneficial is also defined by the user. The best examples of useful mindsets are also correct, those who don’t speed don’t get speeding tickets, etc.

Where things get interesting is where useful and right don’t necessarily agree, or when it’s not currently possible to know what is correct. My particular favorite is the question of free will: It’s not currently known whether or not humans make actual decisions, or if we are merely a product of a deterministic universe.

Whether or not we have free will is probably beyond our current philosophical and scientific knowledge, and it’s certainly beyond mine. But we can analyze what our belief in free will does to our ability. The inputs to this scenario can be put on a 2×2 grid, 2 options for our belief (free will or not), 2 for reality (free will or deterministic), producing 4 options. If we choose to believe and free will, and free will exists we gain our own future. If we choose to believe in free will, and it doesn’t exist, we were predestined to believe as such, ditto if we don’t believe and free will doesn’t exist. But the really painful point is when we choose to not believe in free will and it exists, we end up surrendering our initiative and free well, blaming everything on fate. The obvious choice is to believe in free will, and hope for the best.

The more astute among you will recognize this as Pascal’s wager (which is also a form of the more general Prisoners Dilemna). Personally Pascal’s wager has never swayed my belief, as I’ve found reasons to disagree with its setup (the wager only works if you assume free will, and a vengeful Abrahamic God. If you disagree with this portrayal of a higher being, Pascal’s wager doesn’t work out the same).

So next time you find yourself thinking about your worldview, try using usefulness as another yardstick to compare your beliefs against.

Learn What You Don’t Like

There’s really nothing worse than someone who has lived their life surrounded by people who agree with them fully. People who live this way have confidence that precedes their knowledge, and will treat contradictory knowledge as an attack on their in-group and world view.

While the above mentioned effect is generally most common in religion and politics, it crops up sometimes in engineering in the so called “holy wars”. Linux vs. Mac vs. Windows, Emacs vs. Vim, etc. The more similar the items in question, the more intense the debate.

This is why I think it’s pretty important to try out technologies that you’d formally written off, and re-evaluate your decisions. I’ve been an Emacs user for about 5 years now, and I’d written off learning Vim for anything other than emergency config-editing on a remote server. But at the behest of my colleagues, I tried Vim for a few weeks recently, and I’ve come to a surprising conclusion: I don’t like Vim. But rather than disliking Vim because I’d already invested in Emacs, I’ve learned exactly what I need first and foremost in an editor, and an understanding of the compromises I’m making in order to use Emacs. Basically it toned me down from being a mild Emacs zealot to an Emacs user.

What exactly made me go back? First and foremost, I strongly prefer the way that Emacs uses buffers. The fact that I don’t have to worry about closing a buffer closing Emacs, and the fact that I can purposefully leave buffers unsaved is nice. I also like having multiple shells open to run various programs needed for development purposes, and I found the Vim solution, ConqueTerm, to be insufficient for my needs.

What do I miss from Vim though? While I’m not sold on modality (it’s another thing to track), Vim’s key combinations are truly superior to Emacs’ offering, especially when it comes to repeating and chaining actions. Also, while Vim’s shells are less intuitive than Emacs’, they tend to survive heavy program output better (30+ megs of terminal output to the shell tends to overload Emacs’ poor head). But the most important thing is that I learned what I value most in an editor, so I will know in the future whether or not a new editor will be right for me.

Now that I look back, I see similar Emacs vs. Vim moments in the past few years. One of the more recent moments was retrying Python, which I had written off in my early college years due to a bad experience. In retrying Python, I discovered that it had a few things that I valued highly in a language, and ended up leaving Ruby for it. And I’d never have made the switch if I hadn’t tried out something I didn’t like.

Hands and Feet

When I was a teenager I worked at a large amusement park in the games department, which I guess technically means I was a carnie. One of the more interesting games I worked at was the rope ladder; in order to win the game contestants had to climb up a rope ladder set at a 45 degree angle and attached to swivel points at the bottom. Unsurprisingly I learned to climb it easily (free tries and boredom does wonders for learning), but I witnessed very few winners.

One of the more important lessons I learned was about how to not think about specific tasks. In order to climb the ladder, you need to move opposite limbs: step forward with your right hand and left foot at the same time, etc. This spreads your weight out, and ensures that any movement you make is relatively counter-balanced, reducing the maximum amount you’ll find yourself unbalanced at a given moment. What was particularly interesting to me was the transition from not-making it, to making it. When I first started, I had to concentrate on my hands and feet, and fell constantly. Only once I learned to move my hands and feet without conscious thought, and thus free up my mind to worry about my balance, did I successfully climb the ladder.

This has fairly important implications to my daily work: namely that I cannot concentrate on doing good engineering work without making everything else a subconscious action. Every single non-work related activity breaks concentration for a short period of time, and potentially might take me out of the elusive “zone”. I’ve discovered that If I’m concentrating on your editor, I’m not going to code effectively (if at all). Therefore, It’s absolutely important that every single editor and environment change that you make is judged by balancing how much time/energy it takes to learn and use against its effect. The most powerful Git extension in the world for your editor will do you no good if it takes 2 minutes to commit each time: it would take less time and mindshare to simply type or alias it out on the shell.

For example I use the Magit extension for Emacs to avoid having to go to a terminal to use git; the gains for moderately complicated tasks such as stash manipulation, reverting files, and partially staging a file are truly tremendous. Similarly I use a pylint & nosetests extension for Emacs to avoid having to remember the (quite long) commands it takes to run each command correctly. This also has the tremendous advantage of not linting or testing code that’s outside of my purview, thus running faster and cluttering my mind less with irrelevant results.

The other important take away is to not overdue it. The point of any given addition to your editing environment is to improve the quality and quantity of your engineering output. So if your editor bothers you every 5 minutes with CI notifications that aren’t related to any of your commits, you aren’t really gaining anything. This is why I ended up turning off the Pyflakes/Flymake extension to Emacs: it improved my coding a bit at the time (although most of the time it highlighted code that I hadn’t finished typing yet), but the additional .flymake files it left laying around tripled the amount of time it took for me to work with Git. It simply wasn’t worth the trouble in the long run.

The easiest way to tell whether or not something is worth it is to read the manual and give it a try. Some tools take a while to learn, like Magit, and others only take some time in the setup, like Flymake. Once you’ve exited the learning stage, you should think heavily about whether or not you’re concentrating on the tool instead of your work, and whether or not this has affected the quality or quantity of your output. Remember, there’s no shame in uninstalling a tool if it really doesn’t make your life easier.

Reaction to Kill Your Heroes (and Comments)

Will Larsen had a nice post about what he calls Hero Programmers. Put simply, his theory is that some projects end up in a state where they count on massive effort by a select few heroes on the team. The problem is that this upsets those who aren’t heroes, burns out the heroes, and still won’t save an ailing project.

What was particularly interesting to me was some of the responses on Reddit. In particular one quote from “name_was_taken” stands out to me: “Fire the developers who are useless.”

There is a big assumption on both sides of this discussion that should be cleared. name_was_taken is assuming that the other devs aren’t carrying their own weight, Will Larsen is assuming that the environment is setup to discourage them from doing so.

There are obvious situations where you should fire a developer: misbehavior at work, damaging to team morale, poor performance, low reliability, etc. This is not about the obvious case. This is about when you have decent devs all around, but the distribution of effectiveness is massively unequal.

I think name_was_taken’s attitude comes from an assumption that there’ll be a “Star Programmer” available for this company to hire, that way they can have nothing but heroes. I don’t think that’s possible, and I think that’s a dangerous ideology to run a business on. The simple fact of it is that most programmers are average, and only a small handful are truly stellar. Therefore, any good business should attempt to hire the best, and plan for average.

The truly dangerous thing about the hero mentality is what it does to performance overall. Projects that require heroic attempts at fixing them often exclude team efforts. If one person is running around, massively refactoring everything, in an attempt to hold everything together, there is little to no chance that any other developer will be able to get any work done. They’ll have to keep up with a quickly changing system (that they have no input to), and make sure they don’t step on the heroes toes while working.

So the solution is to get out of this mode. Hero mode will burn out your heroes, and make it almost impossible to use the majority of your dev team. Even if you cull the worst members of your team, you will not be able to reliably hire more heroes in the best of circumstances, and most star programmers can smell and avoid an out of control project a mile away.