The Snowtide Blog

While at dinner with a friend of mine a couple of weekends ago, we got to talking about how certain programming problems, usually the hardest ones we’ve faced, are ones where we ended up having to simply work the problem: stare at the code, stare at the reference / specification / dataset, hunker down, and plow through the tough stuff with more determination and tenacity than engineering or design rigor. While we were talking about this dynamic, I inadvertently summed up that kind of experience and approach with what I think is a pretty apt phrase: sometimes, you just need to “burn it down” (’it’ being the problem at hand).

The phrase “burn it down” has been rattling around in my head incessantly since then, so I’m hoping to exorcise it here. Perhaps the reason why it’s stayed with me so long is that there’s a pretty good chance that that phrase has a lot of applicability to life in general. Looking back over the years, I’ve often found myself in situations where the key to survival and (ostensibly) success has been my ability (tendency?) to wage a war of attrition against that which stands in my way.

The most immediate case is a programming challenge: you’re sitting there trying to solve a problem for the umpteenth time, and the algorithm in your head or on the page in front of you is laughing at your inadequacies. Ideally, you’d like to step up to the whiteboard, and bang out a description of how this data structure should look, a proof for how that graph-walking algorithm can be sped up by an order of magnitude, whatever. I’ve seen programmers do this – they’ll pull out a small pad, and in about 8 minutes, they’ll knock out some pseudocode for an implementation of some algorithm that might take you an hour to comprehend, never mind implement.

Unfortunately, only a vanishing minority can seize hard problems like that on a regular basis; the rest of us are left wanting to be smarter than we clearly are. So, despite all of your hopes, your text editor simply stares blankly at you, the whiteboard is inert, and you’re left with two options: failure, or the last resort of the almost-overwhelmed, unyielding determination.

(This is very simply the same thing that Edison talking about: “Genius is one percent inspiration, ninety-nine percent perspiration.” Of course, I don’t think we’re in genius territory here…competency, perhaps?)

I’ve experienced that exasperation so many times while working on PDFTextStream and other Snowtide products. I’m fond of saying that I’m a math idiot, and while few people believe me, it’s entirely true. There are things in PDFTextStream that required me to slowly, painfully learn more about esoteric corners of computational geometry than I would ever care to know otherwise, and the same goes for various other specialties in mathematics and computer science. For me, understanding formalisms in those fields is like feathering a chicken by running it through a keyhole. The only way I’ve been able to ship a single bloody line of code has been to Burn It Down.

Of course, programming is a very narrow domain, but I think the same strategy is the only tool that us humans have to use in coping with the crush of life and disorder that surrounds us. Through growing up, getting an education, finding a job, starting a business, sustaining a life partnership, raising children, and coping with the march of time, the only common thread of human endeavor is rage. Even that majickal 1% of prodigies and bona fide geniuses have areas of their lives where their only hope of enduring is to simply endure. In a way, it’s comforting to know that despite the pretensions of some, the only sure way to succeed in programming (and likely even loftier domains) is to lay siege to (or perhaps, depending upon your mindset, to practice judo with) whatever challenge is ahead.

The day is young. Burn, baby, burn.

In a thread on the Google Group dedicated to discussing languages hosted on the JVM (i.e. Scala, Groovy, JRuby, et al.), it was asked by a fellow named Jon Harrop whether something like F# (an OCaml / Standard ML derivative that targets the .NET CLR) would find any traction if it were made available for the JVM. Well, some unremarkable discussion ensued about the costs associated with developing languages, how existing efforts attract funding, etc., and then things turned towards the question of “Why not just use Scala?”, since Scala does fold in a lot of functional programming primitives.

Mr. Harrop’s replies centered on various aspects of ML-style languages that he misses in Scala, and aspects of Scala that he finds irritating. All fine and good — hey, everyone has their own preferences — until he unveiled this nugget (emphasis mine):

OCaml and F# have shown that ML’s approach to structured programming using modules, variant types and pattern matching and extensive type inference is almost always preferable to OOP. When given the choice between OOP and FP, seasoned programmers rarely choose OOP.

Zealotry isn’t anything new — you can probably find inverse statements right now in some Smalltalk newsgroups, or someone agitating about the uniform superiority of s-expressions in a Lisp or Scheme forum. The odd thing about this is that Mr. Harrop is not exactly a random troll — he seems fairly well-respected in the F#/OCaml/ML community, is a prolific writer, and looks to be writing a book on F# for Microsoft Press.

Stuff like this makes the whole facade about software development being akin to engineering even more farcical than one might initially imagine. Can we please recognize that there is a difference between spirited advocacy and demagoguery? I’ve certainly been guilty of the latter on occasion (usually much to my later regret), but it’s particularly irksome to find those that are apparently unaware of the distinction at all.

(…or, “Oh, Dear, Wasn’t I Thinking Before?”)

As my friends will attest, I really enjoy programming languages. I’m one of those language fetishists that talk about “expressiveness” and “concision”, and yes, I’m one of those very strange fellows who blurt out bad Lisp jokes while getting odd looks from innocent bystanders. And while my bread and butter is built in Java, I often find myself yearning for a more expressive language while deploying, customizing, or integrating PDFTextStream (there I go again with the “expressiveness” bit). That yearning can reach almost pathological extremes at times, prompting me to go so far as to sponsor projects that make it possible to use Java libraries (including PDFTextStream) from within Python.

Fortunately, things don’t always have to be so hard. Case in point, I recently dove head-first into Scala, a language that combines object orientation and functional programming into one very tasty stew. Scala has a number of characteristics that make it interesting aside from its merging of OO and FP mechanisms:

• it is statically-typed, and provides moderately good type inference that enables one to skip most type declarations and annotations
• it is compiled, which provides a minimum level of performance (sure, it’s actually byte-compiled, but let’s not quibble right now)
• and the real kicker: it compiles down to Java class files (or .NET IL), thereby enabling it to be hosted on a JVM (or .NET’s CLR), and call (and be called by) other Java (or .NET) libraries

There’s a lot to like here, for programmers from many walks of life, and I could go on and on about how Scala has single-handedly created and filled a great niche of delivering most of the raw power of purely functional languages like Haskell and ML within a JVM-hosted environment with respectable performance. But what has really impressed me has been the way that Scala has improved how I work. In short, it’s made really think about development again.

I generally have two working styles. In a classic statically-typed environment (say, Java or C#), I tend to generate pretty clean designs, but my level of productivity is very low. I attribute both of these characteristics to the copious amount of actual work (i.e. finger-typing) that has to go into writing Java or C# code, even with the best of tools. See, while I’m typing (and typing, and typing), I’m thinking two, three, four steps ahead, figuring out the design of the next chunk of code. The verbosity of the language gives me time to reason about the next step while my fingers are working off the previous results.

In a dynamically-typed environment (say, Python or Scheme), I tend to be extraordinarily productive, but unless I consciously step back and purposefully engage in design, the code I write is much more complex. In such environments, there’s less finger-typing going on, so I don’t have a natural backlog allowing me to think about the code before it’s already on the screen. Further, I know I can get from point A to point B relatively easily in many circumstances, so I end up skipping the design step, switching into Cowboy Coder mode, and hacking at things until everything works. Oddly enough, in certain circles, this isn’t so much frowned upon as it is recommended.

Scala is statically-typed, so the naive observer might speculate that my working style in Scala would be much the same as in Java. However, I’ve found that working with Scala has prompted (forced?) me to consciously step back and think about everything, at every step along the way: class hierarchies, type relationships in general, testing strategies, eliminating state where possible…the amount of actual thinking I’ve done while working with Scala has far outstripped the amount of reasoning that typically goes into any similar period of coding. Unsurprisingly, this has led to quite the spike in code quality, which translates into productivity through fewer bugs and less rework.

I attribute this to the strong, static typing that Scala enforces, combined with the type inference that Scala provides. The former forces me to reason about what I’m doing (as it does in Java, for instance), but because the latter eliminates so much of the finger-typing associated with static typing in other environments, I’m given the opportunity to realize that a concrete design phase would yield tremendous benefits, regardless of the scope of code in question. I suspect I would find working in Haskell or ML to be a similar experience, but because those languages don’t easily interoperate with the libraries I need to do my work, I’ve never really given them a chance.

Thankfully, I don’t think I’ll have to. Scala is a great environment, and even more important than its technical merits, its design has led me to engage in a more thoughtful, more conscious development process.

I’m way late in linking to this, but it’s worth it.

Last October, a presentation by Martin Fowler from JAOO 2006 popped up on InfoQ (which does a great job of simulating the actual experience of being at the session with its video/slideshow integration) where he gave a very high-level overview of domain specific languages (DSLs). He really only scratched the surface, but it’s a great introduction for those that haven’t yet thought about DSLs much.

(Of course, that population is getting smaller by the minute thanks to Ruby (and Rails), since it builds in the metaprogramming facilities necessary to implement internal DSLs.)

I recently had occasion to re-watch the presentation. This time around, I took the time to scribble down some thoughts:

1. I think he played up the potential role of DSLs as “configuration code” too much. Yes, you can tailor a DSL to provide primarily configuration data, and that’s very useful as far as it goes. However, internal DSLs (given an appropriately useful host environment) are able to provide levels of abstraction and composability that go way beyond configuration.
2. I think that casting the Java example he showed as a DSL is really over the top, and is a result of overemphasizing the potential configuration role DSLs can play. As Mr. Fowler said, the line between an internal DSL and just a bunch of specific goal-driven coding in the host language is fuzzy. However, a big part of that line (and therefore whether an environment can reasonably host a DSL) is how well the host language’s existing constructs can be recast as sensible constructs in the DSL. The Ruby DSL example fits this criterion well, as its range (4..12, etc) and block constructs mapped well to the domain at hand. On the other hand, the Java example is Java, unapologetically so — the explicit object creation, the function calls, return statements, etc., simply do not map to the domain. The fact that the integers and strings being passed in those function calls can be recast as an actual configuration file should not lead us to think that Java configuration code is a functional DSL.
3. At least in my experience, external DSLs are dead-ends. There’s just too much heavy lifting that needs to be done to consume external DSL “source files” and align their contents with the host language’s environment. True, internal DSLs need to conform to the syntax of their host environment, but the advantages of “symbolic integration” (as Mr. Fowler puts it) and the fact that you get your IDE’s functionality for free are just too compelling to outweigh any nitpicky syntax quibbles that one might have with any DSL-capable language. And, if those syntax quibbles are significant enough, and the problem the DSL is going to solve is significant enough to make you come close to considering building all of the cruft necessary to implement an external DSL, then go find yourself a secondary language/environment that provides a more palatable syntax, and hook everything up with IPC of some kind.