The phrase “burn it down” has been rattling around in my head incessantly since then, so I’m hoping to exorcise it here. Perhaps the reason why it’s stayed with me so long is that there’s a pretty good chance that that phrase has a lot of applicability to life in general. Looking back over the years, I’ve often found myself in situations where the key to survival and (ostensibly) success has been my ability (tendency?) to wage a war of attrition against that which stands in my way.

The most immediate case is a programming challenge: you’re sitting there trying to solve a problem for the umpteenth time, and the algorithm in your head or on the page in front of you is laughing at your inadequacies. Ideally, you’d like to step up to the whiteboard, and bang out a description of how this data structure should look, a proof for how that graph-walking algorithm can be sped up by an order of magnitude, whatever. I’ve seen programmers do this – they’ll pull out a small pad, and in about 8 minutes, they’ll knock out some pseudocode for an implementation of some algorithm that might take you an hour to comprehend, never mind implement.

Unfortunately, only a vanishing minority can seize hard problems like that on a regular basis; the rest of us are left wanting to be smarter than we clearly are. So, despite all of your hopes, your text editor simply stares blankly at you, the whiteboard is inert, and you’re left with two options: failure, or the last resort of the almost-overwhelmed, unyielding determination.

(This is very simply the same thing that Edison talking about: “Genius is one percent inspiration, ninety-nine percent perspiration.” Of course, I don’t think we’re in genius territory here…competency, perhaps?)

I’ve experienced that exasperation so many times while working on PDFTextStream and other Snowtide products. I’m fond of saying that I’m a math idiot, and while few people believe me, it’s entirely true. There are things in PDFTextStream that required me to slowly, painfully learn more about esoteric corners of computational geometry than I would ever care to know otherwise, and the same goes for various other specialties in mathematics and computer science. For me, understanding formalisms in those fields is like feathering a chicken by running it through a keyhole. The only way I’ve been able to ship a single bloody line of code has been to Burn It Down.

Of course, programming is a very narrow domain, but I think the same strategy is the only tool that us humans have to use in coping with the crush of life and disorder that surrounds us. Through growing up, getting an education, finding a job, starting a business, sustaining a life partnership, raising children, and coping with the march of time, the only common thread of human endeavor is rage. Even that majickal 1% of prodigies and bona fide geniuses have areas of their lives where their only hope of enduring is to simply endure. In a way, it’s comforting to know that despite the pretensions of some, the only sure way to succeed in programming (and likely even loftier domains) is to lay siege to (or perhaps, depending upon your mindset, to practice judo with) whatever challenge is ahead.

The day is young. Burn, baby, burn.

And, of course, thanks go to Rich who took the time to make the drive up to Northampton from New York City.  (Fanciful thought: does this mean that the developer’s group constitutes a programmer’s oasis?  Is Western Mass. the new center of gravity for innovative software development in New England? *wink*)

Lou’s notes on the talk itself capture its content far better than I’ll dare to attempt at this point.  Suffice it to say that it was a great presentation by Rich, who clearly has a penchant for teasing apart complex topics and evangelizing Clojure very effectively.  Luckily for the rest of us, there are a number of other talks about Clojure by Rich floating around in the ether.

It should come as no surprise to anyone who knows me at all that I love what Clojure has come to be.  I followed Rich’s prior attempts to marry Lisp and Java (specifically, Foil and jFli), but Clojure tops those efforts handily on essentially every front.

But, what of my fervent love of Scala, so earnestly professed in this very space?  Clearly, I’m not particularly monogamous when it comes to programming languages.  Clojure and Scala have a lot in common, but they are very, very different from each other — although they share the common traits of (a) being better than “straight” Java in so many ways, and (b) enabling functional programming on the JVM (and of course, .NET via ikvm).  You can love both; maybe it’s a right-brain, left-brain thing.  (I can clearly imagine Professor Stillings scowling at me for that one.)

Anyway, again, a big ‘thank you’ to Rich Hickey and everyone else who made last night possible.

Mr. Harrop’s replies centered on various aspects of ML-style languages that he misses in Scala, and aspects of Scala that he finds irritating. All fine and good — hey, everyone has their own preferences — until he unveiled this nugget (emphasis mine):

OCaml and F# have shown that ML’s approach to structured programming using modules, variant types and pattern matching and extensive type inference is almost always preferable to OOP. When given the choice between OOP and FP, seasoned programmers rarely choose OOP.

Zealotry isn’t anything new — you can probably find inverse statements right now in some Smalltalk newsgroups, or someone agitating about the uniform superiority of s-expressions in a Lisp or Scheme forum. The odd thing about this is that Mr. Harrop is not exactly a random troll — he seems fairly well-respected in the F#/OCaml/ML community, is a prolific writer, and looks to be writing a book on F# for Microsoft Press.

Stuff like this makes the whole facade about software development being akin to engineering even more farcical than one might initially imagine. Can we please recognize that there is a difference between spirited advocacy and demagoguery? I’ve certainly been guilty of the latter on occasion (usually much to my later regret), but it’s particularly irksome to find those that are apparently unaware of the distinction at all.

Now, keep in mind that this is my first foray into anything resembling journalism hackery.

• A couple of new faces showed up, including (dear god, I hope I’m getting the names right) Jason from Studio4Technologies, who mostly works with C#, Dan, who lives in Avon, CT, has a day job using C#, but is working through SICP on his own and is interested in all manner of functional programming languages, and Michael (Eger, not McIntosh), who is a jack-of-all-trades consultant.
• Gerard and Lou had already installed Mac OS X 10.5, and sang its praises (for the most part). Highlights include Spaces (a “killer” implementation of virtual desktops), the Exposé/Safari select-and-clip widget maker (very cool feature, although no one can think of a good practical use of the tech), and the salvation that is having the Dock on the side of the screen so as to avoid the absurdities associated with how it’s rendered at the bottom of the screen.
• Lou’s going to a tech convention in Las Vegas next week (thereby missing election day!); it turns out that the whole gang from Atalasoft is going to be sporting snazzy pixelated ties on the convention floor. Very cool, but clip-ons?
• Logo design is always about whatever the guys in the design shop happen to be messing around with in Illustrator the day your order hits their desk. Thus, the cyclic obsession with graceful swoops, cubist abstract styles, and flaming butterflies.
• Gerard took a look at Scala since our last meeting, and found much to his liking. “If I were using Java, I’d definitely be coding in Scala,” said Gerard, in yet another example of how Sun made everything more complicated for everybody by conflating the JVM and the Java programming language for years. Here’s hoping IntelliJ gets moving on some first-class Scala support.
• Dan’s been using Scheme to go through SICP and the Little Schemer (an obvious choice since that’s the language used in those materials), but he’s somewhat overwhelmed by the number of functional programming languages that are available — how can one choose what to use? My response was that they’re all functionally the same (ba-dum-crash!) in terms of the primitives they provide. Which FP language to use in any particular situation generally comes down to the details of one’s deployment and runtime requirements at the time. For example, I’m using Scala right now because it interoperates smoothly with Java and .NET, making it easy to work with PDFTextStream’s existing codebase.
• Michael asked what functional programming is, exactly. From there, I somehow found myself holding forth on definitions of functional programming, why minimizing state is Good™, what continuations are and why they’re handy, and the differences between global continuations and delimited continuations. I’m sure most people were groaning. Lou mentioned Common Lisp’s exception handling (better known in that world as conditions and restarts) as an instance of delimited continuations.
• Jason mentioned that there seems to be some noticeable growth in the number of startups in the area, his Studio4 Technologies being one of them. Maybe it’s time for the group to start tracking how much work is coming into the area, how many new businesses have opened and where, etc.
• Greg pulled out his new Thinkpad, running Linux (though I can’t remember the distro).
• Using OS X Mail to access GMail via IMAP is apparently still a no-go, given Mail’s lack of tag support. Thunderbird was thrown out there as a possible alternative, but some had doubts that GMail’s tags would be properly represented via IMAP.
• The state of social networks is totally out of control — there are too many profiles to manage on too many sites. Some fun was had at the expense of those that are “professional networkers” — those people that have thousands and thousands of LinkedIn connections, etc.
• Michael polled the group for thoughts on XSLT, at which point a wave of grunts was heard ’round the table. Just about everyone panned XSLT as being handy, but inevitably too complex without the right tools. XSLT 2.0 doesn’t exactly make progress on this front. As an aside, I noted that XSLT also happens to be a functional language (perhaps the most widely used FP in industry these days, I’ll wager). [As an aside, Dan's first message to the list arrived with the title "XSLT is like Lisp, but with a punch in the face."]
• Generics were briefly discussed, including some comparison of how they’re implemented in the JVM (erasure) and the CLR (not-erasure, although no one knew the details there).
• ikvm was mentioned as a practical and totally usable way to cross-compile Java libraries for use in .NET environments. [We happen to use it in order to provide PDFTextStream.NET.]
• For those new to the group, the mailing list was mentioned, as well as our usual IRC channel (#wmassdevs on Freenode), which is unfortunately pretty sparse these days!
• A pretty lively exchange was had on the topic of Borders vs Barnes and Noble, especially as it relates to the availability of technology tomes. The B&N in Hadley is tremendously mediocre, and the one in Holyoke isn’t a whole lot better. The general consensus was that Borders generally has a better tech book section than B&N, so there’s reason to hope that the significant presence that Borders looks to be building in the Holyoke Mall will make things a lot easier when one of us just absolutely has to have a particular tech book today. There was also talk of the partnership between Borders and Amazon, which apparently yields various benefits.

That’s all that I can recall. I really should have had a notepad with me, I suppose. If anyone has anything else they want to add, feel free to drop it in the comments.

As my friends will attest, I really enjoy programming languages. I’m one of those language fetishists that talk about “expressiveness” and “concision”, and yes, I’m one of those very strange fellows who blurt out bad Lisp jokes while getting odd looks from innocent bystanders. And while my bread and butter is built in Java, I often find myself yearning for a more expressive language while deploying, customizing, or integrating PDFTextStream (there I go again with the “expressiveness” bit). That yearning can reach almost pathological extremes at times, prompting me to go so far as to sponsor projects that make it possible to use Java libraries (including PDFTextStream) from within Python.

Fortunately, things don’t always have to be so hard. Case in point, I recently dove head-first into Scala, a language that combines object orientation and functional programming into one very tasty stew. Scala has a number of characteristics that make it interesting aside from its merging of OO and FP mechanisms:

• it is statically-typed, and provides moderately good type inference that enables one to skip most type declarations and annotations
• it is compiled, which provides a minimum level of performance (sure, it’s actually byte-compiled, but let’s not quibble right now)
• and the real kicker: it compiles down to Java class files (or .NET IL), thereby enabling it to be hosted on a JVM (or .NET’s CLR), and call (and be called by) other Java (or .NET) libraries

There’s a lot to like here, for programmers from many walks of life, and I could go on and on about how Scala has single-handedly created and filled a great niche of delivering most of the raw power of purely functional languages like Haskell and ML within a JVM-hosted environment with respectable performance. But what has really impressed me has been the way that Scala has improved how I work. In short, it’s made really think about development again.

I generally have two working styles. In a classic statically-typed environment (say, Java or C#), I tend to generate pretty clean designs, but my level of productivity is very low. I attribute both of these characteristics to the copious amount of actual work (i.e. finger-typing) that has to go into writing Java or C# code, even with the best of tools. See, while I’m typing (and typing, and typing), I’m thinking two, three, four steps ahead, figuring out the design of the next chunk of code. The verbosity of the language gives me time to reason about the next step while my fingers are working off the previous results.

In a dynamically-typed environment (say, Python or Scheme), I tend to be extraordinarily productive, but unless I consciously step back and purposefully engage in design, the code I write is much more complex. In such environments, there’s less finger-typing going on, so I don’t have a natural backlog allowing me to think about the code before it’s already on the screen. Further, I know I can get from point A to point B relatively easily in many circumstances, so I end up skipping the design step, switching into Cowboy Coder mode, and hacking at things until everything works. Oddly enough, in certain circles, this isn’t so much frowned upon as it is recommended.

Scala is statically-typed, so the naive observer might speculate that my working style in Scala would be much the same as in Java. However, I’ve found that working with Scala has prompted (forced?) me to consciously step back and think about everything, at every step along the way: class hierarchies, type relationships in general, testing strategies, eliminating state where possible…the amount of actual thinking I’ve done while working with Scala has far outstripped the amount of reasoning that typically goes into any similar period of coding. Unsurprisingly, this has led to quite the spike in code quality, which translates into productivity through fewer bugs and less rework.

I attribute this to the strong, static typing that Scala enforces, combined with the type inference that Scala provides. The former forces me to reason about what I’m doing (as it does in Java, for instance), but because the latter eliminates so much of the finger-typing associated with static typing in other environments, I’m given the opportunity to realize that a concrete design phase would yield tremendous benefits, regardless of the scope of code in question. I suspect I would find working in Haskell or ML to be a similar experience, but because those languages don’t easily interoperate with the libraries I need to do my work, I’ve never really given them a chance.

Thankfully, I don’t think I’ll have to. Scala is a great environment, and even more important than its technical merits, its design has led me to engage in a more thoughtful, more conscious development process.

Last October, a presentation by Martin Fowler from JAOO 2006 popped up on InfoQ (which does a great job of simulating the actual experience of being at the session with its video/slideshow integration) where he gave a very high-level overview of domain specific languages (DSLs). He really only scratched the surface, but it’s a great introduction for those that haven’t yet thought about DSLs much.

(Of course, that population is getting smaller by the minute thanks to Ruby (and Rails), since it builds in the metaprogramming facilities necessary to implement internal DSLs.)

I recently had occasion to re-watch the presentation. This time around, I took the time to scribble down some thoughts:

1. I think he played up the potential role of DSLs as “configuration code” too much. Yes, you can tailor a DSL to provide primarily configuration data, and that’s very useful as far as it goes. However, internal DSLs (given an appropriately useful host environment) are able to provide levels of abstraction and composability that go way beyond configuration.
2. I think that casting the Java example he showed as a DSL is really over the top, and is a result of overemphasizing the potential configuration role DSLs can play. As Mr. Fowler said, the line between an internal DSL and just a bunch of specific goal-driven coding in the host language is fuzzy. However, a big part of that line (and therefore whether an environment can reasonably host a DSL) is how well the host language’s existing constructs can be recast as sensible constructs in the DSL. The Ruby DSL example fits this criterion well, as its range (4..12, etc) and block constructs mapped well to the domain at hand. On the other hand, the Java example is Java, unapologetically so — the explicit object creation, the function calls, return statements, etc., simply do not map to the domain. The fact that the integers and strings being passed in those function calls can be recast as an actual configuration file should not lead us to think that Java configuration code is a functional DSL.
3. At least in my experience, external DSLs are dead-ends. There’s just too much heavy lifting that needs to be done to consume external DSL “source files” and align their contents with the host language’s environment. True, internal DSLs need to conform to the syntax of their host environment, but the advantages of “symbolic integration” (as Mr. Fowler puts it) and the fact that you get your IDE’s functionality for free are just too compelling to outweigh any nitpicky syntax quibbles that one might have with any DSL-capable language. And, if those syntax quibbles are significant enough, and the problem the DSL is going to solve is significant enough to make you come close to considering building all of the cruft necessary to implement an external DSL, then go find yourself a secondary language/environment that provides a more palatable syntax, and hook everything up with IPC of some kind.

Consider: up until last week, I was simply using this space every now and then for some relatively bland navel-gazing related to selected goings-on at Snowtide. Then, a friend of mine decided to put my most recent post (probably the only potentially inflammatory post I’ve ever written) on reddit, and a variety of people weren’t very happy (both in comments to the post itself, on reddit’s comment page, and to a lesser extent on a Joel On Software thread). For someone who can lay only a tenuous claim to being a blogger (never mind the title of A-, B-, C-, or D-list blogger!), it’s been an interesting experience to say the least.

I tried to participate in the discussions that were swirling around, but eventually the comments became too numerous for me to follow in a timely way given the amount of bandwidth I’ve allocated to such things. So, I’m taking the easy/cheap way out with a response post. I know this is frowned upon by many, but c’est la vie.  Here, I will respond in two parts:

1. Python and the Growth “Problem”
2. Sandbox Etiquette

Python and the Growth “Problem”

In reading over all of the commentary, there seem to be three types of responses:

Response Type A: Any lack of growth/”innovation” or a slowing of such growth in Python is good — stability makes it easier to concentrate on customer solutions, and encourages robust library development.

Regardless of your language or platform, if stability and operational continuity is an overriding interest of yours, then lock yourself into a particular build, and stay there as long as you want. This is a significant part of the job of IT organizations in large organizations – to standardize on environments and tools so as to shield the organization from unwanted change and cost.

(As an aside, Ruby’s Matz provides a positive spin on the “Python is stable, and that’s good” attitude, which may or may not be cheeky [it's hard to tell through the translation]: “Perhaps Python has a sense of responsibility.”)

Response Type B: The “significant improvements” I’d like to see in Python are (take your pick): of academic use only; are overhyped genius toys that only make it easier to build overly complex solutions; distractions from other improvements that would be immediately useful to the majority of the Python userbase.

This attitude pops up frequently in any discussion of programming paradigms that are off the beaten track, any technique that is unfamiliar to the commenter, or any anything that the commenter has had problems with in the past. Meek typified this kind of response with:

Python is not growing because you want programmable syntax and “esoteric” features? Features that 99% of software developers should never use. Let me guess, you have never maintained a project written in a language that supports programmable syntax where geniuses abuse meta-programming where simpler alternatives achieve the same goal.

This is a particularly disturbing line of thought, and one that I had always considered to be antithetical to a central principle of Python (at least in my eyes), that the programmer should always be trusted. I’ve always associated this with a variety of Python features, including duck typing, the lack of access controls around class members (modulo the slightly perverse double-underscore notation and associated name mangling of “private” attributes), the composability of namespaces, etc.

Lots of programming features are “esoteric”, depending on who you ask. Pointer access is esoteric to a web app developer and should never be used in such a context, but it’s critical to a C-language device driver programmer. Any number of language features can simultaneously be considered esoteric by some and necessary by others. Not recognizing this, and then implying that “simpler alternatives” could readily take the place of those “genius” toys is evidence of a lack of perspective. 28/w in the JOS thread makes my point better than I ever could:

It’s precisely because I want my projects to be on time that I don’t use assembly language for everything. That’s the same reason I don’t use C++ either. I’m about 5x as productive in Ocaml as I am in C++ i.e., at least 80% of my time spent coding C++ is spent dealing with language issues; it’s the equivalent of spending time making all my function calls out of gotos.

Most likely, 80% of my time coding Ocaml is wasted too, and I just don’t know it.

Bottom line: just because you don’t see a use for a particular language feature doesn’t mean that someone else doesn’t find it absolutely, positively necessary.

Response Type C: Python is growing, and if you were to pay attention, you’d notice. We’re just not working on what you want.

This point has been made by a variety of people, but I should give special attribution to Phillip J. Eby, since he’s a significant Python contributor:

Um, so you don’t think the “with” statement and coroutines were new features?

What about the new metaclass hook that’ll be in Python 3.0 (and maybe 2.6)? It’s actually a pretty significant step forward for implementing Ruby-like DSL’s in Python.

I suppose this is the nut of the problem, at least as far as this discussion has related specifically to the technical aspects of Python: I’m not bowled over by the improvements Phillip cites.  They’re very useful and handy to the vast majority of Python programmers, but they’re not game-changers (which I suppose is what I meant by “significant growth”). I think the description of the metaclass hook as “a pretty significant step forward for implementing Ruby-like DSL’s in Python” is very telling. The facilities for building DSLs in Ruby are good in so far as they make it possible to get the job done, but they’re by no means conceptually complete nor functionally clean (as pointed out by jerf in the reddit comments), so taking a “significant step” towards implementing such facilities isn’t the whole ballgame.

Regardless of that detail, the point is that progress is being made in Python — just not in the vector I need. And, that’s OK. Which brings me to…

Sandbox Etiquette

After all has been said and done, my original post was a mistake, in that I exhibited a similar type and degree of technological selfishness as those who replied with Type A responses.  As some of my friends will attest, I’ve personally been unhappy with Python and its direction for a variety of reasons for months now, especially as I’ve sunk further and further into a class of problems for which Python isn’t particularly well-suited at the moment.  While I had settled on that conclusion some time ago, I’ve obviously been suffering from a mental block that caused me to do drive-bys against Python.  This came to a head with my blog post.

The more mature (and zen) thing to do would have been to simply go looking for a different sandbox, and leave well enough alone with regard to Python.  (It is, after all, a fantastic language and will likely remain my favorite for most common tasks [especially web programming] for a some time hence).  This is especially true given the fact that I am essentially a nobody in the Python community – I’ve contributed in my own small ways, but it’s not like I’m a core hacker or important library author.  Instead, I adopted the Response Type A attitude, but flipped it on its head, claiming that my favorite language should advance itself to suit my requirements, and to hell with the priorities of others.

So, let’s make a deal: I’ll stop sniping on Python, and maybe everyone else can stop making clever comments about “esoteric” language features.  Then we can all spend more time building bigger and better sandcastles.

In understanding Python 3, I think it’s equally important (especially for those of us who might not walk on the beaten path with regard to domains, types of apps, etc) to review PEP 3099, which outlines what won’t be included in Python 3.

Personally, reading PEPs 3099 and 3100 (which Paul references in his post) is depressing. I’ll explain why by example and contrast. Consider this post from GvR from about a year ago. Quote:

But please save your breath. Programmable syntax is not in Python’s future — or at least it’s not for Python 3000. The problem IMO is that everybody will abuse it to define their own language. And the problem with that is that it will fracture the Python community because nobody can read each other’s code any more.

This is predictable, especially given GvR’s prior comments on various topics surrounding more “esoteric” features like multiline anonymous functions, operator overloading, etc., etc., etc. However, compare and contrast that to this slideshow from Ruby’s Matz from about two years ago. I don’t have a money quote, but the difference in attitude and approach is staggering. There’s a line in Annie Hall (I think, or some other Woody Allen movie) where Woody’s character says that relationships are like sharks: they must continue to move forward in order to survive.

Now, I’m not saying that Python is dying somehow — far from it. However, I think it’s safe to say that it has stopped growing (and hasn’t grown significantly since v2.2 with the great class/type unification). Meanwhile, Ruby is “a white-hot nexus of innovation”, according to Tim Bray, anyway; and really, anyone who knows and cares about what Python is lacking in terms of expressiveness and capability would agree.

At this point, you might fairly assume that I’m some kind of Ruby booster, but in fact, about 90% of the coding I do these days is in Python, and that has been the case for almost 3 years. And as much as I enjoy working in Python, it has not (and looks like it will not) grow along with the problems that I need to solve.