(This hasn’t been a secret or anything (for good reason!), but I thought I’d put out an announcement post.)

Dr. Fry will be presenting “Computational Information Design” – a mix of his work in visualization and coding plus a quick introduction to the Processing language and environment.  Processing has had a huge impact on the field of data visualization, and Dr. Fry’s presentation will no doubt be enlightening for anyone who engages in data visualization at any level.

There will be refreshments.  There’s a Google Maps link on this page if you need directions; please note that the presentation will be held in the second-floor conference room, Suite 234.

Small afterthought: the three avid readers of my blog may recall that a similar event was held about a year ago, where we hosted Rich Hickey, creator of the Clojure language.  I think we (meaning Snowtide, Atalasoft, the Western Mass. Developer’s Group, et al.) have a pretty unique combination in this area of outrageously talented people with a collectively broad set of experience and specialties, and a relatively intimate environment where ideas or presentations can be fully fleshed out with lively feedback from everyone involved.  I think there’s some potential to build this foundation up into something very worthwhile; perhaps a regular flow of software wizards to give talks, show off their newest ideas, and recruit evangelists (zealots? ;-)).  Something to think about anyway…

(The impromptu talk happened much after Monday’s formal talks and presentations, and I don’t think that anyone was recording Prof. Sussman’s remarks.  If anyone does have a recording, by all means, post it, and I’ll link to it here — and probably just drop my paraphrasing.)

This is all from memory, so I’ll just apologize ahead of time for any errors or misinterpretations I propagate. If anyone has any corrections, by all means, leave a comment (try to keep your debate reflex in check, though).  In a couple of places, I’ve added notes in italics.  Just to keep things simple and concise, the following is written in first-person perspective:

When we conceived of scheme in the 1970’s, programming was a very different exercise than it is now.  Then, what generaly happened was a programmer would think for a really long time, and then write just a little bit of code, and in practical terms, programming involved assembling many very small pieces into a larger whole that had aggregate (did he say ‘emergent’?) behaviour.  It was a much simpler time.

Critically, this is the world for which scheme was originally designed.  Building larger programs out of a group of very small, understandable pieces is what things like recursion and functional programming are built for.

The world isn’t like that anymore.  At some point along the way (he may have referred to the 1990’s specifically), the systems that were being built and the libraries and components that one had available to build systems were so large, that it was impossible for any one programmer to be aware of all of the individual pieces, never mind understand them.  For example, the engineer that designs a chip, which now have hundreds of pins generally doesn’t talk to the fellow who’s building a mobile phone user interface.

The fundamental difference is that programming today is all about doing science on the parts you have to work with.  That means looking at reams and reams of man pages and determining that POSIX does this thing, but Windows does this other thing, and patching together the disparate parts to make a usable whole.

Beyond that, the world is messier in general.  There’s massive amounts of data floating around, and the kinds of problems that we’re trying to solve are much sloppier, and the solutions a lot less discrete than they used to be.

Robotics is a primary example of the combination of these two factors.  Robots are magnificently complicated and messy, with physical parts in the physical world.  It doesn’t just move forward along the ground linearly and without interruption: the wheels will slip on the ground, the thing will get knocked over, etc.

This is a very different world, and we decided that we should adjust our curriculum to account for that.  So, a committee (here, Prof. Sussman peaked his hands over his head, which I interpreted to indicated pointy-headedness) got together and decided that python was the most appropriate choice for future undergraduate education.  Why did they choose python?  Who knows, it’s probably because python has a good standard library for interacting with the robot.

That is my best paraphrasing of Prof. Sussman’s remarks.  I spoke with him briefly earlier today, primarily to ask his permission for me to post this sort of first-person paraphrasing; he replied: “Sure, as long as you paraphrase me accurately.”  Hopefully I succeeded; I’ll mention again my solicitation for corrections in the comments.

As a short addendum, while I had Prof. Sussman’s ear, I asked him whether he thought that the shift in the nature of a typical programmer’s world minimizes the relevancy of the themes and principles embodied in scheme.  His response was an emphatic ‘no’; in the general case, those core ideas and principles that scheme and SICP have helped to spread for so many years are just as important as they ever were.  However, he did say that starting off with python makes an undergraduate’s initial experiences maximally productive in the current environment.  To that, I suggested that that dynamic makes it far easier to “hook” undergrads on “computer science” and programming, and retaining people’s interest and attracting people to the field(s) is a good thing in general; Prof. Sussman agreed with that tangential point.

FYI, we collected $170 last night.  That covered all of our food expenses and then some — I think once I tally up everything, we’ll have a surplus of ~$40 (and we have a bunch of generic supplies that we can put to use in the future).  Thank you very much to everyone who pitched in in this way, too.  Hopefully we can keep that pot flowing.

Some highlights from the meeting, and random thoughts of mine, in no particular order:

• Doug gave what sounded like a rousing talk about the PHP templating system that he conjured up.  It seemed like most of the group really enjoyed that.
• I generally don’t touch PHP, so I hung back and talked about entrepreneurship and software business models with Lou, Maria, Michael, and….darnit, I can’t recall the name of the other gentleman that joined us.  Sorry, man, I can be bad with names at times.  Keep coming to the meetings, and I’ll straighten out, I promise.
• In the second time slot, I instigated a discussion about the current state of rich client platforms, through the lens of some particular requirements that we have for current/future projects.  That turned out to be pretty entertaining and productive, with a big chunk of time dedicated to people being impressed by the surface features of Titanium/Appcelerator.  That may be a good topic for future blog posts if we end up really digging into it.
• Just about everyone was down on Adobe Flex/AIR as being very unpleasant from an end-user perspective (widgets not behaving as one would expect, etc).  I unintentially sort ended up trashing on JavaFX — or more specifically, the current lack of an integration story between Swing and JavaFX, as well as the oddities of JavaFX script.  In the clear light of day, I feel like I should probably give it a closer look, simply because of our established JVM codebase.
• There was widespread speculation that a “shadow group” got together at Panera, despite all of the chatter and announcements about the change in venue.  Maybe next time (if there’s a next time here @ Snowtide), someone could swing by Panera and gather up those who aren’t as plugged-in to the group’s chatter.
• Gerard walked away with Managing Humans, graciously provided to us by Apress’ developer group book program.
• Will and I ended up holding on to the Terracotta book (also from Apress), though we promise to pass it on to Miles when we’re done!

It seemed like everyone had a good time and that most were pretty happy with the results compared to the usual Panera experience, but I’m clearly biased.  One way or the other, shout out what you liked and didn’t like (either on the mailing list or in the comments below).

FWIW, I’m happy to have Snowtide continue hosting the group’s meetings if people enjoyed the result.  If there’s a next time, get the shared conference room, and see how that works out.

Again, thanks to everyone who came!

Every now and then, though, a podcast lands in my iPod that involves a roundtable of venture capitalists.  VCs are often very dynamic, engaging people that are entertaining to listen to, but just as often, they say the most amazingly absurd things.  A recent roundtable podcast (entitled What is the Next Big Thing) really pegged the absurdity meter, though.

Addressing a gathering of Stanford students, alumni, and associates, three venture capitalists, Tony Perkins, Tim Draper, and Michael Moe discussed the recent economic conditions, with the bottom-line message that it is in “times like these”, when markets and economies look their bleakest, that the most successful and impactful businesses are often forged.  That’s an oldie but goodie — so far, so good.

Things go off the rails around the 13:20 mark, though.  One of the three speakers — I believe it was Tony Perkins, but these things are hard to be sure of in an ensemble podcast — relayed how Marc Andreessen (former founder of Netscape and now also a part-time investor) was talking with Charlie Rose about how the New York Times should just kill their paper version.  That’s no huge new idea, but that got Tony off on a slight tangent that led him straight into the weeds (bold emphasis mine):

A lot of the whole [dot-com] bubble period was based upon a vision of the Internet steamrolling the way people do business and creating what was then called the “New Economy”.  My theory right now is that all of those things we talked about that were going to happen, like the end of television, the end of newspapers, all that stuff that we poured a bunch of money in because we thought it was going to happen ten years ago is actually happening now.

So a lot of the destruction in the market, a lot of the jobs that are being destroyed, are jobs that are being steamrolled — a lot by the Internet — but increasingly by the “green tech” movement because entrepreneurs are looking at how we do everything, and they’re saying “how can I do that same thing in a way that is better for the environment?”.  That’s bringing the Silicon Valley mentality into the whole green space, which is super-exciting.

The reason I share your optimism is because we are the future.  Silicon Valley is the future; a lot of the jobs we’re seeing being destroyed are never going to come back, but it is our world that is causing the destruction, and therefore is going to be the one that creates the jobs.

Hey, I’m essentially a nobody, so maybe Tony’s really got the inside track, and I’m not seeing the forest for the trees.  But wow, the U.S. economy lost 598,000 jobs in January, including:

• 22,000 cut from Caterpillar
• 4,500 from Kodak
• 19,800 cut from Pfizer
• 5,000 cut from Microsoft (the first mass layoff in that company’s history)
• 2,400 cut from EMC
• 13,500 cut from Alcoa

Etc., etc.  Sorry Tony, these job losses aren’t due to Silicon Valley and VC-backed internet and green-tech companies owning the world and replacing Caterpillar’s earth movers and minimizing the need for Alcoa’s aluminum.  There are a lot of theories about why the economy is what it is of late (lending practices, creative derivative strategies, poor Federal Reserve policy, etc.), but honestly it never occurred to me that I’d come across anyone with the chutzpah to say that recent shrinkage (and reversal) of economic growth and the attendant job losses are due to internet and green-tech companies “steamrolling” the Old Economy¹.

Even more crazy to me is the notion that Silicon Valley is going to be singularly responsible for reinvigorating the economy.  It certainly has a role to play, and has had tremendous impact in the past, but from where I sit, Silicon Valley has been far too busy over the past couple of years building Web 2.0 trinkets to be ready with any kind of game-saver anytime in the near future.  Thinking (and saying) otherwise is good marketing within that particular echo-chamber, but it likely sounds like simple self-aggrandizement anywhere else.  (Hopefully there’s a stealth-mode clean energy startup that will prove me wrong on this point.)

I don’t mean to pick on Tony here.  Lots of other VCs have said similar things — it’s just that in this case, the usual VC rhetoric happens to bump up pretty hard into real-world facts and real-world struggle.  Big-picture notions about how entrepreneurship and innovation are the keys to building a stronger economy and a better world are good, but watch out for the odd notions that are borne out of the VC bubble (which seems to have its effect upon almost everyone that steps inside for a time).

My general point is simply that VCs say the darnedest things, and especially as it’s become clear that venture capital isn’t at all required (or even desirable) in many situations, one needs to be careful about how much of the VC worldview one takes to heart.

Footnotes:

¹Wow, typing “Old Economy” right there reminded me of back-in-the-day when Wired was raving on and on about the new economy and introduced The Wired Index consisting of 40 New Economy companies.  That’s classic entertainment.

In case you don’t know, Peter Norvig is the Director of Research at Google.  That’s interesting, but the real reason Dr. Norvig holds sway with me is his classic book, Paradigms of Artificial Intelligence Programming.  If it weren’t for that book, I almost certainly would not be doing what I’m doing today.  Its pages are where I came to understand lisps, and began to imagine what was possible and what I might be able to accomplish in computer science (final results yet to be determined, of course).  For that, I am extraordinarily grateful to him (and others, of course, but I’ll wait to talk about them when they get around to using some of my code! ).

Back to the story.  This morning, I decided to hop onto Google Analytics for a bit to check up on the traffic stats for our various websites.  Lo and behold, in the “top referrals” listing, I saw ‘norvig.com’; “Well,” I thought to myself, “that’s interesting!”   A quick grep of the server logs (is there a screen in Google Analytics that actually provides you with the full referral URLs?) showed the referral URL to be Dr. Norvig’s “post” from last week, An Exercise in Species Barcoding.

A search of my name on that page shows that he needed a way to calculate the Levenshtein distance (also known as the edit distance) between two large strings — his quick implementation (like most) operated in O(n^2) space, which would have required weeks of processing time in his particular case.  So, he looked around for a more efficient implementation, and found one that I wrote in October of 2003 that operated in linear space bounds (and was, ironically enough, my first-ever contribution to an open source project).  With a couple of tweaks to suit his specific needs, the code I wrote worked out nicely for him.

This story is satisfying and funny (for me, anyway) in a couple of different ways:

First, there’s the fact that (what I would now consider) throwaway work of mine floating around the nets six years later.  Remember kids, the Internet never forgets!

Second, it reminded me of what I was doing when I wrote that particular code.  I was building what would later become PDFTextStream’s first ground-truthing system¹ (although I don’t think I knew of that term at the time). It’s a lot more sophisticated now, but back in 2003, I was simply trying to set up a “ground truthing” system where the full (vetted and known-good) extracted text from each PDF document in our nascent test repository would be saved off somewhere, and later builds of PDFTextStream would compare its extracted PDF text to those saved files.

Of course, it wouldn’t be practical to require that PDFTextStream produce identical output forever — some amount of slop had to be allowable, because (for example) if an extracted word was outputted with four spaces before it instead of two, that would generally be sufficient.  For that and other reasons, I wanted to test that current PDF text extracts were the same as the known-good extracts within a defined margin of error.  Unfortunately, I was ground-truthing full document extracts at that time, and most Levenstein functions with their quadratic performance characteristics would take a lot of memory to diff the multi-megabyte strings that were involved.

Solution: write my own Levenshtein function (loosely based off of a pedagogical implementation by Mike Gilleland that had been incorporated into the Apache commons-lang project) that operated in linear space bounds.  Thankfully, I opted to offer the improvement back to the Apache commons-lang project and to Dr. Gilleland — had I not, Dr. Norvig would never had found that code, and I wouldn’t be writing this right now.

Third and finally, this story is satisfying because, hell, Peter Norvig used some of my code.  A person I respect and admire has found it convenient to use some minor thing I created years ago, and was thoughtful enough to say so.  I hope I can follow that example as I go along in my travels.

See, Dr. Norvig, I’m still learning from you.

Footnotes:

¹ Ground truthing is a testing methodology often used in document processing systems where ideal or otherwise known-good output is cataloged, and then actual or current output is compared to it to determine relative accuracy.  PDFTextStream’s current ground-truthing system serves as a semi-rigorous smoke test of its aggregate text extraction accuracy while we’re doing active development, as well as an ironclad regression test for when we’re looking to cut a release.  Thankfully, it’s come a long, long way from the very naive approach I was pursuing in 2003.

That’s cool and all, but we’re (mostly) JVM partisans here, so the inevitable question is: why doesn’t Sun use the same mechanism to drive deployment of the latest and greatest JRE/JavaFX?  Flash functionally has 100% penetration, so it makes tons of sense to use it to make it trivial to get the JRE out there, at which point Java’s “native” update functionality can take over.

I’m not a RIA guru by any stretch, so maybe there’s a good reason why this isn’t done?

Boston just doesn’t have the startup culture that the Valley does. It has more startup culture than anywhere else, but the gap between number 1 and number 2 is huge; nothing makes that clearer than alternating between them.

This has been picked up in a variety of areas by Boston-local entrepreneurs and those that watch that space.  The reaction has been predictable, if you know just how much of an inferiority complex people have vis á vis Silicon Valley (emphasis below mine):

Scott Kirsner:

That’s not just his opinion… it’s reality… and we ought to be addressing it head-on.

Buzz in the HUB:

Paul Graham has long been a critic of the Boston Venture community and their reluctance to invest in his crops of nascent startups. He has now given in to the fact that the Valley is a better place for (web) startups.

Robert Buderi:

…the full-time departure of Graham and Y Combinator is a real loss for the New England innovation community

The most grating reaction I saw was this anonymous comment left on Scott Kirsner’s post:

For people like myself who are on the cusp of creating something new, Boston’s conservative culture translates directly to greater risk and scarcer opportunity. The question always boiles down to: if you have a family, how do you take the plunge, even if your business intention is indeed groundbreaking? Is the risk too great in this town?? And if it is, doesn’t that mean a move to the Valey must be considered.

At this point, I should remind my two readers that I chose to start a software company in Northampton, a town in Western Massachusetts, a solid 80 miles from Boston, thankyouverymuch.  This fact is always disorients my Bostonian acquaintences, who generally have one of three reactions:

• “Heh, so you’re out past the tumbleweeds?”
• “What’s it like, living in the country?”
• “Wait, you mean Northampton Street in Cambridge, right?”

I fully realize that that’s just one way in which I’m outside of the mainstream of the “Boston startup scene” (the other ways include the fact that Snowtide is 100% bootstrapped, and that we’ve been profitable for years — also shocking, I know).  That said, all of the bellyaching about Paul Graham choosing where to spend his summers seems vaguely ridiculous.  It feels very similar in tone to the discussion I witnessed at the end of the 2008 MassTLC Unconference last October, where dozens of people in a room of perhaps 300 of the brightest entrepreneurial minds in the Boston area expressed varying degrees of concern, despair, and panic about how the “Boston scene” has fallen so far behind Silicon Valley.

Now, maybe there are fewer VCs in the Boston area; maybe the “startup culture” is more vibrant (for some?¹) in Silicon Valley; maybe Bostonians are technically and financially more conservative than their west-coast cousins.  I have no way of evaluating the truthiness of those assertions, having never spent any time in Silicon Valley.  However, it’s entirely reasonable to say that all of these assertions are wholly irrelevant to an entrepreneur’s objective: building a sustainable business² (or organization, if your aim is to innovate in the non-profit space).

A business is built out of innovation in a particular market, combining expertise and insight and networking and execution.  Nowhere in that formula is a requirement that some guys from Sand Hill Road need to come down and drop $20M in your lap so you can staff up.  Nowhere in that formula is a requirement that you need to have your offices down the street from 20 other entrepreneurs³.  And in today’s (or yesterday’s!) global economy, nearly all of your customers will be hundreds or thousands of miles away.  Innovative technology startups thrive in places as diverse as Chicago, New York, Austin, Seattle, Liverpool, and yes, even in smaller towns like Champaign, IL and Northampton, MA.

If I were to make a geographical recommendation, I’d say two things:

1. If there is some compelling reason why a particular city or locale would provide a significant advantage to your company, go there.  (e.g. you’re working on tide power, so it makes sense to be near the ocean, or your largest potential customer and first beta site is in Bismark, so you should move in down the street)
2. Otherwise, start your business where you’ll be happy and where you’ll find like-minded people.

If Silicon Valley is where you really want to be, godspeed; however, don’t move anywhere (or really, do anything) just because that’s where the crowd is or because some ostensibly-important person passes gas.  Innovation means thinking for yourself, doing what others think is strange, or foolish, or wrong, and being right in the long term.  Make sure you apply the same kind of independent thinking that will hopefully build your business over the coming years to where you choose to call “home” for that time.

Footnotes:

¹ My impression of Silicon Valley “startup culture” isn’t particularly positive.  Seeing trendy ad-supported websites hit $15 billion paper valuations when they might not be trendy with next year’s incoming class doesn’t make me respect the culture that produces such organizations.  If that’s the sort of culture you’re interested in, fine, but please call it the lottery that it is.

² If you’re an “entrepreneur” in the sense that your business is raising some VC, building buzz about a website, and selling out to Yahoo/Google/Microsoft/etc before you’ve banked a single cent of profit, I’ve nothing to say to you.

³ Although I will grant that such an environment does make for a more enjoyable after-work bar scene.  But, surely cocktail parties aren’t key ingredients in a technology startup?  If so, I’m going to become a carpenter.

That said, one downside of this is that many of our customers realize so much benefit from working with us is that they often are very reluctant to allow us to discuss our relationships with them in public.  So, far more often than not, we can have the satisfaction of realizing that big impact on a customer’s organization, but are barred from talking about it.  Given that, we are doubly appreciative of those that have worked with us to develop case studies on how they use PDFTextStream.

But last week, I discovered that someone we’ve worked with in the past – Neil Gandhi, who was heavily involved with Zinio’s deployment of PDFTextStream – had added a recommendation to my LinkedIn profile.  Now, most recommendations are very pleasant to begin with, but Neil’s was so effusive and unsolicited that I thought I’d share it here (oddly, LinkedIn doesn’t show recommendations on the “public” profile pages, but if you log in and view my profile, you’ll see Neil’s comments):

I worked with Chas during my time at Zinio LLC, a digital publication company that specializes in online and offline delivery of digital magazines. At the time, we were implementing global search functionality but our PDF text extraction solution was really sub-par. We found Chas at Snowtide and worked with him and his team in implementing PDFTextStream; their PDF text extraction solution. We were also testing against a slew of other vendors and open-source solutions to find the best product based on accuracy and service. You can find the case-study here: http://snowtide.com/cs-zinio

Needless to say, PDFTextStream was by far the most accurate solution, but to my surprise, Chas and his team provided the best service a small company like Zinio could ask for. I never had to wait more than half a day for a response and the questions and requests were always answered with a can-do attitude. If something couldn’t be done, Chas always had the time to explain why and also suggested (many times, better) solutions. He could talk the talk as both a CEO and as a developer and could switch back and forth when talking to my Director of Engineering, VP of Technology, and me and was more than competent at all levels. In the end we ended up purchasing the solution for all of our extraction servers, and I made a connection to someone I can always turn to when I need anything PDF related.

Pretty cool.  We’ll obviously be publishing more case studies as the opportunities arise, but it’s hard to beat comments as personal and unsolicited as those.  Thanks, Neil.

Normal view:

“Full Screen” view:

It’s only about 100 pixels or so, but that can be a lot when you’re doing some technical writing and are regularly scrubbing up and down a document as you go along.

Download / install: fogbugz_expand_wiki_editor.user.js

The phrase “burn it down” has been rattling around in my head incessantly since then, so I’m hoping to exorcise it here. Perhaps the reason why it’s stayed with me so long is that there’s a pretty good chance that that phrase has a lot of applicability to life in general. Looking back over the years, I’ve often found myself in situations where the key to survival and (ostensibly) success has been my ability (tendency?) to wage a war of attrition against that which stands in my way.

The most immediate case is a programming challenge: you’re sitting there trying to solve a problem for the umpteenth time, and the algorithm in your head or on the page in front of you is laughing at your inadequacies. Ideally, you’d like to step up to the whiteboard, and bang out a description of how this data structure should look, a proof for how that graph-walking algorithm can be sped up by an order of magnitude, whatever. I’ve seen programmers do this – they’ll pull out a small pad, and in about 8 minutes, they’ll knock out some pseudocode for an implementation of some algorithm that might take you an hour to comprehend, never mind implement.

Unfortunately, only a vanishing minority can seize hard problems like that on a regular basis; the rest of us are left wanting to be smarter than we clearly are. So, despite all of your hopes, your text editor simply stares blankly at you, the whiteboard is inert, and you’re left with two options: failure, or the last resort of the almost-overwhelmed, unyielding determination.

(This is very simply the same thing that Edison talking about: “Genius is one percent inspiration, ninety-nine percent perspiration.” Of course, I don’t think we’re in genius territory here…competency, perhaps?)

I’ve experienced that exasperation so many times while working on PDFTextStream and other Snowtide products. I’m fond of saying that I’m a math idiot, and while few people believe me, it’s entirely true. There are things in PDFTextStream that required me to slowly, painfully learn more about esoteric corners of computational geometry than I would ever care to know otherwise, and the same goes for various other specialties in mathematics and computer science. For me, understanding formalisms in those fields is like feathering a chicken by running it through a keyhole. The only way I’ve been able to ship a single bloody line of code has been to Burn It Down.

Of course, programming is a very narrow domain, but I think the same strategy is the only tool that us humans have to use in coping with the crush of life and disorder that surrounds us. Through growing up, getting an education, finding a job, starting a business, sustaining a life partnership, raising children, and coping with the march of time, the only common thread of human endeavor is rage. Even that majickal 1% of prodigies and bona fide geniuses have areas of their lives where their only hope of enduring is to simply endure. In a way, it’s comforting to know that despite the pretensions of some, the only sure way to succeed in programming (and likely even loftier domains) is to lay siege to (or perhaps, depending upon your mindset, to practice judo with) whatever challenge is ahead.

The day is young. Burn, baby, burn.