Is 'power' really what we want most from a programming language?

Power seems to be a nice thing to have. But what does it really mean? Think of the president of the USA: He's considered quite powerful because (in principle) he is the one who can 'press the button'. But with all his power to destroy, he still can't create peace in the middle-east. That's because power isn't the same as productivity. Power enables one to exert force - but often this simply isn't enough to reach a certain goal.

Now back to programming. Lets take Common Lisp. It's a language which gives you lots of power: You can program in nearly every style you want, forcing your way through all paradigms and programming styles. You have the power to control nearly everything, from the reader to the compiler to the semantics of the language. This language really has power!

But what about productivity? For all this power, the number of applications written in Lisp is quite small. You may argue, that this is a result of political reasons or simply because "people don't get Lisp". But if something is really so much better, then it will wipe away political or psychological inhibitions with ease. Just because people can't afford to give their competition an advantage. But if this is true, why hasn't all this impressive power created equally impressive results?

The answer may be that "power" is simply the wrong metric here. What we need to look at is the "fitness" of a programming language. As we all know, "fitness" is the measure which really counts if lots of things are competing against each other. Fitness is the driving force in evolution, it made the living world the way it is now. And evolution isn't a concept which is limited to biological systems alone: Every system which allows for the concepts of "selection" and "mutation" evolves. Societies, companies in a free market - so why not also programming languages?

But if we now take a look into animals kingdom, it's easy to see that "power" don't seem to play a role in deciding which species succeed and which become extinct. Where are the big and powerful dinosaurs now? What about the big and strong mammoth? Instead of them, it's the small and relatively fragile cockroach which could be considered as a huge evolutional success. So fitness seems to be everything but power.

Back to programming: What determines fitness in the realm of programming languages? A programming languages “lives” if people use it. So what do people really want from a programming language? What makes them use it?

Maybe those three: Productivity, accessibility and usability.

• "Productivity" seems to be a very important one. In the end most people use programming languages to create some kind of application, in other words: "To get the job done". Sure, there are people who use a language just for fun, and this may create a evolutional niche for languages which have properties like "beauty". But living in a niche isn't the same as success. It is "productivity" which determines if a language can be used with commercial success. If it's more a toy or of a tool.


• "Accessibility" determines how easy it is to get started with the language. A powerful, but also very complex and thus rather inaccessible language has it much harder to convince people to use it. Also a language which is only available commercially and costs a lot of money? Its accessibility goes down dramatically. Same for languages without good documentation and tools. It depends on its "accessibility" if people start to use a new language or stay in the one they're familiar with.


• And there is "usability" which determines if using the language feels good or like a nightmare. This has also a lot to do with the language infrastructure: Tools, libs, documentation etc. But also with the syntax and semantics of the language. All this also determines productivity, those three points are kind of overlapping (but still distinct). In the end "usability" decides if people are happy to use a language and stay or if they look around to switch to another as soon as possible.

What about "prevalence"? In fact this is only a secondary property because for a language to become prevalent it first have to succeed without. Once it reached it, a good prevalence rate contributes a lot to let a language stay alive. But don't make the mistake of overestimating this property. Every language starts small and has to compete against more prevalent languages first. PHP had to compete against Perl, Java against C++ and Cobol, Ruby against Java. Every new language which is able to step out of its minority niche had to prove itself first in the previous three areas, so those are the more important. But "prevalence" explains the inertia in the world of programming languages because once reached it can increase the first three factors. But those are still the real important ones.

And where is "power" now? Can it improve those three factors above? Yes, but it also can be a hindrance. So power is a only secondary means, what really counts are the results of power in the above mentioned categories. Like in nature, power is always a question of balance: More power power may make a creature more competitive against more powerful creatures in the same ecological niche - but it also requires more energy throughput which can be fatal in times of scarcity. In programming languages more power can increase complexity or micromanagement which would in turn decrease all the above three factors. So instead of using "power" as a primary means in discussing and comparing programming languages, we should more concentrate on the real important properties above.

With this in mind it's easy to explain why certain languages had big success in the past while others only live in a niche. Like in the biological world, every species survives as long as there is some kind of space where it fits in better that it's competition, so there's always room for lots of programming languages in the world. But only few will be able to break out of their niche and have general success. Let's look at some languages in detail:

Lets start with Common Lisp: Why is it a niche language now? Lets look at the three factors from above: "Accessibility" (what makes people start using a language). Big problem here is the syntax. All those parens... This is no problem after you got used to it, but for the beginner it's simply hard. "Usability" (what make people stay with the language). This depends. For me usability is bad because of the lack of static typing (look here for the reasons. And yes, I know, there are ways to get static typing in Lisp, but I still have to use 3rd party libs which are created without and Lisp simply don't work well if you use static typing. It's simply not “the Lisp way”). And productivity? Not that good either (for me for the same reasons) but of course it depends on the developer. And of course on the money you're able to spend because good Lisp systems (which would allow be work productively) where quite expensive in the past and some aren't really cheap even now.

Now a rather successful one: Java, of course. Java was very accessible from the beginning. It's familiar C-like syntax combined with a rather simple language definition made entry a piece of cake. And it was freely available from the beginning. Sure, usability and productivity weren't perfect right from the start, but things go better fast (especially after MS stopped to mess-up Java and Sun got full control over the language). And after Java 1.3 productivity was very good (mainly because of the huge available libraries and tools and because Java had garbage collection). And usability was great too (mainly because of the powerful Java-IDEs, the good documentation and again because of gc). So it's no wonder why Java became a big success. Things have changed a bit in the meantime because of changes in the technological landscape which starts to hurt Java. But it will stay for some time, simply because of it's prevalence.

Another big OOP-language was Smalltalk. Why wasn't it more successful? Again: Accessibility. This time it was primarily the availability of programming environments which prevented its success. Smalltalk-systems where very expensive and they only worked in their closed 'world'. And usability? Quite good - but only if you were able to shell out lots of money and could afford to neglect OS specific stuff. Smalltalk applications were hard to deploy and always foreign on the desktop. And at the time, Smalltalk has it's high, desktop-apps where 'the thing' (this nearly broke Java's neck, too, but because Java was a little bit later, it could concentrate on the again upcoming market of server-side-programming). So while Smalltalk could be quite usable and productive in principle, in practice it was only in it's small ecological niche while outside other languages (like C and C++ at this time) ruled the market.

Now at last, a look at a more current language: Haskell. It's still to early to make a final judgement, but in the moment it doesn't looks good to me. The language has bad accessibility because of it's complexity. Its usability is also not that good because of bad documentation, lack of good IDEs and certain hard to use language features (I wrote about this some time ago). And productivity doesn't seem to be that good too for the same reasons. The language has for sure big appeal to the research community and to people who like brain-twisters, but this isn't enough to make it a general success.

I could go on and look at a lot more languages this way, but I think you can do this by yourself. Just try to be free of prejudice and don't let your personal preferences blur your view. Can you guess what the "next big thing" may be?

To come to an end: The criteria above may not be the best. Maybe you can up with better ones. But one thing is for sure: "Power" is as much overrated in programming languages as it is in nature.

Static vs 'dynamic typing', part 2: The personality-factor.

After talking about some of the basic nomenclature I will look now into the pros & cons of static typing. But instead of going into pure technical details I want to look at it from a less common angle. What if it's not really a technical but more of a psychological matter?

Let's start by looking at my personal experience: I've used languages without static typing often enough to know that they don't fit into my style of development. I've used Lisps, Ruby, Javascript and played in various other languages without static typing. But it never felt good. And I never was really productive. Sure, when I started a project, everything went fine and I enjoyed the freedom I had without thinking about static types. But as the project grew bigger and bigger, I started to miss static typing more and more and I also needed more and more time for rewrites and 'relearning' my own creation. With static typing I have no problem creating and maintaining programs 10000s of lines in size, but without it seems that above 1000 lines my productivity is dropping dramatically. Like I got stuck in a tar pit.

Of course this is only my personal experience. It may differ for other people but I suspect I'm also not alone with this kind of experience. So what if the question it's not simply a pure technical thing but depends much on the personality of the programmer? That there are personalities which can work well without static typing while others struggle?

Static typing creates some kind of 'safety-net'. It protects you from yourself, forcing you into thoroughly thinking about your design. Without it, it's very tempting to use short cuts instead of solid design (which seems to have always have some overhead in line-count and artificial complexity, at least in the beginning). I am a lazy person and because of this I often take those short cuts - and regret it later.

Now this is also true to a certain degree if I use a statically typed language. If I enter new territory and come across a problem I've never had before, I try to get a working version as fast as possible to try out things and learn more about the problem. It's rather uncommon that I'm able to create something solid in the first try. I could try to work things out on paper, but in the end I've always overlooked crucial things which tend to pop up only in a real implementation. In most cases my first implementation kind of works but is full of bad hacks that it will be rewritten or at least heavily reworked later until it it ready to stay.

But what has this to do with static typing? Static types create global constraints which describes your program in a non-local way. This creates a skeleton you have to fit your program into. If I have to solve a sub-problem which is new territory for me, the rest of my program creates a solid framework in the background which can lead the process while the IDE detects mismatches instantly. From my experiences with languages without static-typing this typed-framework combined with instant-feedback is the thing I'm missing most and which cost most of my productivity.

Static typing also helps a lot if you're refactoring your code. I'm not only talking about automatic refactorings, but also the good old cut&paste-refactorings. Just cut some code out and paste it somewhere else. And the IDE instantly highlights the unresolved names, type-errors etc. which shows where the code has to be changed to fit into the new place. I'm refactoring my code constantly. I rename, change signatures, cut code into pieces to assemble it new elsewhere, extract methods etc. I always 'sculpt' my code, forming it in place. As I write new code I always refactor old code. Because of static typing this nearly always works without introducing new errors and can often even be done semi-automatically by the IDE. If I write code in languages without static-typing I refactor much less. Now every refactoring is a risk, may corrupt my code without noticing it. So it takes more time because I have to be more careful. And even if a mistake shows up by a failing test later I still have to find the source of the error instead of simply checking-off one error-highlight after the other.

The funny thing is that people tend to think that static types hinder exploratory programming. But with my style of work I can explore things quite easily with the abilities to restructure my code again and again, instead of first thinking about how to write for example a macro abstracting things out. So static typing makes exploratory programming even easier for me (but only thanks to the IDE - without it doing all those things by hand would be quite cumbersome) and in the moment I have to work without, I feel bounded and forced to think much more ahead instead of simply exploring things.

Many people praise the 'REPL' ('read-eval-print-loop', a way to execute code immediately in many non-static-typed languages) because of it's way to instantly try out things. But a IDE with instant error-highlighting is even faster in pointing out mistakes - and works with all of your code, even if you don't suspect a mistake.

REPLs are nice and fine, but they need to be used explicitly be the programmer. This requires to enter and write additional code and it also works only if the code kind of works. Instant checks based on static types work automatically and even if the code is still incomplete. So instant checks based on static types can be much more immediate and faster, while 'dynamic checks' like tests (done in the REPL or with testing frameworks) only work later if you're already finished a piece of code.

This leads to the disadvantages of TDD ('test-driven-development'): Tests must be written explicitly and thoroughly or they aren't of much use. This takes time. And if you change your design you also have to rewrite all your tests. And that's is my problem with real TDD: It's again my laziness. Writing and maintaining those lots of tests is essential for this method to succeed - and I'm simply to lazy to do this.

I always let my tests stay behind instead of leading my designs, testing only the most important things instead of doing the kind of fine-grained testing which is necessary to make this methodology work. I always think "hey, this code is simple, why test it, I shouldn't have made any mistakes" - and later I find a typo or something similar stupid. But with a static type-system which let the IDE quickly point out where I made mistakes this happens far less. And it scales well with the size of the program. In small ones the overhead of searching for typos or simple stupidities isn't that big. But as bigger the code-base grows the longer it takes to find those little mistakes. So without static typing I've lost my most important tool to write new working code and it's not wonder why I'm not very successful without it.

The same is true for documentation. Static types in combination with expressive names are often already enough documentation. Sure, for more complex parts of the code, writing additional documentation is mandatory. But again the problem is to hold it up to date with code changes. With static typing I use the types as part of documentation and write only documentation which explains the non-trivial parts of a method or class (like algorithms, how-to-use etc). So static types are an excellent tool for compiler checkable documentation which never gets out of sync with your code. Without static typing the documentation is optional and lazy people as I am tend to omit it to often for their own good.

I think here is also the reason why there are people who can work well without static typing: If you're very disciplined (or work under strict project management which enforces the rules without exceptions), TDD can probably be a feasible alternative to static typing. And if you don't refactor that much as I do (because I am often to lazy to create the right names and interfaces in the first try and have to change it later if the code stays), the advantages of safe refactorings are quite useless for you.

But for lazy people as I am, discipline has to be enforced by the language. And this is what static typing does. It's not perfect because it can't find all kinds of errors. But it forces me to create clean designs and solid interfaces. It points out flaws in my designs and forces me to think about them. It gives me immediate feedback and it is also useful as documentation. It makes refactorings easy and let me clean up my code more often. And the things it can check are not only checked, they are even proved, so I can rely on them (compared with TDD which can check a wider range of things but with much less guarantees because I never know if the tests were correct. I can remember spending hours on 'broken code' discovering later that the code was correct but one of my tests was faulty).

So the prime reason why people can't stop to argue about the topic is probably that people are different and have different ways to work. If you're a disciplined person who writes as much code for tests as for the 'real' program, you will benefit from the freedom a language without static typing gives you, without getting stuck at a certain complexity as I do. If you tend to write working code in the first try, you won't refactor that much and rely more on building different abstractions using the more dynamic nature of a language without static typing. But if you see comprehensive tests as a distraction from the 'real work', while at the same time you're not having problems with thinking more about overall design, creating interfaces and specifications, then static-typing is the way to go because it fits much better to your way to work. Even if it requires additional code and stands sometimes in the way of creating a better abstraction.

And I think that's also the reason why most 'enterprise-programming' is done in static typed languages: With many different people creating and maintaining a project often over decades, it it much to risky to depend on the personality of the individual programmer. And the situation is also asymmetric: People who can work well without static typing can also do productive work in a static typed language. But a programmer who works better with static-typing can totally mess-up a program written in a non-static-typed language. So the risks are much smaller if the language used has static typing. And I'm also quite sure, that the number of programmers who can work well without static-typing is much smaller than vice versa because people in general and programmers in particular are more on the lazy than on the disciplined side.

So if you argue next times over the question 'what is better', please keep in mind that people have different ways to work and to think and that there may be no 'one right way'. To make people most productive, it's important to acknowledge their way to work. Static typing is an additional tool which is very helpful for some people but may also be to restrictive for others. Try out both and see what fits you best. As programmers we all rely on tools. Nobody would be productive today by entering machine-code with switches as in the early days of computing. So there is no disgrace in relying on a compiler or an IDE to do your job. Find the tools which help you most without feeling forced to adopt a certain style which is 'hip' in the moment. It's important to learn new things and try out alternatives constantly - but in the end, all what really matters is to get the job done.