Discussion: Why do most people seem to use inferior programming-languages?

My last article created a relativly high number of comments, which I will try to discuss in this post.

The article was intended as my 'answer' to the topic which is very common in all places where discussions about programming languages takes place: Why do certain ones succeed and others not. I think this is an important question because most people who create new languages want to have some kind of success with it.

"'the market has to be right by definition.' is a wrong assumption" or "the best things seldom win"

I understand why people think this this way. They simply fail to see the 'big picture'. If you only look at certain aspects of a topic you can easily overlook important parts of the reason which are outside your field of view. This is particular true here: If you only look at programming languages by looking only at the language itself, you can easily get a wrong impression. We always have to look at the whole system, not only at a particular part of it.

A often used example is the win of VHS against Beta. Beta was technically better and still VHS succeeded. Isn't that a proof that the best thing will not automatically win? Only if you fail to see the whole picture. Technological superiority is only one part of the whole. Others are prices, license-fees, patents, available support etc. Beta failed because of Sony's licensing strategy. VHS was 'free' and could produced by companies without having to wear the straight jacked Sony tried to put them in. Whole People like 'better things', they don't pay every price for it. High-end fans may hate this, but most people simply don't care this much for quality that they will pay a much higher price for it.

So if a 'inferior' technology succeeds over a seemingly 'superior' one, then there are always good reasons for it. And believe it or not, but this this reason is never stupidity of the users.

"Companies are eager to adopt new things, but the problem is the scarcity of competent workforce."

This is the reason why it always takes some time to adopt new tools. But it can't explain why certain tools never catch on. Remember that the job market is also a market. Unemployment is a risk in most countries and people always want to get a good paid job with good working conditions. So the 'workforce' does compete for the best jobs like the companies compete for the best workers. But its always an optimization process: The average developer has to decide if it really pays to spend lots of time into learning a certain language. People always have to weight the options against each other. Of course this strongly depends on the area you're working in. It's a completely different picture if your working in academics, if you're doing programming as a hobby or if you're working in the industry. If people only look at one of those areas, they can easily come to wrong conclusions.

If now a company tries to adopt a new language, it's like the mentioned car manufacturer which starts to invest in robots: It costs a lot of money and it will take some time for the investments to pay back. Switching languages because of higher productivity also means that there is an initial higher cost for qualified workers and training. But only in the beginning. Remember that people learn their job not in school or in university, they learn it when they do their jobs. Sure, companies would really like to hire experts only, and some may even afford to pay the price for it (like Google in the moment). But thats not a model for the whole industry because most companies simply can't afford it or aren't 'attractive' enough. So companies have to do their share in training their workforce them self or they will never get the personal they need. Most companies really do this. And this is independent from the language you train people in. Some companies are even using obscure in-house languages which are totally unknown outside the company, because in certain domains this can be the most productive way of doing work - but you can't expect that they are able to find programmers with experience in this language, so they have to train them in-house.

And if you hire a 'fresh' Java-programmer for example, you can't expect that he/she is able to do good work from the start. Java may look simply to some people, but doing good work in it isn't that simple either. So the company has to do training and this is the reason why those people start to work as junior programmers which are supervised by senior programmers who know more about the job. And after some years, the junior starts to do good work and can eventually become senior himself. The reason why this seems to look different in other areas of programming (like Lisp, Haskell etc) is that companies which try to adopt a new language are often startups or companies which are in a phase of rapid growth or transition to a different market and need good workers quick. So they pay more to get the seniors directly without having to train them them selfs. But this is only a phase, it will never work for the market as a whole. And this has also happened with Java in the beginning.

And if the company succeeds because of using a different tool, their competitors will notice this and start to use this more productive tools also. This will in turn noticed by the Job marked, creating higher demand for this new language, creating more workers able do do their job in this language with lesser training. And in the end, the industry has again made a transition, even if there are of course still companies who use the old tools for various reasons.

"Lisp/etc. is simple? You must be joking."

No, I'm totally serious here. I know that there is the big fallacy that mainstream languages are more simple than less mainstream ones. This may really be true for some of them, but these are relativly rare exceptions.

Sure, to use a different language, you need time to adopt. But even switching from C++ to Java is much more difficult than certain people assume (and I suspect that those people don't know much about at least one of those languages). While the languages seem quite similar on the outside, to really master them you have to use quite different methods of programming. And if you even switch from C to Java, the step is at least as big as switching from C to Lisp or to Smalltalk. The syntactical similarities between C and Java are quite deceptive. It's a marketing gag, designed to give Java a bit more momentum in the beginning, but after you're over it, you see that both languages really haven't much in common.

Now users of certain non-mainstream languages seem to like the idea that their languages are 'to sophisticated for the masses' giving them a false sense of superiority. This way of thinking may give a good feeling, but it's nonetheless utterly wrong: People don't use Lisp because it's so difficult, they use it because they don't get their job done as good as in other languages. Because of missing tools and (especially some years ago) because of missing performance. The best language isn't useful if you have to reinvent all the wheels you can get for nothing if you use another language. This isn't of much interest for 'fans' of a language, but it's quite important for people who use a language as a tool to reach a certain goal.

If I compare Common Lisp and C++, Common Lisp is more simply in every area: Creating useful macros in C++ is much more difficult using cpp and templates than using Lisp-Macros (which is really a piece of cake compared to C++). Closures are also quite simple, in fact they are very similar to objects (just imagine the captured environment as fields of a class which is created on the fly). What's left? Maybe the syntax, but after you got accustomed to it (may take a week or two) it's hardly a problem anymore (especially if you compare it to the mess, heavy duty C++ template usage can create). In Lisp you don't have to care about memory allocation, you don't have to worry about pointers, you have simple access to basic data structures like lists etc. So if we only count language simplicity I would always choose Common Lisp before C++. So please, Lisp users: If you really think that your language is so difficult, please tell me the reason, why. I simply don't see it.

And Smalltalk? Yes, I've used it (some years ago). It's very simple to learn but like every language you need to learn all the patterns and idioms which takes a lot more time than learning the language itself. The main reason I've not continued using it were: Smalltalk-environments where proprietary and quite expensive at this time. There were only really ugly looking Smalltalky GUI-libs. It was hard to impossible to create stand-alone applications. And the resulting code was much to slow compared to C++ (which I primarily used at this time). Some of those things have changed in the meantime, but if you compare it with Java, Smalltalk is still behind (and it's old. I have the theory that languages make a breakthrough relatively fast or never - and Smalltalk's time is simply gone by).

"What about Ruby. It's slow but it's still a success"

Ruby is still far from being mainstream. The only reason for it's success in a very small domain is 'Ruby on Rails'. And again it's not the language which created the success, it's a tool (a framework in this case). But I think that this still isn't enough in the long term. While RoR can really give you some productivity benefit, that only works in it's small domain. And because of Rubys poor performance it also don't scales well. While using RoR for small scale web-apps can be quite successful, this is only a marginal market. And because the RoR crowd is quite 'loud', it may give wrong impressions on the real spread of RoR usage.

Something similar is true for Python. It's successfully in certain domains, but it's still far from really 'big'. Python also have a big performance problem which limits its success. And while Python IDEs of course exists, they're still far away from Java ones. But the main problem of Python is it's poor performance.

And remember: The perception created by all those bloggers and web-discussions can really lead one astray. Those people (me included) are not 'common programmers'. They do this stuff not only as job, but also as a hobby. And because of this, they are simply 'louder' than the broad masses of normal users who only read about it and do it primarily for work. So academic and hobby users simply have more time and fun to write about the topic and because those people are much more likely to try out new things instead of using the 'boring old stuff', it creates the impression that certain new things (which appeal more to a 'geeky' audience) are more successful than they are in reality.

"Use Erlang"

Some people mentioned Erlang. I think it's to early to talk about it. Erlang really shines in the moment you need massive concurrency - but this isn't really important in most domains yet. But it will change in the coming years and so maybe Erlang will become a 'Big Thing'. But we still need good IDEs and better performance until then: Whats the use of scaling an app to 1000 processors if the result can hardly compete performance wise with a C++ solution running on only 10 processors? Remember, that Erlang is quite slow as long as the program isn't using massive numbers of threads.

"Java isn't slow"

No, it's not slow, but it is still slower than C++. Even if it's in a relative close range. From my tests and other benchmarks (like the shootout) I think that well optimized Java code is about 1.5-2 times slower than well optimized C++ code in general. In certain areas Java can be faster than C++, but I consider this as an exception which seldom shows in a bigger program.

In Java the compiler (the JVM) has to do more runtime checking (Array overflows, Null refs, maintain stack-frames, etc) and this simply shows. Also because of the GC and the jit compilation it needs more memory which reduces performance too. The difference is not that big, but in combination of long startup-times and higher memory consumption, it's still enough to give Java a clear disadvantage in certain domains compared to C++.


"Language XYZ is much better than Java if you want to do ABC"

I know that most existing languages have their niche. But I'm not talking about those niches, I'm talking about the mainstream, about the 'Big Thing'. But Mainstream is something which is 'bad by definition' for certain people, so I'm sure, that every language which reaches the mainstream will always be bashed by those people. Just because it's mainstream and because bashing the mainstream let many people feel superior. Also if something becomes mainstream, it attracts lots of bad programmers. Like every kind of successful 'underground' music-style attracts bad musicians in the moment it starts to become mainstream which then start to produce more and more crap.


"IDEs and libraries followed Java, not the other way around"

Java 1 wasn't really used much and Java really started to catch on maybe from 1.2 upwards. The original idea of sun was to use Java applets to revive the server market which was crumbling because the desktop PC seemed to win. This haven't worked out (main reason was MS which fought back) and so they decided to switch to the server market instead (because of the now more and more successful Internet). Now Java could really start to gain market share and this was around 1999 with Java 1.2. At this time good Java IDEs where already available (NetBeans and VS integration) and IDEA (which later was copied heavily by Eclipse and Netbeans) was on the market around 2000. Creating an IDE for a 'new language' was of course risky, but if you want to make money finding niches is a good idea, even if you always have the risk that the language won't catch on as hoped.

But I think that Sun did a very good job in "bootstrapping Java". They provided comprehensive libs, tools, documentation etc. People or companies who want to make their language a success should take a good look at it.


"The real reason are network effects"

True, this can lead to the eventual success of a language and it's of course necessary in the end. But to make a language a success we first need tools, libs and docs. All those can created by several methods: By the language creator, by the community or by a 3rd party which is interested in promoting the language. Independent developed languages mostly use the 'community method' because the creator of the language isn't able to create all the necessary infrastructure himself. But first you need to have a community and for this you need something to attract them. And you need to attract the 'right' community. Having one which creates lots of 'tempest in a teacup' may work to let people notice your language, but in the end you need a community which actively creates libs, tools and docs or your language will fail.

But in the end network effects aren't so important for programming languages. It's much more important for libraries, interfaces, operating systems or hardware. A programming language is highly invisible: If you have created a product, nobody really cares in which language it was written as long as it does what it should do. Now you may argue, that network effects are more important in maintenance of a program. But I doubt this: Maintaining a product requires often much more time for the maintenance programmer to get to know the program good enough do do its maintenance thing, than it would take to get to learn a new programming language.

"Lisp and Smalltalk have IDEs too"

Sure. I've never implied, that a IDE is the one and only thing required to make a language successful. But it can help a lot. How much depends on the language. For Java an IDE is probably more important than for Ruby - but even Ruby can really profit from a good IDE. That Lisp and Smalltalk haven't succeeded has other reasons.

"Java isn't that productive"

Those big frameworks like J2EE may look a over-engineered today, but remember the time those frameworks where created. There were reasons for it. Of course many mistakes have been made, but that's because this stuff was quite new at those times. Now you may say 'Look how much crap it all is', but this is always easy if you have the advantage of knowing the outcome. Also many of the concepts of worked out well - and those are now integrated into many newer systems, Java based or based on other languages. Of course we've not reached end of history. There will be new systems, new languages and new frameworks, and people in 10 years will laugh about the stuff we call 'high-level' today. Thats natural.

Java was quite productive in the past and it still is. Sure, in certain domains you can even be more productive, for example by using RoR. But what if you expect the system to grow? Can you still recommend RoR then? What if you have to integrate legacy applications or huge data-bases which can't be easily refactored? Then RoR isn't as useful anymore. Similar things are true for many other systems. Java may be the least common denominator, but it works in a very broad range of domains. You may find a better solution in each of those domains, but what if your project touches more than one of those? Or if you're not sure, how you have to scale and extend if afterwards? So it's totally understandable if people choose to use solutions which may be not optimal in one domain but can work in all directions you have to travel later without the need to reimplementing the whole system (that's also the main reason why many people like platform independence so much).

There are lots of 'benchmarks' which try to measure productivity. But all those have one thing in common: They use simple toy examples instead of big real-world ones. And because of this they aren't useful if you need to decide what language to use for your big real-world project.

And people don't like to use many languages at once. It makes life harder and reduce productivity because of all of the additional interfacing you have to do. So if you start a project you choose the language which is overall the best fit and not one which is optimal only for a certain part of the project.

"There are big sites which don't use Java"

Sure. Why not. But having a big site doesn't mean that the language used to implement it has to be mainstream. It's like saying "XYZ has won the Formula 1-WM so XYZ must be the best car manufacturer".


"[This article] explains exactly why people use inferior programming languages, even though the intent was to say that they don't."

The intent of the article was to show that choosing a programming language not only depends on the language, but also on the whole infrastructure. Only looking at the language is like only looking at the engine if you want to choose the right car. So you can buy a car with an inferior engine and still get the best overall product. So people may use an inferior language but still use the superior system.


"People only use Java because they are forced to"

While I have to use Java for certain things because of external constraints in the moment, I'm free to choose the language I want for lots of other things. But despite having tried out lots of different languages, I always fall back to Java. I don't even like the language much because of all the well known limitations, but the system as a whole is still hard to beat. This may of course change in the future. Maybe even tomorrow. But I'm talking about the 'now and here', not about promises which may or may not be fulfilled sometimes.

"It's all the fault of those evil managers"

This is a very subjective view. It's true in certain very big companies, but loses its validity in smaller ones. Of course the 'manager' is always the 'bad guy' to a programmer. Like parents are often the 'bad guys' to the children because they have to order them around. But companies have hierarchical structures because they work best (at least until now, otherwise other structures had succeeded in the past). And this means that nearly every person in a company is ordered around by some other person. And of course mistakes will be made in this process. But everyone who knows it better has the freedom to build his/her own company. Just do it and you will start to experience constraints you may never thought of, when you worked as a programmer. Managers have those constraints too and they also have to live with them.

There are ways to overcome this, but this means that programmers have to 'face reality' and can't only concentrate on programming anymore. But this would also make them less productive as programmers and in the end this may be the reason why we have this separation between management and development. It may be a bad solution, but other solutions seems to work out even worse.


"The article is badly written, the grammar is poor"

Sorry, but I'm not going to write my articles in German and hire a professional translator to translate them into good English. I do my best, incl. using a spell checker to correct mistakes, but English really isn't a language which is easy to master (like probably every natural language). I would really like to see Esperanto or Lojban as 'world-language' instead (to level out the playing field), but in the moment it's English and the native speakers simply have to bear with us non-natives.


To come to an end: It may be possible, that in this moment, Java isn't the most productive language anymore. We will only know this in some years, but it's impossible to know today. And yes, in certain domains Java is for sure not the most productive language - and maybe never was. But that's not the point. The point is that most companies and programmers use Java, because it is (or at least was in the near past) the best overall solution for most programming tasks. If you really think that 'your' language is the 'next big thing' then ask yourself the following questions:

• Can my language compete with Java productivity wise if Java is used by a good Java developer using all the available tools and libs?


• Can my language compete with Java not only in a single domain, but in most domains important to general programming today?


• Has my language comprehensive, easy available documentation which is directed at everyday-use and not only for beginner or 'esoteric' stuff? Which has both concise parts for reference purposes and also explains concrete usage for learning purposes.


• Has my language a GUI lib which enables me to creates good looking GUIs which can compete with professional software at least on the main platforms (Windows, Mac, Linux - ordered by importance)?


• Can I write code which is fast enough to use my program on a bigger scale or use it on the desktop.

If one of the answers is 'no', 'your' language can still become a success, but if there are more no's, it's highly improbable, even if the language is full of cute concepts, has a beautiful syntax and is a pleasure to use.

Why do most people seem to use inferior programming-languages?

If you read discussions about programming languages, one topic is quite common: "Why do people use an inferior language like Java/C++/what-the-hell and not a superior language like Lisp/Haskell/Python/you-name-it"?

The reason is: The language is only a small piece a the big scheme. Todays languages aren't that different at all, productivity wise. While some things have a certain impact (having a garbage collector for example), most things in the language don't really matter in practice to a big degree and are largely overshadowed by the real important things: The availability of tools, libraries, documentation and the performance of the resulting code.

Imagine two competing car-companies A and B. A decides to invest in better automation buying robots, while B shys away from it because if the initial investment and continues to use human workers. After 2 years the investment of A starts to pay out: Productivity increases, the quality of the products gets better and the production costs are reduced. To stay competitive, B now simply has to catch up or lose market share and eventually even disappear.

I don't see a reason why software production is different: If company A can be more productive with a certain language, then the competition has to catch up or lose. That's how capitalism works. As long as we have relativly free markets, all companies have to strife for the best tools to do a job. And this really shows: Not so much time ago, most programming was done in C and later C++. But today Java is the most used language. The industry is obviously able to change to a different language if it really gives an advantage. But why has it moved to Java and not to Lisp, Haskell or Smalltalk?

There are some people who seem to think that most people simply are to stupid to grasp the complexity of those languages and thus using the more simple Java. But this is a fallacy, because Lisp and Smalltalk's simply are not more difficult to use. In fact Smalltalk is a much more easy to use language than Java. You can learn Smalltalk in a day and be quite productive after 2 weeks, something which is impossible in Java.

And Lisp can be quite simple, too (at least in the 'Scheme flavor'). Especially if we take a look at C++, the most used language not long ago, it becomes obvious that simplicity has nothing to do with it: Even Common Lisp is a piece of cake compared to C++. Not only the language is more easy to use, also the concepts used by the language are more simple (People who can use templates won't have problems with Lisp-macros after a short time of accustomization for example).

So if simplicity was really the reason to choose a certain language, nobody would ever have used C++ and the world would probably program in Scheme or Smalltalk today.

The next fallacy always heard of is the stupid manager who choose a language based on the 'marked leader'. But again this doesn't seem to be correct: When Sun created Java, it was no big market leader. The leader was Microsoft and MS used C++ and VB. But still Java made an impact and became the 'big thing'. MS has to actively fight against it, but still partly failed and had to jump on the bandwagon later by creating it's own 'Java' to get market share back. So it's quite short-sighted to use conspiracy theories where the 'market-leaders' make out which the 'next big thing' is and everybody else has to follow. If only a single company would abandon this 'agreement' and start to use a more productive tool, the others have no choice than to catch up, like our car-company B above.

The real reason why Java is still the number one language in the market is that it gives the highest overall productivity. Of course this isn't true in all areas, but it has to be true in the areas where the majority of developers work. But what are the reason for this?

I look at many programming languages and also try out a lot. I've done this for years. And still I use Java as my primary language. You can call me a stupid Java-Joe or whatever you like, but then you will overlook the real reason why I and many others still prefer Java despite the fact that they know most of those new, fancy, shining languages.

Before I used Java, my primary language was C++. I've switched to Java relatively late when version 1.3 came out. But the main reason I switched wasn't the language. I've looked at Java directly after it becomes available - and found it horrible. But years later I tried out JetBrains IDEA and it was unbelievable how much productivity this IDE gave me compared to VC++ I used for C++ development. While VC++ had 'Wizards' for code generation, those Wizards only created 'code-skeletons' which were only useful as a starting point but are no big things over the whole live cycle of a project. But in IDEA all those little helpers were usable in all phases of development. Many of the uglities the Java language has simply disappeared when I used this IDE, and I suddenly could concentrate on the problem instead of fighting with the language.

But maybe the biggest advantage was that the IDE removed the 'documentation-problem' I've to face in almost every other language. If you write code you constantly have to remember names, parameter order etc. By using IDEA with Java this wasn't necessary anymore. The very intelligent code-completion and the integrated documentation-lookup removed the necessity to remember all those things. Finding the right method or class was most often only a key-press away. And because of the real-time error-checking compile errors were nearly eliminated, too. In the end IDEA (and later also Eclipse) made Java a completely different language. Yes, the IDE matters. A lot. Just consider to write Lisp-Code without a REPL or Smalltalk-Code without the Smalltalk-Browser if you're a user of one of those languages.

So the real reason why I switched over to Java was not the language itself, it was the combination of the language, the IDE and the available, well documented and comprehensive libraries. In fact the language was the least important one of those reasons. This hasn't changed: Java without the tools is total crap. Most languages are better. But Java is so 'IDE-friendly' that it was possible to create an IDE like IDEA.

Not every language allows this. The dynamic ones like Python, Ruby, Lisp and Smalltalk always have the problem that it's mostly undecidable at compile-time which methods are available for a certain variable. But hey, that's not a bug, that's the big feature. It's the reason those languages are called 'dynamic', and an Java-IDE is in principle nothing else than an interactive extension to the compiler.

But there are other problems in certain languages. For example Haskell: It's a really nice language to create small, cute programs. The language is like a puzzle, it makes your brain hurt - but in a good way, like your muscles 'hurt good' after a good workout. But if I work to build a house, muscle-ache from to much lifting heavy stuff isn't that funny anymore. And the same is true for doing 'real stuff' in Haskell: Just look at the code of a big project implemented in Haskell, like the sources of the GHC. I don't see 'beauty' anymore in this code. It's hard to read and it's BIG. The type-checker of GHC alone is nearly as big as the whole code of Hugs - but Hugs is written in C while GHC is written in Haskell. Sure, GHC has some more extensions compared to Hugs, but this still should make you think. Also Haskell has a big documentation problem: The Haskell docs are absolutely insufficient on many levels. This alone is enough to ensure that Haskell will never become mainstream unless it changes.

And for other languages there is the performance problem. Yes, performance does matter. People sometimes spend hundreds of bucks to get a 50% performance increase of their hardware. Do you really think they wouldn't care if a program is 100 times slower because of using a certain language? Sure, if you write a script which is used once a week, it really doesn't matter it it takes 0.1sec or 10sec to finish. But if you look at web-apps, a factor of 100 can easily cost millions because you need to buy and maintain 100 times the number of servers.

For desktop application even Java is often still ruled out compared to C++ because of performance reasons. And Java is in average only 1.5-2 times slower than C++! So how could a language like Python or Ruby compete here? The answer is: They simply can't. You may use them in parts of a project, for example for scripting, but the majority work is still done in one of those boring 'previous-generation' languages. And again it's not because the developers are stupid, it's because nobody would buy a program which is as slow as if you made a step back in time to a 486, even if you have invested in a modern dual-core CPU.

And there are the libraries. Libs can make or break a language. And one of the most important one is the dreaded 'GUI-lib': If you want to create a desktop application and a language has no GUI-libs which allows to create good-looking(!!) and easy to use GUIs, the language is ruled out. And since most developers have to create GUI-apps from time to time and don't like to switch languages over and over again, no or only poor GUI-libs often means that the language won't be considered, even for projects which don't seems to need a GUI. This may be inconceivable for some people, but it's a fact of life: A language without good GUI-libs will never become mainstream. This also nearly broke Java's neck, because AWT was horrible and Swing took it's time to become useful, but with 1.3 Swing was decent and from 1.4 up it was good enough (if you use 3rd party look&feel libraries) for many purposes. But even if Swing is relativly good, its deficiencies (especially the non-native looks) still hinders Javas breakthrough on the desktop.

So even the best language can become totally useless if it lacks on of those: Competitive speed, good documentation, comprehensive libraries, necessary tools (like IDEs). If language fails to address those points (and really all of them) it won't become the 'next big thing'.

That's the big secret. Nothing more. Create a language as beautiful as you want. But without the necessary libs and comprehensive documentation: No chance. Slow: No chance. Not competitive if compared to other languages including tools like IDEs: Again, no chance.

So if you really want that your favorite language becomes more used or even mainstream: Help to solve those problems. Lamenting in blog-posts or online-discussions is as useless as spending lots of time writing tutorials explaining concepts from 'your' language to the world. The best tutorial and the most cunning style of discussion is useless in the moment a potential user discovers that there is no library for a certain problem he has to solve (like creating a GUI), or if the novice has to face incomprehensible and lacking documentation in the moment he/she tries to do something real with the language.


[Update:] I've written a follow-up article which discusses some comments and objections to this article. You can read it here .