Beware of Scripting

I recently read an IEEE Computer Society article "In Praise of Scripting". Having been a big Perl fan who ran into big problems with a not even sooo big Perl project I have a comment or two on the article.

In the article (and elsewhere) scripting is displayed as the method of choice for free, creative spirits who develop the groundbreaking, innovative (web!) software of tomorrow while the use of compilers with strict typing is the tool of narrow-minded syntax-nazis who develop unremarkable applications by committee.

I beg to differ, having used Perl heavily for about seven years between 1993 and 2000. Perl does an incredible job when a single programmer has to interpret text files and transform data. This is the kind of job I still often use it for today.

But scripting languages like Perl can be a big problem when a team has to create a really large application with complex data structures and elaborate functionality. During the dot com boom a lot of very successful software was written in Perl. What people might not realize is that maintaining and extending these applications was a nightmare, and I know people who quit their job because they could not get away from having to install and maintain these contraptions.

Perl is excellent for many things including shooting your own foot. Neat features can produce obscure errors that are very hard to find. In example, variables don't need to be declared, and have no type. Of cause it is nice that a variable just springs into existence when you need it (less typing, in example). Or that you don't have to care about conversions from strings to numbers and the other way round. Or that complex data structures can be created, extended, and modified on the fly because associative arrays give you all that.

On the other hand, what if you misspelled a variable name? Or the second parameter of a function was supposed to be the forename of a customer, not the year of his birth? Your program will produce odd results but no error. And you then painfully realize, too, that there is no IDE such as Eclipse to help you track down the mistake you made. The compiler won't help you either, because a = b + c is fine for it whether b and c are strings, numbers, or a combination of the two.

Some find it nice that Perl allows you to program in a number of different styles. You can make the code look like C or Java, or more like Visual Basic. There are a number of constructs that don't exist in other languages that allow you to express yourself even more concise, such as "...unless".

But since these constructs do not exist in other languages, people not familiar with Perl have a hard time understanding what is going on intuitively, and some constructs are just not very intuitive ("do this, this, and this - unless something is so and so" - you first follow that "this and that" will be done, but - surprise! - not under the following condition). Different styles, non-intuitive and surprising developments are great in a book or movie, but not in source code.

This article obviously is not exhausting the subject; I'll give it a shot another time.

Software Industry != Software-Industrialization

In my opinion somebody has been seriously confusing the production of material goods with the development of software.

Let's look at the example of Lego bricks. A lot of effort goes into setting up injection molding machines and a distribution network. Once the complex machinery is in place you produce millions of identical items by employing the same steps over and over, with very little human intervention.

This is not how software is being produced. The multiplication and distribution is the smallest problem. In the case of software created for a specific project and purpose the product might be installed only once. Pretty much all of the effort goes into the specification, design, and development of a single unique product.

Where is the industrialization in that? What is industrialization, anyways? Making a single item is expensive. Constructing a machine for creating a single item is even more expensive. But using that machine to make a lot of items reduces the cost per item enormously.

So what that is software industrialization then? The best analogy I can think is the use of tools like IDEs, debuggers, or maybe even MDA. And using standardized methods, tools, and processes. But the advantages of applying all this only go so far; creating software is done mostly by working with the head, not with tools.

This means that developing software is not an industrial process performed in an automated factory, but rather tool-assisted handcrafting in a manufacture, gladly employing sophisticated methods. This is not a bad thing; better methods and tools will help a lot to improve the quality of software, and to speed up the development process.

Breaking up architecture and development

From practical experience, the emphasis in "breaking up architecture and development" is on "breaking" - it does not work.

A customer I have been working for decided to split software development across two different organizational branches: project management, specification, and architecture on one side and software development and testing on the other. The transition to this organizational structure was made in a "jump into cold water" fashion.

The idea was that the high level and therefore expensive work would be done in Germany and the low level cheap work would be offshored to a "development factory" ("industrialization" was another frequently heard term, but that is another story).

This did not pan out as planned, for a number of obvious and maybe not so obvious reasons:

• Architects in that company often are programmers that are either totally out of touch with current developments, or clearly less than decent programmers, or aren't even programmers at all. IMNSHO an architect must have years of programming experience and must be a really good programmer with clear ideas on what kind of programming style promotes high software quality.
  
• Estimates where done independently by the architecture and the development team. Since both teams had different ideas about how the software would be designed and because these ideas never got synchronized the overall estimate often was the total of planned work for substantially different approaches.
• The architects had no real influence on the programmers. If concepts were not liked the development team simply priced them out of range ("if we do it like this (solution that we don't like), it will cost 30% more"). Project management did not have the necessary insight to call shenanigans or to understand and act on long term consequences.
• When architects and programmers are separate persons a huge amount of information must be transferred in the form of papers, talks, and meetings. It is much more natural and efficient to let the senior developers, possibly with some guidance, develop the architecture, and then let them move on to programming as the project continues.
  
• Developers in India are not just "coders", if that management idea of "low level programmers" would exist. They can and want to do architectural work, too. Intelligent programmers will resist to micro-management on the code level.
  
• The separation felt and worked like contracting development out to an external, untouchable company. Estimates had to be accepted like carved in stone; risk surcharges, profit margins, and baseline costs where added, and there was no direct talk to the programmers. Code reviews weren't even on the agenda.
• And - the whole thing just does not work out financially. The plan was to specify everything down to a pretty low level and then let the cheap "code drones" take over. The problem: You spend 50% of the time or more on expensive experts working out a detailed specification and then try to get your money back by executing the last 50% or less with cheap labor. And then you find that instead of ten programmers you need fourteen plus two group supervisors plus one off-site manager to get the same job done, and your customer is not willing to deal with documents written in English. So much for comparing hourly rates by dividing one through the other and drawing the wrong conclusions.
  

As you might have guessed, the split between architecture and development has been reversed after a year of pain and suffering, and surely a lot of money and enthusiasm lost.