I just published my photos from the European Common Lisp Meeting 2008 in Amsterdam.

BTW: Edi Weitz has a good list of “after-action reports” here.

I am invited as a speaker to the ECLM 2008 in Amsterdam. This is the abstract of my talk:

“Using Common Lisp for Large Internet Systems”

The internet is full of interesting problems for computer scientists. Large internet systems tend to be used by hundreds of thousands to millions ofusers. Many of them “serve the long-tail” with millions of products or a large numbers of other content items like text, video and sound. Always, the main problem to solve is to help the user to find “the needle in the haystack”. Obviously, such systems need intelligent algorithms and efficient implementations.

This raises the question, which programming language is the right tool to implement such systems. Currently most large internet systems are written in Java, which is not exactly famous for its briefness. Neither is C++. And more and more people are starting to use Ruby (on Rails) because “Ruby is an acceptable Lisp”.

Paul Graham said: “Lisp’s power is multiplied by the fact that your competitors don’t get it”.

How could we improve, if we used Common Lisp to build internet systems? This talk provides an in-depth analysis of current and future trends in web/internet programming and shows how to use Common Lisp as a tool to implement intelligent internet systems using the functional paradigm and sometimes even artificial intelligence algorithms. I will reveal what kind of Common Lisp systems we at freiheit.com technologies already have in production and what we are currently working on.

As a follow-up to my posting “Software estimation considered harmful?” I want to talk a little bit about seeing a software project as a combinatorical optimization problem.

Do you know the Knapsack-Problem?

Imagine you are living in Papua New Guinea in the rainforest. You want to trade some of your „cargo“ (see Jared Diamond, „Guns, Germs and Steel“) at the market in the next big city. It is a two-days walk and you have to carry your cargo in a backpack. You separate your cargo in different boxes. Each box has a different weight and a different price per item.

You can‘t take all your cargo with you, because your backpack would be too heavy. So you have to decide which boxes you want to take with you. Of course you want to take the boxes with the lowest weight (i.e. the lowest costs) and the highest possible price (i.e. the highest value).

Which combination of cargo has the lowest costs and the highest value for you?! (To be able to solve problems like these gave our ancestors an advantage in the process of natural selection and evolution of mankind).

Let us now talk about software projects. For each software project you have to answer the following questions:

1. When will the project be finished? (Time between project start and end, delivery date)

2. How much will the project cost? (Costs, mandays and/or money)

3. Which features will be included? (Expected functionality in this time/cost period)

Question number one is almost always fixed. Normally the customer knows in advance until when the software must be finished to be useful. Normally you can‘t change the delivery date. This means, that you have time constraints.

Question number two is almost always fixed, too. The customer has maybe planned an IT budget for a new system. Anyway, nobody has infinite time to deliver a product. So you have some kind of budget constraints.

The answer to question number three requires hard work, systematic analysis and intelligent thinking: You have to go and find out for yourself, what your customer really needs. I don‘t mean you should simply just ask your customers for his software requirements.

You have to dig deep to find out this:

Which combination of features has the lowest cost and the highest value for your customer? (To be able to solve problems like these gives a software company an advantage in the process of natural selection and evolution in the IT-business).

If you have a fixed budget and a fixed delivery date you better find out what has to be delivered to make your customer happy and successful. Your customer expects the best system that can be delivered in time and in budget.

To map this situation to the before mentioned Knapsack-Problem:

The time (working days) between the start and end of the project is your backpack.

The maximum weight you can carry in this backpack is equivalent to the mandays budget for your project. (Number_of_developers * Working_days) < Mandays_budget

And your „cargo“ are the features you have to implement. Each feature has a cost (the mandays needed for implementation) and a value attached to it. The value can be difficult to define. Sometimes you measure the value of a feature in relation to its implementation costs. Sometimes you measure a feature on how much profit it will generate our how much cost savings your customer can achive. Giving features a priority to determine the value is often easier: If a feature has a high priority for your customer, this normally means that the value is high and vice versa.

This is the way how we see software projects. It leads us to a simple planning algorithm:

1. Find and describe all features that the customer needs. (What is a feature?)

2. Estimate each feature. (Minimum estimation granularity 0.25 mandays, maximum 5.0 mandays)

3. Attach a priority to each feature. (Must-have, want-to-have, nice-to-have)

4. Make a releaseplan. (Solve the combinatorical optimization)

5. Implementation. Evolutionary Delivery every three weeks. Never skip or change a delivery date. Monitor your progress and anticipate risks. Check your planned estimations and actuals every day on feature level. This continuously improves your ability to make better estimations. This way, you’ll find problems long before they turn into risks.

To get into the details, I have to write more Blog postings. Stay tuned.

A couple of days ago I read Peter Seibel’s Blog-Post “Software estimation considered harmful?”.

In a nutshell, he writes that he was reading the book “Software Estimation: Demystifying the Black Art”. The complexity and pre-conditions needed to produce software estimations with some of the described techniques really made him think, but in the end he was mostly sure that developers should do estimations.

Then he met his friend Marc (who is running a small internet startup, Wesabe) telling him, that he makes no estimations at all. He just tries to excite his fellow programmers for specific features and they then just do it: “Marc’s point of view, [as] I take it, is that the only reason to do these big projects is because you have to, and if you have to, it doesn’t really matter how long it’s going to take”, Peter writes.

This brings Peter to the fundamental question: “Why estimate?” and from this to the question “Why set targets?”In the end, he sees software estimations more like a communication tool.Funny question, eh?

I started programming in school in 1983 and I did my first professional (for money) programming job in 1987. I studied computer science and systems analysis and I am running my own software company since 1999 with over 50 programmers now in january 2008. In my whole life, I never had a customer who was contracting me or my company without asking for an estimation in advance. And of course they expect that you will deliver as planned. Only if you are your own customer, you maybe can get away without doing estimations.

So the answer for me to the question „Why estimate?“ is „Because your customers will expect this from a professional company“. But the answer to „Why set targets?“ is (maybe surprisingly) „Because it is much more fun to work“!

When you want to write great programs you should look for people who are smart and get things done. Smart is important, because it simplifies communication. You maybe don‘t have to explain fundamentals and you can exchange ideas much easier, because they are understood faster. More signal, less noise. But smart people are not always getting things done. Getting things done is about making stuff work. Finding the most simple and elegant idea/solution to a problem and extending it from that point on. For me „getting things done“ also means „reaching measureable goals“.

Why is programming more fun, when you set and reach measureable goals (targets)?!

It gives you freedom. You don‘t have to spend so much time to actually manage the project. This means less administration and more programming time, because when you set measureable goals and everybody in the team knows what to do, you don‘t have to talk about this so much anymore. You need less coordination, because you don‘t have to dispatch daily work-packages to your team-members. They can do it themselves.

You don‘t have to complete the work of others anymore. If today you know how to get things done, but others in your team don‘t, then maybe you have to complete their work to finish the project in the end. If you set goals and work with people who know how to reach these goals, everybody can concentrate on their own work. Or at least you find out early, if others can‘t meet their goals. This leads to the next point.

You are always in control of the situation and you can adapt to problems very fast. If the time estimation granularity for each goal is not too small and not too big (from my experience less than 5 mandays, 2-3 mandays in the average), then problems in the implementation will pop-up very fast and you can adapt your estimations based on the new experience. You can help people having problems reaching their goals. It‘s some kind of a natural risk-management. Instead of just reacting to problems, you take your destiny in your own hands.

You are more focused. And this makes you faster. When you set goals and you reach them, then you know what‘s behind you, what has already been accomplished. Then you can focus on the future and what still has to be done. This is actually really satisfying. Having a measure of what has already been accomplished and what to do next.

Your customers are happy. Customers are happy, when you deliver as promised. And when your customers are happy, then you will be happy, too. They will recommend your service and they will pay your invoices fast. This leads to a positive cash-flow and this will make you independent. And independence is one ingredient of personal freedom.

This is the normal way to accomplish something: If you want to reach a goal, you should decompose the goal into a hierarchy of sub-goals. You estimate the time/costs required to solve the sub-goals and then you (the team) tackle sub-goal by sub-goal. When all sub-goals are completed, the top-level goal must be completed, too. This is how you plan the birthday party of your kids, it is the way to build a house and to plan a holiday trip. Why do so many people think, that you can‘t plan a software project?

Maybe because they have a misconception about the meaning of estimation and planning or maybe it is because they had a bad experience in their life as a programmer when they were forced by their managers to make unrealistic estimations and stick to them?!

You should see it like this: To estimate a software project does not mean, that you have to foresee the future. It just means that you are making time-based assumptions and predictions. And „Project management“ means, that you check every day if your assumptions and predictions are still correct and that you have to refine your plans if something is going wrong. If you do this every day, your predictions get better and better and you don‘t stumble over so many surprises anymore.

Of course, you should be excited of what you are doing. But I think, that this is a pre-condition for every expert in some area: You are really good at stuff that you really like. I wonder how Mark is exciting his developers at Wesabe about not so exciting features. Excitement is not exactly a prioritization method.

My fellow programmers and myself couldn‘t live without setting goals anymore. We are setting these goals for ourselves, because it is so much more fun to work. And as a by-product we get estimations our customers can rely on: If it works for us, it automatically works for them, too.

In the last 10 years we developed our own software engineering methodology which is based on the assumption, that software project planning and estimation is really a combinatorical optimization problem. Since 1999 we delivered all of our projects on time.

Since about 2001 we have an internal system to manage our projects, features, estimations and releaseplans, called Captain Feature. So we are not only having years and years of experience in doing this, but we also have a big database of completed software projects together with their planned estimations down to the feature level and the actual time needed to do the implementation.

So, we have the data to support my theory: It is possible to estimate and plan a software project. And it can be done in a very natural, programmer- and at the same time customer-compatible way.

I will talk about this in more detail in a later posting. And I guess I also have to write about „The Peopleware Papers“too, because I am in doubt that some of the wisdoms are supported by enough data.

A couple of blocks away from Booksoup we found this nice retro-style advertisement. Maybe for Californian people the high visibility of Yahoo! and other large internet brand names is normal, but in germany it is still not.This is actually what I like most when traveling to California and especially the San Francisco Bay Area: It feels like the internet world is seamlessly integrated with the real world. The first time I traveled to the U.S. was in 1996 or 1997 to visit the first JavaOne conference. I was amazed, that almost all advertisements displayed internet addresses. Now people can’t even remember the time, when there was no internet. 

After visiting Singularity Summit 2007, I spend some days off with my wife in Santa Monica, L.A. We went to BookSoup, a nice Bookstore on Sunset Strip. I spent a lot of time in my life in bookstores and libraries and I really don’t like the big “fast-food chain” bookstores. When you enter a bookstore, you can feel if the owner really likes books. I think this is one of the places.

Inside the shop, there were several post-its sticked on the bookshelves. These were ratings/recommendations of books from the staff and from customers. Especially this one is interesting. It shows two opposite opinions on the work of Noam Chomsky:

This year I went again to the USA to visit the Singularity Summit. (I wrote a Special Report of the first Singularity Summit last year).

We arrived late on the first day, so I couldn’t seen all the talks. Later I downloaded the Podcast of Rodney Brooks Keynote “The Singularity: A Period Not An Event.” Very entertaining. At the end of the Q&A session he is also talking about changes to be made to the geneva convention, when armed robots are deployed in war zones, as “some countries are respecting the geneva convention”. This led to the question, if he should really build robots for the US government… You can find it here.

The second day started with a keynote from Peter Norvig which was quite entertaining, too. Below you can see the essence of his talk and what he thinks what are the prerequisites of Artificial General Intelligence (AGI). In short: Probabilistic First-Order Logic over lots of data. First it sounds like that maybe only Google has access to “this lots of data”, but now they share it with us. You can get approx. one trillion word-tokens (n-grams, from unigrams to 5-grams) of text from publicly available web-pages in 24GB gzip’ed text files over the Linguistic Data Consortium.

In the Q&A session someone asked Peter if he has maybe observed “emergent intelligent behavior” in the large server networks of Google. I am not sure, if this guy was just joking…

The other interesting part was again the american pragmatism: American Venture Capitalists are already interested in the possible business created by the Singularity. In germany, this is unthinkable. I guess most german VCs don’t even know what the Singularity is. And if they knew, they would think about it as science fiction… or scientists gone mad.

So I agree with Rodney Brooks: The Singularity will not be a moment in time. It is a period. And it is not only about intelligent machines, but also about human transformation. Maybe the last people who will be transformed are the german VCs or our government (we shortened the distance to russia and singapore).

In 2006 we launched H.W.A. as a communication platform for news about the freenigma service, our pet project effort to bring e-mail encryption to the masses - together with GnuPGs Werner Koch.

The freenigma service is quite successful - even though it is not the perfect solution for encrypting e-mail, because it manages both the public and the private key in one place. It is for normal people who have not the technical background to install and run things like GnuPG or PGP and who use Web-Mail. These people have no encryption at all or they can use the freenigma service.

Before freenigma service, we developed another pet project, the freenigma appliance. This is a free-software alternative to the proprietary systems from PGP Corp. and Utimaco. We transfered the project and its code to Inmedias, a company well-known in the free software community in germany. Inmedias will market the freenigma appliance in the future.

Now we have room for “new adventures” on H.W.A and we celebrated this with a new layout for this blog which is more close to our corporate design.

We have better quality videos here: AVI and Flash.