Mittwoch, 17. Februar 2010

How a Clojure pet project turned into a full-blown cloud-computing web-app

I just made the slides of my Clojure talk available on Slideshare. The meeting took place at the newthinking store in Berlin and it was sponsored by Franz Lisp. Thank you! And "Thank you" to Hans Hübner for organizing this event!
Next "Berlin Lispers Meetup" will be cool, too. Speakers are Luke Gorrie and Edi Weitz. http://netzhansa.blogspot.com/2010/02/berlin-lispers-meetup-tuesday-march-2.html

Montag, 21. Dezember 2009

Weihnachtszocken at freiheit.com 2009

Last Weekend, it started all over again: We celebrated our traditional "Weihnachtszocken" (zocken is german slang for gaming) in our lovely "Gartenhaus" (a part of the company complex with its own garden).


While initiating the event, there was a new attendee record in the air. Finally we had 23 (!!) registrations for gaming. Of course, on the event day we had some failure based on self-updating Windows-Operating-Systems (damn you!!) and overlong game installations (Next time: start some days earlier ;-) ). Theses failures just have lead to maximum 19 simultanious gamers. But hey, still exceptional!

Our lovely "Gartenhaus" was a perfect gaming location as usual, nearly everybody used the big screens and we had a working network with internet flat and separated from our valuably internal server systems, which should be secured from our windows infected gaming pcs ;-).


Like every year we played on the same great game: Team Fortress 2. It's an ego shooter, which makes it easy for beginner but nevertheless offers a lot of strategic challenges to the more experienced gamer.


To sum up it was a great day with a lot emotions:


Freitag, 13. November 2009

Facebook's changing face over time

I work as a systems administrator at freiheit.com. In order for our developers to test the systems they build with different versions of Internet Explorer they asked me to check out a software called "Internet Explorer Collection" [1].  It installs every Internet Explorer from 1.0 to 8... needless to say I needed to try out current sites with old flavors of IE, here are some screenshots of Facebook in the different browsers:


Hmmm, JavaScript wasn't implemented by Microsoft when IE1 was released. It first appeared 1995 with the Netscape browser. Here, the JS-sources are shown within the page...

On to IE2, still no JavaScript:

IE3: Hooray for JavaScript (at that time called JScript by Microsoft due to trademark issues), but apparently CSS support is only in its beginnings:

Then comes IE4. Basic CSS support is there, but not much going on otherwise:


IE5 to the rescue! CSS support is somewhat better, JavaScript also works. But Facebook kept telling me that my Browser is too old...


Needless to say I wasn't able to login with all of those versions. I won't bother to show you screenshots of IE6 - IE8, I'm sure you can find some machines around that have those versions installed.

When playing with the different versions, I frequently got this message dialog, which shows up every time data is stored in a Cookie (a.k.a. "Internet information stored on your computer"). At that time, this was the default behavior.


And guess what page worked in most of the versions? Of course! Google.com, here is a screenshot from IE3, with the suggest box in the top left corner:

Oh well, the good old times...

1: http://finalbuilds.edskes.net/iecollection.htm

Donnerstag, 3. September 2009

Our corporate WebSite runs on Clojure and Google App Engine!

Today we launched our new corporate website.

Our objective: Being able to post new project stories and other information about our company faster and easier. And we wanted to provide a new navigation system based on a tag cloud to make the usage of the site more fun.

So we first wrote a custom content management system. When we write software for ourselves, we use Common Lisp or Clojure. This time we used Clojure, because we wanted to run the system on Google App Engine (GAE). We are quite happy with the result. :)

Some technical details:
  • Text and Images are stored in the Google Datastore. Forget about relational databases.
  • When an image is uploaded, we scale it automatically with the GAE image service.
  • We use memcached to cache the page content. It is quite fast.
  • We use client-side Google Translation to translate the site into 9 languages.
  • The GAE deployment and versioning is easy and fun.
  • We had very, very few problems with GAE or Clojure.
Visit www.freiheit.com

Check out our post about interactive programming with emacs, clojure and app engine!

Freitag, 28. August 2009

Interactive Programming with Clojure, Compojure, Google App Engine and Emacs

Clojure is lisp running on the JVM. Interactive programming with Clojure is fun. Just hit C-c C-c to compile a defun in Emacs. If you come from Java, you will appreciate this: No long Build-Deploy-Run cycles anymore. But if you want to write Clojure programs for Google App Engine, it looks like interactive programming is not possible. Really?! No! This tutorial shows, how to develop applications interactively lisp-style with Clojure, Compojure, Google App Engine and Emacs.

The Google App Engine SDK includes a development server (based on Jetty) that allows you to test your application in an environment that is close to the real one. This is nice for a pre-flight check but for every change you make in the code you must stop the devserver, recompile and start the server again. You can't develop your code incrementally and interactively as you're used to in a lispy language.

We'll presume that Emacs and Clojure is configured, working with Compojure is familiar and that the App Engine SDK has been downloaded and installed.

John Hume created a nice little clojure binding for appengine that we'll use here.

This is the example servlet:
; a basic application using the google app engine
; when this file is created it will create a class that extends
; javax.servlet.http.HttpServlet which can be mapped in the
; applications web.xml.

(ns helloworld
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.http compojure.html)
  (:require [appengine-clj.datastore :as ds])
  (:import [com.google.appengine.api.datastore Query]))

(defn index
  [request]
  (let [items (ds/find-all (Query. "item"))]
    (html
      [:h1 (str "Hello World. There are " (count items) " in the database.")]
      [:a {:href "/new"} "Create another one"])))

(defn new
  [request]
  (do
    (ds/create {:kind "item" :text "something"})
    (redirect-to "/")))

(defroutes helloworld
  (GET "/" index)
  (GET "/new" new))

(defservice helloworld)


This file compiles to a servlet class that can be used in app engine (see the introduction post about clojure on app engine for a more detailed example). If you navigate your browser to "/" when the devserver is running you'll see a page displaying the amount of "item" entities in the datastore. If you click on the link you'll create a new entity and are redirected to the index page.

In order to develop the code more interactively, we'll create another file that will start a Jetty server. We'll call this file start0.clj:

; starting the application in a jetty.
; the original application is decorated by a function that sets up the
; app engine services.

(ns start0
  (:use helloworld)
  (:use compojure.server.jetty compojure.http compojure.control))

(defn start-it
  []
  (do
    (run-server {:port 9090} "/*" (servlet helloworld))))


Compiling this file with C-c C-k in Emacs and calling the function start-it in the REPL will start the Jetty. The application is available on port 9090.



If you view the application in the browser, there's a problem.



More work is needed. The services aren't initialized yet. We'll fix this in start.clj:

; starting the application in a jetty.
; the original application is decorated by a function that sets up the
; app engine services.

(ns start
  (:use helloworld)
  (:use compojure.server.jetty compojure.http compojure.control))

(defmacro with-app-engine
  "testing macro to create an environment for a thread"
  ([body]
    `(with-app-engine env-proxy ~body))
  ([proxy body]
    `(last (doall [(com.google.apphosting.api.ApiProxy/setEnvironmentForCurrentThread ~proxy)
    ~body]))))

(defn login-aware-proxy
  "returns a proxy for the google apps environment that works locally"
  [request]
  (let [email (:email (:session request))]
    (proxy [com.google.apphosting.api.ApiProxy$Environment] []
      (isLoggedIn [] (boolean email))
      (getAuthDomain [] "")
      (getRequestNamespace [] "")
      (getDefaultNamespace [] "")
      (getAttributes [] (java.util.HashMap.))
      (getEmail [] (or email ""))
      (isAdmin [] true)
      (getAppId [] "local"))))

(defn environment-decorator
  "decorates the given application with a local version of the app engine environment"
  [application]
    (fn [request]
      (with-app-engine (login-aware-proxy request)
      (application request))))

(defn init-app-engine
  "Initialize the app engine services."
  ([]
    (init-app-engine "/tmp"))
  ([dir]
    (com.google.apphosting.api.ApiProxy/setDelegate
    (proxy [com.google.appengine.tools.development.ApiProxyLocalImpl] [(java.io.File. dir)]))))

;; make sure every thread has the environment set up

(defn start-it
  []
  (do
    (init-app-engine)
    (run-server {:port 8080} "/*" (servlet (environment-decorator helloworld)))))


Before running jetty, local test implementations of the services are registered. Also, every request is wrapped by a function that registers the services for the current thread. Now the application runs as expected:



Google includes test implementations of the services in the jar files "appengine-local-runtime.jar" and "appengine-api-stubs.jar". So these two must be in the classpath of the clojure that is run from Emacs/SLIME. Be careful not to include these two jars in the war file that you deploy to the app engine servers. The application won't run!

Now that you've got a setup like this, incremental changes to the code are very simple: Try to change the text in the function "index" and just recompile the function with C-c C-c. The change should appear immediately if you reload the page in your browser.

Now you can work interactivly with Clojure and Google App Engine!

Dienstag, 27. Mai 2008

Photos: ECLM 2008 in Amsterdam

I just published my photos from the European Common Lisp Meeting 2008 in Amsterdam.

BTW: Edi Weitz has a good list of "after-action reports" here.

Dienstag, 22. Januar 2008

European Common Lisp Meeting 2008, April 20, Amsterdam

I am invited as a speaker to the ECLM 2008 in Amsterdam. This is the abstract of my talk:

"Using Common Lisp for Large Internet Systems"

The internet is full of interesting problems for computer scientists. Large internet systems tend to be used by hundreds of thousands to millions ofusers. Many of them "serve the long-tail" with millions of products or a large numbers of other content items like text, video and sound. Always, the main problem to solve is to help the user to find "the needle in the haystack". Obviously, such systems need intelligent algorithms and efficient implementations.

This raises the question, which programming language is the right tool to implement such systems. Currently most large internet systems are written in Java, which is not exactly famous for its briefness. Neither is C++. And more and more people are starting to use Ruby (on Rails) because "Ruby is an acceptable Lisp".

Paul Graham said: "Lisp's power is multiplied by the fact that your competitors don't get it".

How could we improve, if we used Common Lisp to build internet systems? This talk provides an in-depth analysis of current and future trends in web/internet programming and shows how to use Common Lisp as a tool to implement intelligent internet systems using the functional paradigm and sometimes even artificial intelligence algorithms. I will reveal what kind of Common Lisp systems we at freiheit.com technologies already have in production and what we are currently working on.

Software project estimation and planning: Just a combinatorical optimization problem?

As a follow-up to my posting “Software estimation considered harmful?” I want to talk a little bit about seeing a software project as a combinatorical optimization problem.

Do you know the Knapsack-Problem?

Imagine you are living in Papua New Guinea in the rainforest. You want to trade some of your "cargo" (see Jared Diamond, "Guns, Germs and Steel") at the market in the next big city. It is a two-days walk and you have to carry your cargo in a backpack. You separate your cargo in different boxes. Each box has a different weight and a different price per item.

You can't take all your cargo with you, because your backpack would be too heavy. So you have to decide which boxes you want to take with you. Of course you want to take the boxes with the lowest weight (i.e. the lowest costs) and the highest possible price (i.e. the highest value).

Which combination of cargo has the lowest costs and the highest value for you?! (To be able to solve problems like these gave our ancestors an advantage in the process of natural selection and evolution of mankind).

Let us now talk about software projects. For each software project you have to answer the following questions:

  1. When will the project be finished? (Time between project start and end, delivery date)
  2. How much will the project cost? (Costs, mandays and/or money)
  3. Which features will be included? (Expected functionality in this time/cost period)

Question number one is almost always fixed. Normally the customer knows in advance until when the software must be finished to be useful. Normally you can't change the delivery date. This means, that you have time constraints.

Question number two is almost always fixed, too. The customer has maybe planned an IT budget for a new system. Anyway, nobody has infinite time to deliver a product. So you have some kind of budget constraints.

The answer to question number three requires hard work, systematic analysis and intelligent thinking: You have to go and find out for yourself, what your customer really needs. I don't mean you should simply just ask your customers for his software requirements.

You have to dig deep to find out this:

Which combination of features has the lowest cost and the highest value for your customer? (To be able to solve problems like these gives a software company an advantage in the process of natural selection and evolution in the IT-business).

If you have a fixed budget and a fixed delivery date you better find out what has to be delivered to make your customer happy and successful. Your customer expects the best system that can be delivered in time and in budget.

To map this situation to the before mentioned Knapsack-Problem:

The time (working days) between the start and end of the project is your backpack.

The maximum weight you can carry in this backpack is equivalent to the mandays budget for your project. (Number_of_developers * Working_days) <>

And your "cargo" are the features you have to implement. Each feature has a cost (the mandays needed for implementation) and a value attached to it. The value can be difficult to define. Sometimes you measure the value of a feature in relation to its implementation costs. Sometimes you measure a feature on how much profit it will generate our how much cost savings your customer can achive. Giving features a priority to determine the value is often easier: If a feature has a high priority for your customer, this normally means that the value is high and vice versa.

This is the way how we see software projects. It leads us to a simple planning algorithm:

  1. Find and describe all features that the customer needs. (What is a feature?)
  2. Estimate each feature. (Minimum estimation granularity 0.25 mandays, maximum 5.0 mandays)
  3. Attach a priority to each feature. (Must-have, want-to-have, nice-to-have)
  4. Make a releaseplan. (Solve the combinatorical optimization)
  5. Implementation. Evolutionary Delivery every three weeks. Never skip or change a delivery date. Monitor your progress and anticipate risks. Check your planned estimations and actuals every day on feature level. This continuously improves your ability to make better estimations. This way, you’ll find problems long before they turn into risks.

To get into the details, I have to write more Blog postings. Stay tuned.

Donnerstag, 3. Januar 2008

Software estimation considered harmful?

A couple of days ago I read Peter Seibel's Blog-Post Software estimation considered harmful?.

In a nutshell, he writes that he was reading the book "Software Estimation: Demystifying the Black Art". The complexity and pre-conditions needed to produce software estimations with some of the described techniques really made him think, but in the end he was mostly sure that developers should do estimations.

Then he met his friend Marc (who is running a small internet startup, Wesabe) telling him, that he makes no estimations at all. He just tries to excite his fellow programmers for specific features and they then just do it: "Marc's point of view, [as] I take it, is that the only reason to do these big projects is because you have to, and if you have to, it doesn't really matter how long it's going to take", Peter writes.

This brings Peter to the fundamental question: "Why estimate?" and from this to the question "Why set targets?" In the end, he sees software estimations more like a communication tool. Funny question, eh?

I started programming in school in 1983 and I did my first professional (for money) programming job in 1987. I studied computer science and systems analysis and I am running my own software company since 1999 with over 50 programmers now in january 2008. In my whole life, I never had a customer who was contracting me or my company without asking for an estimation in advance. And of course they expect that you will deliver as planned. Only if you are your own customer, you maybe can get away without doing estimations.

So the answer for me to the question "Why estimate?" is "Because your customers will expect this from a professional company". But the answer to "Why set targets?" is (maybe surprisingly) "Because it is much more fun to work"!

When you want to write great programs you should look for people who are smart and get things done. Smart is important, because it simplifies communication. You maybe don't have to explain fundamentals and you can exchange ideas much easier, because they are understood faster. More signal, less noise. But smart people are not always getting things done. Getting things done is about making stuff work. Finding the most simple and elegant idea/solution to a problem and extending it from that point on. For me "getting things done" also means "reaching measureable goals".

Why is programming more fun, when you set and reach measureable goals (targets)?!

It gives you freedom. You don't have to spend so much time to actually manage the project. This means less administration and more programming time, because when you set measureable goals and everybody in the team knows what to do, you don't have to talk about this so much anymore. You need less coordination, because you don't have to dispatch daily work-packages to your team-members. They can do it themselves.

You don't have to complete the work of others anymore. If today you know how to get things done, but others in your team don't, then maybe you have to complete their work to finish the project in the end. If you set goals and work with people who know how to reach these goals, everybody can concentrate on their own work. Or at least you find out early, if others can't meet their goals. This leads to the next point.

You are always in control of the situation and you can adapt to problems very fast. If the time estimation granularity for each goal is not too small and not too big (from my experience less than 5 mandays, 2-3 mandays in the average), then problems in the implementation will pop-up very fast and you can adapt your estimations based on the new experience. You can help people having problems reaching their goals. It's some kind of a natural risk-management. Instead of just reacting to problems, you take your destiny in your own hands.

You are more focused. And this makes you faster. When you set goals and you reach them, then you know what's behind you, what has already been accomplished. Then you can focus on the future and what still has to be done. This is actually really satisfying. Having a measure of what has already been accomplished and what to do next.

Your customers are happy. Customers are happy, when you deliver as promised. And when your customers are happy, then you will be happy, too. They will recommend your service and they will pay your invoices fast. This leads to a positive cash-flow and this will make you independent. And independence is one ingredient of personal freedom.

This is the normal way to accomplish something: If you want to reach a goal, you should decompose the goal into a hierarchy of sub-goals. You estimate the time/costs required to solve the sub-goals and then you (the team) tackle sub-goal by sub-goal. When all sub-goals are completed, the top-level goal must be completed, too. This is how you plan the birthday party of your kids, it is the way to build a house and to plan a holiday trip. Building software is different from building a house, but it is possible to do planning. Why do so many people think, that you can't plan a software project?

Maybe because they have a misconception about the meaning of estimation and planning or maybe it is because they had a bad experience in their life as a programmer when they were forced by their managers to make unrealistic estimations and stick to them?!

You should see it like this: To estimate a software project does not mean, that you have to foresee the future. It just means that you are making time-based assumptions and predictions. And "Project management" means, that you check every day if your assumptions and predictions are still correct and that you have to refine your plans if something is going wrong. If you do this every day, your predictions get better and better and you don't stumble over so many surprises anymore.

Of course, you should be excited of what you are doing. But I think, that this is a pre-condition for every expert in some area: You are really good at stuff that you really like. I wonder how Mark is exciting his developers at Wesabe about not so exciting features. Excitement is not exactly a prioritization method.

My fellow programmers and myself couldn't live without setting goals anymore. We are setting these goals for ourselves, because it is so much more fun to work. And as a by-product we get estimations our customers can rely on: If it works for us, it automatically works for them, too.

In the last 10 years we developed our own software engineering methodology which is based on the assumption, that software project planning and estimation is really a combinatorical optimization problem. Since 1999 we delivered all of our projects on time.

Since about 2001 we have an internal system to manage our projects, features, estimations and releaseplans, called Captain Feature. So we are not only having years and years of experience in doing this, but we also have a big database of completed software projects together with their planned estimations down to the feature level and the actual time needed to do the implementation.

So, we have the data to support my theory: It is possible to estimate and plan a software project. And it can be done in a very natural, programmer- and at the same time customer-compatible way.

I will talk about this in more detail in a later posting.

Mittwoch, 19. September 2007

Yahoo! retro-style advertisement on Sunset Blvd.

A couple of blocks away from Booksoup we found this nice retro-style advertisement. Maybe for Californian people the high visibility of Yahoo! and other large internet brand names is normal, but in germany it is still not.This is actually what I like most when traveling to California and especially the San Francisco Bay Area: It feels like the internet world is seamlessly integrated with the real world. The first time I traveled to the U.S. was in 1996 or 1997 to visit the first JavaOne conference. I was amazed, that almost all advertisements displayed internet addresses. Now people can't even remember the time, when there was no internet.