Sunday, October 30, 2011

Waterfall development is like shooting a big rocket

I had a good day in Hamburg, joining the ThoughtWorks workshop about Continuous Delivery.

Vlad, the main trainer, gave the following statement:
Agile development puts the business in the driver seat.
Got it, and for me it's worth to add:
... and the driver is not blindfold.

So, how to interpret "Waterfall development" in the context of this nice analogy? For me its
Waterfall development is..... You spot a target, a few kilometers away. Then you load your big rocket, which takes ages. By the way you do not take a look at the target while loading the rocket. Then you fire the rocket, and you almost hit the target. Just 600m off since the big rocket is not very precise, and furthermore, the target moved since you last took a look at it. 
And of course the team required to handle the rocket is 10 times larger than the agile team, and the rocket costs millions.

Wednesday, May 25, 2011

Realizing Build Pipelines: The First Pipeline

So, for my little post series about How to realize a Continuous Delivery Pipeline with tool X? I am looking for an appropriate first example pipeline.

I am trying to start with a fairly easy, but nevertheless realistic kind of pipeline... The choice is influenced by some pipelines I saw working in production. Of course, it's simplified.

Pipeline #1. Simple pipeline w/ artifact and manual trigger
  • commit stage -- compiles the code, executes unit tests and produces a war artifact
  • deploy to staging -- deploys the war file to a staging test server
  • acceptance test -- executes the 
  • deploy to production -- deploys the war file to production
Ok, a fairly easy one. But how it should be configured in detail?

  • triggering -- The last step is to be started manually, all others are triggered automatically.
  • version control - The first build uses a version control checkout for the application code, the 2nd and 4th one a checkout from a deployment code repo and the tests are located in another repository. 
Of course, it's more a kind of anti-pattern to have production code and tests separated from each other. Anyway, often this is the case and I would like to have scenario which does not have two jobs which require to checkout the same version/revision of the source code.

Sunday, May 15, 2011

Drag-and-Drop with SVG with Rails Persistence (Part 1)

As a spike for my pipeline dashboard project I would like to have a page with some drawn rectangles which I can move around. Furthermore, the position of the moved objects should be persisted.


Since I like to get more hands on experience with Ajax websites which communicate with the server using JSON, the outline of my solution is the following:

  • The rectangles are drawn using SVG.
  • The rectangles are moved around using drag-and-drop.
  • There is a Rails application which provides a RESTful web service to retrieve the positions of the rectangles and of course, to persist changes of the positions. (This is inspired by an example in "RESTful web services" by Richardson and Ruby.)
  • To simplify the usage the example page is provided as the index page of the Rails web application.

Creating the RESTful service

Creating a RESTful web service with rails is very simple. You will see, it takes just a few minutes.

$ rails spike-dnd-with-svg-and-rails
      create  app/controllers
      create  app/helpers
      create  app/models
      create  app/views/layouts
      create  config/environments
      create  config/initializers
      create  config/locales
      create  db
      create  doc
      create  lib
      create  lib/tasks
      create  log
      create  public/images
      create  public/javascripts
      create  public/stylesheets
      create  script/performance
      create  test/fixtures
      create  test/functional
      create  test/integration
      create  test/performance
      create  test/unit
      create  vendor
      create  vendor/plugins
      create  tmp/sessions
      create  tmp/sockets
      create  tmp/cache
      create  tmp/pids
      create  Rakefile
      create  README
      create  app/controllers/application_controller.rb
      create  app/helpers/application_helper.rb
      create  config/database.yml
      create  config/routes.rb
      create  config/locales/en.yml
      create  db/seeds.rb
      create  config/initializers/backtrace_silencers.rb
      create  config/initializers/inflections.rb
      create  config/initializers/mime_types.rb
      create  config/initializers/new_rails_defaults.rb
      create  config/initializers/session_store.rb
      create  config/environment.rb
      create  config/boot.rb
      create  config/environments/production.rb
      create  config/environments/development.rb
      create  config/environments/test.rb
      create  script/about
      create  script/console
      create  script/dbconsole
      create  script/destroy
      create  script/generate
      create  script/runner
      create  script/server
      create  script/plugin
      create  script/performance/benchmarker
      create  script/performance/profiler
      create  test/test_helper.rb
      create  test/performance/browsing_test.rb
      create  public/404.html
      create  public/422.html
      create  public/500.html
      create  public/index.html
      create  public/favicon.ico
      create  public/robots.txt
      create  public/images/rails.png
      create  public/javascripts/prototype.js
      create  public/javascripts/effects.js
      create  public/javascripts/dragdrop.js
      create  public/javascripts/controls.js
      create  public/javascripts/application.js
      create  doc/README_FOR_APP
      create  log/server.log
      create  log/production.log
      create  log/development.log
      create  log/test.log
 $ cd spike-dnd-with-svg-and-rails

Now we use the scaffolding to create our domain object "item", which represents a rectangle to be drawn. An item has x and y coordinates, a name and an id.

$ ruby script/generate scaffold item id:integer name:text x:integer y:integer 
      exists  app/models/
      exists  app/controllers/
      exists  app/helpers/
      create  app/views/items
      exists  app/views/layouts/
      exists  test/functional/
      exists  test/unit/
      create  test/unit/helpers/
      exists  public/stylesheets/
      create  app/views/items/index.html.erb
      create  app/views/items/show.html.erb
      create  app/views/items/new.html.erb
      create  app/views/items/edit.html.erb
      create  app/views/layouts/items.html.erb
      create  public/stylesheets/scaffold.css
      create  app/controllers/items_controller.rb
      create  test/functional/items_controller_test.rb
      create  app/helpers/items_helper.rb
      create  test/unit/helpers/items_helper_test.rb
       route  map.resources :items
  dependency  model
      exists    app/models/
      exists    test/unit/
      exists    test/fixtures/
      create    app/models/item.rb
      create    test/unit/item_test.rb
      create    test/fixtures/items.yml
      create    db/migrate
      create    db/migrate/20110514212212_create_items.rb

Before starting the service we init the database as follows.

$ rake db:migrate
(in /Users/alex/SWDEV/spike-dnd-with-svg-and-rails)
==  CreateItems: migrating ====================================================
-- create_table(:items)
   -> 0.0014s
==  CreateItems: migrated (0.0015s) ===========================================

The service is started with a script contained in the folder script.

$ script/server --daemon
=> Booting Mongrel
=> Rails 2.3.5 application starting on

What we have so far?

We have a very simple web application running on http://localhost:3000 which allows for managing our domain objects. Its index page shows some welcome message with some hints. The interesting page is http://localhost:3000/items. It returns a list of our domain objects, which we called "item". Of course the list is empty.

To create our first entry we klick on "New item" and fill the form with id=1 and some fantasy. Now http://localhost:3000/items returns a list containing one item. Beside the HTML pages we can also request XML data:

$ curl -i http://localhost:3000/items.xml
TTP/1.1 200 OK
Connection: close
Date: Sun, 15 May 2011 14:00:00 GMT
ETag: "8fc46ac5da86d8127536a43e164a4878"
Content-Type: application/xml; charset=utf-8
X-Runtime: 7
Content-Length: 339
Cache-Control: private, max-age=0, must-revalidate

<?xml version="1.0" encoding="UTF-8"?>
<items type="array">
    <created-at type="datetime">2011-05-14T21:40:20Z</created-at>
    <id type="integer">1</id>
    <name>my first rect</name>
    <updated-at type="datetime">2011-05-15T10:39:21Z</updated-at>
    <x type="integer">3</x>
    <y type="integer">4</y>

Using the verbose option -i of curl tells us a lot more details about the request, especially its return code.

As the Rails application provides a RESTful service we can access the "item" with id 1 using
$ curl -i http://localhost:3000/items/1.xml
HTTP/1.1 200 OK
Connection: close
Date: Sun, 15 May 2011 14:01:24 GMT
ETag: "a388de3aec9d892bc1b1af2f81194701"
Content-Type: application/xml; charset=utf-8
X-Runtime: 7
Content-Length: 293
Cache-Control: private, max-age=0, must-revalidate

<?xml version="1.0" encoding="UTF-8"?>
  <created-at type="datetime">2011-05-14T21:40:20Z</created-at>
  <id type="integer">1</id>
  <name>my first rect</name>
  <updated-at type="datetime">2011-05-15T10:39:21Z</updated-at>
  <x type="integer">3</x>
  <y type="integer">4</y>

What about JSON?

Cool, out of the box we got HTML and XML. Let's try to use json. We either can tell cURL that we would like to get JSON results or we append the postfix ".json" to the URL,

$ curl -i http://localhost:3000/items.jsonl
HTTP/1.1 406 Not Acceptable
Connection: close
Date: Sun, 15 May 2011 14:10:41 GMT
Content-Type: text/html; charset=utf-8
X-Runtime: 5
Content-Length: 1
Cache-Control: no-cache

$ curl -i -H "Accept: application/json" http://localhost:3000/items
HTTP/1.1 406 Not Acceptable
Connection: close
Date: Sun, 15 May 2011 14:13:27 GMT
Content-Type: text/html; charset=utf-8
X-Runtime: 5
Content-Length: 1
Cache-Control: no-cache

Both calls are returning 406 and no content. So, there is something to do to enable JSON support.

How to ad JSON support?

To provide JSO we have to extend our controller ./app/controllers/items_controller.rb. The controller has code like this:
def index
    @items = Item.all

    respond_to do |format|
      format.html # index.html.erb
      format.xml  { render :xml => @items }

To add json support we simply copy each "format.xml" line and exchange "xml" with "json". That's it. (Of course, it would be very nice if an option for scaffolding could enable the json support.)

Let's verify whether the JSON support works.
$ curl http://localhost:3000/items/1.json
 {"item":{"name":"my first rect","created_at":"2011-05-14T21:40:20Z","x":3,"y":4,"updated_at":"2011-05-14T21:40:20Z","id":1}}


Cool. we have out first RESTful web service with JSON, XML (and also HTML) support...  written in less than 10 minutes. Not so bad.

With the next post I try to cover the manipulation of data using PUT and DELETE requests rather than GET requests.

Friday, April 29, 2011

How to realize a Continuous Delivery Pipeline with tool X?

In the last years the idea of Build Pipelines was established, especially in the context of Continuous Deployment or Continuous Delivery. Roughly speaking, a that's Continuous Integration v2.0. Rather than working with a collection of CI builds (often with additional nightly builds plus release builds), there is a sequence of builds which models the workflow of all steps which are required to verify a commit before it can go to production Every single commit can travel thru this pipeline. With some good luck it passes all tests and ends up in production.

For Continuous Integration there are a tons of tools around. Wikipedia lists 20+ tools. So, my question is:
What about their support for Build Pipelines? Are any of these tools able to realize Build Pipelines?

Of course, the answer is YES. -- so far it sounds good. ;)

Unfortunately, more in detail the answer is more something like: 
Up to some degree most tools are able to realize some pipelines in some sense.

Uffs. To many "somes", a very vague and disappointing statement. I am not amused.

Ok, maybe I am too rude. Most tools have some features to "tie builds together" in some sense. Realizing build pipelines requires a lot of different features, depending on the nature of our specific build pipeline scenario. But no features provides some valuable documentations and no one gives any good examples. Especially with Jenkins/hudson the situation is a little bit unpleasant. The tool out of the box does not help to realize pipelines. In the universe of jenkins plugins (note: the jenkins project on github contains 500+ repositories) there roughtly 10 plugins which help to realize pipelines, but in most cases you need more than one plugin, the documentation vacous rather than helpful and it takes a log time to figure out how to solve the puzzle.

For the next month I plan to investigate how to realize build pipelines with different tools. I try to consider different scenarios with different kinds of pipelines which are demanding different features.

In order to provide something which has at least a tiny chance to help others, I try to follow two rules: (1) the scenarios are described scenarios using BDD techniques, Given-When-Then, and (2) I am going to provide the example code on github, including example code for the pipeline as well as the configuration for the CI tool (unless its using a db).

I tend to focus on the following list of tools:
  • Jenkins (formerly known as Hudson) as the most famous and vital open source tool.
  • TeamCity by JetBrains
  • Bamboo by Atlassian
  • Pulse by Zutubi, compare this post
  • Last but not least ThoughtWorks GO! which focuses on realizing pipelines.

Please kick me if I do not continue this series...

Monday, March 7, 2011

Zutubi Pulse - The forgotten CI system

About two years ago I evaluated a lot of Continuous Integration systems. The list of candidates included known ones like Atlassian Bamboo, TeamCity by JetBrains and of course, hudson (which is now know as jenkins). But another system was in the list, and I was not able to rember its name, or to find it using a google search.

Now I got it. Cleaning up old notes helped to find an old piece of paper with the comparision matrix of the CI systems.... and it shows the name of the forgotten CI product. It was Pulse by Zutubi.

The list of features now includes a nice view for upstream dependencies of my build. Compare

I am going to give it another trial...

The META Acceptance Criteria

GIVEN a story described by the PO and diskussed by the whole team
WHEN the team implements the story and presents it during the sprint demo
  • the PO is happy
  • the customer is happy
  • the story is DONE with respect to given definition of done of the team