Javascript is a special language with somewhat unusual features. Often you see people struggling with the concept of “this”, “new” and “prototype”. This is my attempt to shed some light on these mysterious concepts. If you are coming from a statically typed language like C# or Java, you will quickly realise that they are not what they seem to be.

This and that

A common mistake is to think that the “this” keyword in Javascript refers to the instance of the object you are in. This is the common use in C# and Java.

You can think of the “this” keyword as an implicit argument to the function. In a browser this implicit parameter will be set to the root object by default, e.g. the window object: 

function foo(){
	return this;
}
	
alert(foo()===window) //true

However, you can pass the “this” reference to the function you are calling by using the call (or apply) function:

	
function foo(){
	return this;
}
	
//calling foo with “this” assigned to the value “hello”
var result = foo.call(“hello”) 

alert(result) //”hello”

As you can see, the “this” keyword depends on the context in which you call the function foo.

The new keyword

To construct objects in Javascript you use the “new” keyword together with a constructor function. Functions that are used as a constructor usually starts with an upper case letter (best practice)

	
function Foo(){
	this.helloWorld = function(){return "Hello world"}
}
	
var obj = new Foo();

alert(obj.helloWorld()) //”Hello world”

When you call a function with the “new” keyword, a new empty object is created and then the constructor function is called with “this” set to the new object. In the example above, this will cause the new object to get the method “helloWorld” assigned to it. The prototype hierarchy is also set up, but I’ll come back to that later.

The following code demonstrates another way to get  the same effect as the new keyword (without the prototype hierarchy)

function foo(){
	this.helloWorld = function(){return "Hello world"}
}

//emulate "new"
var obj = {};
foo.call(obj);

alert(obj.helloWorld()) //"Hello world"

Forgetting the new keyword

If you “forget” to use the new keyword on a function that is meant to be used as a constructor you will get strange behaviour:

	
function Foo(){
	this.helloWorld = function(){return "Hello world"}
}
	
var obj = Foo();

alert(obj.helloWorld()) //fail
alert(window.helloWorld!==undefined) //true

You just assigned the function helloWorld to the window object. Probably not the intention :)

The prototype hierarchy

When you use the new keyword, the prototype hierarchy is set up. This means that the new object will have  the methods in the prototype object of the Foo function:

function Foo(){
	this.helloWorld = function(){return "Hello world"}
}

//define a method on the prototype
Foo.prototype.helloWorld2 = function(){return "Hello world 2"}


var obj = new Foo();

alert(obj.helloWorld()); //"Hello world"
alert(obj.helloWorld2()); //"Hello world 2"

The new object will also have methods defined higher up in the prototype hierarchy.

Well. That’s it for now.

Comments F Share

This post is just about me sharing some of the stuff I have looked at in the open source community lately. There might be some ruby stuff in here. First out is rack.

Rack

Imagine for a minute that there were a lot of different web servers out there like IIS,  Apache and perhaps even Kayak. Also imagine that there were a lot of different web frameworks out there, like Fubu MVC, ASP.Net Web Forms (don’t get me started) and Asp.Net MVC.

Wouldn’t it be cool if we could put any web server together with any web framework just by putting a common abstraction in between ?

The web frameworks would only need to implement an adapter working with the abstraction and the same for would go for the web server. To get n web frameworks working with m different web servers, we would only need to implement n+m adapters as opposed to n*m without the abstraction. We could pick our favorite server and our favorite web framework and start developing with a smile on our face. (Ignore for a minute that this is not really possible with the servers and frameworks I just mentioned)

What would this abstraction look like ? Well, a web server is at its hart a very simple thing. You get a request and return a response. A response is simply a status, a set of headers, and a body.

In .NET a very simple implementation of a web “framework” adhering to the abstraction would look something like this.

public Response Call(Environment env)
{
  return new Response(
    status: 200,
    headers: new Dictionary<string, string> { { "Content-Type", "text/plain" } },
    body: new List<string> { "Hello world" });
}

or the more terse ruby version

def call(env)
  [200, {"Content-Type" => "text/plain"}, ["Hello world!"]]
end

Now I finally came to the point of this entire rant. In ruby they actually have this abstraction in place. It’s called rack. You can mix and match web servers and web frameworks to make your coding life a dance on roses.

I like it.

You would think the story ends here, but it doesn’t. There is also something called rack middleware. It takes advantage of the common abstraction between the web server and the web framework and offers you the ability to put logic in between the two. It is entirely possible to create rack middleware for debugging, logging, authentication, the list goes on. You now have reusable components for your web site that is independent of both the web server and the web framework you have chosen. Cool huh?

I like it.

(This part of the post was mostly inspired by this introduction to rack)

Sinatra

Sinatra is a rack enabled web framework. Its implementation is very small and the sample application is even smaller. I will not say much about Sinatra, because the web site says it all.

require 'sinatra'
get '/hi' do
  "Hello World!"
end

Ahhh, the beauty of simplicity. Describe the URL, define the action, and you’re done.

Since Sinatra supports rack, you get the power of rack middleware in addition to the simplicity of Sinatra. I like it.

Tilt

Any serious web framework needs a nice templating engine. In ruby you have a lot of choises like erb, haml and my current favorite mustache. It would be a pain (in the ass) to implement support for all these engines in every web framework, so the ruby guys have (mostly) agreed on tilt. Tilt is a common abstraction that goes between the web framework and the templating engine. Sounds familiar ? To use a tilt enabled templating engine, you need to supply the template and some context for the template (the data). Easy easy.

template = Tilt::ERBTemplate.new('templates/foo.erb')
joe = Person.find('joe')
output = template.render(joe)

Nice stuff.

Well, that’s it for now. See you later.

Comments F Share

NDC 2010 was great. This post is inspired by Tom Gilbs ”Lean quality assurance” talk:

Often we set out to build high quality software without really considering what that entails. How can you really know, when project has finished, that you have created quality software?

Clean code, test coverage and executable specifications created iteratively together with the primary stakeholders seem to be the main focus for most agile projects, at least, in agile theory. There are other concerns that can easily be overlooked when people go into “agile mode”.

Numeric entry and exit

As a stakeholder in a project you don’t really care about testing and executable specifications. You care about the ilities. Usability, Maintainability, Reliability, Security and Scalability are some that come to mind.  Often requirements emerge as the software is developed, but many of these are well known in advance, at least to the stakeholders.

How can we identify these requirements and ensure that they are met at the end of the project? 

A coder will leave it to chance. A software engineer will measure and verify that the requirements are met.

To identify the requirements you first need to identify the stakeholders. This is a process in its own right. Later you can start identifying the requirements. 

According to Tom Gilb you can measure *any* requirement numerically. Later in the process you can use these numbers to evaluate the quality of the software (or some other item) you are producing. A requirement is considered done when your software meets the numeric target for that requirement.   “Numeric entry and exit” is a term that describes this way of thinking.

How can you measure things like usability of a feature? The answer is obviously “it depends”. One metric can be the percentage of grandmothers that understand the feature on the first try. If 80% get it, you have a usable product, and you have met your target for that specific requirement.

Obviously dissecting your requirements this way requires a lot of time and effort. For some projects doing it this way makes no sense, for some it’s too cumbersome and expensive, and for some it totally makes sense.

You get what you measure

Later.

Comments F Share

I guess I need to supply some context here: I’m still in the JavaScript world and I’m still writing desktop-like applications that run in the browser. I only load the DOM once. Here we go:

Every test you add to your application creates a dependency on it. More tests mean more dependencies. Dependencies spell trouble, especially if you depend on code that changes a lot, a.k.a. unstable components.

If you have a test to verify the correctness of a feature, the test should touch a minimal amount code, and it should ideally only touch stable components.

A common problem with integration tests is that they break for reasons having nothing to do with the feature they are testing. You often run in to this problem if you are testing thru the UI. The UI is very unstable (it changes a lot). Minor design changes can break tests. If you cannot easily change your application without creating extra work, the ROI of the integration tests is diminishing.

One solution is to ignore the outer layer of the UI and instead test one layer in. When I write desktop-like web applications I have a thin view layer that encapsulates all the UI interaction (http://bjarte.com/post/effective-javascript-unit-testing). This means that I can invoke every user action thru the view layer.

Let’s look at the following example of a search box:

The view would look something like this:

function SearchView(callback) {
  
  $("#search").click(function () {
    callback($("#q").val());
  });

  function doSearch(){
    $("#search").click();
  }

  return {
    doSearch: doSearch
  }
}

Somewhere else in my code I will make all my views visible globally thru a “views” collection. From my tests I can now invoke behaviour by talking directly to the SearchView:

views.searchView.doSearch(); 

I actually use WatiN to test thru the browser. The only line of code I need in my test (actually specification) to invoke the behaviour is

Browser.Eval("views.searchView.doSearch();");

As you can see, some of my application code is there only to support integration tests (e.g. the doSearch method on the view). I don’t have  a problem with that.You can plug a computer into a modern car to diagnose it. Why shouldn’t you be able to diagnose software?

The point here is that my tests depend on more stable code. The layout can change. The button can move, or perhaps become an image. The search behaviour will stay the same. Hopefully.

Obviously you are not testing EVERYTHING when you don’t run the tests thru every layer of the application. But you know what ? You cannot test everything anyways.  You come to a point where writing more tests will not give you more benefits. At some point more tests become a problem, a maintenance problem. Where you draw this line is dependent, not only on your technical challenges, but also the people in your team, the culture, the process and many other factors. Context, context, context.

Later.

Comments F Share

Most JavaScript testing frameworks run in the browser and are tightly coupled to the DOM.

I don’t care about the DOM and I definitely don’t care about the browser, at least not when I am (unit) testing my JavaScript components.

This probably needs some context; I’m not writing web sites with 5 lines of JavaScript in them you see.  I’m writing desktop like applications that run in the browser. This means I have a lot of widgets and screens that need to talk to each other. I am also doing XMLHTTP calls to the server. All the interaction needs to be tested.

I don’t want to run the tests thru the GUI. It’s to brittle. Changing an a-tag to an input-tag or moving a component on the page can easily break a test. Testing is supposed to improve productivity, not slow it down.

I use the Model-View-Presenter design pattern to design my application layer. This means I have a thin layer, call “the view”,  over my HTML that abstracts away both the DOM and with it, the browser.

The View

Let’s say I have the following component in my application. (It would probably say “search” and not “Google search”, but you get it)

The corresponding HTML would look something like this (ignoring the lucky-button)

<input id="q" type="text" />
<input id="search" type="button" />

I would now create a SearchView module to put an abstraction on top of this HTML (using some JQuery magic)

function SearchView(callback){
	$("#search").click(function(){
		callback($("#q").val());
	});
}

Now the rest of my code will depend on the SearchView function, and only the SearchView function will depend on the browser DOM. But that’s no problem. We can always create a fake SearchView function to act as a stand in for testing purposes.

The key here is that even if I build a very complex application with a lot of interaction, it is only the thin view layer that is depending on the browser DOM. The rest of the code is pure JavaScript.

Testing the code (on the Windows platform)

So I have A LOT of JavaScript code to test, and it does not depend on the browser DOM, so why would I need a browser to test the code ? Wouldn’t it be much better to run the tests thru a command line JavaScript interpreter? Or even better, have the tests run inside Visual Studio ? I definitely think so. Opening up a browser to run a unit  test is way too much friction for me.

There is a JavaScript interpreter build into Windows. You can run it thru the cscript command line tool. To run a file, you do the following:

cscript myFile.js

Obviously if you are doing any meaningful testing you need to load multiple files into the environment, the testing framework, the tests, and the system under test. Luckily you can run multiple files at the same time by creating an xml file. Let's call the file test.wsf.

<?xml version="1.0" encoding="utf-8" ?>
<package>
   <job id="1">
     <script language="JScript" src="TestGui.js" />
     <script language="JScript" src="TestRunner.js" />
     <script language="JScript" src="Foo.js" />
     <script language="JScript" src="FooTests.js" />
   </job>
</package>

You can now run all the files

cscript test.wsf

We can now use this command line tool inside Visual Studio by creating an “external tool”

Tools -> External tools

By adding a shortcut to the external tool, I can run my JavaScript tests from anywhere in the Solution, only using a single keystroke. AND IT’S FAST!

The test runner / framework

As I mentioned previously, most JavaScript testing frameworks out there are tightly coupled to the browser DOM. That will not fly when you are using the Windows Script Host. I chose to build my own little test framework to run the tests. It is very small, and only have three method calls that are coupled to the environment it is running in.

//write to console
WScript.Echo(s); 	
...	
//quit application with error (which will break the build on TeamCity)
WScript.Application.Quit(1);	
...
//write to StdErr
WScript.StdErr.WriteLine(s);    

As you might tell from the code sample above, I run these tests on our CI server as well. No problem there.

My home-made testing framework is in an early stage. You can look at it if you want to, but don’t expect too much: http://github.com/BjartN/BigUnit.

Well, that’s it for now.  Later.

Comments F Share

JavaScript has gotten a bad reputation in the software industry. Most of it is related to the browser DOM and incompatibility across browsers. This problem is starting to go away due to great frameworks like Jquery and Prototype. Web 2.0 sites with heavy AJAX functionality forces browser manufacturers to improve their JavaScript interpreters. JavaScript is even becoming a contender on the server side.

Designing objects

If you google ”object oriented design (OOD) using JavaScript” you will find that most people build JavaScript objects using prototype and the “new” keyword. Here is how you normally would create an object:

var Address = function (street) {
  this._street = street;
};

Address.prototype = {
  setStreet: function (street) {
    this._street = street
  },

  getStreet: function(){
    return this._street;
  }
}

I create the function Address. This function will work as my constructor. Then I extend Address’ prototype with two functions, setStreet and getStreet. I can now use the “new” keyword to create an address object with the two functions “setStreet” and “getStreet” baked in

var addr = new Address("Elm Street");

If I want to, I can change the street name in the address object

addr.setStreet("Old Berry Vale");

This is all good. However, there are some problems with this approach.

This and that

If you come from OO languages like C# or Java, you are tempted to believe that the “this” keyword refers to the current instance of the Address class. However that’s not always true. The “this” keyword actually refers to the context in which the function is invoked. If somebody for some reason invokes the setStreet function in a different contex you will get into trouble. This usually happens when we pass around function pointers. I can use the "call" function on the "setStreet" function to illustrate this by calling the function "setStreet" in the context of an empty object ({})

addr.setStreet("Old Berry Vale"); 
areEqual("Old Berry Vale", addr.getStreet()); //nothing strange here

addr.setStreet.call({}, "Cotton Acres"); //calling method from different context
areEqual("Old Berry Vale", addr.getStreet());  //hey.. the name didn't change

Are you protecting your privates ?

Well, are you?

addr._street = "Pleasant Island Stead"; //I can change internal state
areEqual("Pleasant Island Stead", addr.getStreet()); 

Ai ai ai. The “_street” instance variable isn't really private. Not having private instance variable when you create objects like this is problematic.

Closures. The road to a better approach.

From the description above you might believe that JavaScript is troublesome. However, as with any sharp tool, you need to use it right if you don’t want to get hurt.

JavaScript has the fine notion of closures. And yes, now it’s time to concentrate:

function main(foo) {

  runFunction(sayHello);

  function sayHello() {
    alert(foo);
  }
}

function runFunction(callback) {
  callback();
}

main("Hello world"); //=> “Hello world”

In JavaScript you can define functions within functions. This might seem a little bit strange, but you get used to it. The ”runFunction” is straight forward. It takes a function pointer as an argument, and executes that function . The “main” function is more fun, but still pretty simple. It just passes the “sayHello” function to the “runFunction” function.

The interesting thing here is that when the “runFunction” invokes the “sayHello” function, the sayHello function knows the value of “foo”. How can this be? The answer is that even after you pass the “sayHello” function it still has access to the parent scope where it was defined. The “sayHello” function is a closure. Cool huh ?

Another thing to mention here is that the “sayHello” function is private within the “main” function.

A better approach

By using closures and a couple of additional tricks you can create “objects” that doesn’t have the problems outlined in the previous solution. This is how you do it:

var Address = function (street) {

  function setStreet(s) {
    street = s;
  }

  function getStreet() {
    return street;
  }

  //explicitly make private methods public
  return {
    getStreet:getStreet,
    setStreet:setStreet
  };

}

You can now create an “instance” of the “object” this way:

var addr = Address("Elm Street");

Notice we are not using the “new” keyword, and we are not using “prototype”.  We are basically calling the Address function which returns an object containing the getStreet and the setStreet function. The “street” variable acts as an instance variable. The function getStreet and setStreet has access to this variable since they are closures.

The setStreet and getStreet functions are initially private so we need to return them in an object to make them public. The act of making functions public is explicit, which I think is a good thing.

We get the expected behaviour:

var addr = Address("Elm Street");
addr.setStreet("Old Berry Vale");
areEqual("Old Berry Vale", addr.getStreet());

addr.setStreet.call({}, "Cotton Acres"); //different context doesn't create problems
areEqual("Cotton Acres", addr.getStreet());

addr.street = "Pleasant Island Stead"; //private variables cannot be tampered with
areEqual("Cotton Acres", addr.getStreet());
 

Good good. One other thing to notice is that the code is now visually much cleaner. Well, that’s it for now.  [UPDATE] Look at the “yahoo javascript module pattern” if you want to learn more [/UPDATE]

A small note on methods and functions

In mathematics a function always returns the same value on the same input. In this post I have used the word in a more loose fashion because the of the “function” keyword in JavaScript. Methods and functions are not the same. Just so you know :)

Comments F Share

Did I get your attention? Good.

To many of my fellow programmers what I am about to say  is second nature. However, taking a recent discussion into consideration it is not second nature to everybody.

I don’t really think my point of view is very controversial. It basically goes like this:

Write comments in your code if  it adds value

Obviously this is a very vague guideline. When does a comment really add value?

Comments on classes

In well written object oriented software your objects play one or more roles. I find it valuable to add a comment to the class declaration explaining the objects role in the system. This helps you see the big picture, which usually cannot be described by a simple class name. Sometimes it can.

It’s easier to explain my attitude towards writing comments by going thru the cases where I don’t write comments, as opposed to talking about the ones where I do.

Comments on and inside methods

In my opinion you should usually not comment your methods and the code inside them. Public APIs is an exception.

In well written code a method should only have one or two responsibilities. This means that by carefully selecting a method name you can give the reader a good idea of what is going on inside the method. Now, obviously you cannot explain the nitty gritty details, but that is what code is for. If you want the details, you should read the code. Method level comments cannot really add value. Putting a short description of what the method does in the comment is redundant because the method name is already doing this. Putting details in the comment gives you a maintenance hell when you need to keep the comment in sync with your code. If you want the details you better read the code.

Inside a method your often feel the urge to insert some comments to make the code more readable. Why don’t you just make the actual code readable instead? If you have a section of code that is doing some complicated calculation you can always put the calculation in its own method and give that method a meaningful name. Problem fixed.

Obviously there are some exceptions to my above comments. They are few. If you find yourself commenting a method or code inside a method usually it is a problem with your code. Comments are often a code smell.

Another problem with comments is the fact that you need to maintain them as you change your code. Up to date comments is not a big problem. Out of date comments is a big one.

When it comes to the title of this post, it’s not completely accurate.Today I’m pretending I’m running a tabloid.

(Please see my comments in the context of good old OOD. They might not be totally suitable for functional programming)

Comments F Share

In software development we often end up with local optimization. Developers focus on getting the development work done, testers focus on getting the testing done, and the analysis team focus on getting their work done. This is a problem, because local optimizations does not optimize the whole.

The development process should be optimized for global productivity. Work should flow thru the development pipeline without batches of work accumulating within the pipeline.

In the following diagram imagine the work flowing from left to right. First it enters the analysis team, then it’s developed, then tested, then finally it is delivered to the customer. A work item is indicated using the letter “x”.

This shows a situation where the testers are a accumulating work because they cannot process all the work the developers are handing over. Notice every team is working with top efficiency. This way of working introduces problems.

Problems caused by large batches

What are these problems we are creating ? If you look  at the entire development process as a queue of work going from left to right, you will see that more work in process creates a longer queue. A longer queue means that the time it takes for one item of work to pass thru the entire queue is longer. This means that the delivery time of our software increases ! When work in progress increases, the delivery time increases. This is obviously a problem for many reasons that I will not go into right now.

There are other problems with the above work flow as well. Developers continue developing before their previous work is validated by the testing team. This means that they are creating work based on possible false assumptions. When this propagates thru the system it creates even more problems.

Limit the work in progress

The key is to limit the work in progress. In the previous scenario the testing stage of the software development process is the bottleneck. To avoid the accumulation of work in progress we need to increase the thruput in the test team, or lower the rate at which work is arriving. One solution could be to move some of the developers over to the testing team. Another would be to slow down the work upstream (to the left) of the test team, by giving the craftsmen there some “slack”.

Having people not doing the work they should be doing is often a problem for managers. They measure success by how efficient their  team is. Suddenly we are back to local optimization. Having slack is actually a good thing. In the right enviroment it creates opportunities for learning and innovation.

In rigid  functional teams, it is hard to do anything to modify the flow. People are happy with local optimization. In cross-functional teams it is easier to modify the process to accommodate the flow.

Some additional thoughts

1) It is easy to think of work in an software organisation as small items of work flowing thru the pipeline. In reality the work items varies in size as they flow thru the pipe. Also the issue of rework puts a little brake on the happy path. Keep this in mind as you take your next steps to improve the flow :)

2) There are things to be said about having the testing team upstream from the development team and making the batches of work smaller. That’s something for another post.

3) Focusing only on the work in progress withouth thinking about keeping the team members happy from a social point of view is a good way to fail in the long term.

Feel free to comment :)

Inspired by:
Kanban Chalk-Talk  (http://vimeo.com/7814701)

Comments F Share

Yesterday I was at a .NET User Group meeting starring Kent Beck. He did a presentation on the different considerations you need to make when increasing the speed of deployment.

Most companies deploy their software on a yearly to quarterly basis. The trend is going towards shorter intervals. Why would you deploy on a more regular basis? The reason is somewhat obvious. It makes the business much more flexible. If you can deploy faster, you can respond faster to the changing business requirements. Also I believe that companies that are ably to deploy on short intervals, are companies with a small amount of overhead.

Moving from annual to quarterly deployments

Add: Automated acceptance tests, Refactoring, Continuous Integration, and Subscription based model

When moving from annual to quarterly deployments you need to add automated acceptance tests. Making software ready for production by manual acceptance tests takes too much time (if you do it the right way). If you move to quarterly deployments, the time it takes to manually test the software will easily overshadow the development time. However if you are doing annual deployments the manual testing is not an overwhelming part of the development effort, and you can live with it.

You need to practice code refactoring at all times when developing for quarterly deployments. In the business world the term refactoring is (mis)used to mean changing the entire code base in one go. This is not what I mean by refactoring. Refactoring is changing the structure of the code one small step at a time. This is a continuous effort that ensures that the code base always is optimized for changeability.

The above considerations are technical. From a business perspective you can consider moving to a subscription based model for your software. If your software is released on a annual basis you would send your sales team out to sell “the big upgrade”. This model does not work as well if you are deploying every three months. Moving to a subscription based model makes more sense. This model also ensures a steady income.

Moving from quarterly to monthly deployments

Add: Developer testing, “Stand up meetings”, “Cards on a wall” and Pay per use

Remove: QA Department, Multiple deployed versions, Design document, Change request, Analysis team and the Build team

When moving to monthly deployments the handoffs between the developers and the QA department starts to become a bottleneck. You need developer testing to prevent bugs moving further down the production line. You could still have a QA team, but this team is much closer to the development team, and the software they are testing is of a much higher quality.

The communication bandwidth needs to increase between the team members. This can be done by adding visual aids like cards on the wall and by frequent short meetings.

Also you need to remove the heavy weight handovers that occurs between the developer team, the test team, the build team and the analysis team. The overhead caused by these handovers will start, again, to overwhelm the development effort. The team needs to be cross functional.

Having multiple versions of your software deployed will easily become maintenance problem. You cannot afford this, so you must have a smooth upgrade process for your software. This will ensure that you only have one version deployed at all times.

From a business perspective you have now the ability to introduce the “pay per use” model. With shorter deployment cycles you have the ability to adjust and change your software as the revenue varies. With longer deployment cycles you are stuck with your software longer and a “pay per use” model would not work very well because it takes a long time to change and adapt.

Moving from monthly to weekly deployments

Add: Live 2-way data migrations, Defect zero, Short lived branches, Keystoning, Kanban, Bootstrap financing

Remove: Test team, One-way data migration, Release branch, Patches, Up-front usability design, Venture capital

When moving to weekly deployments you are possibly deploying 52 times per year. This means that there is a good chance that some of the deployments will fail. You need to be able to roll out and roll back your changes automatically. Doing this manually is too much effort. In other words, you need, among other things 2-way data migrations.

Defect rates must be close to zero. Due to the sort deployment intervals, smaller defects can be fixed in the next deployment. Ideally they don’t occur at all.

Having code that live for weeks on a separate branch is a no-go. The cost of merging is too high. Branches must be short and sweet and usually merged into the main branch during the day.

Weekly iterations will not work. You need to be able to continuously change the priority of the work and work in a continuous manner. This can be done by applying Kanban. Developers will pull a task from the task list when the previous task is completed. There is no hard limit to the amount of work that should be done within a week. Developing the product is a smooth continuous effort.

Note: The idea of Kanban is much more involved than what I have outlined here. Do the Google dance if you want to know more. I strongly recommend this approach for developing software.

When deploying weekly and possibly using a subscription based model you are much closer to the revenue stream. For this reason alone this approach if very tempting to bootstrapped businesses.

Moving from weekly to daily deployments

Add: Immunization, A/B Testing

Remove: Staging, Operations team, Stand-up meetings

Is it getting hot in here? Did you just say daily deployments? Well. Yes :)

Immunization is the idea of automating rollout and rollback by using metrics. For companies with a constant revenue stream you can monitor the money flow on the servers you have upgraded. If you see drops in performance you automatically roll back to the previous version. There are also other metrics you can look at, e.g. the response time of the server.

Daily deployments enable a high level of A/B testing. By deploying different versions of your interface and monitoring the performance of each version, you can easily find the better solution.

“Inside the building there are only opinions”

…you need to test your software our there to really know what is the best solution.

Considerations

If you look at what is happening as you move to a shorter deployment cycle, you will see that all we do is removing friction. An organization that is only able to deploy on a yearly basis has a lot of waste in the system.

Moving to a shorter deployment cycle means learning new technical skills and having organizational backing. If your sales team is selling “big-upgrades”, having weekly deployments makes no sense. The technical skills also need to improve. The old software needs to support the new technical practices. This is not always trivial. You should move slowly to the next level, and not plunge into it.

I will finish this with a somewhat unrelated Kent Beck gem

“Your relationship with your customer is your most important asset”

Comments F Share

I am a firm believer that everything that can be easily automated should be.
Making a human responsible for repetitive IT work is not only inhumane, but also extremely inefficient and a waste of money.

Lately I have been working with automating deployment of new applications within our company. Deploying automatically to our customers is not within reach (yet).

The only thing that should be required for a developer in order to show his latest work to the rest of the company is to check in the code. The tests run and the application(s) should be deployed immediately after. A deployment package should also be created.

We are using TeamCity as our continuous build server, and it's doing a great job. The easiest way for us to build our code continuously, is to use the "sln2008" build runner. This means that TeamCity pulls the code from our code repository (Subversion) and compiles it. In addition we have set up the build runner to run our NUnit tests as well.

If we want to create an IIS application and deploy our code to it, we need to go for another option. Some common ones in .NET are NAnt, MSBuild, Rake and probably a lot more.

As a hardcore .NET guy my choice fell on MSBuild. I need to get started somewhere right?

An introduction to MSBuild

MSBuild is not that magical. Running MSBuild is as simple as running MsBuild.exe on the command line together with the path of your MSBuild script. The MSBuild script is an XML file. Actually the Visual Studio project files are MSBuild scripts. When you compile in Visual Studio, MSBuild is runs in the background.

The structure of an MSBuild script is pretty simple. It consists of a project. Inside a project you can have targets and inside each target you have one or more tasks:

<Project>
	<Target>
		<Task></Task>
		..
	</Target>
	..
</Project>

A task is some action you want to perform, like creating a directory or deleting an IIS application.

A target is just a collection of tasks to be executed sequentially.

A project is just a wrapper for a set of targets.
The following build script will write a “hello world" message when you run it on the command line using "msbuild fooscript.msbuild"

<Project DefaultTargets="FooTarget">
	<Target Name="FooTarget">
		<Message Text="Hello world" />
	</Target>
</Project>

Notice that the project defines "DefaultTargets", which is a list of targets to run. Inside the FooTarget there is only a single task "Message" that will output "Hello world" to the console.

You can make multiple targets and let them run depending on the success of the previous target.

<Project DefaultTargets="FooTarget">
	<Target Name="FooTarget" DependsOnTargets="FiiTarget">
		<Message Text="Hello world" />
	</Target>
	<Target Name="FiiTarget" >
		<Message Text="Hello world" />
	</Target>
</Project>

In the script above the default target is FooTarget, but since FooTarget depends on FiiTarget, FiiTarget will run before FooTarget. If FiiTarget succeeds, which it probably will, the FooTarget will run.

In MsBuild scripts you often have the need for defining variables that you can use later in the script. A variable is referred to using the $-sign. You can also define lists which are referred to using the @-sign. The following example demonstrates use of a variable:

<Project DefaultTargets="FooTarget">
	<PropertyGroup>
    		<FooVariable>Hello world</FooVariable>	
	</PropertyGroup>
	
	<Target Name="FooTarget">
		<Message Text="$(FooVariable)" />
	</Target>
</Project>

The above script will output "Hello World" (again) :)

Sometimes you don't want to put all of your targets in a single file. Maybe you want to use targets defined by third parties, like Microsoft or open source. I have used open source targets from tigris.org (http://msbuildtasks.tigris.org/). They have targets for copying files over FTP and editing XML-files.  Both are valuable for automatic deployment.
To use targets defined in other files, you import them using the import statement:

<Project DefaultTargets="FooTarget">
	<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />
	
	<Target Name="FooTarget">
		<FtpUpload .... />
	</Target>
</Project>

Here I import the MSBuild file located at C:\Program Files (x86)\MSBuild\MSBuildCommunityTasks. MSBuild uses the built in variable $(MSBuildExtensionsPath) to find the MSBuild root directory. By doing this I make the task "FtpUpload" available to my script. The built-in Microsoft tasks defined here (http://msdn.microsoft.com/en-us/library/7z253716(v=VS.90).aspx) are always available.

If you are moving a script from one machine to another, you need to make sure that you also move the imported scripts. Usually you can find these in the MSBuild directory. In my case it is C:\Program Files (x86)\MSBuild.

Now we are ready to look at how I actually implemented our MSBuild script to automatically deploy our web applications.

Creating a continuous integration script using MSBuild.

To deploy a web site you basically need to build it, create an IIS application and deploy the application to the physical location of the web site. In addition you would run your tests using the NUnit task, but I will not go into that here.

To create the build script for my web site I installed “Web Deployment Projects” for Visual Studio. You can add this project to you Visual Studio solution the same way you add any other project. (http://www.microsoft.com/downloads/details.aspx?familyId=0AA30AE8-C73B-4BDD-BB1B-FE697256C459&displaylang=en)

The web deployment project creates a .wsproj file which is actually an MSBuild script. Surprise.  This script is configured to build the web application. I have removed a lot of noise from the script:

<Project DefaultTargets="Build">
  <PropertyGroup>...<PropertyGroup>
 
  <Import Project="$(MSBuildExtensionsPath)\Microsoft\WebDeployment\v9.0\Microsoft.WebDeployment.targets"/>

  <Target Name="BeforeBuild"> </Target>
  <Target Name="BeforeMerge"> </Target>
  <Target Name="AfterMerge"></Target>
   <Target Name="AfterBuild">...</ Target>
</Project>

Notice that the default target is “Build”, but “Build” is not actually defined in this script. The “Build” target is actually defined in the imported script Microsoft.WebDeployment.targets. This script was put in the MSBuild folder when I installed the Web Deployment Project. It is the “Build” target that is responsible for building my web site. I will not go into the details of that script here. Suffice to say that Microsoft has done a good job.  Notice however that when the “Build” task finishes, the final web site is put in the “./source” folder. In an MSBuild script it is common practice to operate with relative paths. This causes the least amount of hassle.

Next step is to deploy the web site to the IIS server. I did this by putting a set of tasks into the “AfterBuild” target in the .wsproj file. (Same as above, but with some details removed and some added)

<Project ...>
   <PropertyGroup>
      <Customer>$(FOO_CUSTOMER)</Customer>
      <DeploymentDirectory>C:\ContinuousIntegration\$( Customer)</DeploymentDirectory>
      <WebSite>Default Web Site</WebSite>
      <AppName>$(Customer)</AppName>
      <FullAppName>$(WebSite)/$(AppName)</FullAppName>
  </PropertyGroup>  

....
  <Target Name="AfterBuild">
   <ItemGroup>
       <File Include=".\source\**\*.*" />
    </ItemGroup>

    <RemoveDir Directories="$(DeploymentDirectory)" />
    <MakeDir Directories="$(DeploymentDirectory)"/>

    <Copy 
      SourceFiles="@(File)" 
      DestinationFolder="$(DeploymentDirectory)\%(RecursiveDir)">
    </Copy>

     <Exec Command="appcmd delete app /app.name:"$(FullAppName)"" ContinueOnError="true"></Exec>
    <Exec Command="appcmd add app /site.name:"$(WebSite)" /path:/$(AppName) /physicalPath:$(DeploymentDirectory)"></Exec>
    <Exec Command="appcmd set app /app.name:"$(FullAppName)" /applicationPool:FooAppPool"></Exec>
   </ Target>
</Project>

First I copy the compiled files over to the deployment directory, making sure I remove previous files.  Then I use the Exec task to run the command line tool “AppCmd”. AppCmd is used to create an IIS 7 application pointing to the same physical location as the compiled files.

Well that’s it. I have compiled the web site, moved the web site to a deployment directory and created an IIS application pointing at the web site.

A side note

I have omitted talking about the problems I had figuring out the difference between compiling a Web Application and a Web Site. Not sure I have the big picture on this question yet. It seems like you easily can build a Web Application Project using only a MSBuild task (within MSBuild)

<MSBuild
      Projects="$(ProjectFile)"
      StopOnFirstFailure="true"
      Targets="ResolveReferences;_CopyWebApplication;_BuiltWebOutputGroupOutput"
 Properties="WarningLevel=0;DefineConstants=TRACE;Configuration=Debug;Platform=AnyCPU;Optimize=true;DebugSymbols=false;OverwriteReadOnlyFiles=true;WebProjectOutputDir=$(DeploymentDir)\;OutDir=$(DeploymentDir)\bin\;DocumentationFile=;"/>

That's it. For now. In a later post I guess I will talk about getting the script to run in TeamCity.

Comments F Share