Polymorphic rants

Mark Needham recently wrote a blog post on how his team worked around a Javascript asynchronous unwanted behaviour.

They want to iterate over a collection, executes some code for each element of the collection and only when all the collection has been traversed execute a final step.

Obviously this didn't work. Due to the asynchronous nature of Javascript the do something with grid block is invoked before the grid itself is filled and nothing interesting happens.

 

Their final approach is solving the unwanted behaviour by mostly removing any possible event driven behaviour and handling control in a full imperative way, explicitly calling functions in the expected sequence.

This works but it's not leveraging Javascript nature, and the code itself results harder then what it should be.

How the same problem can be solved embracing asynchronous thinking ?

The way to achieve control flow in Javascript is with events.

When a block is completed it emits an event. All the interested parties will be listening to the specific event and execute their specific job, contributing to our overarching business flow.

This alternative solution is probably also more idiomatic and as a consequence more coincise and easier to read.

In the previous blog post I described the architecture of the firehose we built in Forward with NodeJS.

At the lowest level each node has to tail a log file.

Tom Hall and I couldn't find any useful cross-platform (i.e. not relying on the unix tail command) node module for that task, so I ended up writing node-tail:

Using node-tail is very simple:

node-tail is also available via npm, just install it with:

> npm install tail

 

In Forward we handle a huge stream of real time data and we are always looking for interesting ways to use that stream.

We already have a Hadoop cluster for high latency analysis (mostly reporting), but recently we started building a set of tools that can give us a near real-time view of what's going on.

With this goal in mind I have been recently involved in building a data firehose with NodeJS.

 

 

The result is the following:

 

 

Javascript has become in the last years the language for rich web applications,  but still it rarely receives the level of attention it deserves. Libraries such jQuery boost our productivity but the code we often end up writing tend  to be a big ball of mud, with an entangled mix of presentation logic, busines logic and server side interaction,  all incredibly hard to test and to maintain. It's time to move away from this approach and start writing better quality Javascript. The very first step required to avoid Javascript spaghetti code is start thinking to Javascript as a first class language, and start dealing with it with the same mindset and approach we would use for any server side language. With this new approach in the same way we identify roles and integration points  in server side code we want to start building abstractions in our Javascript codebase. With the right abstractions in place we are defining clear boundaries between different parts of the system, and as a consequence our code is simpler, we promote reuse  and the DRY principle, and  we are finally enabling better testing. Let's look at any standard Web 2.0 Javascript code from this new perspective: now the DOM and HTTP are two clear integration points. Our javascript code manipulates the DOM adding  nodes or changing existing nodes content in the same way any other language would interact with a database. Each call to a server over HTTP via Ajax is exactly the same of calling a web server from our server side code. With these very two first abstractions in mind, we can start rewriting our code, isolating these interactions behind clearly defined objects. Now the javascript code is not a ball of mud anymore, but a network of objects that collaborate. With these smaller objects  in place we can now favour interaction based tests, moving away from any dependency on the DOM and on the HTTP protocol. The identified abstractions let me actually mock  and stub things out and test if the different tiers in my javascript code are exchanging the right messages. This separation of concerns keep the business logic nicely isolated from the user interface transformations, enabling also a cleaner state based testing for that specific part of the code (there are no more dependencies on the DOM).

Friday, 20th of November 2009, the 6th Italian Agile Day will be held in Bologna. It's a great opportunity to share experiences and practices, learn, discuss and meet other praticioners part of the  Agile community...and moreover it's free :-) I'll see you there !

The beginning of a project is always a hectic period where several things have to be put in place in order to be able to start the actual development from a solid position. Interestingly enough, I see that what is important to have ready for ThoughtWorkers most of the time is not what has to be ready for other people. Given that obviously every project is different and deserve specific attention, here is my list of things that has to be ready before the kick off of iteration 1. Infrastructure A repository, a Continuos Integration Environment with Cruise (or suitable alternatives) installed and working, a QA environment where we can deploy every successful build whenever we want, a basic build script (i.e. build/run tests/package). Pairing boxes, each one with exactly the same configuration. Architecture Just identify the core services/components, there's no need to go into detail for each one at this time of the project. If we identify what is the main responsibility of a service is good enough for now, the details will be discovered later in the project. If we can easily identify the way services will communicate (web service ? message broker ?) good, if not through good OO principles we can abstract the low level mechanism and reduce the cost of change later, deferring the final decision to the point in the project where we have more understanding of the technological constraints. Patterns Pairing and frequent pair rotation will help in spreading the knowledge of the approach to solve specific problems and in maintaining consistency throughout the code base, so most of the time there is no need of defining a sort of "project dictionary" of valid patterns at day 1. If we're introducing new approaches to solve a specific problem, it's important to highlight pros and cons of the approach so that people know what they are doing once pairing. Most of the time these are probably enough to start Iteration 1. All the other decisions can (and sometimes should...) be deferred to a later stage.

In a lot of companies  the productivity  of a QA is measured through the number of defects she raised: the  more defects she finds, the harder she works, therefore the better she is. This approach has an interesting corollary:  if you have really good QAs you don't have good developers. If your QAs are finding a lot of defects this means that your developers are quite poor and are regularly introducing defects in the code base. This approach isn't obviously team oriented: QA and Development are seen as two separate and independent entities that interact through a specific contract. If we truly believe in collaboration the most important productivity metrics are not related to a specific role/set of skills but to the team capability of deliver quality software in time and in budget. We need to ask ourselves why the defects are finding their way into the application deployed in the live environment (incomplete acceptance criteria ? not enough analysis or understanding of the domain ? gap in the test suite ?), and fix the issue as a team, not just using that number as the boundary between two streams of work.

Delivering software is a pretty complex activity that requires interaction between people with different skill sets. One of the cornerstone of Agile Development is continuous improvement, and one of the tool often used to learn and improve  is the retrospective. In a context where the collaboration is not effective, people tend to look for local optimization instead of seeing the big picture, and you have things such  the "QA retrospective"  or the  "Developer retrospective". In this scenario a "QA retrospective" (as the Dev's one, or a BA's one)  is probably more harmful than anything else; the specific issues that will be identified won't address the whole activity of delivering software but will only be focused on that specific step ("we need x to do y"). But what is the impact of that step in the overall process ? How that step fits into the chain of event that will take a business idea to be delivered as a software artifact (hopefully in a timely and quality fashion ) ? Don't get me wrong: it's definitively through specific improvements that you improve your overall process but if  don't frame your changes in the big picture, it's more likely that your changes will impact your process in the wrong direction and actually cause an additional waste. Each attempt of optimization should therefore start from the clear analysis of what is wrong from a high level point of view and only then it's time to shift our attention to the specific details of each step.

Acceptance Testing is a fundamental practice: it gives you confidence that your application behaves as expected from the end customer point of view. In the Flex world there are some projects that are currently emerging in the Acceptance Testing space, each one with specific advantages and weak points. Let's have a look to some of these. FlexMonkey FlexMonkey Open source but based on closed source API (the Automation API are released only with FlexBuilder and not with the open source Flex SDK), it's based on the record-playback approach and as far I  have seen is not easily integrable with a Continuous Integration server. Flash-Selenium Flash-Selenium is open source, and it works as an extension of SeleniumRC; the tests can be written in Java, .Net, Ruby or Phyton and the integration with a Continuous Integration server is therefore quite out of the box. SeleniumFLex API SeleniumFlex is another open source project. it's an extension of SeleniumIDE, test should be written in Selenese. FunFX FunFX is based on the automation API (therefore you can use it only if you're owner of FlexBuilder), the fixtures are  written in Ruby. At the moment there isn't a  de-facto solution: the community is quite dynamic and all these different tools are trying to find their space. Which one is  my favourite ? It's really hard to say, I think that the context matters a lot. I  don't like the solutions that require the Automation API simply because this tights you to a vendor just for testing; it's also true that the pure open source solutions requires some extra-hack (like exposing methods through  ExternalInterface) that are less than ideal. Looking at the two pure open source tools I like Flash-Selenium for its out of the box integration with Continuous Integration server, while I prefer Selenium-Flex approach for handling the necessary ExternalInterface configuration.

Walking in an office and just looking around for 10 minutes is enough to have a feelingof the level of communication in the environment. When:

• co-located people communicate mainly via IMs, mails, or comments on online collaborative tools instead of face-to-face conversation
• a heavyweight tool is the preferred way for driving the work flow instead of using it only for backup and tracking
• on the whiteboard you can read the outcomes of the retrospective of 7 months before

There's definitively something wrong. What about your team ? How many of the above bullet points are you ticking off ?