Polymorphic rants

  • Control flow in Javascript

    • 26 Sep 2011
    • 0 Responses
    •  views
    • javascript
    • Edit
    • Delete
    • Tags
    • Autopost

    Mark Needham recently wrote a blog post on how his team worked around a Javascript asynchronous unwanted behaviour.

    They want to iterate over a collection, executes some code for each element of the collection and only when all the collection has been traversed execute a final step.

    Obviously this didn't work. Due to the asynchronous nature of Javascript the do something with grid block is invoked before the grid itself is filled and nothing interesting happens.

     

    Their final approach is solving the unwanted behaviour by mostly removing any possible event driven behaviour and handling control in a full imperative way, explicitly calling functions in the expected sequence.

    This works but it's not leveraging Javascript nature, and the code itself results harder then what it should be.

    How the same problem can be solved embracing asynchronous thinking ?

    The way to achieve control flow in Javascript is with events.

    When a block is completed it emits an event. All the interested parties will be listening to the specific event and execute their specific job, contributing to our overarching business flow.

    This alternative solution is probably also more idiomatic and as a consequence more coincise and easier to read.

    • Tweet
  • Introducing node-tail: a NodeJS tail library

    • 15 Sep 2011
    • 0 Responses
    •  views
    • Javascript forward node
    • Edit
    • Delete
    • Tags
    • Autopost

    In the previous blog post I described the architecture of the firehose we built in Forward with NodeJS.

    At the lowest level each node has to tail a log file.

    Tom Hall and I couldn't find any useful cross-platform (i.e. not relying on the unix tail command) node module for that task, so I ended up writing node-tail:

    Using node-tail is very simple:

    node-tail is also available via npm, just install it with:

    > npm install tail

     

    • Tweet
  • Building a firehose with NodeJS

    • 12 Sep 2011
    • 2 Responses
    •  views
    • Edit
    • Delete
    • Tags
    • Autopost

    In Forward we handle a huge stream of real time data and we are always looking for interesting ways to use that stream.

    We already have a Hadoop cluster for high latency analysis (mostly reporting), but recently we started building a set of tools that can give us a near real-time view of what's going on.

    With this goal in mind I have been recently involved in building a data firehose with NodeJS.

     

     

    The result is the following:

    Firehose

     

    The lowest layer of the firehose is a thin component installed on each server that tails the log file we care about and publishes each log entry to a collector (called firehose-master) via ZeroMQ. The master collects the log entries from all the nodes and republishes everything to the rest of our software ecosystem as a single stream via a single ZeroMQ end point.

    With this architecture we easily preserve the horizontal scalability of our main service, in fact adding a new node to the firehose is as simple as installing the tail component on the new server and adding its IP address to the master configuration file.

    This stream can now be the core foundation of clients that consume the firehose for different purposes, from real-time trends visualisation to HDFS data bulk load.

     

    • Tweet
  • About

    1232 Views
  • Archive

    • 2011 (4)
      • September (4)

    Get Updates

    Follow this Space »
    You're following this Space (Edit)
    You're a contributor here (Edit)
    This is your Space (Edit)
    Follow by email »
    Get the latest updates in your email box automatically.
    Loading...
    Subscribe via RSS