If your brain works the way mine does, you often find yourself doing a variety of related but not necessary tasks along the way to fulfilling a larger goal. For some reason, the history of which I’m not entirely sure, this is called “yak shaving.” Some of my finest work has been done that way, so I’m happy to embrace it, even if I do add hours to the job at hand.

This week’s yak shaving has neared epic proportions, but proved eminently satisfying. I’ve been working on collecting together an overview of big data products. It’s early days yet for Hadoop centered offerings, but 2012 will be the year that Microsoft and Oracle will join IBM and EMC Greenplum in having launched products in the marketplace.

Surveying who-offers-what is a bit like writing code. I naively started off with a spreadsheet, but I quickly realized that real life data doesn’t fit into grids. Even if you can coerce it that way, the effort required to get there is massive. Instead I wanted a structured description of the products that I could amend and refactor as the whole landscape came into perspective.

To get this done, I revived an old interest. Some years ago I contributed a little to Dave Beckett’s Redland RDF project, and so I returned to this technology. RDF is simply a machine readable way of writing descriptions about things.

The task at hand was to collect knowledge about each vendor and their products, and have this in a format that could be used to create feature tables and comparison charts.

In my little project, each vendor has a file in which I’ve written a machine-readable description of their products. Here’s a little excerpt from my description of Cloudera:

<http://www.cloudera.com/hadoop/>
    a :HadoopDistribution ;
    dc:title "Cloudera's Distribution including Apache Hadoop" ;
    :homePage <http://www.cloudera.com/hadoop/> .