The Information Staircase

With the Big Data wave rolling over us these days, it seems everyone is trying to wrap their heads around how these new components fit into the overall information architecture of the enterprise.

Not only that, there are also organisational challenges on how to staff the systems drinking the big data stream. We are hearing about new job roles such as "Data Scientist” being coined (the banks have had them for a long time, they call them Quants) and old names being brought back like “Data Steward”.

While thinking of these issues, I have tried to put together a visual representation of the different architecture layers and the roles interacting with them:

The Big Picture – EDW/DW architecture

Now that the cat is out of the bag on the Kimball forum, I figured it would be a good idea to present the full architecture that I will be arguing for. I was hoping to build up to it slowly, by establishing each premise on its own before moving on to the conclusion.

But perhaps it is better to start from the conclusion and then work my way down to the premises and show each one in turn.

Data Warehouse Introduction

In this blog, I will argue for and against key architectural choices that face the data modeler who wants to build a big data warehouse. I will do my utmost to avoid hand waving and references to “Logical Models”. Indeed, I find that talking about a “Logical Model”, whatever that is nowadays, distracts from the real issues that must be solved by the designer and that such model provide very little value in a real life implementations. I will therefore move directly back and forth from between conceptual (business user understandable) design and the actual implementation.