DW and Big Data
One of my specialties is large warehouses and the new trend of big data maps very nicely into this.
My basic architectural argument is that you should target data warehouse products to systems that HAVE a data model and that Big Data is best used where the model is more loosely defined and near schema-less – which include many of the data structures traditionally used for auditing and history tracking.
If you think of the data flow at a very high level, from a source to the warehouse, this is my approach:
I have written quite a few blog posts that I hope you will find are coming together in a coherent theme of design guidelines. To make it easier to locate them, I have provided the index below.
Keys and how to handle them
In order to deliver clean and integrated data, the proper and high performance handlings of keys is crucial. I have blogged extensively about this here:
- Why Integer Keys are the Right Choice
- Good keys, what are they like?
- An Overview of Source Key Pathologies
- Transforming Source Keys to Real Keys (two parts)
- Physically Placing the Maps in the architecture
- Why Surrogate Keys are not Good Keys
Warehouse Knowledge
Information about large databases and how they work.
Myth Busting
I like to put “common sense” recommendations from the database community to the test. Here are the posts I have written about this so far:
-
July 6, 2012 at 20:26Why Integer Keys are the Right Choice! « Thomas Kejser's Database Blog
-
July 6, 2012 at 20:34An Overview of Source Key Pathologies « Thomas Kejser's Database Blog
-
July 6, 2012 at 20:35Transforming Source Keys to Real Keys – Part 1: Introducing Map tables « Thomas Kejser's Database Blog
-
July 6, 2012 at 20:38Transforming Source Keys to Real Keys – Part 2: Using Maps To Fix Key Problems « Thomas Kejser's Database Blog
-
July 6, 2012 at 20:43Physically Placing the Maps in the architecture « Thomas Kejser's Database Blog
-
July 6, 2012 at 20:46Why Surrogate Keys are not Good Keys « Thomas Kejser's Database Blog
-
July 6, 2012 at 20:49Boosting INSERT Speed by Generating Scalable Keys « Thomas Kejser's Database Blog
-
July 6, 2012 at 21:03Why “Date BETWEEN FromDate AND ToDate” is a dangerous join criteria « Thomas Kejser's Database Blog
-
July 6, 2012 at 21:07Don’t Become a One-trick Architect « Thomas Kejser's Database Blog
-
July 6, 2012 at 21:52Defining the Good Data Model « Thomas Kejser's Database Blog
-
July 6, 2012 at 22:09Intermezzo–Data Modeling « Thomas Kejser's Database Blog
-
July 6, 2012 at 23:44Good keys, what are they like? « Thomas Kejser's Database Blog