What Is This Shit Called?
I was going to say all this sounds pretty boring but actually if you put it this way it sounds like something out of an action movie!
Extract, Transform, Load. It might seem mundane and it probably is but somehow, I seem to get a lot of satisfaction thinking about how to elegantly and efficiently program solutions for problems of this kind. Unfortunately though I didn’t know that the stuff that I was doing was called ETL (Extract, Transform, Load) when I started out. Had I known that, maybe I could have used some of the work done by people before me but unfortunately I was oblivious. I guess my membership with the GIP was still valid at that time.
So That’s What It’s Called
As luck would have it though, salvation came about the time that I was working on the second incarnation of my homefuckedup framework. At this time, Ayende started posting about ETL (there are probably a lot of older posts but this was the oldest I could find quickly) and I learned a great deal. Most importantly, that this business was called ETL. Extract, Transform, Load… What the fuck. Who would’ve figured that huh? I was just calling it data transports or pipelines or something but “extract, transform, load”? I bet the guy who came up with that can make taking a shit sound like the greatest thing in the world (sometimes it is though – when the ritual succeeds perfectly).
Sidenote
By a strange fuckup of coincidence, the same crazy ass shit happened to me a few weeks ago when, after fucking 5 years of plowing, I finally found out that the crazy ass database shit we’ve been doing is called Entity-Attribute-Value modelling (EAV). Yeah that’s right: EAV. It’s some kind of screwup of the late 80′s when people were investigating how to store objects in relational databases. But even though EAV is old and outdated, I find this technology strangely cool.
Stuff and Rx
Rhino.ETL is the bomb and if you are sick of SSIS, DTS of whatever you call it, please, check it out. It saved me a lot of headaches because it allows me to do all the stuff in code. I don’t want to program ETL packages by playing a game of click, drag and fill in the properties. I just want to say what I want done and Rhino.ETL allows this. You can use the DSL based on Boo but I found more joy in running compiled assemblies as ETL packages. And when the default driver doesn’t suit, you can easily implement your own.
Nowadays, there is just so much cool stuff coming out that seems usable to investigate. Take for example the work of Erik Meijer and team. One of our unsung heroes and mind you – he is on the SQL Server team so don’t worry mates: Erik is on the job!. For me though, there is some kind of intangibleness with his latest creation, Rx (Reactive Extensions). It’s cool as Pluto but somehow you wonder how it can help you in less contrived scenarios.
Still, I love writing the sort of code that it encourages you to write. With the continuations and the lambda’s and fluent interfaces that make everything fit together like it was meant to be that way. I even like the nested generics that pop up sometimes in the intellisense. When you actually call it though, it’s a lot less brutal (all thanks to type inference) but you can’t help to marvel about those nested generic type definitions. Dutch people would say: “Het is mooi van lelijkheid.” Which roughly translates to something like: “It’s beautiful by its ugliness.” Then again, you can sculpt it anyway you like.
And of course, you also know: “if Erik spend so much time designing this, it has to be useful.” But of course, not every secret is told. We are like apes. We are given a tool that is very powerful but not a lot of people are able to use it. We wonder: “Why has the great one invented a tool like this?” “What use can it have?”
