Creating custom metadata solutions continues to be a popular topic. Last time I wrote about it, I asked whether it was wise to focus on building your own solution, but it seems if you want to manage your metadata, you're going to have to bite the bullet and customize to some extent.
At least, that's what I've gathered from a recent LinkedIn Group discussion on the topic.
That's the bad news. The good news is that you don't have to start from scratch and an ETL tool with the right capabilities can get you a long way down the road, according to Paige Roberts, a software engineer who works in Pervasive's Product Marketing, Integration Division.
The metadata management needs of individual enterprises are so widely varied that trying to create a one-size-fits-all metadata solution borders on the impossible. But writing the whole thing from scratch isn' t the answer either. Custom coding huge chunks of technology that have already been built by a dozen companies seems like picking up a hammer and chisel to invent a wheel because you want a custom car.
In particular, as Daniel Doiron, a project manager based in Canada, pointed out during the LinkedIn discussion, you want a tool that provides data lineage and data profiling. Posted Doiron:
Data Lineage is, along with data profiling, the most significant metadata application. You get those right and 85 percent of your ETL related metadata management problems are gone. If you wish to maintain business definitions and glossaries, it's cool and mandatory for end users. Your ETL tool will pick them up if they exist.
Both Roberts' post and the LinkedIn Group discussion are great resources for learning more about metadata solutions. You'll have to join the group TDWI's Business Intelligence and Data Warehousing Discussion to read the answers. It's a great group, and worth joining even if you don't care about this particular discussion. Plus, joining both the group and LinkedIn at the basic membership are free. If it's not at the top of the discussion page, you should be able to search the group for "Metadata without a tool" to find it.
On a related note, expressor's VP of marketing also wrote an interesting piece on why metadata is critical to integration and creating an effective ETL development environment.
And if you found yourself going "meta What?" during this post, I highly recommend Gartner Managing VP David McCoy's recent blog post for a quick and fun explanation of metadata.