Two Steps to Designing Your Own Big Data Architecture

Loraine Lawson
Slide Show

Big Data: Not Just for Big Business Anymore

If one of your goals in the New Year is to move toward using Big Data, then it’s time to move beyond the theoretical discussion to the nitty-gritty of implementations.

That doesn’t mean you should ignore your strategic goals, of course: It just means filling in the integration blanks between having Big Data and using Big Data.

TechTarget recently published a good starting point by excerpting chapter 10 from “Data Warehousing in the Age of Big Data,” written by Krish Krishnan, who is a Chicago-based executive consultant with Daugherty Business Solutions and a TDWI faculty member.


The two-part excerpt, like most technology books, is no-nonsense and on the dry side. Still, it’s laser-focused on the integration layer, so it’s definitely worth reading if you’re a CIO, architecture or a technology decision-maker who is likely to be involved in a Big Data project.

Krishnan breaks it down into two overarching steps: The data-driven architecture, which covers all the head-work you need to do before you build. The second step is the physical architecture implementation, which is where you get to buy and build things.

Each part has several action items. (You didn’t think it’d be that easy, did you?)

Everybody always wants to skip ahead to that last one, but don’t. We all know you have to do your homework first.

“Finalizing the data architecture is the most time-consuming task that, once completed, will provide a strong foundation for the physical implementation,” Krishnan writes. “The physical implementation will be accomplished using technologies from the earlier discussions, including big data and RDBMS systems.”

Then Krishnan hunkers down to the type of down-and-dirty-integration discussion we’ve wanted for years now, such as how you know what kind of tools you need, including whether a Big Data appliance would be a good fit for your organization.

First, a Word About Data Integration

In the past, integration was pretty straightforward: You need batch processing, and then get ETL. You need incremental data loads, so look at change data capture.

With Big Data, you’ll likely use a variety of integration approaches, as well as master data management, semantic technologies and so on.

That’s why Krishnan recommends you make the integration layer “data-driven,” which means your first order of business will be to identify all the different data types you want to use. That includes everything from enterprise data warehouse data to sensor and semi-structured data.

Don’t worry: Krishnan’s included a list to help you. It’s comprehensive and in no way what you would call “short.”

Only after you’ve figured that out should you move on to action item two: Designing the architecture. This does not by any stretch mean you’re almost ready to start buying things, by the way.

No: It means you start defining master data elements, data types, the data owners and the data stewards. In other words, this is where you start the data quality and data governance discussion.

Action step three is workload management, which Krishnan writes is the biggest need for processing Big Data. That’s a pretty long, dense read, so I’ll let you go to the source for that.

Finally, the last action step for building the data architecture is pinpointing your analytical processing requirement. You can almost taste the RFPs.

Now You Buy and Build

Most of the second excerpt deals with the physical parts of your integration layer and architecture, which includes discussions on:

  • Data loading options, including MapReduce, appliances and batch processing
  • Data availability and special issues related to Big Data
  • Data volumes and problem areas such as retention, compliance and legal issues in acquiring data
  • Storage performance

As I said, it’s not an easy read, but it is an extremely useful read. If you’d like to hear more about Big Data from Krishnan, I recommend following him @datagenius on Twitter.



Add Comment      Leave a comment on this blog post
Dec 19, 2013 10:22 PM sunitha sunitha  says:
Great post, Learn and train on Big data at Jan 24-26 Big Data Bootcamp-Santa Clara Register at http://globalbigdataconference.com/bigdata-bootcamp-sca-january-2014.php use promo code BLOG to get $200 discount expires on Dec 23 Reply

Post a comment

 

 

 

 


(Maximum characters: 1200). You have 1200 characters left.

 

 

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.


 
Resource centers

Business Intelligence

Business performance information for strategic and operational decision-making

SOA

SOA uses interoperable services grouped around business processes to ease data integration

Data Warehousing

Data warehousing helps companies make sense of their operational data


Thanks for your registration, follow us on our social networks to keep up-to-date