Is Big Data a business initiative or an IT initiative? For decades, businesses have been trying to make sense of disorganized transaction data sprinkled throughout the enterprise, relying heavily on human resources to analyze data via data warehouses. Steps traditionally include collecting, cleaning, conforming, consolidating and organizing the data for business analysts to perform ad-hoc queries to answer key questions such as “How are my sales, by region?” or “How is my inventory, by product?” Of course, to query data, it must be structured in a data warehouse and browsed via business intelligence tools, correct? Well, not anymore.
Big Data has changed the rules. Before embarking on a new Big Data project, a policy for handling this data must be in place. Joe Caserta, founder and CEO at Caserta Concepts, a consulting and technology services firm that specializes in data warehousing, business intelligence and Big Data analytics, and co-author of the industry best seller ‘The Data Warehouse ETL Toolkit,’ shares four steps to a Big Data strategy for success.
Click through for four steps to create a successful Big Data strategy, as identified by Joe Caserta, founder and CEO at Caserta Concepts.
A Big Data project requires planning and sophisticated orchestration. Why? Because it disrupts and introduces new hardware, software, resources and data sets. It will involve technical toolsets never experienced by the business or IT, and bring together data sets never before integrated. New policies, procedure, training and project planning need to be carefully provisioned.
Big Data solutions include data warehouse data, raw transaction data and unstructured log data. A financial Big Data solution may have trade data, market data, position data, news feeds, customer reference data, weblogs and system logs. Repeatable processes must be established for the consumption of each data source. Techniques inherited from traditional data warehousing such as change-data-capture, micro-batch processing and real-time data streaming still apply.
The paradigm shift to Big Data introduces a new role in the corporate organization: the data scientist. This role requires deep understanding of advanced mathematics, system engineering, data engineering and domain (business) expertise. In practice, it’s common to utilize a data science team, where statisticians, technologists and business subject matter experts collectively solve problems and provide solutions.
Every Big Data strategy must include continuous monitoring and maintenance of the technical solution. As data volume and analytic requirements increase, the configuration of the solution must evolve and grow. The distributed system will need to have nodes added, data redistributed/balanced, replication adjusted, and the configuration for all of the above continuously fine-tuned for optimal performance.
Before a Big Data project is launched, a strategic readiness test should be performed to assess the adoption of the new paradigm. Business analysts will need to be retrained or repurposed. The goal of shifting to a Big Data platform may include changing from reactive analysis (did that campaign work?) to proactive (what should be our next campaign offer?). Now we can proactively influence non-buyers to follow behavior patterns of loyal customers, or re-stimulate active customers when their behavior pattern begins to look like a lost customer.
Now armed with a complete Big Data ecosystem, including recommendations created by data scientists, it’s possible to close the loop – feed the results of the analysis into the engine that creates the customer experience: your website, marketing department, sales force, product development and customer service. Moreover, the Big Data machine can now consume recommendations provided as a result of its analytics correlated to new customer behavior patterns and quantify its effectiveness.
Lastly, remember that with any new initiative, there’s risk when implementing a new Big Data project. A tool, a language or a platform alone does not make a solution.