Hill Climbing: From Artificial Intelligence to Business Intelligence

David Selinger

Machine Learning as Management Strategy

Innovators like Google and Amazon have been data-centric since their inception, with a repeatable formula that drives business value through exploration, innovation, disruption, and agility. Rather than decision-making by HIPPO (the Highest Paid Person’s Opinion), ideas are tested and judged on efficacy — and everyone is encouraged to test early and often. This focus on metrics enables meritocracy: a virtuous circle that empowers continual innovation throughout the organization.

In 2003, my Data Mining and Personalization team at Amazon brought Jeff Bezos a hypothesis he hated; nonetheless, he let us try it. A small test on the live website confirmed that running ads was extremely profitable. His openness to enabling testing in the first place, and his willingness to choose results over instinct, led to Amazon’s single most profitable project to date.

After a decade observing the most innovative and profitable tech companies, now as a founder and CEO myself, I’ve confirmed that most 21st-century juggernauts share this mindset. I’ve distilled these tenets into a management approach called Hill Climbing (after the computer science thought experiment of the same name). Like the optimization technique that postulates continuous improvement of a solution from any starting point, Hill Climbing enables any business to adopt the following as a framework for decision-making and begin to benefit immediately:

  1. Data democratization (make it accessible to everyone in the organization)
  2. Consistent measurement (compare apples to apples throughout the organization)
  3. Employee empowerment to test ideas (at every level of the organization)

Amazon is a Hill Climbing organization because Jeff Bezos’s decision to test wasn’t a fluke or an outlier. Their innovations don’t stop at web design or product features. A bias toward using data (rather than hunches) permeates the entire company. Amazon analyzes and tests every piece of data — not just the products it sells, but how Finance, HR, and Operations processes operate. That’s how they became a behemoth: Building a metrics-oriented culture meant employees could grasp any metrics — about their own projects and departments, as well as the activities of other teams — in an easily visualized form (which included alerts), so continuous improvement was possible.

Ultimately, by building a numbers-oriented approach in an optimization-focused culture with the infrastructure to execute tests consistently, quickly, and at low cost:

  • Senior management talks less about the IT pipeline and more about strategy. (Rather than spending three-quarters of their time negotiating “the IT Roadmap,” they can focus on “What is core?” and “Do we understand our customer?”)
  • Team members feel empowered to make decisions, execute tests and implement results.
  • The organization as a whole focuses better on core issues than on commodity issues.

At RichRelevance everyone optimizes. How we hire product managers and engineers, for instance, exemplifies our own testing culture. In a typical engineering interview, we often ask candidates to write code to solve a problem. Even if they get it right immediately, how do they know it’s right? How do they test and verify their answers? We want to know about their thought process: How do they validate their own ideas? How do they measure success? What data points do they check to see if what they’ve done is correct?

Customers now demand real-time feedback as well as consistency and ease of experience among touch-points, which have expanded from in-store, POS and call center to include mobile, web click-stream and social preferences. Data increases by an order of magnitude while “the Single View of the Customer” increases in importance to management. Seamless data integration is vital, but legacy technology often cannot deliver real-time decisioning, storage costs for all that new data are soaring, and traditional data analytics and warehousing technologies are reaching their limits.

Given that IT has its hands full with challenges of scalability, availability and security, even as brands must test and learn faster, the key infrastructure enablers of a successful Hill Climbing organization are:

  • Testing. It must be easy to execute a test — whether A/B or MVT — and 95 percent of them should not require IT intervention.
  • Targeting. Marketing and merchandising associates must be able to create a piece of content and — without involving IT — be able to target this content for deployment on the site. Targeting infrastructure is designed to enable non-technical personnel to make rapid, frequent course corrections.
  • Measurement. Measuring the results of a piece of content and its targeting rules must be as simple as pushing a button. Segmentation tools (multi-channel data) and multi-channel reporting must be available. First, the system must measure discrete items of interest in the decision-making process (not just “How much did I sell today?”). Second, how the system responds to test results and your subsequent decisions must be automated.

As systems are selected, built and integrated, testability and flexibility are included as meta-requirements. Rather than initial specs-like elements driving designs too rigidly, flexibility enables nimble changes later — as business conditions change (which they will). Marketing and Merchandising can independently test and update to achieve their goals, just as IT is freed up to meet its myriad challenges.

Along the way, if a department or employee suggests an idea that seems silly, odd or downright distasteful … encourage your firm’s leadership to pause before passing on it. The next technology game-changer, industry transformation or billion-dollar innovation could start with a simple test, enabled by personalities secure enough to tolerate — or even encourage — disruption.

David Selinger, CEO and co-founder, RichRelevance, first garnered international recognition as an expert in the field of eCommerce data analytics and personalization with his groundbreaking work leading the research and development arm of Amazon’s Data Mining and Personalization team. In that role, David increased Amazon’s annual profit by over $50 million (25% of US profit, 2003) setting the industry standard for personalization.In addition, David’s contributions to e-commerce range from developing the first Javascript-powered ad delivery system for Flycast—the same technology used by Google AdSense, DoubleClick and Overture—to leading Overstock’s personalization efforts as Vice President of Software Development and Data Mining. David also co-founded Redfin, a venture backed, online real estate search/brokerage company that set the path for how homes are promoted and sold online. David holds a broad collection of awards and standing patents for online personalization in customer segmentation, analytics and data mining, including over a dozen pioneered at RichRelevance. David’s accolades also include the coveted PC Magazine Editor’s Choice Award (won for 2Roam Wireless, acquired by Air2Web) and most recently, the San Francisco Business Times’ “40 Under 40″ recognition of emerging Silicon Valley business leaders in 2012. David received his BS in Computer Science from Stanford University.

Add Comment      Leave a comment on this blog post

Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.