A Marketers Guide to Real Time (Streaming) Analytics

The speed at which data is generated, processed, and analyzed today is increasing at an unbelievably rapid pace. A broad range of industries are struggling to deal with increasingly large, complex, and rapidly changing data. Retailers, Social media, ad tech, gaming, network service providers and Internet of Things (IoT), and even airline operators are working to deal with increasingly large, high velocity data. Traditional Big Data solutions based on Data Warehousing or Apache Hadoop are not well-suited for most real time data processing. What is needed is a real time (Streaming) Analytics.

The Apache Software Foundation is a leading non-profit dedicated to sponsoring open source projects, and now sponsors a dozen projects focused on streaming data. Streaming data refers to a never ending sequence of records, typically coming from multiple sources, and is growing in importance for marketing professionals.

Applications of Streaming Big Data:

Streaming Big Data solutions complement and doesn’t replace Big Data analytic systems. Customer segmentation, campaign performance, customer renewals, and other studies rely on historical results over time. Streaming Analytics, on the other hand, reveals how a campaign is performing today and delivers personalized real time engagement with customers.

Streaming Big Data systems are growing rapidly in a variety of uses. Financial service providers rely on real time data processing to detect and prevent fraud. Retailers deliver personalized offers to shoppers, and make dynamic pricing updates. Customer engagement strategies drive improved cross sell, customer renewals, and automated customer service can lower support costs.

Streaming data systems also play a growing role in shaping customer’s digital experience. Insurers and financial service providers use streaming data processing to detect and respond to attempted fraud. Stock trades are routed for order processing using smart algorithms. Airlines combine weather and engine performance in-flight, to optimize performance and on-time arrival. Commuters appreciate updated driving routes based on real time traffic.

If you haven’t already become acquainted with Streaming Data for marketing, now is the time!

An Overview of Streaming Big Data Systems:

Streaming data systems combine events from multiple sources and often include online transactions, social media posts, web browsing sessions, and wireless calls. Specific events are targeted, which are formatted by “event listeners” for efficient processing. Events are routed to an “ingestion” service such as Amazon’s Kinesis, or directly to the event processing system.

Customer profile information is required for events to be acted upon. To respond to customers in real time, the customer name and contact methods are needed (ie., “enrichment data”). While collecting events in real time is challenging, the greater challenge is combining events into scenarios to drive real time actions. The creation of scenarios, the ability to adapt or update scenarios, and manage hundreds of scenarios, is a key challenge in Streaming Analytics.

Streaming Data Applied: a Customer Engagement Scenario

The first step in designing a Streaming Data system is to identify events for real time actions. In the example below we imagine how a traveler’s experience is enhanced with Streaming Analytics. In these scenarios real time data is used to keep the customer informed, and improve the customer experience. A branded mobile application would significantly enhance the travel experience by:

1. Informing travelers in real time of flight schedule changes, and taking into account commute times from their residence are provided recommended departure times and routes to the airport.
2. Travelers are informed of parking options, and presented with updated parking offers.
3. Travelers who miss check-in are placed into an outbound call queue for rebooking
4. Operations can see in real time if a check-in kiosk stops functioning
5. A Bluetooth enabled beacon adjacent to check-in kiosks could identify travelers who need expedited check-in or security clearance services.
6. Travelers are informed when baggage is loaded. In the event that baggage misses making a connection, the traveler is notified, and can instruct the airline on delivery without having to visit a “customer service” counter.
7. Operations prepares efficiently for jet maintenance, based on in-flight maintenance alerts.
8. Travelers are provided connecting flight information, and directions to Uber pick up locations and other services.

Streaming Analytics Applied for Airline Traveler Customer Engagement

Streaming Analytics isn’t limited to customer engagement, but is also widely used for operations.In-flight routes are optimized for fuel consumption and on-time performance, and maintenance jobs can be relayed in during the flight to allow for efficient on-ground service, and minimizing unplanned downtime.

Conclusions and Next Steps:

The capabilities that we’ve explored here are practical and affordable, and lower risk than many Big Data strategies. A system capable of supporting over 1 million users, with up to a dozen scenarios can be implemented in less than a month, and can be hosted on a modest Amazon AWS cloud based machine. The advantage of Streaming Analytics systems is that real time actions generate immediate and easily measured results.

The Gartner Group recognizes the following vendors as leaders in Streaming Analytics: Apache Foundation, EVAM, Microsoft, Oracle, SAP, SAS, Software AG, and Tibco Software (Source: Gartner Hype Cycle for Data Science, July 25, 2016).I recently co-authored a comprehensive Guide to Evaluating Streaming Analytics solutions, co-sponsored by EVAM.