Real-time Personalization: Choosing the right tools
Real-time personalization is the pathway to better user experiences. But it often feels like you must choose between complex DIY and expensive SaaS. Here's the happy middle path.
Picture this: You’re an engineer working at a fast-paced company selling products or services in the digital realm. Your goal? To quickly build a system that will place the right offer, product, or experience in front of your company’s customers at the exact right time. The results? Increased order value! Less churn! Better customer experiences! And maybe even a raise, bonus, or promotion for you.
What I’ve just described is called Real-time Personalization. And for online retailers, sports betting and gaming companies, transportation and travel agencies, social media companies, advertisers, and other such digitally-focused enterprises, real-time personalization is the gold standard. We’re all trying to build it, and fast.
But here’s the catch: Building a Real-time Personalization system comes with tradeoffs. You either have to choose complex, potentially unproven technologies like vector databases and machine learning or turn to some off-the-shelf software or system like Marketo or IBM Watson which command a hefty price tag and lack the flexibility that you really want in your data stack.
In this blog post, you'll learn how Real-time Personalization can be built using real-time data platforms, which eliminate the overhead of managing complex infrastructure without sacrificing the power and flexibility to build custom real-time personalization logic.
In this post, you’ll learn a new path: Building real-time personalization systems without complex tools or bloated software. Instead, you’ll learn how to put the right real-time data architecture in place so you can quickly and cost-effectively build real-time personalization systems that scale.
What is Real-time Personalization?
Real-time personalization is the practice of dynamically tailoring user experiences based on real-time data about individual user preferences, behaviors, and contextual information like real-time inventory.
On the surface this sounds simple, but under the hood is a real-time data pipeline and feedback loop that captures user activity events, analyzes their contents, and returns hyper-personalized recommendations in milliseconds.
How is real-time personalization different from web personalization?
Traditional web personalization typically involves segmenting users into predefined groups (or “cohorts”, as they’re often called in the advertising world) and delivering static content based on those segments, often through customized URLs or “smart content” that updates based on offline data calculated before a user begins a web session.
Traditional web personalization isn’t bad, per se, but it misses out on data that’s being collected right now, especially data that a user is generating within their current session. When used for personalization, this data can be a powerful boon to conversion rates and improving user satisfaction.
Traditional web personalization uses offline data to serve static content. Real-time personalization injects dynamic content into a user session based on data collected during that session.
Real-time personalization makes use of this kind of data. Instead of relying on static content and offline segmentation, real-time personalization uses in-session customer engagement data and metrics paired with up-to-date product or inventory data to update landing pages in real time, injecting new, customized content directly into the session based on the streaming output of the recommendation engine.
Consider a practical and common example of an online clothing retailer. A visitor arrives after clicking an advertisement. Traditional web personalization might use smart content to serve a customized landing page based on the keyword the visitor searched. While this can improve conversion rates for that landing page, what happens if the user continues their session? Maybe they find another product they like better, but it’s out of stock, so they abandon their cart.
With real-time personalization, the retailer can use a real-time recommendation engine and automation to place an offer for a similar product in front of them during or before checkout. Done well, this results in lower cart abandonment rates, more conversions, and even upsells.
What are the benefits of real-time personalization?
The benefits of real-time personalization can be summarized as follows:
- Increased customer satisfaction and loyalty
- Increased time on site and reduced bounce rates
- Increased conversion rates
- Increased average order value
- Increased relevance to customers
- Increased brand preference over competitors
- Increased marketing ROI
These benefits are made clear in consumer preferences towards personalization. According to a study conducted by Epsilon and GBH Insights, 80% of consumers express a stronger affinity towards brands that personalize their experiences. That same study shows that personalization is highly effective in increasing the lifetime value of a customer through more conversions, larger sales, and longer-lasting brand affinity.
80% of consumers express stronger affinity towards brands that personalize their user experiences.
At the core of any real-time personalization model is the idea that you should give the customer what they want when they want it. If you’re able to update your customer profiles in real time based on behavioral data, purchase histories, and customer journey mapping, you’re more likely to attract, convert, upsell, and retain customers on the strength of the user experiences your brand offers.
In addition, websites that employ real-time personalization are able to keep customers on the website longer, increasing the touchpoints for potential conversions.
The cherry on top of all this is that it will cost less to attract and retain customers, especially as you use real-time personalization to update your advertising models in real-time based on active customer data.
All of this equates to higher revenues at lower costs.
Real-time Personalization examples
Some examples of real-time personalization in various industries include:
- Offering personalized products during checkout in e-commerce
- Offering unique travel experiences to hotel bookers
- Offering special lines or betting incentives for new sports bets
- Creating personalized content feeds for social media users
- Optimizing marketing campaigns with personalized ads
Let’s dig a little bit deeper into these use cases and how they work.
Real-time personalization in e-commerce
Consider an online retailer that tracks a customer’s behavior while the customer is on their website. This first-party data, paired with a smart inventory management system (also powered by real-time data pipelines) enables the retailer to place personalized product recommendations in front of customers in real-time, increasing the likelihood of purchase and upsell opportunities.
Imagine you’re the shopper, browsing their site, looking for sunglasses for your upcoming beach vacation. You search for “sunglasses”. You find the ones you like, but alas, they’re out of stock. You decide to look elsewhere, but go ahead and put a new shirt in the cart and proceed to checkout.
Online retailers use real-time personalization to increase conversion rates and average order values.
In the background, the retailer’s real-time personalization engine has determined that you want sunglasses, you’re willing to buy (because you put a shirt in the cart), and they know the type of sunglasses and style that you are likely to prefer, based on the sunglasses that customers like you bought when they bought that same shirt.
Now, they compare your current browsing and cart data with customers who match a similar profile as you. Then they cross-check it with their current inventory.
As you check out (or perhaps even before), you see an offer: Similar sunglasses as the ones you wanted, with a 15% discount. Sold!
This personalized experience happens in a matter of seconds or milliseconds. Thanks to real-time personalization, it’s a win-win. You, the shopper, are better equipped for your beach vacation, and the retailer has both increased the value of your order and created a happy customer in the process. This is real-time personalization.
Real-time personalization in travel and transportation
Consider a booking agency that helps hoteliers sell rooms. They track competitor hotel prices, third-party booking sites, and user behavior in real time. The goal? Turn “lookers” into “bookers” and increase hotel on-site conversion rates without relying on third-party booking services that take a cut.
Imagine you’re preparing to go on vacation, browsing a hotel’s website looking for a room. You’re also looking at third-party sites to compare rates. You’re making a last-minute booking based on a Trip Advisor referral, and you want breakfast included.
Travel booking companies use real-time personalization to turn "lookers" into "bookers".
Normally, this hotel doesn’t offer free breakfast. But based on your profile and their understanding of your lifecycle data, the hotel calculates and creates a personalized offer just for you: “Coming from Trip Advisor? See what other Trip Advisor customers booked. Oh, and by the way, breakfast is on us!”
Meanwhile, the price is 5% cheaper than the third-party booking site. Sold!
Based on real-time customer behavior data and real-time pricing data from third-party sites, this hotel has a new booking and an opportunity to build lifetime value with a new customer.
Real-time personalization in sports betting and gaming
Betting is inherently risky. Sports betting companies strive to de-risk bets for their users, especially new users. Winning early drives customer satisfaction, which drives retention.
Consider a sports betting site that assesses, in real-time, the types of sports a customer is interested in, the bets they’ve researched most thoroughly, and their likely risk appetite.
Sports betting companies use real-time personalization to decrease the time to first bet.
Imagine you’re a football fan with $50 to drop on the next Champions League match. You’re considering a bet on Barca, but the line is tight.
The sports betting site sees your hesitancy and offers a match: “Place a $50 bet, and we’ll give you special odds.” You win. You feel good. And you’re likely to bet again.
This is real-time personalization in action.
Real-time personalization in social media
Ah, the almighty feed, built with real-time personalization. Social media sites like Twitter, LinkedIn, and TikTok are well-known for their ability to assess user behavior in real time and provide a highly tailored feed of content.
Imagine you’re a TikTok user, and you watch an entire video to the end and then react. Similar users just did the same thing 15 minutes ago, and a few of them absolutely loved the next video that TikTok showed them, sticking around for the entire thing.
So, TikTok shows you that video too.
Social media platforms and news feeds use real-time personalization to serve the most relevant content and increase user retention.
That’s real-time personalization through both content filtering and collaborative filtering, showing you the content you want based on the content you, and others like you, enjoyed. Nearly every social media platform and many news feed aggregators utilize real-time personalization to ensure you stay engaged with your feed.
Real-time personalization in marketing and advertising
Marketing and advertising were some of the first industries to adopt personalization, and real-time personalization takes marketing campaigns to a whole new level.
With real-time technology, marketers and advertisers can update content and ad creative automatically based on user search data, cohort analysis, and other demographic and behavioral data.
The result is lower spending on advertisements, higher clickthrough rates, and more ROI on marketing efforts.
But real-time personalization doesn’t just stop at advertising. For example, imagine you’re a marketer working for a SaaS company, and you’re hoping to improve account-based marketing tactics. You use tools to monitor when visitors from certain high-value companies come to your site. You notice a visitor from your top target, Foo.com, who views the multiple pages on your site during a 5-minute session. In particular, they take a look at your pricing page.
Using real-time personalization, you trigger a personalized marketing email to users in your CRM who have a @foo.com email address, explaining why your pricing makes sense and the potential ROI of your product.
Two days later, you book a meeting with the VP of Finance at Foo, and your pipeline gets a nice bump. This is real-time personalization.
The essential components of a Real-time Personalization system
So, now that you know what real-time personalization is and what it can be used for, you’re wondering: how is it built?
Let’s start by laying out the core components of a real-time personalization system. Real-time personalization systems are a type of real-time recommendation engine. They capture streaming data using event streaming platforms and enrich that data with facts and dimensions stored in databases, data warehouses, and data lakes. Then, real-time personalization engines use real-time analytics or machine learning (or both!) to analyze and calculate a personalization outcome. Finally, these systems expose the recommendation through a real-time API that supports real-time visualizations in user-facing applications.
Let’s dig into each of these components.
Event data streaming
Real-time personalization systems are supported by event-driven architectures. These systems generate and place user events data into an event streaming platform or message queue, either by directly modifying backend application code to interact with event streaming APIs, or by using techniques such as real-time change data capture to track changes in application or service databases and propagate them to downstream systems.
Real-time data processing
Real-time personalization systems process streaming data on the fly, generating metrics or “features” that can be used by artificial intelligence and machine learning systems or other heuristic models to compute the proposed recommendation. Real-time data processing can be applied with many different types of tools, ranging from stream processing engines to real-time databases or real-time data platforms.
Real-time API layer
Fundamental to the concept of real-time personalization is using real-time data to insert a recommendation back into the product, often seconds or milliseconds after the data itself has been generated and processed.
A successful e-commerce site will have many users shopping at the same time, and they want the same personalized experience. That experience shouldn’t degrade during big sales just because more people are accessing the site and more personalized experiences are being generated.
A low-latency, high-concurrency API layer ensures that real-time personalization systems can support this kind of demand.
Keep in mind that many engineers use caching to improve the performance of their API layer. But for real-time personalization, caching won’t do. Why? Caching relies on occasionally loading pre-calculated personalization data into a server-side key-value database. This makes APIs faster, but the batch approach to computing personalizations violates the real-time nature of real-time personalization. You’ll get fast APIs, but not real-time personalization.
A real-time personalization API must serve requests for personalized content in milliseconds based on data that is just seconds old (or less).
A real-time personalization system involves event data streaming to capture user activity events, a real-time data processing engine to filter, aggregate, and enrich streaming data, and a real-time API layer to expose personalized content back to the website or application. Stitching these components together piecemeal can be time-consuming, but off-the-shelf SaaS can be expensive and inflexible. Fortunately, there's an alternative path.
Now, these components can be stitched together piecemeal by a dedicated team of engineers hoping to build a bespoke real-time personalization system. But this is time-consuming, as it requires a lot of overhead to set up dedicated infrastructure, set up and maintain best-of-breed tooling for each component, and integrate each of them together. And the result often has undesirable latency due to the many custom integrations and technical handoffs.
Alternatively, teams can purchase commercial off-the-shelf (COTS) personalization systems to try to handle these functions without the overhead. But this software is often both expensive and inflexible, and it doesn’t enable the types of cutting-edge personalization systems that many engineering teams want and need to build. What you gain in reduced overhead you give up in ROI and flexibility.
There is a happy middle ground between these two approaches. Engineering teams can leverage dedicated real-time data platforms to build highly flexible and performant real-time personalization engines. These platforms provide the necessary infrastructure to handle real-time data ingestion and processing, build real-time personalization business logic, and publish APIs to support user-facing personalization features.
With real-time data platforms, engineers and product teams can invest in building the business logic for highly customized, cost-effective real-time personalization use cases while outsourcing the technical overhead to the platform.
Build a Real-time Personalization pipeline with simple tools
For those who want to build highly customizable real-time personalization solutions without worrying about infrastructure, consider a real-time data platform like Tinybird.
Tinybird is a serverless real-time platform for data and engineering teams to quickly and cost-effectively build real-time personalization systems.
Tinybird supports native Connectors to streaming data sources like Apache Kafka, Confluent Cloud, Amazon Kinesis, Google Pub/Sub, and data warehouses like BigQuery and Snowflake. It can also be used to ingest streaming data directly from an application thanks to its HTTP streaming endpoint.
Furthermore, Tinybird’s SQL engine can be used to build powerful real-time personalization logic. With SQL, engineers can join and enrich event streams with product or user dimensions stored in databases or data warehouses.
And best of all, Tinybird instantly publishes real-time personalization pipelines as low-latency, high-concurrency APIs that can be used within user-facing websites and applications.
Tinybird is a real-time data platform that has been used by companies like FanDuel, The Hotels Network, and large eRetailers to build real-time personalization systems.
In fact, companies like The Hotels Network, FanDuel, and one of the largest eRetailers in the world have used Tinybird to build real-time personalization systems and bring them to market very quickly.
To learn more about Tinybird, you can visit the website, check out the product documentation, or even get started for free with no credit card or time limit.
If you have questions about how Tinybird works for real-time personalization, or how to build real-time data pipelines and APIs with Tinybird, feel free to join us on Slack.