Meta's Multi-Stage Framework Enhances Ads Ranking Efficiency and User Modeling

Meta's latest advancements in advertisement recommendations illustrate the potential of effective sequence modeling technologies. Each day, its platforms handle billions of user interactions, providing vital temporal signals that reveal individual preferences and intents across various products, ads, and content. Previously, Meta's approach to user action dynamics heavily relied on static and manually engineered features; however, recent innovations emphasize modeling the order and timing of user actions to create richer, sequence-aware representations of user interests and ad preferences.

In this context, Meta's introduction of two architectural breakthroughs markedly improves the scalability of their sequence learning models, enabling predictable scaling akin to large language model (LLM) scaling laws. The improvements include: (1) a multi-stage sequence model that separates the intensive offline user modeling from lightweight online ranking tasks, and (2) a learning method that integrates dense tokenization with target-aware attention, allowing for efficient learning of feature interactions directly from data.

The Historical Challenges of Sequence Modeling

Advertisement recommendation systems necessitate quick retrieval and ranking of thousands of ads, processing millions of candidates each second. Traditional sequence model approaches have often utilized hybrid configurations, comprising distinct models for processing user event sequences and handling sparse feature interactions. Though effective in meeting production demands, this hybrid setup introduces specific trade-offs, including:

• Fragmented knowledge transfer between components.
• Ongoing reliance on manual feature engineering.
• Limitations in scalability due to potential interference between ranking and sequence model components.

This structure often creates bottlenecks in enhancing user ad experiences and limiting advertisers' campaign performance. To address these issues, Meta developed two key innovations: the multi-stage sequence model, which decouples offline user modeling from online ranking, and a more efficient dense tokenization and learning paradigm that improves the model's performance and scalability.

Introducing the Multi-Stage Sequence Model

The newly developed multi-stage model features a compute-efficient structure enabling the scaling of transformer-based sequence models. By dividing the sequence model into two complementary stages—upstream/offline user modeling and downstream/online ranking—Meta ensures that improvements in model capacity do not lead to proportionate increases in computational resources.

Figure 1 illustrates this setup: the offline user model processes extended user histories asynchronously to create cached embeddings that reflect deep behavioral patterns, while the online ranking model synchronizes these representations with real-time ad candidate signals to generate the final ranking. This separation allows for enhanced complexity in the offline user model without increasing costs in the online ranking model.

Two Key Stages of the Model

First Stage: Offline User Model

The first stage employs advanced transformer models to asynchronously process user features, enabling the generation of cached embeddings that are independent of specific ad candidates. This architecture allows for greater scaling to accommodate user history lengths that reach into the thousands.

Second Stage: Online Ranking Model

This stage utilizes the precomputed representations from the offline model while integrating real-time user signals and ad candidate information to facilitate immediate ranking. Engineered for speed, this model adheres to strict latency requirements while maximizing the benefits of deep embeddings created in the previous stage.

Sequence Model Architecture Innovations

Dense Tokenization

Meta's approach to tokenization merges sparse features with sequential behavioral data into a dense vocabulary, allowing the attention mechanisms to autonomously identify interactions. Unlike earlier methods that depended on predefined feature representations, this approach enables the model to learn valuable interactions directly from the data, leading to improved performance.

Target-Aware Multi-Head Attention

By combining tokenized sparse features with user behavior sequences, the model can weigh a user's past activities against the specific ad being evaluated. This dual processing capability allows for the capture of more nuanced interactions, producing progressively refined representations of the historical behaviors influencing ad performance.

A Predictable Scaling Curve

LLM-Style Scaling Law

Operating on real-world ad traffic, the multi-stage sequence model has revealed scalable laws reminiscent of those observed in LLMs. The relationship between computational resources and model performance has demonstrated a log-linear correlation, showcasing significant enhancements in efficiency compared to other transformer-based designs. Figure 2 exemplifies these properties, outlining performance metrics across several axes, including model depth, content enrichment, and sequence length.

Levers for Scaling

While LLMs handle dense text data, ad recommendation systems must also incorporate sparse identifiers alongside temporal sequences, indicating that the emergence of LLM-style scaling is a promising indicator of effective architectural fits for advancing sequence learning applications. Four levers identified for enhancing scaling include:

1. Balanced Model Shape

Optimal output necessitates coordinated growth across model dimensions. Scaling on varied axes prevents diminishing returns and aligns with findings from LLM scaling law research—termed the scaling synergy principle.

2. Multi-Stage Tunability

The architecture's design allows for focused scaling of the offline or online models. Enhancing the online ranking model yields more immediate performance improvements, while the offline component can be incrementally expanded without latency hurdles.

3. Sequence Composition

Insights indicate that diverse action types aligned in sequences outperform homogenous ones. Incorporating a variety of engagement types promotes richer behavioral representations compared to sequences limited to high-signal actions.

4. Semantic Feature Representation

Incorporating semantic content features can significantly bolster traditional collaborative filtering, enhancing ad performance despite limited historical data, especially for new advertisers or campaigns.

The Impact of Multi-Stage Sequence Modeling

This architecture brings substantial improvements on multiple fronts:

Deeper User Representation

Through modeling thousands of user event sequences, the offline model generates nuanced user representations that elevate ad relevance. Meta reported cumulative increases in conversions of 6% on Instagram and 3-3.5% on Facebook due to these advancements.

Scaling Efficiency

The design enhances performance while maintaining computational efficiency, with initial evaluations confirming successful improvements in ad ranking quality and minimal resource impact. This illustrates that model complexity can coexist with production efficiency.

Platform Integration

This sequence learning model is integral to the Generative Ads Recommendation Model (GEM), designed for flexibility across various ranking tasks, thereby minimizing adaptation overhead.

Current Work: Continued Scaling

With initial successes, the predictive scaling laws show promising longevity. Leveraging previously successful LLM techniques may continuously yield optimal performance gains while balancing efficiency demands.

Read the Paper

A comprehensive technical exploration of this model architecture is accessible in the paper “LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation.”

The original post From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking was published on Engineering at Meta.