Revamping Data Ingestion at Meta: Strategies for a Seamless Migration

Meta's recent revamp of its data ingestion system marks a pivotal shift towards improved reliability and efficiency at a massive scale. Designed to support the company's vast data needs, this system plays a crucial role in creating up-to-date snapshots of its social graph, all while managing several petabytes of data daily. The transition from the legacy system involved comprehensive strategies aimed at ensuring a smooth migration without sacrificing data integrity.

The upgraded architecture moves away from customer-owned pipelines, which proved effective at smaller scales, to a self-managed data warehouse service capable of operating efficiently at hyperscale. This transformation not only enhances performance but also greatly increases the system’s availability, as Meta has completely deprecated the old infrastructure.

The Migration Challenge

As Meta's operations expanded, the legacy system struggled with stability, mainly due to increasingly stringent data landing time requirements. Acknowledging this challenge, the engineering teams faced the daunting task of migrating numerous data ingestion jobs while ensuring seamless execution throughout the process.

Ensuring a Seamless Transition

To achieve a seamless migration, it was essential to track the lifecycle of thousands of jobs meticulously. Setting rigorous rollout and rollback protocols allowed for quick responses to potential issues during the migration. This comprehensive approach protected data integrity and improved operational reliability.

The Migration Lifecycle

The team established a well-defined migration job lifecycle. Each job undergoing the transition needed to fulfill success criteria before progressing to the next stage:

• No data quality issues: A thorough comparison of row counts and checksums ensured the data from both the legacy and new systems matched perfectly.
• No landing latency regression observed: The new system was required to enhance or at least match the performance of the previous one.
• No resource utilization regression observed: Comparable, if not improved, compute and storage metrics were mandatory.
• Additional migration criteria were defined for critical table migrations in concurrence with dependent teams.

Phase 1: The Shadow Phase

The initial phase involved creating shadow jobs that mirrored production jobs but directed output to separate shadow tables. This setup allowed the team to assess real production data with a safety net, revealing any issues while retaining the capacity to troubleshoot and adapt quickly.

Continuous monitoring of row counts and checksums between these job classes highlighted discrepancies for immediate investigation, aiding in the refinement process. The resource allocation for shadow jobs was likewise evaluated to ensure that production resources would be sufficient for the full migration.

Phase 2: The Reverse Shadow Phase

Once both production and shadow jobs verified stability, the process transitioned to the reverse shadow phase. Here, data flowed from shadow jobs into the production table while the original production job began to function as the shadow job. This approach not only maintained data quality oversight through comparisons but also allowed for a rapid rollback if any discrepancies emerged.

Phase 3: Migration Cleanup

The final phase formalized the transition by monitoring both job outputs. If inconsistencies were not detected, the old job was entirely phased out, concluding the migration process.

Custom Data Quality Analysis Tooling

The successful migration relied heavily on comprehensive debugging tools to enable teams to swiftly identify and rectify issues. Central to this was a data quality analysis tool that monitored shadow table partitions and compared them against production counterpart tables.

Any inconsistencies were logged into Meta's data management system, Scuba, allowing team members quick access to detailed debugging information and root cause analysis for issues that arose.

Handling Rollout and Rollback

Both the old and new ingestion systems utilized change data capture (CDC), facilitating incremental data ingestion into target tables. This methodology posed a significant challenge; if prior data had issues, those problems could cascade into new datasets. To mitigate risks, the migration strategy emphasized maintaining early signals for potential data issues and establishing rapid rollback protocols.

Early Signals After Rollout

Instead of waiting for consumers to report issues with the new data, the approach included proactive monitoring of migration success post-rollout. The reverse shadow phase that followed immediately triggered backfills on both job types, allowing ongoing comparisons to signal successful migrations, and preventing erroneous data from impacting consumers.

Stopping Bad Data Propagation Quickly During Rollback

Given the dynamic nature of CDC, stopping the spread of faulty data quickly was essential. This applied not just to batches that had already begun landing, but also to halting problematic partitions immediately after detection. Each afflicted partition was marked in metadata, ensuring that further data landings were halted, with alerts dispatched to relevant team members. The system could then easily prioritize correcting these issues through metadata queries, facilitating efficient backfills.

Executing Large-Scale Migration

With successful small-scale migrations boosting confidence, the engineering team prepared for a comprehensive migration. They identified two key challenges: monitoring a large number of simultaneous job migrations and conducting effective shadow testing amid limited resources.

Monitoring With Automated Tooling

To streamline the migration of tens of thousands of jobs, the team developed automated tools that eased the process and enhanced reliability. Using a clearly defined migration job lifecycle, the system continuously fed job status information, including lifecycle promotion criteria, to Scuba for oversight.

External migration tools monitored these real-time signals, automatically transitioning jobs between migration lifecycle stages based on their overall performance. System and job-level dashboards also facilitated quick access to migration progress and aided in debunking any individual job issues.

Planning With Limited Capacity

Migration capacity constraints necessitated a batch approach to job migrations. Selection criteria based on job throughput, priority, and unique aspects guided the grouping of jobs for each migration round. Teams prepared the environment beforehand to prevent noise caused by unresolved issues and prioritize mission-critical jobs while keeping communication open for those relying on the service.

Addressing known issues before creating shadow jobs prevented excessive resource consumption due to unnecessary full dumps. Once bugs were resolved, subsequent migrations could be executed more smoothly. Clever solutions, such as utilizing snapshot partitions from the old system as initial datasets, helped alleviate full dump pressures and further increased migration efficiency.

Acknowledgements

We would like to thank all the team members and the leadership that contributed to making this project a success for Meta.

The post Migrating Data Ingestion Systems at Meta Scale appeared first on Engineering at Meta.