Senior Software Engineer, Data Governance & Foundations

About the Team

Instacarts Data Governance & Foundations team builds and operates the core systems that power the company's data ecosystem — a modern data lakehouse at scale, spanning ingestion, stream processing, analytical compute, and self-serve tooling. You'll join a collaborative team of 6–7 engineers responsible for keeping a highly reliable production platform running today while architecting the infrastructure that will serve the business for the next 3–5 years.

This is a high-ownership, high-autonomy role. Architectural decisions carry both technical and financial weight, and you'll be expected to drive direction, not just execute it. You'll work closely with engineering leadership and cross-functional partners across Data Science, ML Platform, Ads Infrastructure, Finance Engineering, and senior stakeholders throughout the organization.

About the Job

• Define and drive multi-year architecture roadmaps for large-scale data ingestion and processing infrastructure, setting technical direction that balances reliability, scalability, and cost.
• Own end-to-end platform initiatives — from build vs. buy decisions and migration design through production rollout and risk management — across Kafka-based streaming and Postgres-based systems.
• Partner with vendors (Snowflake, Databricks, Confluent) on technical integration, contract evaluation, and TCO modeling to inform infrastructure investment decisions.
• Collaborate with various teams to embed governance and compliance controls (SOX, CPRA, GDPR) directly into platform architecture and data lifecycle management.
• Optimize infrastructure spend at scale: identify cost reduction opportunities across compute, storage, and pipeline efficiency; manage multi-million dollar infrastructure budgets.
• Write compelling architecture documents, strategy memos, and proposals that drive alignment with engineering leadership and senior stakeholders across the organization.
• Mentor engineers on the team, model strong engineering culture, and help grow a high-performing data infrastructure organization.
• Collaborate with Data Science, ML Platform, Ads Infrastructure, Finance Engineering, and Product teams to ensure the platform meets evolving needs.

About You

• 5+ years of software engineering focused on data infrastructure or distributed systems at scale, in a high-growth, data-intensive environment.
• Experience in modern data lakehouse architectures and open table formats — Apache Iceberg, Delta Lake, Hudi — with strong understanding of compute/storage trade-offs.
• Hands-on experience with distributed query and compute systems (Trino, Spark, ClickHouse) including performance tuning and production reliability work.
• Proven depth in event-driven infrastructure: Kafka for high-throughput data ingestion and Flink (or equivalent) for stream processing at scale.
• Track record owning and executing major platform transitions, including migration design, phased rollout, and risk management under production constraints.
• Experience building business cases for infrastructure investments: cost-benefit analysis, TCO modeling, and presenting recommendations to leadership.
• Exceptional written technical communication — clear architecture docs, strategy memos, and cross-team proposals that drive decisions and alignment.
• Strong ownership and comfort operating in ambiguity; ability to drive large, multi-team initiatives from concept to production with organizational influence.

PREFERRED QUALIFICATIONS

• Familiarity with data governance and compliance frameworks (SOX, CPRA, GDPR) and experience designing governance controls into platform architecture.
• Experience with FinOps and data platform cost optimization, including managing large infrastructure budgets and negotiating enterprise vendor contracts.
• Deep SQL expertise and strong proficiency in Python or Scala for systems-level work.
• Experience with orchestration (Apache Airflow) and transformation pipelines (dbt) in large-scale production environments.
• Bachelor's, Master's, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience.