Senior Software Engineer II, Data Platform

About the Role

Our backend systems power the clients used by millions of customers every year to buy their groceries online. These systems must also support tight integration with the largest retailers in the US and Canada. Engineering at Instacart provides the opportunity to work on challenging scaling problems while also designing the features that will define our industry. You will learn how to build in an open collaborative environment serving millions of requests daily.

As a Senior Software Engineer in our Data Platform team, you will be the technical force that shapes our data infrastructure. You will design, develop, and maintain comprehensive access controls and governance frameworks to safeguard the integrity and privacy of our data. Your work will elevate the data utilization across several departments, enabling valuable insights, informed decision-making, and driving business value.

About the Team

The Data Platform team provides a robust and cutting edge platform to process petabytes of data daily using industry best practices. We work closely with Product/Data Science/ML teams to understand their needs and provide high quality solutions. The team is still small and many of our bigger initiatives are at an early stage. We expect you to work closely with stakeholders and shape these systems from design, technical decisions, project management to execution. Your input will be critical for driving the Data Strategy and building the platform that Instacart data lake will be built upon. Instacart's technology is constantly changing and adapting. Some of the technologies you would use at Instacart include: Scala, Python, Postgres, Snowflake, DeltaLake, Iceberg, Clickhouse, Spark, DBT, Flink Kafka, Airflow If you have experience with these technologies, you will have the opportunity to dive deeper. If you haven't used these technologies you will have the chance to learn from the collective experience of our team.

About the Job

Your responsibilities will include:

• Designing, developing, and sustaining the comprehensive access controls and governance systems that improve the integrity and privacy of our data.
• Ensuring the reliability, scalability, and security of the data platform.
• Collaborating with various stakeholders and actively involve in the data infrastructure.

Minimum Qualifications

We are looking for someone who:

• Has 5+ years of experience in software engineering.
• Exhibits an in-depth understanding of distributed systems, with proven experience with data processing technologies such as DBT and Airflow, and common web frameworks such as Rails.
• Proficiently uses SQL for writing and reviewing complex queries for data analysis and debugging.
• Can design for scale with the entire system in mind.
• Capably communicates and is comfortable seeking and receiving feedback.
• Possesses strong analytical and debugging skills.
• Takes a strong sense of ownership while working with large codebases and diverse suite of products.
• Embraces a collaborative mindset to partner with engineers, designers, and PMs from multiple teams to co-create impactful solutions while supporting system contributions.
• Communicates clearly, presents ideas well, and can influence key stakeholders at manager, director, and VP levels.

Preferred Qualifications

We would love it if you:

• Holds a Bachelor’s degree in Computer Science, Software Engineering, or a related field, or can demonstrate equivalent industry experience (4+ years).
• Have prior work experience in Data Platforms.
• Hold experience with big data technologies such as Spark, Hadoop, Flink, Hive, or Kafka, and with both streaming and batching data pipelines.
• Have proven experience with distributed system designs.
• Possess strong general programming and algorithm skills.
• Show strong attention to detail and accuracy in your implementation.
• Have a strong experience writing complex and optimized SQL queries.
• Appreciate a data-driven mindset.