Technical Lead II – Data Infrastructure

POS-16866

HubSpot is seeking a Technical Lead II to join the Data Infrastructure—Batch Processing Team.

About the team

The Batch Processing Team makes processing large datasets at HubSpot unified, fast, and easy. We provide a suite of tools, platforms, and processes based on Spark, Hadoop, and Iceberg.

Our platform supports thousands of processing jobs daily across hundreds of Spark and Hadoop clusters, processing a petabyte of customer data per day. We deliver this platform on AWS through a Kubernetes control plane across multiple data centers and regions. Through supporting tools and libraries, we enable teams across the engineering organization by defining, developing, and advocating for best practices that help them process data effectively, reliably, and with high performance.

Typical projects include developing tools and client libraries, as well as implementing core orchestration and deployment functionality within our suite of Kubernetes operators. This past year, we introduced an Iceberg lakehouse platform that's driving substantial improvements in interoperability and cost savings as new ML and AI workloads ramp up.

What we're looking for

This role offers a unique opportunity to work at scale on a team pushing the boundaries of a business-critical system. As an engineer, you'll design and solve problems in large, distributed systems and develop software in Java. As a technical lead, you'll own the roadmap and strategy for these platforms, balance competing demands across a portfolio of technologies, and craft the platform into a true product that provides leverage for hundreds of engineers at HubSpot.

Role Responsibilities:

• Lead a team of 3–5 engineers as a player-coach
• Design and implement scalable backend services, libraries, tools, and automation for data store operations and usage
• Collaborate with engineering teams to reliably integrate data processing technologies into their stack
• Support the uptime and reliability of database systems
• Work in a dynamic, agile environment to deliver comprehensive solutions that address user needs
• Conduct code reviews, contribute to internal tooling, and maintain high coding standards for application reliability and performance
• Coach the team to work iteratively, experiment with new ideas, and embrace a startup mentality of learning fast and adapting quickly
• Continuously learn and adapt to new technologies and frameworks to keep our solutions at the forefront of innovation