Senior Data Engineer

Project Description The Data Engineer will join a data-infrastructure team responsible for a scalable, robust, self-service analytics platform. Responsibilities: Maintaining an ingestion pipeline processing billions of events daily against a defined SLA Supporting teams that build and optimize complex data pipelines Identifying platform problems with other engineering teams Developing tools and frameworks for the data platform Supporting data scientists and analysts with production workloads Contributing to ML-platform and streaming initiatives Establishing software-engineering and data-development practices Requirments: Must have: 3+ years of experience in data engineering Experience working with an orchestration tool (e.g., Airflow) Experience working with a large data set Experience building complex ETL pipelines Experience working with cloud providers such as GCP, AWS, or Azure Strong programming skills in Spark with Scala and Python Experience working with CI/CD tools like Jenkins and Git Strong understanding of Software Engineering practices and principles Excellent problem-solving and communication skills Self-motivated and have a proven ability to take the initiative to own the problems that come up and solve them Nice to have: Preferred experience with Kubernetes Kafka or another messaging system, streaming platforms BigQuery or Redshift data-warehouse administration.