Senior Software Engineer - Data Engineering

About the position

Bloomberg runs on data. As the Data Management & Analytics team within Engineering, we support our organization's needs around managing data efficiently and enabling everyone across the company to make informed decisions using self service data analytics tools. We are responsible for ingesting and preparing massive amounts of data and ensuring that the data is properly modeled, governed, accurate and protected for reporting and advanced analytics. A key objective of this role is for you to help build and support company wide data analytics programs leveraging data engineering technologies such as Hadoop, Pyspark, Flink, Airflow and traditional MPP Data Warehouse/ETL technologies.

Responsibilities

• Proactively drive the vision for Data Analytics across multiple business areas, and define and execute on a plan to achieve that vision.
• Build high-quality, scalable and robust data engineering applications with the goal of providing end users with self-service analytics.
• Be proficient at developing, architecting, standardizing and supporting technology platforms using open source technologies such as Pyspark, Airflow and industry leading ETL solutions.
• Build data pipelines by ingesting petabytes of data using real-time streaming tools like Kafka, Flink. Maintain and support these data pipelines and ensure they are scalable and highly available.
• Incorporate exceptional handling, logging and auditing procedures within data pipelines codebase.
• Leverage open source frameworks around workflow automation, orchestration and job scheduling.
• Integrate data management capabilities such as data modeling, data quality, data anonymization throughout the data lifecycle - data in transit and data at rest.
• Collaborate and build cross functional relationships with Product Owners, Data Scientists and Software Engineers to understand business and deliver on those needs.
• Embrace agile framework with iterative product development and continuous improvement mindset.
• Stay up to date with market trends, bring new ideas on-board and evaluate tooling for future needs.
• Be highly motivated to drive innovations across engineering.

Requirements

• 7+ years of experience designing and developing complex data engineering solutions.
• 5+ years of experience building ETL frameworks using Python/Pyspark and Hadoop/Hive.
• 2+ years of experience on any Industry Standard Orchestration tools, e.g. Tidal, Airflow, Autosys.
• 3+ years of experience in distributed parallel computing databases for Data Warehousing needs.
• 3+ years of experience using real-time streaming frameworks such as Kafka, Flink.
• Strong understanding of data warehousing methodologies, ETL processing and dimensional data modeling.
• Advanced SQL proficiency and data modeling experience. Knowledge of database design techniques.
• Demonstrated experience and ability to work with business users to gather requirements and manage scope.
• Strong communication, presentation, problem-solving, and trouble-shooting skills.
• BA, BS, MS, PhD in Computer Science, Engineering or related technology field.

Nice-to-haves

• Experience working with extremely large data volumes, large databases and data warehouse implementations.
• Understanding of VLDB performance aspects, such as table partitioning, sharding, table distribution and optimization techniques.
• Prior working experience with ETL tools such as Informatica/IDMC.
• Knowledge of reporting tools such as QlikSense, Tableau, PowerBI.

Benefits

• Paid holidays
• Paid time off
• Medical insurance
• Dental insurance
• Vision insurance
• Short and long term disability benefits
• 401(k) with match
• Life insurance
• Various wellness programs