Software Engineer Lead - Senior Software Engineer (Java & Spark)

About the position

As Senior Software Engineer, you will work on one of the world's largest social media platforms which deals with a few petabytes of data coming to the system daily. You will contribute as part of an R&D self-organized team working in a challenging, innovative environment for our client. Investigate, create, and implement the solutions for many technical challenges using cutting-edge technologies, including building/enhancing Data processing platform enabling work of software used by hundreds of millions of users.

Responsibilities

• Obtains tasks from the project lead or Team Lead (TL), prepares functional and design specifications, approves them with all stakeholders.
• Ensures that assigned area/areas are delivered within set deadlines and required quality objectives.
• Provides estimations, agrees task duration with the manager and contributes to project plan of assigned area.
• Evaluating existing data systems.
• Developing best practices for data coding to ensure consistency within the system.
• Updating and optimizing local and metadata models.
• Evaluating implemented data systems for variances, discrepancies, and efficiency.
• Troubleshooting and optimizing data systems.
• Understand the business drivers and analytical use-cases and translate these to data products.
• Design, implement and maintain pipelines that produce business critical data reliably and efficiently using cloud technology.
• Addresses area-level risks, provides and implements mitigation plan.
• Reports about area readiness/quality, and raise red flags in crisis situations which are beyond his/her AOR.
• Initiates and conducts code reviews, creates code standards, conventions, and guidelines.
• Suggests technical and functional improvements to add value to the product.
• Constantly improves his/her professional level.
• Collaborates with other teams.
• Design and implement Fine Grained Access Control system for Client's big data platform (Spark, Flink, PyTorch, Ray and other ML/AI frameworks).
• Improve and customize the internals of open source Spark to meet our challenges at scale and actively participate in open source community.
• Build and scale big data processing frameworks and infrastructure to process petabytes-scale datasets.

Requirements

• 5 years of experience as Software Engineer with Solid Java coding skill.
• 3 years of experience of building and support large scalable big data infrastructure/platform.
• Strong experience with Spark.
• Knowledge of big data technologies (e.g. Spark, Hadoop, Parquet/ORC, Flink).
• Ability to organize and document solutions.
• A passion for streamlining systems and processes to make the difficult trivial with Strong OOP skill.

Nice-to-haves

• Experience in AWS & K8S.

Benefits

• Collaborating with teams of creative, fun, and driven colleagues.
• Flexible work options enabling time and location-based flexibility.
• Company-provided home office equipment.
• Virtual collaboration and productivity tools to enable hybrid teams.
• Comprehensive benefits program (Health, Welfare, Retirement and Paid time off).
• Other perks and wellness benefits like discount programs, and gym/studio access.
• Paid Parental Leave and coaching, baby welcome gift, and family care/illness days.
• Back-up childcare/elder care, childcare discounts, and subsidized virtual tutoring.
• Tuition assistance and weekly hot skill development opportunities.
• Experiential, high-impact learning series events.
• Access to mental health resources and mindfulness programs.
• Access to join Capgemini Employee Resource Groups around communities of interest.