PURPOSE OF THE JOB
As Senior Software Engineer, you will work on one of the world's largest social media platform which deals with a few petabytes of data coming to the system daily. You will contribute as part of R&D self-organized team working in a challenging, innovative environment for our client.
Investigate, create, and implement the solutions for many technical challenges using cutting edge technologies, including building/enhancing Data processing platform enabling work of software used by hundreds of millions of users.
MAIN TASKS AND RESPONSIBILITIES
• Obtains tasks from the project lead or Team Lead (TL), prepares functional and design specifications, approves them with all stakeholders.
• Ensures that assigned area/areas are delivered within set deadlines and required quality objectives.
• Provides estimations, agrees task duration with the manager and contributes to project plan of assigned area.
• Evaluating existing data systems.
• Developing best practices for data coding to ensure consistency within the system.
• Updating and optimizing local and metadata models.
• Evaluating implemented data systems for variances, discrepancies, and efficiency.
• Troubleshooting and optimizing data systems.
• Understand the business drivers and analytical use-cases and translate these to data products
• Design, implement and maintain pipelines that produce business critical data reliably and efficiently using cloud technology
• Addresses area-level risks, provides and implements mitigation plan.
• Reports about area readiness/quality, and raise red flags in crisis situations which are beyond his/her AOR.
• Initiates and conducts code reviews, creates code standards, conventions, and guidelines.
• Suggests technical and functional improvements to add value to the product.
• Constantly improves his/her professional level.
• Collaborates with other teams.
• Design and implement Fine Grained Access Control system for Client’s big data platform (Spark, Flink, PyTorch, Ray and other ML/AI frameworks)
• Improve and customize the internals of open source Spark to meet our challenges at scale and actively participate in open source community
• Build and scale big data processing frameworks and infrastructure to process petabytes-scale datasets
REQUIRED EDUCATION AND EXPERIENCE
Must have:
• 5+ years of experience as Software Engineer with Solid Java coding skill .
• 3+ years of experience of building and support large scalable big data infrastructure/platform
• Strong experience with Java
• Knowledge of big data technologies (e.g. Spark, Hadoop, Parquet/ORC, Flink)
• Ability to organize and document solutions.
• A passion for streamlining systems and processes to make the difficult trivial
• Strong OOP skills
• Effective communication (oral & written), collaboration, and interpersonal skills
• Result oriented approach
Would be a plus:
• Experience with AWS & K8S.
Availability to work in 100% remote in EST Timezone.
