About the position
Joining NVIDIA's DGX Cloud Team means contributing to the infrastructure that powers our innovative AI research. This team focuses on optimizing efficiency and resiliency of ML workloads, as well as developing scalable AI infrastructure tools and services. Our objective is to deliver a stable, scalable environment for AI researchers, providing them with the vital resources and scale to champion innovation. We are seeking a distributed software engineer to join our team! As a Senior engineer, you'll be instrumental in developing and optimizing AI infrastructure services to bring peak AI performance and high resiliency for DGX Cloud. Your expertise in cloud services software architecture that drives the full resilience stack that will deliver an AI platform that redefines what is possible. This is an exceptional opportunity to push the boundaries of technology and shape the future of AI and the Cloud, and work with a world-class team of like-minded engineers.
Responsibilities
- Developing solutions at the intersection of machine learning, distributed systems, and high-performance computing, supplying to the advancement of AI technologies.
- Designing, developing, and optimizing (micro-)services orchestrated by Kubernetes to provide large-scale AI training workflows on AI training supercomputers located at major CSPs, with resiliency and efficiency.
- Co-designing and implementing the APIs that allow these services to integrate vertically with NVIDIA's resiliency stacks, ranging from tier-0 telemetry services to break/fix automation services to checkpoint and execution systems.
- Crafting a submission abstraction that enables model engineers and training platforms/frameworks to seamlessly submit long-running training jobs while hiding the complexity of handling infrastructure failures, running job lifecycles with auto-restarts on failure, ensuring full efficiency, and promptly advising users.
- Crafting these services to be modular, enabling them to be coordinated with and deployed onto on-premises AI clusters that apply NVIDIA Hardware and Cloud services.
Requirements
- A Bachelor's degree or higher in Computer Science or a related technical field (or equivalent experience).
- 5+ years of hands-on experience in backend development, preferably with Python, Go, C/C++, or similar high-performance languages.
- Consistent track record of building and scaling large-scale distributed systems.
- Experience with cloud computing platforms such as AWS, Azure, and GCP, as well as container technologies like Docker and Kubernetes, and HPC/AI platforms such as Slurm.
Nice-to-haves
- Real world experience in DL frameworks, orchestrators like PyTorch, TensorFlow, JAX, and Ray.
- Experience in developing a framework plugin architecture that allows the framework to be integrated with the cluster scheduler visibly to the users.
- Strong understanding of NVIDIA GPUs, network technologies, and their failure patterns.
- Experience with AI models and AI based tools.
- Provide references to your code contributions.
Benefits
- Equity and benefits.
