AI Platform Reliability Engineer

Project

Work done for a product or program.

• Build monitoring, alerting and dashboards for accelerators and serving stacks.
• Run incident response and write blameless post-mortems.
• Perform capacity planning and load tests before major launches.
• Partner with silicon teams on firmware and driver defects found in production.

Function

Work that keeps the discipline running: methods, tools, suppliers and people.

• Define service level objectives for training throughput and inference latency.
• Automate failure recovery for node, network and job failures.
• Reduce operational toil through automation.

Project

1. Produce and hand over the role's key deliverables
   • SLO definitions
   • Monitoring and alerting stack
   • Post-mortem reports
   • Automation runbooks
2. Support programsGeneral
   • Agree what each program needs
   • Deliver in time for milestones
3. Manage issuesGeneral
   • Track requests to closure
   • Escalate risks

Function

1. Run the functionGeneral
   • Maintain policies and processes
   • Keep records and controls
2. ImproveGeneral
   • Report performance
   • Lead improvement work