What to Expect When You Go On Call

This document was created as a primer for the Core Product organization at Sentry.io.

Going on-call and carrying a pager for the first time can feel stressful, but it doesn't have to! This document lays out our on-call philosophies and outlines expectations, advantages, cultural norms we'd like to encourage, and some best practices.

Why should I be on call?

Having full ownership of our software solutions creates virtuous cycles:

  • Teams are empowered to manage their production infrastructure and don't need to wait on SRE to get things done.
  • We are incentivized to not cause incidents (by releasing stable, well tested code that we have high confidence in).
  • We are incentivized to design our systems to scale from day one.
  • We are incentivized to have rich instrumentation, a strong mental model of how our system is operating in production, and what the failure modes are.

Historically, we have offloaded much of this responsibility to SRE and been worse off for it.

How will I get prepared?

  • Read the SRE team's The Purpose of On-Call
  • Read the SRE team's Expectations of On-Call as well
  • Attend SRE incident management role-playing and on-call training
  • Understand the overall architecture of the systems for which you are responsible
  • Familiarize yourself with service runbooks provided by your team

Expectations

  • Teams will have the tools they need to diagnose and fix production issues in the systems for which they are responsible.
  • Teams will have monitors and SLOs that reflect reality (they alarm if and only if systems they own are having problems that impact users).
  • Teams will set up a primary and secondary rotation in PagerDuty that is triggered by those monitors.
    • If teams are small, and they wish to be on-call less frequently, they can join forces to cover both teams' domains. E.g. a group of 12 would be on-call approximately once per quarter.
  • Individuals will be on-call 24/7 for Tier 0 and Tier 1 services that their team(s) own(s). These are the services that are critical for the healthy functioning of Sentry as a business.
  • Teams will continue to maintain Tier 2+ services during normal business hours.
  • Teams should re-evaluate service tiering periodically and ensure the registry is up to date.
  • Teams will invest significant time in improving our processes and stability in an effort to minimize the burden of being on-call. This work will be done in lieu of project work while on call. Making less (or even zero) progress on project work while on-call is acceptable as long as we're doing this instead.
  • There will always be an SRE on-call as well to coordinate with when necessary.

Cultural Norms

  • We should aim to never be paged. If we are paged, something has gone wrong. We should treat this as a bug and aggressively fix it so it can't happen again, just like any incident follow-up.
  • Be courteous. Don't be the one to cause your colleague to be paged.
  • If you do get paged, you need to be available. You don't need to be chained to your desk or laptop, but you should be near enough to wifi to jump online and engage quickly (< ~10m). Remember, if a T0 service is down, so is most of Sentry. If you can't be available, you need to make sure your secondary is informed and can cover for you.
  • We are reasonable people. If you got paged in the middle of the night, no one will mind if you sleep in or skip coming to the office the next day. We also have an unlimited vacation policy, so coordinate recovery time with your manager.
  • When things are going wrong, we help one another. If you're on-call, the responsibility is yours, but if other people are online, they're encouraged to help out. There's always your secondary, an Incident Manager On-Call and an on-call SRE available to back you up.

Best Practices

  • PagerDuty alerts should link to runbooks to diagnose and address common problems.
    • Caveat: Common problems should have automated solutions
  • Retro on-call weeks. Collect a backlog of ways to make being on-call better and have people work on that during their on-call weeks.

How will we roll this out?

Each team or set of teams will experience a "bear hug" from SRE over the course of 2-3 months to ensure they are well equipped to take over on-call responsibilities.

Proposed order:

  1. Issue Detectors & Crons/Uptime
  2. Billing + AI/ML
  3. Issue workflows & Ecosystem