PagerDuty
PagerDuty can open an incident from an email, and each service can be given its own integration address. That makes email the practical route from a Watch alert to an on-call page - PagerDuty's Events API authenticates with a routing key inside a fixed payload shape we do not produce, so the webhook is not the way in.
The same pattern applies to any tool with an inbound email address: ServiceNow inbound email actions, a Jira mail handler, a Zendesk support address.
Creating the integration address
- In PagerDuty, go to Services > Service Directory and select the service that should be paged.
- Open the Integrations tab and add an Email integration.
- Copy the generated integration email address.
By default PagerDuty opens a new incident for every email received at that address. Email Filters can check the subject, body and from address against regex patterns if you want to narrow that, and Email Management Rules give finer routing.
Getting alerts to that address
Watch product email alerts are on by default, controlled by the Send email alerts toggle in each watcher's Config modal, and they are sent to your account's notification address.
There is currently no field in Report URI for sending alerts to a different address, so route them at your mail provider - either forward messages from Report URI to the PagerDuty integration address, or make the account's notification address a distribution list that includes it.
Filter on the sender. Watch alerts arrive from a per-product address at our domain, and the subject is of the form New <product> <items> Detected, so a rule matching those is precise enough to avoid paging on unrelated mail.
Things to watch for
- Every email opens an incident by default. A watcher with a threshold of
1on a busy site can page repeatedly. Raise the watcher's alert threshold, add a PagerDuty email filter, or both. - Email is not instant. It is right for "somebody should look at this", not for a hard latency target.
- Forwarding rules are easy to set and easy to forget. If they are removed or the mailbox changes, alerts stop with nothing in Report URI to indicate it - so include the path in whatever you use to review alerting.
- Turning email alerts off for a watcher stops the page too, since it is the same feed.
A note on the webhook route
PagerDuty's Events API v2 takes its routing key in the request body rather than a header, and our payload shape is fixed, so we cannot populate it. Sending the webhook through a small relay that rewrites the body into PagerDuty's format is possible - see Security Operations - but for most teams the email integration reaches the same outcome without hosting anything.
Official documentation
PagerDuty - Email Integration Guide
Start Monitoring with Report URI
Already using us? Add an Email integration to your PagerDuty service, then route your alert emails to it.
New to Report URI? Watch products are available on Business plans and higher.