Incident Handling - Revision history

Jakobbuis: /* Non-Critical incidents */

2026-03-23T12:35:28Z

Non-Critical incidents

← Older revision		Revision as of 05:35, 23 March 2026
Line 151:		Line 151:

	== Non-Critical incidents ==		== Non-Critical incidents ==
	* Non-critical incidents are acknowledged within ~~9 hours~~ and resolved within ~~one week~~.		Non-critical incidents are acknowledged within one working day and resolved within three weeks.

	=== Acknowledging ===		=== Acknowledging ===

Jakobbuis: /* Non-Critical Incidents */

2026-03-23T12:35:02Z

Non-Critical Incidents

← Older revision		Revision as of 05:35, 23 March 2026
Line 37:		Line 37:

	=== Non-Critical Incidents ===		=== Non-Critical Incidents ===
	Non-critical incidents must be acknowledged within ~~9 hours~~ and resolved within ~~1 week~~.		Non-critical incidents must be acknowledged within one working day and resolved within 3 weeks.

	# Acknowledge in Zabbix thread		# Acknowledge in Zabbix thread

Jakobbuis: /* Checklist */

2026-01-12T14:57:55Z

Checklist

← Older revision		Revision as of 07:57, 12 January 2026
Line 29:		Line 29:
	## Mark Zulip topic as resolved if no other incidents for the host.		## Mark Zulip topic as resolved if no other incidents for the host.
	## Check for related triggers and resolve them.		## Check for related triggers and resolve them.
	## If there were any SRE - Report threads		## If there were any SRE - Report threads:
	### post a summary describing the high-level incident, that it is resolved and how it was resolved.		### post a summary describing the high-level incident, that it is resolved, and how it was resolved.
	### post that summary message to any client channels such as Slack too.		### post that summary message to any client channels such as Slack too.
	### close the thread in SRE - Report		### close the thread in SRE - Report

			Note: we do not accept discussions on the how or why of incident response in the SRE - Report channel; those should be redirected to either Retro or Organisational channels. The only reason to reopen a thread in SRE - Report should be to report that there's still impact and the incident has been resolved prematurely.

	=== Non-Critical Incidents ===		=== Non-Critical Incidents ===

Jakobbuis at 14:55, 12 January 2026

2026-01-12T14:55:49Z

← Older revision		Revision as of 07:55, 12 January 2026
Line 14:		Line 14:
	# Check if the incident is still ongoing.		# Check if the incident is still ongoing.
	# Determine whether the incident is ongoing		# Determine whether the incident is ongoing
			# If this report came in via SRE - Report:
			## keep that thread open until the incident is resolved
			## post a link to the SRE - Report thread to any underlying technical threads in SRE # Critical, SRE ## Non-critical, or SRE ### Informational that is related
	# Determine whether clients are potentially affected, if so:		# Determine whether clients are potentially affected, if so:
	## notify the affected clients (Slack preferred)		## notify the affected clients (Slack preferred if available)
	## share the message sent to the client in the incident Zulip thread		## share the message sent to the client in the incident Zulip thread
	# Document all actions taken in the Zulip topic.		# Document all actions taken in the Zulip topic.
Line 26:		Line 29:
	## Mark Zulip topic as resolved if no other incidents for the host.		## Mark Zulip topic as resolved if no other incidents for the host.
	## Check for related triggers and resolve them.		## Check for related triggers and resolve them.
			## If there were any SRE - Report threads
			### post a summary describing the high-level incident, that it is resolved and how it was resolved.
			### post that summary message to any client channels such as Slack too.
			### close the thread in SRE - Report

	=== Non-Critical Incidents ===		=== Non-Critical Incidents ===

Jakobbuis: remove common issues in imperative checklist, better listed below

2025-10-21T11:13:58Z

remove common issues in imperative checklist, better listed below

← Older revision		Revision as of 04:13, 21 October 2025
Line 26:		Line 26:
	## Mark Zulip topic as resolved if no other incidents for the host.		## Mark Zulip topic as resolved if no other incidents for the host.
	## Check for related triggers and resolve them.		## Check for related triggers and resolve them.

	~~Common issues that have occurred previously, and ''could'' occur again:~~
	* SSH down: Check MaxStartups throttling, apply custom SSH config
	* No backup: Verify backup process is running, check the devteam email
	* HTTPS down on Sunday: this can be due to GitLab updates

	=== Non-Critical Incidents ===		=== Non-Critical Incidents ===

Jakobbuis: /* Deviating from the process */

2025-10-21T11:10:24Z

Deviating from the process

← Older revision		Revision as of 04:10, 21 October 2025
Line 3:		Line 3:

	= Deviating from the process =		= Deviating from the process =
	You may deviate from the process at any moment. A deviation should be communicated to the dev-team, preferably in the Zulip topic about the applicable incident, ~~ideally~~ as soon as possible after deciding to deviate.		You may deviate from the process at any moment. A deviation should be communicated to the dev-team, preferably in the Zulip topic about the applicable incident, as soon as possible after deciding to deviate.

	= Checklist =		= Checklist =

Jakobbuis at 11:09, 21 October 2025

2025-10-21T11:09:52Z

← Older revision		Revision as of 04:09, 21 October 2025
Line 1:		Line 1:
	= This is the process =		= This is the process =
	This document ~~describes~~ the process. This document supersedes all prior documents on process.		This document is an authoritative description of the process. This document supersedes all prior documents on the process.

	= Deviating from the process =		= Deviating from the process =

Jakobbuis: /* Handover */

2025-10-17T13:52:57Z

Handover

← Older revision		Revision as of 06:52, 17 October 2025
Line 196:		Line 196:

	== Handover ==		== Handover ==
	~~When handing over the responsibility of '''first responder''' (FR), the following needs to happen:~~		See [[Handover]]
	* The handover can be initiated by both the upcoming FR or the acting FR
	* Acting FR adds the upcoming FR to the IPA sla-first-responder user group and enables Zabbix calling for the upcoming FR if they have that set by going to Zabbix > Configuration > Actions > [~~https://status.delftinfra.net/zabbix/actionconf.php?eventsource=0# Trigger actions]~~
	* Before the handover, the acting FR must ensure that all active incidents are acknowledged (this includes alert emails or opened topics in SRE General, etc...), updated with the latest status, and properly documented.
	* The upcoming FR makes sure they are aware of the state of the SLA and knows what questions they wants to ask the acting FR.
	* The upcoming FR makes sure they are subscribed to the right channels.

	~~The following steps can be done async or in person:~~
	* The acting FR announces/informs the upcoming FR has been added to the sla-first-responder group (In Zulip's [https://chat.dsinternal.net/#narrow/stream/13-Organisational Organisational channel] if asynq).
	* If the acting FR wants to hand over responsibility for any ongoing incident they also state which incidents they want the upcoming FR to take over.
	* If there are any particularities the upcoming FR needs to be aware of, those are shared.
	* The upcoming FR asks their questions until they are satisfied and able to take over the FR
	* The upcoming FR ensures they are subscribed to the following channels on Zulip: [https://chat.dsinternal.net/#narrow/stream/23-SRE---General SRE - General], [https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical SRE # Critical] and if part of the SRE team [https://chat.dsinternal.net/#streams/4/SRE%20##%20Non-critical SRE ## Non-Critical] and [https://chat.dsinternal.net/#streams/5/SRE%20###%20Informational SRE ### Informational].
	* The upcoming FR announces/informs that they are now the acting FR over Zulip's [~~https://chat.dsinternal.net/#narrow/stream/13-Organisational Organisational channel~~]
	* The now acting FR removes the previous FR from IPA the sla-first-responder user group and disables Zabbix calling for the previous FR if they had that enabled by going to Zabbix > Configuration > Actions > [https://status.delftinfra.net/zabbix/actionconf.php?eventsource=0# Trigger actions]

Jakobbuis: /* Additional context */

2025-10-17T12:54:51Z

Additional context

← Older revision		Revision as of 05:54, 17 October 2025
Line 145:		Line 145:

	When you are done, there should be no more critical triggers firing in Zabbix or open in the Zabbix-Mattermost integration, for which no-one has taken responsibility or you have taken responsibility for and are not actively handling.		When you are done, there should be no more critical triggers firing in Zabbix or open in the Zabbix-Mattermost integration, for which no-one has taken responsibility or you have taken responsibility for and are not actively handling.

	~~===Additional context===~~
	* Critical incidents are posted in [https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical '''SLA - Critical'''].

	== Non-Critical incidents ==		== Non-Critical incidents ==

Jakobbuis: /* Additional context */

2025-10-17T12:54:11Z

Additional context

← Older revision		Revision as of 05:54, 17 October 2025
Line 148:		Line 148:
	===Additional context===		===Additional context===
	* Critical incidents are posted in [https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical '''SLA - Critical'''].		* Critical incidents are posted in [https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical '''SLA - Critical'''].
	* <s>When it is being tracked on GitLab a heavy check mark is added to the message.</s>
	* <s>Responses on the thread and on GitLab are automatically synced (to some extend)</s>
	* <s>When you reply with '''I agree that this has been fully resolved''' eventually our Zabbix-Mattermost integration will pick this up and a green check mark is added to the message.</s>

	== Non-Critical incidents ==		== Non-Critical incidents ==