Delft Solutions - User contributions [en]

Error/OpusSafety::UnprocessableEntityError

2025-09-11T11:02:07Z

SleeplessByte: Created page with "'''Unprocessable Entity Error''' is an error Opus Compliance Cloud raises when the object you provided did not adhere to all constraints. This usually sets the response to a status code [https://http.cat/status/422 422: Unprocessable Entity]. == Common cause(s) == This error is seen when the server understands the request, and there are no authentication, authorization, conflict, or precondition errors. This generally means the error is with the actual content of the r..."

'''Unprocessable Entity Error''' is an error Opus Compliance Cloud raises when the object you provided did not adhere to all constraints. This usually sets the response to a status code [https://http.cat/status/422 422: Unprocessable Entity].

== Common cause(s) ==

This error is seen when the server understands the request, and there are no authentication, authorization, conflict, or precondition errors. This generally means the error is with the actual content of the request.

== Common resolution(s) ==

The error message generally indicates the issue with the request. When the error message is generic, double check the request body and try again.

Error/OpusSafety::RequestFormatError

2025-09-11T10:59:56Z

SleeplessByte: Created page with "'''Request Format Error''' is an error Opus Compliance Cloud raises when The request has an invalid, incorrect, or incomplete format. This usually sets the response to a status code [https://http.cat/status/400 400: Bad Request]. == Common cause(s) == Usually caused because of the ''format'' of the request body (including URL query parameters), causing the system to be unable to ''parse'' the request. If the request is generally parseble but incorrect, a Error/OpusS..."

'''Request Format Error''' is an error Opus Compliance Cloud raises when The request has an invalid, incorrect, or incomplete format. This usually sets the response to a status code [https://http.cat/status/400 400: Bad Request].

== Common cause(s) ==

Usually caused because of the ''format'' of the request body (including URL query parameters), causing the system to be unable to ''parse'' the request. If the request is generally parseble but incorrect, a [[Error/OpusSafety::UnprocessableEntityError|OpusSafety::UnprocessableEntityError]] is more common.

== Common resolution(s) ==

The error message generally indicates the issue with the request. When the error message is generic, double check the request body and try again.

Incident Handling

2024-11-06T15:36:44Z

SleeplessByte: /* Critical incidents */

== Zulip migration ==
Due to a migration to Zulip, the integration as was available on Mattermost is not available yet on Zulip. This leads to the following process changes:
* Acknowlegements and triggers resolving are not posted to Zulip by Zabbix
* Triggers are grouped in a topic on Zulip per host
* When an incident has been fully resolved, mark the topic as resolved, when any other incidents reported for the host are resolved
* There's no `?ongoing`, instead for now we can track open incidents by checking for unresolved topics
* The posting of incidents is less smart (only posting when not posted yet), so in order to prevent an incident from not being reported due to network issues or the likes, a message is posted after an inteval (8 hours for non-critical and lower, 1 hour for critical and above) while the incident has not been acknowleged.
* Incidents can be manually tracked by creating a topic by hand and reporting the problem
* There is no automatic gitlab issue creation or syncing anymore.

Finally, where this process says to do something on Mattermost, you should now do so on Zulip. The updates in the process chapters themselves are WIP.

== Critical incidents ==
'''Critical incidents are resolved within 16 hours.'''

As first responder you take on the responsiblity of seeing an incident resolved. This does not mean that you are the person required to do all the work. You can attempt to involve other to help you (often referred to as escalating the incident), but since other are not on-call, they are not obliged to help you, especially outside of normal working hours.
Involving multiple people can quickly be required if multiple critical incidents with different causes occur simultaniously. In that case, the First Responder usually takes on a more information management role and steers those that are brought on into resolving the issues. (Example: if a server crashes, several critical triggers can fire, but the underlying cause can quite quickly be determined to be a single issue, the crashed server. So you wouldn't need to call in people to manage each incident. But a client's service being down in one cluster while in a different cluster a different VM no longer boots is likely to be to different issues, so in order to resolve them on time you'd want to call in help to resolve the incident in time).

=== Process ===
The general process is made up of the folowing steps. Each step has additional information on how to handle/execute them in the sections below.
# Take responsibility for seeing the incident resolved
# Determine if incident is still ongoing
# If ongoing: Communicate to affected clients that the issue is being investigated
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# Resolve incident + cleanup

During working on an incident it is expected that all communication is done in the incident's thread. This means all information to a problem can be found in a clear a predictable place. Sometimes an incident can be resolved by work done in another incident. In that case, it is required to post a link to that thread in the incident's thread with the comment that the resolution is done in that thread.

==== Acknowledge the incident on Zabbix ====
The first step is to take responsibility for seeing the incident resolved by acknowledging the incident on Zabbix. Simply acknowledging the trigger suffices. It is however entirely possible that multiple critical incidents are firing at the same time. This can be a coincidence, or can be because of a share cause of failure. For example, a server crashing will cause server VM' to reboot, or the router having an connectivity issue will lead to most other VM's having connectivity issues as well. If there are multiple critical incidents, it is advised to quickly observe what's ongoing, Zabbix is the best source of firing triggers for this, and pick the incident that is likely the root cause to

* Acknowledging an incident on Zabbix will stop Zabbix from calling the First Responder to notify them of the ongoing incident. And stops Zabbix from posting reminders on Zulip.

==== Determine if incident is still ongoing ====
The next step is to check if the reported problem is still ongoing. Depending on the observations made here your process to follow and steps needed to resolve the incident can change. There are three options:
# The trigger resolved itself and the problem cannot be observed. Example: HTTPS is down for a site, but the FR can access the site through HTTPS without incident.
# The trigger resolved itself and the problem can still be observed.
# The trigger is still firing but the problem cannot be observed: Our triggers might not be perfect, so it could be that something else is causing it to fire. A simple example would be that Zabbix reports that the the DNS for a site can't be resolved, but in reality there's a bug in the script we wrote that checks if the DNS resolves and the DNS resolves fine. Final note: keep in mind that an 'it works on my machine' does not necessarily mean it works for most other people, so depening on the trigger you need to do some evaluations if your tests suffice.

In order to make sure you are actually trying to observe the same thing as the trigger is looking for, make sure to check the trigger definition and the current data of the associated item(s). Some triggers might fire if one of multiple conditions is met (Such as a trigger that monitors the ping response time firing if the value exceeds a certain threshold, or if no data for a certain period of time was observed).

Make sure to report your findings in the incident's thread. It's advised to post a screenshot of the relevant item(s) and your own observations. (Continuing the ping example, you would post a screenshot of the relevant values, state your conclusion why the trigger is firing, and your own observations/pings)

==== Communicate to affected clients ====
If the incident is still ongoing and the service is down, we need to communicate to affected clients that we are aware of the problem and that we are investigating it. This is because critical incident usually mean the service is down, something the clients can notice/are affected by, so we to be transparent that something is going on. There are some additional notes to this though:
* If an incident has already resolved itself and the problem is no longer observable, we don't communicate anything. Doing so might only cause confusion, and since the client has not reported any issues, they have not had a noticeable problem with it themselves.
* Although a critical incident generally means that the client service is down or experiencing reduced service, not all critical incidents are of that nature. Some are more administrative, or are only an issue for Delft Solutions itself. As of writing I don't have an exhaustive list, but here is those I can think of:
** SSH Service is down: We don't have any clients that SSH into their services, so it's generally not a problem. But SSH is mostly used for SRE maintenance and publishing new builds. The SRE maintenance is an internal problem, so no need to communicate to the client. The publishing is done to Kaboom, preventing new builds from being published, and the two SM VM's.
** No backup for x days: Clients don't notice it if a backup is running late, so no need to communicate with clients. Just need to make sure the backup gets completed
** SSL certificate is expiring in < 24 hours: This is a bit dependent on how soon this incident is being handled, but if it handled quickly, the certificate never actually expired, and there has not been any disruption to the client's service, so no need for communicating about it.
* Determining which clients are being affected can be done by looking at the host's DNS in the trigger, and/or looking up the VM in Proxmox and checking the tags of the VM's for client names. In the case that this issue is causing multiple other critical triggers to fire, you would have to check for which clients are affected by those incidents.
* Communicating to DS about ongoing incidents is usually assumed to be automaticly have been done by the fact that the incident was reported on Zulip.

As always, report the decisions taken and actions maded in the incident thread. (e.g.: I've sent a message in the Slack to let Kaboom know that we aware of problem x, and that we are investigating it)

==== Communicate plan/next steps + Communicate findings/results of executed plan ====
This is the main part of handling an incident. There are several actions you can take in these steps, but at the basis they consist of sharing your next steps, performing those, and reporting the results. The reason all this needs to be reported is to ensure that all known information about a problem is logged, making it easier for someone else to be onboarded into the issue, for later reference if a similar issue is encountered, and even for use during the incident itself in case an older configuration needs to be referenced after you changed it.
The objective from these steps is determining what is actually wrong and how to resolve it. Depending on the observations made earlier on whether the incident is still ongoing and is (still) observable your investigation can go into different directions. (e.g. Find the underlying cause for a trigger, or determining why the trigger is firing while it likely shouldn't, and then how to resolve that underlying cause or how to update the trigger to work better)

There are three main types of steps defined, but you are not limited to these:
# Hypothesis: If you have an idea what could be causing it, you would state your hypothesis and your next step would be to prove that hypothesis. For example, for an incident 'SSH service is down on X' your hypothesis could be that this is due to 'MaxStartups' throttling, which can be proven by 'grep'ing journalctl for that, and compare the start and end times of throttling with the timestamps of the item reporting the status of the SSH service.
# Information gathering: Sometimes it just helps to get some facts about the situation collected. What is usefull information that is relevant depends on the triggers, but some examples are: The syslog/journalctl of the host from around the time of the incident (it can contain a reference to the an underlying problem in various levels of explicitness), the ping response from several hosts on the route to a host or a traceroute (this helps with networking issues). The gathered information is usually intended to help you come up with an hypothesis on what's wrong.
# Investigative: The most rigorous of process. The full process is described here originally [https://docs.google.com/document/d/1AQYJM1Q9l2Tyk6zfCVaQ2aEq-dpbfUH5okE88bpKkhw/edit#heading=h.5fq2skijqbdc Drive - Final Coundown - General Investigative Process]. To summarize, when you don't know why something is failing, and/or don't have any decent hypotheses to follow up, you can follow this process to systematicly find the problem.

Regarding the resolution to an incident: The resolution to any incident is usually one of two things:
# Fix the underlying problem.
# Fix the trigger itself.
Fixing the trigger is relavively straightforward, but do make sure document in the thread what you changed to which trigger.
Fixing the underlying problem can be more complex. A trade-off needs to be made sometimes between resolving technical debt, or simply patching the current system to resolve the issue. We usually look for a resolution that ensures that the problem won't re-occur soon, or makes it unexpected/unlikely for the problem to re-occur. Taking into account the timeframe that is available to resolve the incident you can make some trade-offs. An example would be: normal backups of VM's are failing due to the Proxmox backup server being down/unreachable and it is determined that this cannot be resolved at that moment. We can set up automatic backups to local storage temporary to resolve the immediate problem and ensure we keep our SLO's versus setting up a new Proxmox Backup server at a different location. Since we don't have much time to resolve the problem, the resolution would be to set up the automatic backups to local storage, and set up a new Proxmox Backup Server later as a seperate issue.

Some know issues and their resolutions:
* SSH service is down: The internet is a vile place. There's constant port scanning and hacking attempts ongoing to any machine connected to the internet (mostly IPv4). Due to this, SSH has a throttling functionality build in to prevent a system from being DDOS'ed by the amount of malicious SSH requests. This throttling can cause the Zabbix server from being denied an SSH connection, of which several failures fire this trigger. This hypothesis can be proven with a `journalctl -u ssh | grep 'MaxStartupsThrottling'` (you probably want to select a relevant time period with `--since "2 hours ago"` or something similar to prevent having to process a month of logging). You can then compare the throttling start and end times with the timestamps of the item data itself. The resolution for the issue is to add our custom ssh configuration [https://chat.dsinternal.net/#narrow/stream/23-SRE---General/topic/DS.20Whitelisted.20Custom.20SSH.20configuration/near/1620 Custom SSH Configuration].
* No backup for 3 days: Are S3 backup is very slow. Not much to prove as an underlying issue here. What needs to be done is check that the backup process is ongoing. The Zabbix latest data can be checked to verify that backups are running by checking that that days backups were done for the smaller buckets. The devteam email can be checked for if the backup process could not start on day due to it already running (it takes 24+ hours, and an attempt to start it is done each day by cron).
* git.* HTTPS is down: On Sunday mostly, Gitlab gets automaticly updated, but this incurs some downtime as the service is restarted. This is usually short enough to not be reported to Zulip as per our settings, but sometimes it's longer. If the service does not stay down, the issue can be just resolved.

==== Resolve incident + cleanup ====
When you've executed and verified the resolution in the previous steps we can proceed resolving the issue in our Mattermost integration. Resolving an incident can be done by doing the following:
# Verify that the trigger is no longer firing. An incident will be immediatly re-opened if the trigger is still firing, and the incident cannot be considered resolved if the trigger is still firing. If the trigger is still firing but you're sure that you've resolved the problem, you might need to force the item the trigger depends on to update. This can be done by finding the item in the host's configuration on Zabbix and selecting 'Execute Now', after a short period this should force Zabbix to re-execute the item. You can check the timestamps in the latest data of an item to check if it was updated.
# Close the incident by marking the topic as resolved, when there are no other triggers firing for the host.

Unfortunatly, some problems cause multiple critical and non-critical triggers to fire. This means we have to check Zabbix and Zulip for other fired triggers and ongoing incidents. The goal is to identify critical and non-critical incidents that were caused by the incident/underlying issue you just resolved.
# First, these incidents need to be acknowledged on Zabbix, and in the acknowledgement message you mention the incident/problem that caused this.
# Next, check the incident tracked by the integration on Mattermost using the `?ongoing` command. Resolve incidents that were (re-)opened by this incident by executing the following steps. If the first two fail (problem still persists, trigger is still firing), the incident needs to considered it's own issue and the relevant process needs to be followed (critical or non-critical depending on criticality).
## Ensuring the mentioned problem is no longer observable
## The trigger has resolved (You might need to force an update with `Execute Now`).
## Posting a link to the main incident you resolved with the comment that the underlying problem was resolved in that topic.
## Closing the incident by marking the topic as resolved, when there are no other triggers firing for the host.

When you are done, there should be no more critical triggers firing in Zabbix or open in the Zabbix-Mattermost intergration, for which no-one has taken responsibility or you have taken responsibility for and are not actively handling.

===Additional context===
* Critical incidents are posted in [https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical '''SLA - Critical'''].
* <s>When it is being tracked on GitLab a heavy check mark is added to the message.</s>
* <s>Responses on the thread and on GitLab are automatically synced (to some extend)</s>
* <s>When you reply with '''I agree that this has been fully resolved''' eventually our Zabbix-Mattermost integration will pick this up and a green check mark is added to the message.</s>

== Non-Critical incidents ==
* Non-critical incidents are acknowledged within 9 hours and resolved within one week.

=== Acknowledging ===
Fully acknowledging a non-critical incident requires the following tasks to have been completed:
* Acknowledging the incident on Zabbix
* Add the non-critical incident as a milestone in the metrics sheet
** Start date is the date of the incident
** DoD states what needs to be true for the non-critical incident to be consider resolved
* Add the non-critical incident to Lynx as a project
** Tasks need to be added
** Final tasks needs to have the SLO deadline set as 'contraint'
** Project priority is set to 20 (as a default)
** The tasks are estimated for SP
* The Lynx project ID is reported in the non-critical incident's topic on Zulip
* A Kimai activity is created in Kimai for the non-critical incident

Checklist (outdated)
# Acknowledge on Zabbix and state who is responsible for resolving this in the description
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# If there is no resolution to the incident, evaluate if the trigger needs updating/disabling
# Resolve incident

== Informational incidents ==
* Informational incidents are acknowledged within 72 hours

Checklist
# Acknowledge on Zabbix
# Sanity check the event, post result in thread
# If action needed, perform action

== If an incident is reported by other means than the Zabbix-Zulip intergration ==
# Acknowledge receipt.
# Classify the incident as critical, non-critical, or informational.
# Create an topic in the relevant SRE channel, stating the problem and that you is responsible for resolving it.
# Proceed to treat the incident according to the criticality you just classified it as. (So for a critical incident, it means you now start the critical incident handling process)

== Handover ==
When handing over the responsibility of '''first responder''' (FR), the following needs to happen:
* The handover can be initiated by both the upcoming FR or the acting FR
* Acting FR adds the upcoming FR the the IPA sla-first-responder user group and enables Zabbix calling for that the upcoming FR if they have that set by going to Zabbix > Configuration > Actions > [https://status.delftinfra.net/zabbix/actionconf.php?eventsource=0# Trigger actions]
* The upcoming FR makes sure they are aware of the state of the SLA and knows what questions they wants to ask the acting FR.
* The upcoming FR makes sure they are subscribed to the right channels

The following steps can be done async or in person:
* The acting FR announces/informs the upcoming FR has been added to the sla-first-responder group (In Zulip's [https://chat.dsinternal.net/#narrow/stream/13-Organisational Organisational channel] if asynq).
* If the acting FR wants to hand over responsibility for any ongoing incident they also state which incidents they want the upcoming FR to take over.
* If there are any particularities the upcoming FR needs to be aware of, those are shared.
* The upcoming FR asks their questions until they are satisfied and able to take over the FR
* The upcoming FR ensures they are subscribed to the following channels on Zulip: [https://chat.dsinternal.net/#narrow/stream/23-SRE---General SRE - General], [https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical SRE # Critical] and if part of the SRE team [https://chat.dsinternal.net/#streams/4/SRE%20##%20Non-critical SRE ## Non-Critical] and [https://chat.dsinternal.net/#streams/5/SRE%20###%20Informational SRE ### Informational].
* The upcoming FR announces/informs that they are now the acting FR over Zulip's [https://chat.dsinternal.net/#narrow/stream/13-Organisational Organisational channel]
* The now acting FR removes the previous FR from IPA the sla-first-responder user group and disables Zabbix calling for the previous FR if they had that enabled by going to Zabbix > Configuration > Actions > [https://status.delftinfra.net/zabbix/actionconf.php?eventsource=0# Trigger actions]

===Checklist for Handing Over First Responder (FR) Responsibilities===

When transferring the role of First Responder (FR), follow these steps:

1. Initiate the Handover:
- Acting FR or Upcoming FR*: Initiate the handover process.

2. Add Upcoming FR to User Group and Enable Zabbix Calling:
- Acting FR:
- Add the upcoming FR to the IPA `sla-first-responder` user group.
- Enable Zabbix calling for the upcoming FR (if applicable) by navigating to:
- Zabbix> Configuration > Actions > [Trigger actions](https://status.delftinfra.net/zabbix/actionconf.php?eventsource=0#).

3. Announce Addition to User Group:
- Acting FR:
- Inform the upcoming FR that they have been added to the `sla-first-responder` group.
- If communicating asynchronously, announce this in Zulip's [Organisational channel](https://chat.dsinternal.net/#narrow/stream/13-Organisational).

4. Communicate Ongoing Incidents and Particularities:
- Acting FR:
- Share details of any ongoing incidents to be handed over.
- Inform the upcoming FR of any specific details they need to be aware of.

5. Review SLA State and Prepare Questions:
- Upcoming FR:
- Familiarize yourself with the current state of the SLA.
- Prepare any questions for the acting FR.

6. Clarify and Confirm Understanding:
- Upcoming FR:
- Ask all necessary questions until you are confident in taking over the FR role.

7. Subscribe to Relevant Zulip Channels:
- Upcoming FR:
- Subscribe to the following channels:
- [SRE - General](https://chat.dsinternal.net/#narrow/stream/23-SRE---General)
- [SRE # Critical](https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical)
- If part of the SRE team, also subscribe to:
- [SRE ## Non-Critical](https://chat.dsinternal.net/#streams/4/SRE%20##%20Non-critical)
- [SRE ### Informational](https://chat.dsinternal.net/#streams/5/SRE%20###%20Informational)

8. Announce Acceptance of FR Role:
- Upcoming FR (now Acting FR):
- Announce that you are now the acting FR in Zulip's [Organisational channel](https://chat.dsinternal.net/#narrow/stream/13-Organisational).

9. Remove Previous FR from User Group and Disable Zabbix Calling:
- Now Acting FR:
- Remove the previous FR from the IPA `sla-first-responder` user group.
- Disable Zabbix calling for the previous FR by navigating to:
- Zabbix> Configuration > Actions > [Trigger actions](https://status.delftinfra.net/zabbix/actionconf.php?eventsource=0#).
- Ensure the previous FR's phone number is disabled in Zabbix calling settings.

10. Confirm Completion of Handover:
- Now Acting FR:
- Verify that all steps have been completed and the handover is fully executed.

MediaWiki:Mobile.css

2024-07-21T23:31:54Z

SleeplessByte: Fix double close

/* All CSS here will be loaded for users of the mobile site */

body {
display: flex;
flex-direction: column;
min-height: 100vh;
justify-content: start;
padding-top: 70px;
}

#mw-panel {
position: relative;
top: 0;
width: 100%;
background: beige;
border-bottom: 1px solid black;
box-shadow: 2px 2px 0 rgba(0, 0, 0, .2);
}

#mw-head, #p-personal {
position: relative;
}

#left-navigation, #right-navigation, .vector-menu-tabs-legacy, .vector-search-box {
float: none;
}

#left-navigation {
margin-left: 0;
margin-top: 0;
}

#p-personal ul {
padding-left: 0;
}

#footer, #content {
margin-left: 0;
}

MediaWiki:Mobile.css

2024-07-21T23:30:55Z

SleeplessByte: Remove indentation for menu

MediaWiki:Mobile.css

2024-07-21T23:28:44Z

SleeplessByte: Stop floating all the things

MediaWiki:Mobile.css

2024-07-21T23:25:59Z

SleeplessByte: Probably make the site work on mobile

MediaWiki:Mobile.css

2024-07-21T23:19:02Z

SleeplessByte: Make menu available on mobile

/* All CSS here will be loaded for users of the mobile site */

#mw-head {
display: none;
}

#mw-panel {
position: relative;
top: 0;
width: 100%;
background: beige;
border-bottom: 1px solid black;
box-shadow: 2px 2px 0 rgba(0, 0, 0, .2);
}

#footer, #content {
margin-left: 0;
}

Retrospectives

2024-07-05T17:27:40Z

SleeplessByte: /* Process */

At Delft Solutions most teams will hold a retrospective each week to see where process can be improved.

== Process ==

During the week, issues with the current process (for example a process not being followed by someone, or a process not taking care of a specific incident) will be listed in the [https://chat.dsinternal.net/#narrow/stream/15-Retro Retro channel]. Once a week, generally on Friday, these points are discussed in a timeboxed meeting. There is ''always'' an outcome to each item, where everyone in the team agrees that is how we wil move forward. That means that we cannot just do nothing, but we can ''decide'' to not change anything (and have it happen again next time). In most cases the resolution will be to amend the process at hand.

This also means that you cannot diverge from a current process ''during'' the week, but only after discussing it at the Retrospective.

Incident Handling

2024-07-05T17:23:16Z

SleeplessByte: Add retrospective point information about required channels.

== Zulip migration ==
Due to a migration to Zulip, the integration as was available on Mattermost is not available yet on Zulip. This leads to the following process changes:
* Acknowlegements and triggers resolving are not posted to Zulip by Zabbix
* Triggers are grouped in a topic on Zulip per host
* When an incident has been fully resolved, mark the topic as resolved, when any other incidents reported for the host are resolved
* There's no `?ongoing`, instead for now we can track open incidents by checking for unresolved topics
* The posting of incidents is less smart (only posting when not posted yet), so in order to prevent an incident from not being reported due to network issues or the likes, a message is posted after an inteval (8 hours for non-critical and lower, 1 hour for critical and above) while the incident has not been acknowleged.
* Incidents can be manually tracked by creating a topic by hand and reporting the problem
* There is no automatic gitlab issue creation or syncing anymore.

Finally, where this process says to do something on Mattermost, you should now do so on Zulip. The updates in the process chapters themselves are WIP.

== Critical incidents ==
* Critical incidents are resolved within 16 hours.

As first responder you take on the responsiblity of seeing an incident resolved. This does not mean that you are the person required to do all the work. You can attempt to involve other to help you (often referred to as escalating the incident), but since other are not on-call, they are not obliged to help you, especially outside of normal working hours.
Involving multiple people can quickly be required if multiple critical incidents with different causes occur simultaniously. In that case, the First Responder usually takes on a more information management role and steers those that are brought on into resolving the issues. (Example: if a server crashes, several critical triggers can fire, but the underlying cause can quite quickly be determined to be a single issue, the crashed server. So you wouldn't need to call in people to manage each incident. But a client's service being down in one cluster while in a different cluster a different VM no longer boots is likely to be to different issues, so in order to resolve them on time you'd want to call in help to resolve the incident in time).

=== Process ===
The general process is made up of the folowing steps. Each step has additional information on how to handle/execute them in the sections below.
# Take responsibility for seeing the incident resolved
# Determine if incident is still ongoing
# If ongoing: Communicate to affected clients that the issue is being investigated
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# Resolve incident + cleanup

During working on an incident it is expected that all communication is done in the incident's thread. This means all information to a problem can be found in a clear a predictable place. Sometimes an incident can be resolved by work done in another incident. In that case, it is required to post a link to that thread in the incident's thread with the comment that the resolution is done in that thread.

==== Acknowledge the incident on Zabbix ====
The first step is to take responsibility for seeing the incident resolved by acknowledging the incident on Zabbix. Simply acknowledging the trigger suffices. It is however entirely possible that multiple critical incidents are firing at the same time. This can be a coincidence, or can be because of a share cause of failure. For example, a server crashing will cause server VM' to reboot, or the router having an connectivity issue will lead to most other VM's having connectivity issues as well. If there are multiple critical incidents, it is advised to quickly observe what's ongoing, Zabbix is the best source of firing triggers for this, and pick the incident that is likely the root cause to

* Acknowledging an incident on Zabbix will stop Zabbix from calling the First Responder to notify them of the ongoing incident. And stops Zabbix from posting reminders on Zulip.

==== Determine if incident is still ongoing ====
The next step is to check if the reported problem is still ongoing. Depending on the observations made here your process to follow and steps needed to resolve the incident can change. There are three options:
# The trigger resolved itself and the problem cannot be observed. Example: HTTPS is down for a site, but the FR can access the site through HTTPS without incident.
# The trigger resolved itself and the problem can still be observed.
# The trigger is still firing but the problem cannot be observed: Our triggers might not be perfect, so it could be that something else is causing it to fire. A simple example would be that Zabbix reports that the the DNS for a site can't be resolved, but in reality there's a bug in the script we wrote that checks if the DNS resolves and the DNS resolves fine. Final note: keep in mind that an 'it works on my machine' does not necessarily mean it works for most other people, so depening on the trigger you need to do some evaluations if your tests suffice.

In order to make sure you are actually trying to observe the same thing as the trigger is looking for, make sure to check the trigger definition and the current data of the associated item(s). Some triggers might fire if one of multiple conditions is met (Such as a trigger that monitors the ping response time firing if the value exceeds a certain threshold, or if no data for a certain period of time was observed).

Make sure to report your findings in the incident's thread. It's advised to post a screenshot of the relevant item(s) and your own observations. (Continuing the ping example, you would post a screenshot of the relevant values, state your conclusion why the trigger is firing, and your own observations/pings)

==== Communicate to affected clients ====
If the incident is still ongoing and the service is down, we need to communicate to affected clients that we are aware of the problem and that we are investigating it. This is because critical incident usually mean the service is down, something the clients can notice/are affected by, so we to be transparent that something is going on. There are some additional notes to this though:
* If an incident has already resolved itself and the problem is no longer observable, we don't communicate anything. Doing so might only cause confusion, and since the client has not reported any issues, they have not had a noticeable problem with it themselves.
* Although a critical incident generally means that the client service is down or experiencing reduced service, not all critical incidents are of that nature. Some are more administrative, or are only an issue for Delft Solutions itself. As of writing I don't have an exhaustive list, but here is those I can think of:
** SSH Service is down: We don't have any clients that SSH into their services, so it's generally not a problem. But SSH is mostly used for SRE maintenance and publishing new builds. The SRE maintenance is an internal problem, so no need to communicate to the client. The publishing is done to Kaboom, preventing new builds from being published, and the two SM VM's.
** No backup for x days: Clients don't notice it if a backup is running late, so no need to communicate with clients. Just need to make sure the backup gets completed
** SSL certificate is expiring in < 24 hours: This is a bit dependent on how soon this incident is being handled, but if it handled quickly, the certificate never actually expired, and there has not been any disruption to the client's service, so no need for communicating about it.
* Determining which clients are being affected can be done by looking at the host's DNS in the trigger, and/or looking up the VM in Proxmox and checking the tags of the VM's for client names. In the case that this issue is causing multiple other critical triggers to fire, you would have to check for which clients are affected by those incidents.
* Communicating to DS about ongoing incidents is usually assumed to be automaticly have been done by the fact that the incident was reported on Zulip.

As always, report the decisions taken and actions maded in the incident thread. (e.g.: I've sent a message in the Slack to let Kaboom know that we aware of problem x, and that we are investigating it)

==== Communicate plan/next steps + Communicate findings/results of executed plan ====
This is the main part of handling an incident. There are several actions you can take in these steps, but at the basis they consist of sharing your next steps, performing those, and reporting the results. The reason all this needs to be reported is to ensure that all known information about a problem is logged, making it easier for someone else to be onboarded into the issue, for later reference if a similar issue is encountered, and even for use during the incident itself in case an older configuration needs to be referenced after you changed it.
The objective from these steps is determining what is actually wrong and how to resolve it. Depending on the observations made earlier on whether the incident is still ongoing and is (still) observable your investigation can go into different directions. (e.g. Find the underlying cause for a trigger, or determining why the trigger is firing while it likely shouldn't, and then how to resolve that underlying cause or how to update the trigger to work better)

There are three main types of steps defined, but you are not limited to these:
# Hypothesis: If you have an idea what could be causing it, you would state your hypothesis and your next step would be to prove that hypothesis. For example, for an incident 'SSH service is down on X' your hypothesis could be that this is due to 'MaxStartups' throttling, which can be proven by 'grep'ing journalctl for that, and compare the start and end times of throttling with the timestamps of the item reporting the status of the SSH service.
# Information gathering: Sometimes it just helps to get some facts about the situation collected. What is usefull information that is relevant depends on the triggers, but some examples are: The syslog/journalctl of the host from around the time of the incident (it can contain a reference to the an underlying problem in various levels of explicitness), the ping response from several hosts on the route to a host or a traceroute (this helps with networking issues). The gathered information is usually intended to help you come up with an hypothesis on what's wrong.
# Investigative: The most rigorous of process. The full process is described here originally [https://docs.google.com/document/d/1AQYJM1Q9l2Tyk6zfCVaQ2aEq-dpbfUH5okE88bpKkhw/edit#heading=h.5fq2skijqbdc Drive - Final Coundown - General Investigative Process]. To summarize, when you don't know why something is failing, and/or don't have any decent hypotheses to follow up, you can follow this process to systematicly find the problem.

Regarding the resolution to an incident: The resolution to any incident is usually one of two things:
# Fix the underlying problem.
# Fix the trigger itself.
Fixing the trigger is relavively straightforward, but do make sure document in the thread what you changed to which trigger.
Fixing the underlying problem can be more complex. A trade-off needs to be made sometimes between resolving technical debt, or simply patching the current system to resolve the issue. We usually look for a resolution that ensures that the problem won't re-occur soon, or makes it unexpected/unlikely for the problem to re-occur. Taking into account the timeframe that is available to resolve the incident you can make some trade-offs. An example would be: normal backups of VM's are failing due to the Proxmox backup server being down/unreachable and it is determined that this cannot be resolved at that moment. We can set up automatic backups to local storage temporary to resolve the immediate problem and ensure we keep our SLO's versus setting up a new Proxmox Backup server at a different location. Since we don't have much time to resolve the problem, the resolution would be to set up the automatic backups to local storage, and set up a new Proxmox Backup Server later as a seperate issue.

Some know issues and their resolutions:
* SSH service is down: The internet is a vile place. There's constant port scanning and hacking attempts ongoing to any machine connected to the internet (mostly IPv4). Due to this, SSH has a throttling functionality build in to prevent a system from being DDOS'ed by the amount of malicious SSH requests. This throttling can cause the Zabbix server from being denied an SSH connection, of which several failures fire this trigger. This hypothesis can be proven with a `journalctl -u ssh | grep 'MaxStartupsThrottling'` (you probably want to select a relevant time period with `--since "2 hours ago"` or something similar to prevent having to process a month of logging). You can then compare the throttling start and end times with the timestamps of the item data itself. The resolution for the issue is to add our custom ssh configuration [https://chat.dsinternal.net/#narrow/stream/23-SRE---General/topic/DS.20Whitelisted.20Custom.20SSH.20configuration/near/1620 Custom SSH Configuration].
* No backup for 3 days: Are S3 backup is very slow. Not much to prove as an underlying issue here. What needs to be done is check that the backup process is ongoing. The Zabbix latest data can be checked to verify that backups are running by checking that that days backups were done for the smaller buckets. The devteam email can be checked for if the backup process could not start on day due to it already running (it takes 24+ hours, and an attempt to start it is done each day by cron).
* git.* HTTPS is down: On Sunday mostly, Gitlab gets automaticly updated, but this incurs some downtime as the service is restarted. This is usually short enough to not be reported to Zulip as per our settings, but sometimes it's longer. If the service does not stay down, the issue can be just resolved.

==== Resolve incident + cleanup ====
When you've executed and verified the resolution in the previous steps we can proceed resolving the issue in our Mattermost integration. Resolving an incident can be done by doing the following:
# Verify that the trigger is no longer firing. An incident will be immediatly re-opened if the trigger is still firing, and the incident cannot be considered resolved if the trigger is still firing. If the trigger is still firing but you're sure that you've resolved the problem, you might need to force the item the trigger depends on to update. This can be done by finding the item in the host's configuration on Zabbix and selecting 'Execute Now', after a short period this should force Zabbix to re-execute the item. You can check the timestamps in the latest data of an item to check if it was updated.
# Close the incident by marking the topic as resolved, when there are no other triggers firing for the host.

Unfortunatly, some problems cause multiple critical and non-critical triggers to fire. This means we have to check Zabbix and Zulip for other fired triggers and ongoing incidents. The goal is to identify critical and non-critical incidents that were caused by the incident/underlying issue you just resolved.
# First, these incidents need to be acknowledged on Zabbix, and in the acknowledgement message you mention the incident/problem that caused this.
# Next, check the incident tracked by the integration on Mattermost using the `?ongoing` command. Resolve incidents that were (re-)opened by this incident by executing the following steps. If the first two fail (problem still persists, trigger is still firing), the incident needs to considered it's own issue and the relevant process needs to be followed (critical or non-critical depending on criticality).
## Ensuring the mentioned problem is no longer observable
## The trigger has resolved (You might need to force an update with `Execute Now`).
## Posting a link to the main incident you resolved with the comment that the underlying problem was resolved in that topic.
## Closing the incident by marking the topic as resolved, when there are no other triggers firing for the host.

When you are done, there should be no more critical triggers firing in Zabbix or open in the Zabbix-Mattermost intergration, for which no-one has taken responsibility or you have taken responsibility for and are not actively handling.

===Additional context===
* Critical incidents are posted in [https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical '''SLA - Critical'''].
* <s>When it is being tracked on GitLab a heavy check mark is added to the message.</s>
* <s>Responses on the thread and on GitLab are automatically synced (to some extend)</s>
* <s>When you reply with '''I agree that this has been fully resolved''' eventually our Zabbix-Mattermost integration will pick this up and a green check mark is added to the message.</s>

== Non-Critical incidents ==
* Non-critical incidents are acknowledged within 9 hours and resolved within one week.

Checklist
# Acknowledge on Zabbix and state who is responsible for resolving this in the description
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# If there is no resolution to the incident, evaluate if the trigger needs updating/disabling
# Resolve incident

== Informational incidents ==
* Informational incidents are acknowledged within 72 hours

Checklist
# Acknowledge on Zabbix
# Sanity check the event, post result in thread
# If action needed, perform action

== If an incident is reported by other means than the Zabbix-Zulip intergration ==
# Acknowledge receipt.
# Classify the incident as critical, non-critical, or informational.
# Create an topic in the relevant SRE channel, stating the problem and that you is responsible for resolving it.
# Proceed to treat the incident according to the criticality you just classified it as. (So for a critical incident, it means you now start the critical incident handling process)

== Handover ==
When handing over the responsibility of '''first responder''' (FR), the following needs to happen:
* Acting FR adds the upcoming FR the the IPA sla-first-responder user group (Optional: and enables Zabbix calling for that person if they have that set by going to Zabbix > Configuration > Actions > [https://status.delftinfra.net/zabbix/actionconf.php?eventsource=0# Trigger actions])
* The upcoming FR makes sure they are aware of the state of the SLA and knows what questions they wants to ask the acting FR.
* The upcoming FR makes sure they are subscribed to the right channels

The following steps can be done async or in person:
* The acting FR announces/informs the upcoming FR has been added to the sla-first-responder group (In Zulip's [https://chat.dsinternal.net/#narrow/stream/13-Organisational Organisational channel] if asynq).
* If the acting FR wants to hand over responsibility for any ongoing incident they also state which incidents they want the upcoming FR to take over.
* If there are any particularities the upcoming FR needs to be aware of, those are shared.
* The upcoming FR asks their questions until they are satisfied and able to take over the FR
* The upcoming FR ensures they are subscribed to the following channels on Zulip: [https://chat.dsinternal.net/#narrow/stream/23-SRE---General SRE - General], [https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical SRE # Critical] and if part of the SRE team [https://chat.dsinternal.net/#streams/4/SRE%20##%20Non-critical SRE ## Non-Critical] and [https://chat.dsinternal.net/#streams/5/SRE%20###%20Informational SRE ### Informational].
* The upcoming FR announces/informs that they are now the acting FR over Zulip's [https://chat.dsinternal.net/#narrow/stream/13-Organisational Organisational channel]
* The now acting FR removes the previous FR from IPA the sla-first-responder user group (Optional: and disables Zabbix calling for that person if they had that enabled by going to Zabbix > Configuration > Actions > [https://status.delftinfra.net/zabbix/actionconf.php?eventsource=0# Trigger actions])

Incident Handling

2024-07-05T17:17:46Z

SleeplessByte: /* Additional context */

== Zulip migration ==
Due to a migration to Zulip, the integration as was available on Mattermost is not available yet on Zulip. This leads to the following process changes:
* Acknowlegements and triggers resolving are not posted to Zulip by Zabbix
* Triggers are grouped in a topic on Zulip per host
* When an incident has been fully resolved, mark the topic as resolved, when any other incidents reported for the host are resolved
* There's no `?ongoing`, instead for now we can track open incidents by checking for unresolved topics
* The posting of incidents is less smart (only posting when not posted yet), so in order to prevent an incident from not being reported due to network issues or the likes, a message is posted after an inteval (8 hours for non-critical and lower, 1 hour for critical and above) while the incident has not been acknowleged.
* Incidents can be manually tracked by creating a topic by hand and reporting the problem
* There is no automatic gitlab issue creation or syncing anymore.

Finally, where this process says to do something on Mattermost, you should now do so on Zulip. The updates in the process chapters themselves are WIP.

== Critical incidents ==
* Critical incidents are resolved within 16 hours.

As first responder you take on the responsiblity of seeing an incident resolved. This does not mean that you are the person required to do all the work. You can attempt to involve other to help you (often referred to as escalating the incident), but since other are not on-call, they are not obliged to help you, especially outside of normal working hours.
Involving multiple people can quickly be required if multiple critical incidents with different causes occur simultaniously. In that case, the First Responder usually takes on a more information management role and steers those that are brought on into resolving the issues. (Example: if a server crashes, several critical triggers can fire, but the underlying cause can quite quickly be determined to be a single issue, the crashed server. So you wouldn't need to call in people to manage each incident. But a client's service being down in one cluster while in a different cluster a different VM no longer boots is likely to be to different issues, so in order to resolve them on time you'd want to call in help to resolve the incident in time).

=== Process ===
The general process is made up of the folowing steps. Each step has additional information on how to handle/execute them in the sections below.
# Take responsibility for seeing the incident resolved
# Determine if incident is still ongoing
# If ongoing: Communicate to affected clients that the issue is being investigated
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# Resolve incident + cleanup

During working on an incident it is expected that all communication is done in the incident's thread. This means all information to a problem can be found in a clear a predictable place. Sometimes an incident can be resolved by work done in another incident. In that case, it is required to post a link to that thread in the incident's thread with the comment that the resolution is done in that thread.

==== Acknowledge the incident on Zabbix ====
The first step is to take responsibility for seeing the incident resolved by acknowledging the incident on Zabbix. Simply acknowledging the trigger suffices. It is however entirely possible that multiple critical incidents are firing at the same time. This can be a coincidence, or can be because of a share cause of failure. For example, a server crashing will cause server VM' to reboot, or the router having an connectivity issue will lead to most other VM's having connectivity issues as well. If there are multiple critical incidents, it is advised to quickly observe what's ongoing, Zabbix is the best source of firing triggers for this, and pick the incident that is likely the root cause to

* Acknowledging an incident on Zabbix will stop Zabbix from calling the First Responder to notify them of the ongoing incident. And stops Zabbix from posting reminders on Zulip.

==== Determine if incident is still ongoing ====
The next step is to check if the reported problem is still ongoing. Depending on the observations made here your process to follow and steps needed to resolve the incident can change. There are three options:
# The trigger resolved itself and the problem cannot be observed. Example: HTTPS is down for a site, but the FR can access the site through HTTPS without incident.
# The trigger resolved itself and the problem can still be observed.
# The trigger is still firing but the problem cannot be observed: Our triggers might not be perfect, so it could be that something else is causing it to fire. A simple example would be that Zabbix reports that the the DNS for a site can't be resolved, but in reality there's a bug in the script we wrote that checks if the DNS resolves and the DNS resolves fine. Final note: keep in mind that an 'it works on my machine' does not necessarily mean it works for most other people, so depening on the trigger you need to do some evaluations if your tests suffice.

In order to make sure you are actually trying to observe the same thing as the trigger is looking for, make sure to check the trigger definition and the current data of the associated item(s). Some triggers might fire if one of multiple conditions is met (Such as a trigger that monitors the ping response time firing if the value exceeds a certain threshold, or if no data for a certain period of time was observed).

Make sure to report your findings in the incident's thread. It's advised to post a screenshot of the relevant item(s) and your own observations. (Continuing the ping example, you would post a screenshot of the relevant values, state your conclusion why the trigger is firing, and your own observations/pings)

==== Communicate to affected clients ====
If the incident is still ongoing and the service is down, we need to communicate to affected clients that we are aware of the problem and that we are investigating it. This is because critical incident usually mean the service is down, something the clients can notice/are affected by, so we to be transparent that something is going on. There are some additional notes to this though:
* If an incident has already resolved itself and the problem is no longer observable, we don't communicate anything. Doing so might only cause confusion, and since the client has not reported any issues, they have not had a noticeable problem with it themselves.
* Although a critical incident generally means that the client service is down or experiencing reduced service, not all critical incidents are of that nature. Some are more administrative, or are only an issue for Delft Solutions itself. As of writing I don't have an exhaustive list, but here is those I can think of:
** SSH Service is down: We don't have any clients that SSH into their services, so it's generally not a problem. But SSH is mostly used for SRE maintenance and publishing new builds. The SRE maintenance is an internal problem, so no need to communicate to the client. The publishing is done to Kaboom, preventing new builds from being published, and the two SM VM's.
** No backup for x days: Clients don't notice it if a backup is running late, so no need to communicate with clients. Just need to make sure the backup gets completed
** SSL certificate is expiring in < 24 hours: This is a bit dependent on how soon this incident is being handled, but if it handled quickly, the certificate never actually expired, and there has not been any disruption to the client's service, so no need for communicating about it.
* Determining which clients are being affected can be done by looking at the host's DNS in the trigger, and/or looking up the VM in Proxmox and checking the tags of the VM's for client names. In the case that this issue is causing multiple other critical triggers to fire, you would have to check for which clients are affected by those incidents.
* Communicating to DS about ongoing incidents is usually assumed to be automaticly have been done by the fact that the incident was reported on Zulip.

As always, report the decisions taken and actions maded in the incident thread. (e.g.: I've sent a message in the Slack to let Kaboom know that we aware of problem x, and that we are investigating it)

==== Communicate plan/next steps + Communicate findings/results of executed plan ====
This is the main part of handling an incident. There are several actions you can take in these steps, but at the basis they consist of sharing your next steps, performing those, and reporting the results. The reason all this needs to be reported is to ensure that all known information about a problem is logged, making it easier for someone else to be onboarded into the issue, for later reference if a similar issue is encountered, and even for use during the incident itself in case an older configuration needs to be referenced after you changed it.
The objective from these steps is determining what is actually wrong and how to resolve it. Depending on the observations made earlier on whether the incident is still ongoing and is (still) observable your investigation can go into different directions. (e.g. Find the underlying cause for a trigger, or determining why the trigger is firing while it likely shouldn't, and then how to resolve that underlying cause or how to update the trigger to work better)

There are three main types of steps defined, but you are not limited to these:
# Hypothesis: If you have an idea what could be causing it, you would state your hypothesis and your next step would be to prove that hypothesis. For example, for an incident 'SSH service is down on X' your hypothesis could be that this is due to 'MaxStartups' throttling, which can be proven by 'grep'ing journalctl for that, and compare the start and end times of throttling with the timestamps of the item reporting the status of the SSH service.
# Information gathering: Sometimes it just helps to get some facts about the situation collected. What is usefull information that is relevant depends on the triggers, but some examples are: The syslog/journalctl of the host from around the time of the incident (it can contain a reference to the an underlying problem in various levels of explicitness), the ping response from several hosts on the route to a host or a traceroute (this helps with networking issues). The gathered information is usually intended to help you come up with an hypothesis on what's wrong.
# Investigative: The most rigorous of process. The full process is described here originally [https://docs.google.com/document/d/1AQYJM1Q9l2Tyk6zfCVaQ2aEq-dpbfUH5okE88bpKkhw/edit#heading=h.5fq2skijqbdc Drive - Final Coundown - General Investigative Process]. To summarize, when you don't know why something is failing, and/or don't have any decent hypotheses to follow up, you can follow this process to systematicly find the problem.

Regarding the resolution to an incident: The resolution to any incident is usually one of two things:
# Fix the underlying problem.
# Fix the trigger itself.
Fixing the trigger is relavively straightforward, but do make sure document in the thread what you changed to which trigger.
Fixing the underlying problem can be more complex. A trade-off needs to be made sometimes between resolving technical debt, or simply patching the current system to resolve the issue. We usually look for a resolution that ensures that the problem won't re-occur soon, or makes it unexpected/unlikely for the problem to re-occur. Taking into account the timeframe that is available to resolve the incident you can make some trade-offs. An example would be: normal backups of VM's are failing due to the Proxmox backup server being down/unreachable and it is determined that this cannot be resolved at that moment. We can set up automatic backups to local storage temporary to resolve the immediate problem and ensure we keep our SLO's versus setting up a new Proxmox Backup server at a different location. Since we don't have much time to resolve the problem, the resolution would be to set up the automatic backups to local storage, and set up a new Proxmox Backup Server later as a seperate issue.

Some know issues and their resolutions:
* SSH service is down: The internet is a vile place. There's constant port scanning and hacking attempts ongoing to any machine connected to the internet (mostly IPv4). Due to this, SSH has a throttling functionality build in to prevent a system from being DDOS'ed by the amount of malicious SSH requests. This throttling can cause the Zabbix server from being denied an SSH connection, of which several failures fire this trigger. This hypothesis can be proven with a `journalctl -u ssh | grep 'MaxStartupsThrottling'` (you probably want to select a relevant time period with `--since "2 hours ago"` or something similar to prevent having to process a month of logging). You can then compare the throttling start and end times with the timestamps of the item data itself. The resolution for the issue is to add our custom ssh configuration [https://chat.dsinternal.net/#narrow/stream/23-SRE---General/topic/DS.20Whitelisted.20Custom.20SSH.20configuration/near/1620 Custom SSH Configuration].
* No backup for 3 days: Are S3 backup is very slow. Not much to prove as an underlying issue here. What needs to be done is check that the backup process is ongoing. The Zabbix latest data can be checked to verify that backups are running by checking that that days backups were done for the smaller buckets. The devteam email can be checked for if the backup process could not start on day due to it already running (it takes 24+ hours, and an attempt to start it is done each day by cron).
* git.* HTTPS is down: On Sunday mostly, Gitlab gets automaticly updated, but this incurs some downtime as the service is restarted. This is usually short enough to not be reported to Zulip as per our settings, but sometimes it's longer. If the service does not stay down, the issue can be just resolved.

==== Resolve incident + cleanup ====
When you've executed and verified the resolution in the previous steps we can proceed resolving the issue in our Mattermost integration. Resolving an incident can be done by doing the following:
# Verify that the trigger is no longer firing. An incident will be immediatly re-opened if the trigger is still firing, and the incident cannot be considered resolved if the trigger is still firing. If the trigger is still firing but you're sure that you've resolved the problem, you might need to force the item the trigger depends on to update. This can be done by finding the item in the host's configuration on Zabbix and selecting 'Execute Now', after a short period this should force Zabbix to re-execute the item. You can check the timestamps in the latest data of an item to check if it was updated.
# Close the incident by marking the topic as resolved, when there are no other triggers firing for the host.

Unfortunatly, some problems cause multiple critical and non-critical triggers to fire. This means we have to check Zabbix and Zulip for other fired triggers and ongoing incidents. The goal is to identify critical and non-critical incidents that were caused by the incident/underlying issue you just resolved.
# First, these incidents need to be acknowledged on Zabbix, and in the acknowledgement message you mention the incident/problem that caused this.
# Next, check the incident tracked by the integration on Mattermost using the `?ongoing` command. Resolve incidents that were (re-)opened by this incident by executing the following steps. If the first two fail (problem still persists, trigger is still firing), the incident needs to considered it's own issue and the relevant process needs to be followed (critical or non-critical depending on criticality).
## Ensuring the mentioned problem is no longer observable
## The trigger has resolved (You might need to force an update with `Execute Now`).
## Posting a link to the main incident you resolved with the comment that the underlying problem was resolved in that topic.
## Closing the incident by marking the topic as resolved, when there are no other triggers firing for the host.

When you are done, there should be no more critical triggers firing in Zabbix or open in the Zabbix-Mattermost intergration, for which no-one has taken responsibility or you have taken responsibility for and are not actively handling.

===Additional context===
* Critical incidents are posted in [https://chat.dsinternal.net/#narrow/stream/24-SRE-.23-Critical '''SLA - Critical'''].
* <s>When it is being tracked on GitLab a heavy check mark is added to the message.</s>
* <s>Responses on the thread and on GitLab are automatically synced (to some extend)</s>
* <s>When you reply with '''I agree that this has been fully resolved''' eventually our Zabbix-Mattermost integration will pick this up and a green check mark is added to the message.</s>

== Non-Critical incidents ==
* Non-critical incidents are acknowledged within 9 hours and resolved within one week.

Checklist
# Acknowledge on Zabbix and state who is responsible for resolving this in the description
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# If there is no resolution to the incident, evaluate if the trigger needs updating/disabling
# Resolve incident

== Informational incidents ==
* Informational incidents are acknowledged within 72 hours

Checklist
# Acknowledge on Zabbix
# Sanity check the event, post result in thread
# If action needed, perform action

== If an incident is reported by other means than the Zabbix-Zulip intergration ==
# Acknowledge receipt.
# Classify the incident as critical, non-critical, or informational.
# Create an topic in the relevant SRE channel, stating the problem and that you is responsible for resolving it.
# Proceed to treat the incident according to the criticality you just classified it as. (So for a critical incident, it means you now start the critical incident handling process)

== Handover ==
When handing over the responsibility of first responder (FR), the following needs to happen:
* Acting FR adds the upcoming FR the the IPA sla-first-responder user group (Optional: and enables Zabbix calling for that person if they have that set by going to Zabbix > Configuration > Actions > Trigger actions)
* The upcoming FR makes sure they are aware of the state of the SLA and knows what questions they wants to ask the acting FR.
The following steps can be done async or in person:
* The acting FR announces/informs the upcoming FR has been added to the sla-first-responder group (In Zulip's organisational channel if asynq).
* If the acting FR wants to hand over responsibility for any ongoing incident they also state which incidents they want the upcoming FR to take over.
* If there are any particularities the upcoming FR needs to be aware of, those are shared.
* The upcoming FR asks their questions until they are satisfied and able to take over the FR
* The upcoming FR announces/informs that they are now the acting FR over Zulip's organisational channel
* The now acting FR removes the previous FR from IPA the sla-first-responder user group (Optional: and disables Zabbix calling for that person if they had that enabled by going to Zabbix > Configuration > Actions > Trigger actions)

Stack

2024-05-28T17:22:46Z

SleeplessByte:

We tend to use "whatever tool for the job makes most sense" for greenfield projects.

== Mobile applications ==

''Programming Languages
* Bash
* CSS
* HTML
* Kotlin
* Swift
* TypeScript

''Common Libraries
* [https://atlassian.design/components/pragmatic-drag-and-drop/core-package Pragmatic Drag and Drop]
* [https://reactnative.dev/ React-Native]
* [https://docs.swmansion.com/react-native-reanimated/ React-Native Reanimated]
* [https://tanstack.com/query/latest TanStack Query]
* [https://tanstack.com/table/latest TanStack Table]

''UI frameworks
* [https://www.nativewind.dev/ Nativewind]
* [https://reactnativepaper.com/ React-Native paper]

''Frameworks
* [https://expo.dev/ Expo]

== Full stack applications ==

''Programming Languages
* Bash
* CSS
* HTML
* Ruby
* TypeScript

''Common Libraries
* [https://github.com/SleeplessByte/media-types-ruby Media Types]
* [https://github.com/XPBytes/media_types-serialization/tree/master Media Types Serialization]
* [https://postcss.org/ PostCSS]
* [https://stimulus.hotwired.dev/ Stimulus]
* [https://uppy.io/docs/uppy/ Uppy]

''UI frameworks
* [https://m3.material.io/ Material Design]
* [https://tailwindcss.com/ Tailwind CSS]
* [https://tailwindui.com/ Tailwind UI]

''Frameworks
* [https://rubyonrails.org/ Ruby on Rails]

== Front-endish web applications ==
''Programming Languages
* Bash
* CSS
* HTML
* TypeScript

''Libraries
* [https://react.dev React]
* [https://tanstack.com/query/latest TanStack Query]
* [https://tanstack.com/table/latest TanStack Table]

''UI frameworks
* [https://m3.material.io/ Material Design]
* [https://tailwindcss.com/ Tailwind CSS]
* [https://tailwindui.com/ Tailwind UI]

''Frameworks
* [https://remix.run/ Remix]

Stack

2024-05-28T17:22:23Z

SleeplessByte: Created page with "We tend to use "whatever tool for the job makes most sense" for greenfield projects. == Mobile applications == ''Programming Languages * Bash * CSS * HTML * Kotlin * Swift * TypeScript ''Common Libraries * [https://atlassian.design/components/pragmatic-drag-and-drop/core-package Pragmatic Drag and Drop] * [https://reactnative.dev/ React-Native] * [https://docs.swmansion.com/react-native-reanimated/ React-Native Reanimated] * [https://tanstack.com/query/latest TanStack..."

We tend to use "whatever tool for the job makes most sense" for greenfield projects.

== Mobile applications ==

''Programming Languages
* Bash
* CSS
* HTML
* Kotlin
* Swift
* TypeScript

''Common Libraries
* [https://atlassian.design/components/pragmatic-drag-and-drop/core-package Pragmatic Drag and Drop]
* [https://reactnative.dev/ React-Native]
* [https://docs.swmansion.com/react-native-reanimated/ React-Native Reanimated]
* [https://tanstack.com/query/latest TanStack Query]
* [https://tanstack.com/table/latest TanStack Table]

''UI frameworks
* [https://www.nativewind.dev/ Nativewind]
* [https://reactnativepaper.com/ React-Native paper]

''Frameworks
* [https://expo.dev/ Expo]

== Full stack applications ==

''Programming Languages
* Bash
* CSS
* HTML
* Ruby
* TypeScript

''Common Libraries
* [https://github.com/SleeplessByte/media-types-ruby Media Types]
* [https://github.com/XPBytes/media_types-serialization/tree/master Media Types Serialization]
* [https://postcss.org/ PostCSS]
* [https://stimulus.hotwired.dev/ Stimulus]
* [https://uppy.io/docs/uppy/ Uppy]

''UI frameworks
* [https://m3.material.io/ Material Design]
* [https://tailwindcss.com/ Tailwind CSS]
* [https://tailwindui.com/ Tailwind UI]

''Frameworks
* [https://rubyonrails.org/ Ruby on Rails]

== Front-endish web applications ==
* Bash
* CSS
* HTML
* TypeScript

''Libraries
* [https://react.dev React]
* [https://tanstack.com/query/latest TanStack Query]
* [https://tanstack.com/table/latest TanStack Table]

''UI frameworks
* [https://m3.material.io/ Material Design]
* [https://tailwindcss.com/ Tailwind CSS]
* [https://tailwindui.com/ Tailwind UI]

''Frameworks
* [https://remix.run/ Remix]

Internal

2024-05-28T17:02:36Z

SleeplessByte: /* Other */

== Finance ==

=== Exact ===

* [[booking bonus|Booking bonus]]
* [[booking wages|Booking wages]]
* [[new receipt|Enter a new receipt]]
* [[reconciliation|Reconciliation of transaction]]
* [[invoicing|Send an invoice]]
* [[payment reminders|Send payment reminder]]
* [[invoice approval|Process for approving invoices (/filed receipts)]]

=== Bunq ===

* [[top up account|Top up expense account]]

== Work Process ==

* [[Definition of done|Definition of Done]]
* [[Incident Handling|Incident Handling]]

== Internal Process ==

* [[12 percent|12% time]]
* [[Annual leave|Annual leave]]
* [[Bonus allocation|Bonus allocation]]
* [[Calamity leave|Calamity leave]]
* [[Overtime|Overtime]]
* [[Retrospectives|Retrospectives]]
* [[Sick leave|Sick leave]]
* [[Training and self-study|Training and Self-Study]]

== Projects ==

* Era Inventory [[project_era_inventory_api|API Description]]

== SRE ==

''To be further populated with guide from drive''
* [[create gitlab runner host|Create a GitLab runner host]]
* [[vm setup|Create a (Debian) VM]]
* [[border reboot|Reboot border without downtime]]
* [[WS Proxmox node reboot|Reboot WS Proxmox node without downtime]]
* [[Resize VM Disk]]
* [[SRE tools]]

== Other ==

* [[stack|Greenfield stack]]
* [[standard tools|Standard Tools]]
* [[list of unfurl debuggers|List of unfurl debuggers]]

SRE tools

2024-05-28T16:57:33Z

SleeplessByte: Created page with "* Ceph * Debian * FreeIPA * HTTPS/TLS * Kerberos * Keycloak * LDAP * MAKE * Netbox * Proxmox * Zabbix * Zulip ''' Handbooks * [https://www.debian.org/doc/manuals/maint-guide/ Debian maintainers handbook] * Debian users handbook"

* Ceph
* Debian
* FreeIPA
* HTTPS/TLS
* Kerberos
* Keycloak
* LDAP
* MAKE
* Netbox
* Proxmox
* Zabbix
* Zulip

''' Handbooks
* [https://www.debian.org/doc/manuals/maint-guide/ Debian maintainers handbook]
* Debian users handbook

Internal

2024-05-28T16:53:16Z

SleeplessByte: /* SRE */

== Finance ==

=== Exact ===

* [[booking bonus|Booking bonus]]
* [[booking wages|Booking wages]]
* [[new receipt|Enter a new receipt]]
* [[reconciliation|Reconciliation of transaction]]
* [[invoicing|Send an invoice]]
* [[payment reminders|Send payment reminder]]
* [[invoice approval|Process for approving invoices (/filed receipts)]]

=== Bunq ===

* [[top up account|Top up expense account]]

== Work Process ==

* [[Definition of done|Definition of Done]]
* [[Incident Handling|Incident Handling]]

== Internal Process ==

* [[12 percent|12% time]]
* [[Annual leave|Annual leave]]
* [[Bonus allocation|Bonus allocation]]
* [[Calamity leave|Calamity leave]]
* [[Overtime|Overtime]]
* [[Retrospectives|Retrospectives]]
* [[Sick leave|Sick leave]]
* [[Training and self-study|Training and Self-Study]]

== Projects ==

* Era Inventory [[project_era_inventory_api|API Description]]

== SRE ==

''To be further populated with guide from drive''
* [[create gitlab runner host|Create a GitLab runner host]]
* [[vm setup|Create a (Debian) VM]]
* [[border reboot|Reboot border without downtime]]
* [[WS Proxmox node reboot|Reboot WS Proxmox node without downtime]]
* [[Resize VM Disk]]
* [[SRE tools]]

== Other ==

* [[standard tools|Standard Tools]]
* [[list of unfurl debuggers|List of unfurl debuggers]]

Incident Handling

2024-05-27T16:27:02Z

SleeplessByte: /* Handover */

== Critical incidents ==
* Critical incidents are resolved within 16 hours.

As first responder you take on the responsiblity of seeing an incident resolved. This does not mean that you are the person required to do all the work. You can attempt to involve other to help you (often referred to as escalating the incident), but since other are not on-call, they are not obliged to help you, especially outside of normal working hours.
Involving multiple people can quickly be required if multiple critical incidents with different causes occur simultaniously. In that case, the First Responder usually takes on a more information management role and steers those that are brought on into resolving the issues. (Example: if a server crashes, several critical triggers can fire, but the underlying cause can quite quickly be determined to be a single issue, the crashed server. So you wouldn't need to call in people to manage each incident. But a client's service being down in one cluster while in a different cluster a different VM no longer boots is likely to be to different issues, so in order to resolve them on time you'd want to call in help to resolve the incident in time).

=== Process ===
The general process is made up of the folowing steps. Each step has additional information on how to handle/execute them in the sections below.
# Take responsibility for seeing the incident resolved
# Determine if incident is still ongoing
# If ongoing: Communicate to affected clients that the issue is being investigated
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# Resolve incident + cleanup

During working on an incident it is expected that all communication is done in the incident's thread. This means all information to a problem can be found in a clear a predictable place. Sometimes an incident can be resolved by work done in another incident. In that case, it is required to post a link to that thread in the incident's thread with the comment that the resolution is done in that thread.

==== Acknowledge the incident on Zabbix ====
The first step is to take responsibility for seeing the incident resolved by acknowledging the incident on Zabbix. Simply acknowledging the trigger suffices. It is however entirely possible that multiple critical incidents are firing at the same time. This can be a coincidence, or can be because of a share cause of failure. For example, a server crashing will cause server VM' to reboot, or the router having an connectivity issue will lead to most other VM's having connectivity issues as well. If there are multiple critical incidents, it is advised to quickly observe what's ongoing, Zabbix is the best source of firing triggers for this, and pick the incident that is likely the root cause to

* Acknowledging an incident on Zabbix will stop Zabbix from calling the First Responder to notify them of the ongoing incident.

==== Determine if incident is still ongoing ====
The next step is to check if the reported problem is still ongoing. Depending on the observations made here your process to follow and steps needed to resolve the incident can change. There are three options:
# The trigger resolved itself and the problem cannot be observed. Example: HTTPS is down for a site, but the FR can access the site through HTTPS without incident.
# The trigger resolved itself and the problem can still be observed.
# The trigger is still firing but the problem cannot be observed: Our triggers might not be perfect, so it could be that something else is causing it to fire. A simple example would be that Zabbix reports that the the DNS for a site can't be resolved, but in reality there's a bug in the script we wrote that checks if the DNS resolves and the DNS resolves fine. Final note: keep in mind that an 'it works on my machine' does not necessarily mean it works for most other people, so depening on the trigger you need to do some evaluations if your tests suffice.

In order to make sure you are actually trying to observe the same thing as the trigger is looking for, make sure to check the trigger definition and the current data of the associated item(s). Some triggers might fire if one of multiple conditions is met (Such as a trigger that monitors the ping response time firing if the value exceeds a certain threshold, or if no data for a certain period of time was observed).

Make sure to report your findings in the incident's thread. It's advised to post a screenshot of the relevant item(s) and your own observations. (Continuing the ping example, you would post a screenshot of the relevant values, state your conclusion why the trigger is firing, and your own observations/pings)

==== Communicate to affected clients ====
If the incident is still ongoing and the service is down, we need to communicate to affected clients that we are aware of the problem and that we are investigating it. This is because critical incident usually mean the service is down, something the clients can notice/are affected by, so we to be transparent that something is going on. There are some additional notes to this though:
* If an incident has already resolved itself and the problem is no longer observable, we don't communicate anything. Doing so might only cause confusion, and since the client has not reported any issues, they have not had a noticeable problem with it themselves.
* Although a critical incident generally means that the client service is down or experiencing reduced service, not all critical incidents are of that nature. Some are more administrative, or are only an issue for Delft Solutions itself. As of writing I don't have an exhaustive list, but here is those I can think of:
** SSH Service is down: We don't have any clients that SSH into their services, so it's generally not a problem. But SSH is mostly used for SRE maintenance and publishing new builds. The SRE maintenance is an internal problem, so no need to communicate to the client. The publishing is done to Kaboom, preventing new builds from being published, and the two SM VM's.
** No backup for x days: Clients don't notice it if a backup is running late, so no need to communicate with clients. Just need to make sure the backup gets completed
** SSL certificate is expiring in < 24 hours: This is a bit dependent on how soon this incident is being handled, but if it handled quickly, the certificate never actually expired, and there has not been any disruption to the client's service, so no need for communicating about it.
* Determining which clients are being affected can be done by looking at the host's DNS in the trigger, and/or looking up the VM in Proxmox and checking the tags of the VM's for client names. In the case that this issue is causing multiple other critical triggers to fire, you would have to check for which clients are affected by those incidents.
* Communicating to DS about ongoing incidents is usually assumed to be automaticly have been done by the fact that the incident was reported on Mattermost.

As always, report the decisions taken and actions maded in the incident thread. (e.g.: I've sent a message in the Slack to let Kaboom know that we aware of problem x, and that we are investigating it)

==== Communicate plan/next steps + Communicate findings/results of executed plan ====
This is the main part of handling an incident. There are several actions you can take in these steps, but at the basis they consist of sharing your next steps, performing those, and reporting the results. The reason all this needs to be reported is to ensure that all known information about a problem is logged, making it easier for someone else to be onboarded into the issue, for later reference if a similar issue is encountered, and even for use during the incident itself in case an older configuration needs to be referenced after you changed it.
The objective from these steps is determining what is actually wrong and how to resolve it. Depending on the observations made earlier on whether the incident is still ongoing and is (still) observable your investigation can go into different directions. (e.g. Find the underlying cause for a trigger, or determining why the trigger is firing while it likely shouldn't, and then how to resolve that underlying cause or how to update the trigger to work better)

There are three main types of steps defined, but you are not limited to these:
# Hypothesis: If you have an idea what could be causing it, you would state your hypothesis and your next step would be to prove that hypothesis. For example, for an incident 'SSH service is down on X' your hypothesis could be that this is due to 'MaxStartups' throttling, which can be proven by 'grep'ing journalctl for that, and compare the start and end times of throttling with the timestamps of the item reporting the status of the SSH service.
# Information gathering: Sometimes it just helps to get some facts about the situation collected. What is usefull information that is relevant depends on the triggers, but some examples are: The syslog/journalctl of the host from around the time of the incident (it can contain a reference to the an underlying problem in various levels of explicitness), the ping response from several hosts on the route to a host or a traceroute (this helps with networking issues). The gathered information is usually intended to help you come up with an hypothesis on what's wrong.
# Investigative: The most rigorous of process. The full process is described here originally [https://docs.google.com/document/d/1AQYJM1Q9l2Tyk6zfCVaQ2aEq-dpbfUH5okE88bpKkhw/edit#heading=h.5fq2skijqbdc Drive - Final Coundown - General Investigative Process]. To summarize, when you don't know why something is failing, and/or don't have any decent hypotheses to follow up, you can follow this process to systematicly find the problem.

Regarding the resolution to an incident: The resolution to any incident is usually one of two things:
# Fix the underlying problem.
# Fix the trigger itself.
Fixing the trigger is relavively straightforward, but do make sure document in the thread what you changed to which trigger.
Fixing the underlying problem can be more complex. A trade-off needs to be made sometimes between resolving technical debt, or simply patching the current system to resolve the issue. We usually look for a resolution that ensures that the problem won't re-occur soon, or makes it unexpected/unlikely for the problem to re-occur. Taking into account the timeframe that is available to resolve the incident you can make some trade-offs. An example would be: normal backups of VM's are failing due to the Proxmox backup server being down/unreachable and it is determined that this cannot be resolved at that moment. We can set up automatic backups to local storage temporary to resolve the immediate problem and ensure we keep our SLO's versus setting up a new Proxmox Backup server at a different location. Since we don't have much time to resolve the problem, the resolution would be to set up the automatic backups to local storage, and set up a new Proxmox Backup Server later as a seperate issue.

Some know issues and their resolutions:
* SSH service is down: The internet is a vile place. There's constant port scanning and hacking attempts ongoing to any machine connected to the internet (mostly IPv4). Due to this, SSH has a throttling functionality build in to prevent a system from being DDOS'ed by the amount of malicious SSH requests. This throttling can cause the Zabbix server from being denied an SSH connection, of which several failures fire this trigger. This hypothesis can be proven with a `journalctl -u ssh | grep 'MaxStartupsThrottling'` (you probably want to select a relevant time period with `--since "2 hours ago"` or something similar to prevent having to process a month of logging). You can then compare the throttling start and end times with the timestamps of the item data itself. The resolution for the issue is to add our custom ssh configuration [https://chat.empiresmod.com/era/pl/oxiu4ark4t8e5paueftr981iyo Custom SSH Configuration].
* No backup for 3 days: Are S3 backup is very slow. Not much to prove as an underlying issue here. What needs to be done is check that the backup process is ongoing. The Zabbix latest data can be checked to verify that backups are running by checking that that days backups were done for the smaller buckets. The devteam email can be checked for if the backup process could not start on day due to it already running (it takes 24+ hours, and an attempt to start it is done each day by cron).
* git.* HTTPS is down: On Sunday, Git gets automaticly updated, but this incurs some downtime. This is usually short enough to not be reported to Mattermost as per our settings, but sometimes it's longer. If the service does not stay down, for more then 20 minutes, the issue can be just resolved.

==== Resolve incident + cleanup ====
When you've executed and verified the resolution in the previous steps we can proceed resolving the issue in our Mattermost integration. Resolving an incident can be done by doing the following:
# Verify that the trigger is no longer firing. An incident will be immediatly re-opened if the trigger is still firing, and the incident cannot be considered resolved if the trigger is still firing. If the trigger is still firing but you're sure that you've resolved the problem, you might need to force the item the trigger depends on to update. This can be done by finding the item in the host's configuration on Zabbix and selecting 'Execute Now', after a short period this should force Zabbix to re-execute the item. You can check the timestamps in the latest data of an item to check if it was updated.
# Type the magic string 'I agree that this has been fully resolved' in the thread. During the next iteration of the Zabbix-Mattermost integration the incident will be closed.

Unfortunatly, some problems cause multiple critical and non-critical triggers to fire. This means we have to check Zabbix and Mattermost for other fired triggers and ongoing incidents. The goal is to identify critical and non-critical incidents that were caused by the incident/underlying issue you just resolved.
# First, these incidents need to be acknowledged on Zabbix, and in the acknowledgement message you mention the incident/problem that caused this.
# Next, check the incident tracked by the integration on Mattermost using the `?ongoing` command. Resolve incidents that were (re-)opened by this incident by executing the following steps. If the first two fail (problem still persists, trigger is still firing), the incident needs to considered it's own issue and the relevant process needs to be followed (critical or non-critical depending on criticality).
## Ensuring the mentioned problem is no longer observable
## The trigger has resolved (You might need to force an update with `Execute Now`).
## Posting a link to the main incident you resolved with the comment that the underlying problem was resolved in that thread.
## Closing the incident with the magic string `I agree that this has been fully resolved`.

When you are done, there should be no more critical triggers firing in Zabbix or open in the Zabbix-Mattermost intergration, for which no-one has taken responsibility or you have taken responsibility for and are not actively handling.

===Additional context===
* Critical incidents are posted in '''Infrastructure'''.
* When it is being tracked on GitLab a heavy check mark is added to the message.
* Responses on the thread and on GitLab are automatically synced (to some extend)
* When you reply with '''I agree that this has been fully resolved''' eventually our Zabbix-Mattermost integration will pick this up and a green check mark is added to the message.

== Non-Critical incidents ==
* Non-critical incidents are acknowledged within 9 hours and resolved within one week.

Checklist
# Acknowledge on Zabbix and state who is responsible for resolving this in the description
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# If there is no resolution to the incident, evaluate if the trigger needs updating/disabling
# Resolve incident

== Informational incidents ==
* Informational incidents are acknowledged within 72 hours

Checklist
# Acknowledge on Zabbix
# Sanity check the event, post result in thread
# If action needed, perform action

== If an incident is reported in the SRE channel by other means than the Zabbix-Mattermost intergration ==
# Acknowledge receipt.
# Classify the incident as critical, non-critical, or informational.
# Create an issue using the `[https://chat.empiresmod.com/era/pl/xt669m4fcprfbciwee9iiq5xte ?track]` command and state that you've done so.
# Proceed to treat the incident according to the criticality you just classified it as. (So for a critical incident, it means you now start the critical incident handling process)

== Handover ==
When handing over the responsibility of first responder (FR), the following needs to happen:
* Acting FR adds the upcoming FR the the IPA sla-first-responder user group (Optional: and enables Zabbix calling for that person if they have that set by going to Zabbix > Configuration > Actions > Trigger actions)
* The upcoming FR makes sure they are aware of the state of the SLA and knows what questions they wants to ask the acting FR.
The following steps can be done async or in person:
* The acting FR announces/informs the upcoming FR has been added to the sla-first-responder group (In Mattermost's organisational channel if asynq).
* If the acting FR wants to hand over responsibility for any ongoing incident they also state which incidents they want the upcoming FR to take over.
* If there are any particularities the upcoming FR needs to be aware of, those are shared.
* The upcoming FR asks their questions until they are satisfied and able to take over the FR
* The upcoming FR announces/informs that they are now the acting FR over Mattermost's organisational channel
* The now acting FR removes the previous FR from IPA the sla-first-responder user group (Optional: and disables Zabbix calling for that person if they had that enabled by going to Zabbix > Configuration > Actions > Trigger actions)

MediaWiki:Common.css

2024-05-20T14:54:33Z

SleeplessByte: Revert

/* CSS placed here will be applied to all skins */

:root {
--shadow-color: 246deg 73% 41%;
--shadow-elevation-low:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
0.4px 0.8px 1px -1.2px hsl(var(--shadow-color) / 0.34),
1px 2px 2.5px -2.5px hsl(var(--shadow-color) / 0.34);
--shadow-elevation-medium:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.36),
0.8px 1.6px 2px -0.8px hsl(var(--shadow-color) / 0.36),
2.1px 4.1px 5.2px -1.7px hsl(var(--shadow-color) / 0.36),
5px 10px 12.6px -2.5px hsl(var(--shadow-color) / 0.36);
--shadow-elevation-high:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
1.5px 2.9px 3.7px -0.4px hsl(var(--shadow-color) / 0.34),
2.7px 5.4px 6.8px -0.7px hsl(var(--shadow-color) / 0.34),
4.5px 8.9px 11.2px -1.1px hsl(var(--shadow-color) / 0.34),
7.1px 14.3px 18px -1.4px hsl(var(--shadow-color) / 0.34),
11.2px 22.3px 28.1px -1.8px hsl(var(--shadow-color) / 0.34),
17px 33.9px 42.7px -2.1px hsl(var(--shadow-color) / 0.34),
25px 50px 62.9px -2.5px hsl(var(--shadow-color) / 0.34);
}

body {
background-image: linear-gradient(245deg, rgba(89,149,255,1) 0%, rgba(104,88,255,1) 100%);
background-color: rgb(104,88,255);
}

html, body {
height: auto;
min-height: 100%;
}

.vectorTabs, #mw-head .vectorMenu h3 {
background-image: none;
}

#mw-page-base {
background: transparent;
}

.mw-body, .parsoid-body {
background-color: white;
}
.mw-body {
border-color: rgb(104,88,255);
border-radius: 0 0 0 14px;
box-shadow: var(--shadow-elevation-low);
}

.vector-menu-tabs-legacy {
padding-left: 0;
margin-left: 1px;

border-radius: 14px 0 0 0;
overflow: hidden;
}

.vectorTabs ul li {
background: #293590;
}

.vectorTabs span {
background-image: none;
border-left: none;
border-right: none;
}

pre, .mw-code {
background-color: #F0F0F2;
}
.vectorTabs li.selected span {
border: none;
}

.vectorTabs li.selected {
background-image: none;
background-color: white;
}

.vectorTabs li a {
color: white;
}

.vectorTabs li a:visited {
color: #F0F0F2;
}

.vectorTabs li.new a, .vectorTabs li.new a:visited {
color: #5E7EFF;
}

#p-logo {
padding-top: 0.5em;
}

#p-logo a {
background-size: contain;
width: 8em;
margin: auto;
}

#mw-panel .portal .body li a {
color: white;
}

#mw-panel .portal .body li a:visited {
color: #E0E1E4;
}

a.new, #p-personal a.new {
color: #F05C6C;
}

a {
color: white;
}

a:visited {
color: #F0F0F2;
}

.mw-parser-output a.external {
color: #D15E54;
}

#content a {
color: #5E7EFF;
text-decoration: underline;
}

#content a:visited {
color: #192369;
}

#content a.new {
color: #D15E54;
}

#footer ul li,
#mw-panel .portal h3,
.vectorMenu h3 span {
color: #F0F0F2;
}

.vector-menu.vector-menu-portal a[href]:link, .vector-menu.vector-menu-portal a[href]:visited {
color: white;
transition: color 150ms ease, opacity 150ms ease;
}

.vector-menu.vector-menu-portal a[href]:hover {
opacity: 0.7;
}

.vector-menu.vector-menu-portal a[href]:active {
opacity: 0.9;
}

@media screen and (max-width: 999px) {
#p-personal,
.vector-user-menu-legacy {
position: absolute;
top: 0;
left: 0;
right: 0;
}

.vector-user-menu-legacy .vector-menu-content {
display: block;
width: 100%;
padding: 8px 12px;
overflow: auto;
box-sizing: border-box;
}

#p-personal ul,
.vector-user-menu-legacy .vector-menu-content-list {
white-space: nowrap;
display: block;
padding-left: 0;
}

.vector-user-menu-legacy .mw-list-item {
float: none;
display: inline-block;
}
}

MediaWiki:Mobile.css

2024-05-20T14:54:14Z

SleeplessByte: Created page with "/* All CSS here will be loaded for users of the mobile site */ #mw-head, #mw-panel { display: none; } #footer, #content { margin-left: 0; }"

/* All CSS here will be loaded for users of the mobile site */

#mw-head, #mw-panel {
display: none;
}

#footer, #content {
margin-left: 0;
}

MediaWiki:Common.css

2024-05-20T14:52:46Z

SleeplessByte: Maybe responsive?

/* CSS placed here will be applied to all skins */

:root {
--shadow-color: 246deg 73% 41%;
--shadow-elevation-low:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
0.4px 0.8px 1px -1.2px hsl(var(--shadow-color) / 0.34),
1px 2px 2.5px -2.5px hsl(var(--shadow-color) / 0.34);
--shadow-elevation-medium:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.36),
0.8px 1.6px 2px -0.8px hsl(var(--shadow-color) / 0.36),
2.1px 4.1px 5.2px -1.7px hsl(var(--shadow-color) / 0.36),
5px 10px 12.6px -2.5px hsl(var(--shadow-color) / 0.36);
--shadow-elevation-high:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
1.5px 2.9px 3.7px -0.4px hsl(var(--shadow-color) / 0.34),
2.7px 5.4px 6.8px -0.7px hsl(var(--shadow-color) / 0.34),
4.5px 8.9px 11.2px -1.1px hsl(var(--shadow-color) / 0.34),
7.1px 14.3px 18px -1.4px hsl(var(--shadow-color) / 0.34),
11.2px 22.3px 28.1px -1.8px hsl(var(--shadow-color) / 0.34),
17px 33.9px 42.7px -2.1px hsl(var(--shadow-color) / 0.34),
25px 50px 62.9px -2.5px hsl(var(--shadow-color) / 0.34);
}

body {
background-image: linear-gradient(245deg, rgba(89,149,255,1) 0%, rgba(104,88,255,1) 100%);
background-color: rgb(104,88,255);
}

html, body {
height: auto;
min-height: 100%;
}

.vectorTabs, #mw-head .vectorMenu h3 {
background-image: none;
}

#mw-page-base {
background: transparent;
}

.mw-body, .parsoid-body {
background-color: white;
}
.mw-body {
border-color: rgb(104,88,255);
border-radius: 0 0 0 14px;
box-shadow: var(--shadow-elevation-low);
}

.vector-menu-tabs-legacy {
padding-left: 0;
margin-left: 1px;

border-radius: 14px 0 0 0;
overflow: hidden;
}

.vectorTabs ul li {
background: #293590;
}

.vectorTabs span {
background-image: none;
border-left: none;
border-right: none;
}

pre, .mw-code {
background-color: #F0F0F2;
}
.vectorTabs li.selected span {
border: none;
}

.vectorTabs li.selected {
background-image: none;
background-color: white;
}

.vectorTabs li a {
color: white;
}

.vectorTabs li a:visited {
color: #F0F0F2;
}

.vectorTabs li.new a, .vectorTabs li.new a:visited {
color: #5E7EFF;
}

#p-logo {
padding-top: 0.5em;
}

#p-logo a {
background-size: contain;
width: 8em;
margin: auto;
}

#mw-panel .portal .body li a {
color: white;
}

#mw-panel .portal .body li a:visited {
color: #E0E1E4;
}

a.new, #p-personal a.new {
color: #F05C6C;
}

a {
color: white;
}

a:visited {
color: #F0F0F2;
}

.mw-parser-output a.external {
color: #D15E54;
}

#content a {
color: #5E7EFF;
text-decoration: underline;
}

#content a:visited {
color: #192369;
}

#content a.new {
color: #D15E54;
}

#footer ul li,
#mw-panel .portal h3,
.vectorMenu h3 span {
color: #F0F0F2;
}

.vector-menu.vector-menu-portal a[href]:link, .vector-menu.vector-menu-portal a[href]:visited {
color: white;
transition: color 150ms ease, opacity 150ms ease;
}

.vector-menu.vector-menu-portal a[href]:hover {
opacity: 0.7;
}

.vector-menu.vector-menu-portal a[href]:active {
opacity: 0.9;
}

@media screen and (max-width: 999px) {

body.skin--responsive {
#mw-head, #mw-panel {
display: none;
}

#footer, #content {
margin-left: 0;
}
}

#p-personal,
.vector-user-menu-legacy {
position: absolute;
top: 0;
left: 0;
right: 0;
}

.vector-user-menu-legacy .vector-menu-content {
display: block;
width: 100%;
padding: 8px 12px;
overflow: auto;
box-sizing: border-box;
}

#p-personal ul,
.vector-user-menu-legacy .vector-menu-content-list {
white-space: nowrap;
display: block;
padding-left: 0;
}

.vector-user-menu-legacy .mw-list-item {
float: none;
display: inline-block;
}
}

MediaWiki:Common.css

2024-05-18T15:52:22Z

SleeplessByte:

MediaWiki:Common.css

2024-05-18T15:50:56Z

SleeplessByte:

/* CSS placed here will be applied to all skins */

:root {
--shadow-color: 246deg 73% 41%;
--shadow-elevation-low:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
0.4px 0.8px 1px -1.2px hsl(var(--shadow-color) / 0.34),
1px 2px 2.5px -2.5px hsl(var(--shadow-color) / 0.34);
--shadow-elevation-medium:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.36),
0.8px 1.6px 2px -0.8px hsl(var(--shadow-color) / 0.36),
2.1px 4.1px 5.2px -1.7px hsl(var(--shadow-color) / 0.36),
5px 10px 12.6px -2.5px hsl(var(--shadow-color) / 0.36);
--shadow-elevation-high:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
1.5px 2.9px 3.7px -0.4px hsl(var(--shadow-color) / 0.34),
2.7px 5.4px 6.8px -0.7px hsl(var(--shadow-color) / 0.34),
4.5px 8.9px 11.2px -1.1px hsl(var(--shadow-color) / 0.34),
7.1px 14.3px 18px -1.4px hsl(var(--shadow-color) / 0.34),
11.2px 22.3px 28.1px -1.8px hsl(var(--shadow-color) / 0.34),
17px 33.9px 42.7px -2.1px hsl(var(--shadow-color) / 0.34),
25px 50px 62.9px -2.5px hsl(var(--shadow-color) / 0.34);
}

body {
background-image: linear-gradient(245deg, rgba(89,149,255,1) 0%, rgba(104,88,255,1) 100%);
background-color: rgb(104,88,255);
}

html, body {
height: auto;
min-height: 100%;
}

.vectorTabs, #mw-head .vectorMenu h3 {
background-image: none;
}

#mw-page-base {
background: transparent;
}

.mw-body, .parsoid-body {
background-color: white;
}
.mw-body {
border-color: rgb(104,88,255);
border-radius: 0 0 0 14px;
box-shadow: var(--shadow-elevation-low);
}

.vector-menu-tabs-legacy {
padding-left: 0;
margin-left: 1px;

border-radius: 14px 0 0 0;
overflow: hidden;
}

.vectorTabs ul li {
background: #293590;
}

.vectorTabs span {
background-image: none;
border-left: none;
border-right: none;
}

pre, .mw-code {
background-color: #F0F0F2;
}
.vectorTabs li.selected span {
border: none;
}

.vectorTabs li.selected {
background-image: none;
background-color: white;
}

.vectorTabs li a {
color: white;
}

.vectorTabs li a:visited {
color: #F0F0F2;
}

.vectorTabs li.new a, .vectorTabs li.new a:visited {
color: #5E7EFF;
}

#p-logo {
padding-top: 0.5em;
}

#p-logo a {
background-size: contain;
width: 8em;
margin: auto;
}

#mw-panel .portal .body li a {
color: white;
}

#mw-panel .portal .body li a:visited {
color: #E0E1E4;
}

a.new, #p-personal a.new {
color: #F05C6C;
}

a {
color: white;
}

a:visited {
color: #F0F0F2;
}

.mw-parser-output a.external {
color: #D15E54;
}

#content a {
color: #5E7EFF;
text-decoration: underline;
}

#content a:visited {
color: #192369;
}

#content a.new {
color: #D15E54;
}

#footer ul li,
#mw-panel .portal h3,
.vectorMenu h3 span {
color: #F0F0F2;
}

.vector-menu.vector-menu-portal a[href]:link, .vector-menu.vector-menu-portal a[href]:visited {
color: white;
transition: color 150ms ease, opacity 150ms ease;
}

.vector-menu.vector-menu-portal a[href]:hover {
opacity: 0.7;
}

.vector-menu.vector-menu-portal a[href]:active {
opacity: 0.9;
}

@media screen and (max-width: 999px) {
.vector-user-menu-legacy {
position: absolute;
top: 0;
left: 0;
right: 0;
}

.vector-user-menu-legacy .vector-menu-content {
display: block;
width: 100%;
padding: 8px 12px;
overflow: auto;
box-sizing: border-box;
}

.vector-user-menu-legacy .vector-menu-content-list {
whitespace: nowrap;
display: block;
}

.vector-user-menu-legacy .mw-list-item {
float: none;
display: inline-block;
}
}

MediaWiki:Common.css

2024-05-18T15:44:52Z

SleeplessByte: Better responsive top-menu

/* CSS placed here will be applied to all skins */

:root {
--shadow-color: 246deg 73% 41%;
--shadow-elevation-low:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
0.4px 0.8px 1px -1.2px hsl(var(--shadow-color) / 0.34),
1px 2px 2.5px -2.5px hsl(var(--shadow-color) / 0.34);
--shadow-elevation-medium:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.36),
0.8px 1.6px 2px -0.8px hsl(var(--shadow-color) / 0.36),
2.1px 4.1px 5.2px -1.7px hsl(var(--shadow-color) / 0.36),
5px 10px 12.6px -2.5px hsl(var(--shadow-color) / 0.36);
--shadow-elevation-high:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
1.5px 2.9px 3.7px -0.4px hsl(var(--shadow-color) / 0.34),
2.7px 5.4px 6.8px -0.7px hsl(var(--shadow-color) / 0.34),
4.5px 8.9px 11.2px -1.1px hsl(var(--shadow-color) / 0.34),
7.1px 14.3px 18px -1.4px hsl(var(--shadow-color) / 0.34),
11.2px 22.3px 28.1px -1.8px hsl(var(--shadow-color) / 0.34),
17px 33.9px 42.7px -2.1px hsl(var(--shadow-color) / 0.34),
25px 50px 62.9px -2.5px hsl(var(--shadow-color) / 0.34);
}

body {
background-image: linear-gradient(245deg, rgba(89,149,255,1) 0%, rgba(104,88,255,1) 100%);
background-color: rgb(104,88,255);
}

html, body {
height: auto;
min-height: 100%;
}

.vectorTabs, #mw-head .vectorMenu h3 {
background-image: none;
}

#mw-page-base {
background: transparent;
}

.mw-body, .parsoid-body {
background-color: white;
}
.mw-body {
border-color: rgb(104,88,255);
border-radius: 0 0 0 14px;
box-shadow: var(--shadow-elevation-low);
}

.vector-menu-tabs-legacy {
padding-left: 0;
margin-left: 1px;

border-radius: 14px 0 0 0;
overflow: hidden;
}

.vectorTabs ul li {
background: #293590;
}

.vectorTabs span {
background-image: none;
border-left: none;
border-right: none;
}

pre, .mw-code {
background-color: #F0F0F2;
}
.vectorTabs li.selected span {
border: none;
}

.vectorTabs li.selected {
background-image: none;
background-color: white;
}

.vectorTabs li a {
color: white;
}

.vectorTabs li a:visited {
color: #F0F0F2;
}

.vectorTabs li.new a, .vectorTabs li.new a:visited {
color: #5E7EFF;
}

#p-logo {
padding-top: 0.5em;
}

#p-logo a {
background-size: contain;
width: 8em;
margin: auto;
}

#mw-panel .portal .body li a {
color: white;
}

#mw-panel .portal .body li a:visited {
color: #E0E1E4;
}

a.new, #p-personal a.new {
color: #F05C6C;
}

a {
color: white;
}

a:visited {
color: #F0F0F2;
}

.mw-parser-output a.external {
color: #D15E54;
}

#content a {
color: #5E7EFF;
text-decoration: underline;
}

#content a:visited {
color: #192369;
}

#content a.new {
color: #D15E54;
}

#footer ul li,
#mw-panel .portal h3,
.vectorMenu h3 span {
color: #F0F0F2;
}

.vector-menu.vector-menu-portal a[href]:link, .vector-menu.vector-menu-portal a[href]:visited {
color: white;
transition: color 150ms ease, opacity 150ms ease;
}

.vector-menu.vector-menu-portal a[href]:hover {
opacity: 0.7;
}

.vector-menu.vector-menu-portal a[href]:active {
opacity: 0.9;
}

@media screen and (max-width: 999px) {
.vector-user-menu-legacy .vector-menu-content {
display: block;
width: 100vw;
padding: 0 8px 12px;
overflow: auto;
box-sizing: border-box;
}

.vector-user-menu-legacy .vector-menu-content .mw-list-item {
float: none;
}
}

Training and self-study

2024-05-18T15:28:58Z

SleeplessByte: Created page with "At Delft Solutions we welcome training and self-study, if you think it will benefit you in your role or the company as a whole. It is often possible to: * attend conferences related to your work * purchase books (physical and epubs) * follow courses There is no strict yearly budget, but please ask management. If a case can be made that it's helpful, it is generally approved."

At Delft Solutions we welcome training and self-study, if you think it will benefit you in your role or the company as a whole.

It is often possible to:

* attend conferences related to your work
* purchase books (physical and epubs)
* follow courses

There is no strict yearly budget, but please ask management. If a case can be made that it's helpful, it is generally approved.

Sick leave

2024-05-18T15:26:46Z

SleeplessByte: Created page with "Delft Solutions follows [https://business.gov.nl/running-your-business/staff/terms-of-employment/sick-leave-your-employee-calls-in-sick/ the Dutch law regarding sick leave]. == Process == * If you get ill, you must report this as soon as possible, at least before the start of the working day, or as it happens during the day. * If you get ill, and you scheduled your week beforehand, you can log those hours as sick leave. * If you get ill, and you work adhoc / have not..."

Delft Solutions follows [https://business.gov.nl/running-your-business/staff/terms-of-employment/sick-leave-your-employee-calls-in-sick/ the Dutch law regarding sick leave].

== Process ==

* If you get ill, you must report this as soon as possible, at least before the start of the working day, or as it happens during the day.
* If you get ill, and you scheduled your week beforehand, you can log those hours as sick leave.
* If you get ill, and you work adhoc / have not scheduled your week beforehand, the first day is ignored. This is the default for those who choose to work irregularly.
* If you get ill during paid holiday leave, please report this so your holidays can be reimbursed.
* You are free to share your reason for illness, but you do not need to. You will never be asked by management.
* Instead, after a period of time, a company doctor will make an inquiry and advice both you and management what the next steps are.
* If possible, hand-over any work both when falling ill and when returning to work. However, if circumstances make this hard or impossible, do not strain yourself further trying to hand-over work.

At Delft Solutions various people are in regular contact with immunocompromised people. For this reason, and in order to prevent spreading illness, you will work from home if you feel or expect to be or become symptomatic of any illness, unless you are certain it cannot be spread (like hay fever). Please talk to management to ensure you have a workable environment at home, as they can help you (including financially) to set this up if it's lacking.

Bonus allocation

2024-05-18T15:15:52Z

SleeplessByte: Created page with "After each quarter has ended there is a company wide Quarterly update meeting in which we talk about the past and upcoming quarter. This is generally scheduled 1 to 2 weeks after the quarter's end. Divergence from the bonus calculation is shared during this meeting. == Process == * Under regular circumstances in order to calculate the bonus we take the quarterly throughput and deduct the bonus threshold. * The remainder is capped at the maximum bonus throughput and mu..."

After each quarter has ended there is a company wide Quarterly update meeting in which we talk about the past and upcoming quarter. This is generally scheduled 1 to 2 weeks after the quarter's end. Divergence from the bonus calculation is shared during this meeting.

== Process ==

* Under regular circumstances in order to calculate the bonus we take the quarterly throughput and deduct the bonus threshold.
* The remainder is capped at the maximum bonus throughput and multiplied against the bonus amount.
* If this yields a nonzero amount, the bonus is awarded to each team member that has logged at least one working hour, prorated to the average number of hours a week worked (which should be 32 by default for everyone).
* The bonus amount is communicated to the accountants and appears on the next wage slip.
* After the bonus is calculated, the amount in liquid funds is taken and checked against the desired working capital.
* Any funds after deducting the desired working capital and quarterly bonus costs is the quarterly dividend to be shared amongst the shareholders.

Internal

2024-05-16T13:09:31Z

SleeplessByte: /* Process */

== Finance ==

=== Exact ===

* [[booking bonus|Booking bonus]]
* [[booking wages|Booking wages]]
* [[new receipt|Enter a new receipt]]
* [[reconciliation|Reconciliation of transaction]]
* [[invoicing|Send an invoice]]
* [[payment reminders|Send payment reminder]]
* [[invoice approval|Process for approving invoices (/filed receipts)]]

=== Bunq ===

* [[top up account|Top up expense account]]

== Work Process ==

* [[Definition of done|Definition of Done]]
* [[Incident Handling|Incident Handling]]

== Internal Process ==

* [[12 percent|12% time]]
* [[Annual leave|Annual leave]]
* [[Bonus allocation|Bonus allocation]]
* [[Calamity leave|Calamity leave]]
* [[Overtime|Overtime]]
* [[Retrospectives|Retrospectives]]
* [[Sick leave|Sick leave]]
* [[Training and self-study|Training and Self-Study]]

== Projects ==

* Era Inventory [[project_era_inventory_api|API Description]]

== SRE ==

''To be further populated with guide from drive''
* [[create gitlab runner host|Create a GitLab runner host]]
* [[vm setup|Create a (Debian) VM]]
* [[border reboot|Reboot border without downtime]]
* [[WS Proxmox node reboot|Reboot WS Proxmox node without downtime]]
* [[Resize VM Disk]]

== Other ==

* [[standard tools|Standard Tools]]
* [[list of unfurl debuggers|List of unfurl debuggers]]

Retrospectives

2024-05-16T13:09:21Z

SleeplessByte: Created page with "At Delft Solutions most teams will hold a retrospective each week to see where process can be improved. == Process == During the week, issues with the current process (for example a process not being followed by someone, or a process not taking care of a specific incident) will be listed in the [https://chat.empiresmod.com/era/channels/retro ~Retro channel]. Once a week, generally on Friday, these points are discussed in a timeboxed meeting. There is ''always'' an outc..."

At Delft Solutions most teams will hold a retrospective each week to see where process can be improved.

== Process ==

During the week, issues with the current process (for example a process not being followed by someone, or a process not taking care of a specific incident) will be listed in the [https://chat.empiresmod.com/era/channels/retro ~Retro channel]. Once a week, generally on Friday, these points are discussed in a timeboxed meeting. There is ''always'' an outcome to each item, where everyone in the team agrees that is how we wil move forward. That means that we cannot just do nothing, but we can ''decide'' to not change anything (and have it happen again next time). In most cases the resolution will be to amend the process at hand.

This also means that you cannot diverge from a current process ''during'' the week, but only after discussing it at the Retrospective.

Overtime

2024-05-16T13:06:19Z

SleeplessByte: /* Process */

At Delft Solutions we don't like overtime because everyone is entitled to off-time such as evenings or weekends. Structural / recurring overtime is not allowed without a written request and approval and will only be granted under specific circumstances.

== Process ==

* If you accidentally work longer than expected (say an hour), you can self-manage it the next week by working an hour shorter. Hours registered going over your weekly amount (of 32 hours), will automatically be added to your available holiday/vacation hours.
* If you intend to work a significant amount of time on top of your regular hours, for example because you want to meet a deadline, you must request this by informing management. Under normal circumstances requests like these will not be granted. Instead, management will often help you communicate to clients that delays are to be expected, or find a way to make the deadline without overtime. In the case the overtime is granted, because of the above, your hours are reimbursed at a 1:1 rate automatically.

Overtime

2024-05-16T13:05:02Z

SleeplessByte: Created page with "At Delft Solutions we don't like overtime because everyone is entitled to off-time such as evenings or weekends. Structural / recurring overtime is not allowed without a written request and approval and will only be granted under specific circumstances. == Process == * If you accidentally work longer than expected (say an hour), you can self-manage it the next week by working an hour shorter. Hours registered going over your weekly amount (of 32 hours), will automatica..."

Calamity leave

2024-05-16T13:01:05Z

SleeplessByte: Created page with "We followed the official rules and guidance by the Dutch government regarding short-term and long-term care leave. '''Short-term care leave is always granted.''' == Resources == * [https://business.gov.nl/regulation/leave-schemes/ business.gov.nl] * [https://www.rijksoverheid.nl/onderwerpen/zorgverlof/vraag-en-antwoord/duur-zorgverlof rijksoverheid.nl (Dutch)]"

We followed the official rules and guidance by the Dutch government regarding short-term and long-term care leave.

'''Short-term care leave is always granted.'''

== Resources ==

* [https://business.gov.nl/regulation/leave-schemes/ business.gov.nl]
* [https://www.rijksoverheid.nl/onderwerpen/zorgverlof/vraag-en-antwoord/duur-zorgverlof rijksoverheid.nl (Dutch)]

Annual leave

2024-05-16T12:57:36Z

SleeplessByte: Created page with "Everyone is entitled to annual leave and Delft Solutions encourages everyone to take their vacation days. == Process == In general everything will be approved, as long as we can ensure SLOs. Please request your annual leave via the [https://chat.empiresmod.com/era/channels/company ~Organisational channel]. In case of a conflict meeting the SLOs, if no one is willing to give up the overlap, the first one who made the request following this process will be able to take t..."

Everyone is entitled to annual leave and Delft Solutions encourages everyone to take their vacation days.

== Process ==

In general everything will be approved, as long as we can ensure SLOs. Please request your annual leave via the [https://chat.empiresmod.com/era/channels/company ~Organisational channel]. In case of a conflict meeting the SLOs, if no one is willing to give up the overlap, the first one who made the request following this process will be able to take their annual leave that moment.

* When your annual leave is coming up, please ensure a handover of all outstanding tasks.
* When someone is on annual leave, please ensure you try everything else before contacting them. This time is supposed to be sacred. If we treat it as such for everyone, we also know that getting a call during annual leave can be considered a last effort.
* When you are on annual leave and you get a call, '''you are entitled to ignore it''', and emergency work done during annual leave is reimbursed as holiday not taken.
* When you are on annual leave and you fall ill, please communicate this to management so we can reimburse the sick days.

== Management ==

Max and Derk-Jan will not be taken leave at the same time.

12 percent

2024-05-16T12:52:11Z

SleeplessByte:

At Delft Solutions 12% of your paid working hours can be spend on anything you like, preferably useful to either your role or the company. This can include learning more, or developing software tools for in house use, or writing blog posts for the company website.

''It's "something you think is useful, but doesn't go on the backlog".''

If you feel like it, during [[Retrospectives|the retrospective]] you can demo what you've been working on.

12 percent

2024-05-16T12:51:49Z

SleeplessByte: Created page with "At Delft Solutions 12% of your paid working hours can be spend on anything you like, preferably useful to either your role or the company. This can include learning more, or developing software tools for in house use, or writing blog posts for the company website. ''It's "something you think is useful, but doesn't go on the backlog".'' If you feel like it, during Retrospectives you can demo what you've been working on."

At Delft Solutions 12% of your paid working hours can be spend on anything you like, preferably useful to either your role or the company. This can include learning more, or developing software tools for in house use, or writing blog posts for the company website.

''It's "something you think is useful, but doesn't go on the backlog".''

If you feel like it, during [[the retrospective|Retrospectives]] you can demo what you've been working on.

Internal

2024-05-16T12:38:41Z

SleeplessByte: /* Process */

== Finance ==

=== Exact ===

* [[booking bonus|Booking bonus]]
* [[booking wages|Booking wages]]
* [[new receipt|Enter a new receipt]]
* [[reconciliation|Reconciliation of transaction]]
* [[invoicing|Send an invoice]]
* [[payment reminders|Send payment reminder]]
* [[invoice approval|Process for approving invoices (/filed receipts)]]

=== Bunq ===

* [[top up account|Top up expense account]]

== Process ==

* [[definition of done|Definition of Done]]
* [[Incident Handling|Incident Handling]]

== Projects ==

* Era Inventory [[project_era_inventory_api|API Description]]

== SRE ==

''To be further populated with guide from drive''
* [[create gitlab runner host|Create a GitLab runner host]]
* [[vm setup|Create a (Debian) VM]]
* [[border reboot|Reboot border without downtime]]
* [[WS Proxmox node reboot|Reboot WS Proxmox node without downtime]]
* [[Resize VM Disk]]

== Other ==

* [[standard tools|Standard Tools]]
* [[list of unfurl debuggers|List of unfurl debuggers]]

Incident Handling

2024-05-07T15:17:49Z

SleeplessByte: Visual status changes

== Critical incidents ==
* Critical incidents are resolved within 16 hours.

===Checklist===
# Acknowledge on Zabbix and state who is responsible for resolving this in the description
# Determine affected clients
# Communicate to affected clients that the issue is being investigated
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# Resolve incident

===Additional context===
* Critical incidents are posted in '''Infrastructure'''.
* When it is being tracked on GitLab a heavy check mark is added to the message.
* Responses on the thread and on GitLab are automatically synced (to some extend)
* When you reply with '''I agree that this has been fully resolved''' eventually Zabbix will pick this up and a green check mark is added to the message.

== Non-Critical incidents ==
* Non-critical incidents are acknowledged within 9 hours and resolved within one week.

Checklist
# Acknowledge on Zabbix and state who is responsible for resolving this in the description
# Communicate plan/next steps (even if that is gathering information)
# Communicate findings/results of executed plan, go back to previous step if not resolved
# If there is no resolution to the incident, evaluate if the trigger needs updating/disabling
# Resolve incident

== Informational incidents ==
* Informational incidents are acknowledged within 72 hours

Checklist
# Acknowledge on Zabbix
# Sanity check the event, post result in thread
# If action needed, perform action

== If an incident is reported in the SRE channel by a human ==
* Acknowledge receipt.
* Classify the incident as critical, non-critical, or informational.
* Create an issue and state that you've done so.

== Handover ==
When handing over the responsibility of first responder (FR), the following needs to happen:
* Acting FR adds the upcoming FR the the IPA sla-first-responder user group, and enables Zabbix calling for that person
* The upcoming FR makes sure he is aware of the state of the SLA and knows what questions he wants to ask the acting FR
The following steps can be done async or in person:
* The acting FR announces/informs the upcoming FR has been added to the sla-first-responder group (In Mattermost's organisational channel if asynq).
* If the acting FR wants to hand over responsibility for any ongoing incident he also states which incidents he want the upcoming FR to take over.
* If there are any particularities the upcoming FR needs to be aware of, he shares them then.
* The upcoming FR asks his questions until he is satisfied and able to take over the FR
* The upcoming FR announces/informs that he is now the acting FR over Mattermost's organisational channel
* The now acting FR removes the previous FR from IPA the sla-first-responder user group, and disables Zabbix calling for that person

Error/OpusSafety::NotFoundError

2024-03-09T12:25:09Z

SleeplessByte: Created page with "You encountered this error on [https://cloud.opus-safety.co.uk/ Opus Safety Cloud]. * If this happened after clicking a link on the Opus Safety Cloud platform, please report it [https://sites.google.com/opus-safety.co.uk/opus-help/home via the support center]. * If this happened after opening a bookmark, or following a link on a different platform or email, please update the bookmark or let the sender know."

You encountered this error on [https://cloud.opus-safety.co.uk/ Opus Safety Cloud].

* If this happened after clicking a link on the Opus Safety Cloud platform, please report it [https://sites.google.com/opus-safety.co.uk/opus-help/home via the support center].
* If this happened after opening a bookmark, or following a link on a different platform or email, please update the bookmark or let the sender know.

List of unfurl debuggers

2024-03-07T20:38:50Z

SleeplessByte: Created page with "* Facebook: [https://developers.facebook.com/docs/sharing/webmasters#markup debugger] [https://developers.facebook.com/docs/sharing/webmasters#markup docs] * Slack: [https://api.slack.com/tools/unfurl-debugger debugger] [https://api.slack.com/reference/messaging/link-unfurling docs]"

* Facebook: [https://developers.facebook.com/docs/sharing/webmasters#markup debugger] [https://developers.facebook.com/docs/sharing/webmasters#markup docs]
* Slack: [https://api.slack.com/tools/unfurl-debugger debugger] [https://api.slack.com/reference/messaging/link-unfurling docs]

Internal

2024-03-07T20:35:03Z

SleeplessByte: /* Other */

Standard tools

2024-02-16T11:44:58Z

SleeplessByte: Created page with "== Images == * One-off optimisations of PNG, JPEG via [https://squoosh.app/ squoosh.app] * One-off optimisations of SVG via [https://jakearchibald.github.io/svgomg/ SVGOMG] * Optimise PNGs via [https://packages.debian.org/bookworm/optipng optipng] e.g. ''optipng -o7 -preserve'' * Optimise JPEGs via [https://packages.debian.org/bookworm/jpegoptim jpegoptim] e.g. ''jpegoptim --strip-all -m76'' * Process images via [https://packages.debian.org/bookworm/libvips-dev libvips]..."

== Images ==

* One-off optimisations of PNG, JPEG via [https://squoosh.app/ squoosh.app]
* One-off optimisations of SVG via [https://jakearchibald.github.io/svgomg/ SVGOMG]
* Optimise PNGs via [https://packages.debian.org/bookworm/optipng optipng] e.g. ''optipng -o7 -preserve''
* Optimise JPEGs via [https://packages.debian.org/bookworm/jpegoptim jpegoptim] e.g. ''jpegoptim --strip-all -m76''
* Process images via [https://packages.debian.org/bookworm/libvips-dev libvips]

== Audio ==

* Processing, encoding, or recoding via [https://packages.debian.org/bookworm/ffmpeg ffmpeg]

== Video ==

* Processing, encoding, or recoding via [https://packages.debian.org/bookworm/ffmpeg ffmpeg]

== Formatting ==

* JavaScript, TypeScript, Markdown, YAML, and CSS can be automatically formatted using [https://prettier.io/ prettier]

Internal

2024-02-16T11:10:57Z

SleeplessByte:

== Finance ==

=== Exact ===

* [[booking wages|Booking wages]]
* [[new receipt|Enter a new receipt]]
* [[reconciliation|Reconciliation of transaction]]
* [[invoicing|Send an invoice]]
* [[payment reminders|Send payment reminder]]

=== Bunq ===

* [[top up account|Top up expense account]]

== Process ==

* [[definition of done|Definition of Done]]

== Projects ==

* Era Inventory [[project_era_inventory_api|API Description]]

== SRE ==

''To be further populated with guide from drive''
* [[create gitlab runner host|Create a GitLab runner host]]
* [[vm setup|Create a (Debian) VM]]
* [[border reboot|Reboot border without downtime]]

== Other ==

* [[standard tools|Standard Tools]]

Category:Terminology

2023-07-11T11:33:10Z

SleeplessByte: Created blank page

MediaWiki:Common.css

2023-07-11T11:30:29Z

SleeplessByte:

MediaWiki:Common.css

2023-07-11T11:24:00Z

SleeplessByte:

/* CSS placed here will be applied to all skins */

:root {
--shadow-color: 246deg 73% 41%;
--shadow-elevation-low:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
0.4px 0.8px 1px -1.2px hsl(var(--shadow-color) / 0.34),
1px 2px 2.5px -2.5px hsl(var(--shadow-color) / 0.34);
--shadow-elevation-medium:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.36),
0.8px 1.6px 2px -0.8px hsl(var(--shadow-color) / 0.36),
2.1px 4.1px 5.2px -1.7px hsl(var(--shadow-color) / 0.36),
5px 10px 12.6px -2.5px hsl(var(--shadow-color) / 0.36);
--shadow-elevation-high:
0.3px 0.5px 0.7px hsl(var(--shadow-color) / 0.34),
1.5px 2.9px 3.7px -0.4px hsl(var(--shadow-color) / 0.34),
2.7px 5.4px 6.8px -0.7px hsl(var(--shadow-color) / 0.34),
4.5px 8.9px 11.2px -1.1px hsl(var(--shadow-color) / 0.34),
7.1px 14.3px 18px -1.4px hsl(var(--shadow-color) / 0.34),
11.2px 22.3px 28.1px -1.8px hsl(var(--shadow-color) / 0.34),
17px 33.9px 42.7px -2.1px hsl(var(--shadow-color) / 0.34),
25px 50px 62.9px -2.5px hsl(var(--shadow-color) / 0.34);
}

body {
background-image: linear-gradient(245deg, rgba(89,149,255,1) 0%, rgba(104,88,255,1) 100%);
background-color: rgb(104,88,255);
}

html, body {
height: auto;
min-height: 100%;
}

.vectorTabs, #mw-head .vectorMenu h3 {
background-image: none;
}

#mw-page-base {
background: transparent;
}

.mw-body, .parsoid-body {
background-color: white;
}
.mw-body {
border-color: rgb(104,88,255);
}

.vectorTabs ul li {
background: #293590;
}

.vectorTabs span {
background-image: none;
border-left: none;
border-right: none;
}

pre, .mw-code {
background-color: #F0F0F2;
}
.vectorTabs li.selected span {
border: none;
}

.vectorTabs li.selected {
background-image: none;
background-color: white;
}

.vectorTabs li a {
color: white;
}

.vectorTabs li a:visited {
color: #F0F0F2;
}

.vectorTabs li.new a, .vectorTabs li.new a:visited {
color: #5E7EFF;
}

#p-logo {
padding-top: 0.5em;
}

#mw-panel .portal .body li a {
color: white;
}

#mw-panel .portal .body li a:visited {
color: #E0E1E4;
}

a.new, #p-personal a.new {
color: #F05C6C;
}

a {
color: white;
}

a:visited {
color: #F0F0F2;
}

.mw-parser-output a.external {
color: #D15E54;
}

#content a {
color: #5E7EFF;
text-decoration: underline;
}

#content a:visited {
color: #192369;
}

#content a.new {
color: #D15E54;
}

#footer ul li,
#mw-panel .portal h3,
.vectorMenu h3 span {
color: #F0F0F2;
}

.vector-menu.vector-menu-portal a[href]:link, .vector-menu.vector-menu-portal a[href]:visited {
color: white;
transition: color 150ms ease, opacity 150ms ease;
}

.vector-menu.vector-menu-portal a[href]:hover {
opacity: 0.7;
}

.vector-menu.vector-menu-portal a[href]:active {
opacity: 0.9;
}

MediaWiki:Common.css

2023-07-11T11:21:35Z

SleeplessByte:

/* CSS placed here will be applied to all skins */
body {
background-image: linear-gradient(245deg, rgba(89,149,255,1) 0%, rgba(104,88,255,1) 100%);
background-color: rgb(104,88,255);
}

html, body {
height: auto;
min-height: 100%;
}

.vectorTabs, #mw-head .vectorMenu h3 {
background-image: none;
}

#mw-page-base {
background: transparent;
}

.mw-body, .parsoid-body {
background-color: white;
}
.mw-body {
border-color: rgb(104,88,255);
}

.vectorTabs ul li {
background: #293590;
}

.vectorTabs span {
background-image: none;
border-left: none;
border-right: none;
}

pre, .mw-code {
background-color: #F0F0F2;
}
.vectorTabs li.selected span {
border: none;
}

.vectorTabs li.selected {
background-image: none;
background-color: white;
}

.vectorTabs li a {
color: white;
}

.vectorTabs li a:visited {
color: #F0F0F2;
}

.vectorTabs li.new a, .vectorTabs li.new a:visited {
color: #5E7EFF;
}

#p-logo {
padding-top: 0.5em;
}

#mw-panel .portal .body li a {
color: white;
}

#mw-panel .portal .body li a:visited {
color: #E0E1E4;
}

a.new, #p-personal a.new {
color: #F05C6C;
}

a {
color: white;
}

a:visited {
color: #F0F0F2;
}

.mw-parser-output a.external {
color: #D15E54;
}

#content a {
color: #5E7EFF;
text-decoration: underline;
}

#content a:visited {
color: #192369;
}

#content a.new {
color: #D15E54;
}

#footer ul li,
#mw-panel .portal h3,
.vectorMenu h3 span {
color: #F0F0F2;
}

.vector-menu.vector-menu-portal a[href]:link, .vector-menu.vector-menu-portal a[href]:visited {
color: white;
transition: color 150ms ease, opacity 150ms ease;
}

.vector-menu.vector-menu-portal a[href]:hover {
opacity: 0.7;
}

.vector-menu.vector-menu-portal a[href]:active {
opacity: 0.9;
}

MediaWiki:Common.css

2023-07-11T11:20:30Z

SleeplessByte:

/* CSS placed here will be applied to all skins */
body {
background-image: linear-gradient(245deg, rgba(89,149,255,1) 0%, rgba(104,88,255,1) 100%);
background-color: rgb(104,88,255);
}

html, body {
height: auto;
min-height: 100%;
}

.vectorTabs, #mw-head .vectorMenu h3 {
background-image: none;
}

#mw-page-base {
background: transparent;
}

.mw-body, .parsoid-body {
background-color: white;
}
.mw-body {
border-color: rgb(104,88,255);
}

.vectorTabs ul li {
background: #293590;
}

.vectorTabs span {
background-image: none;
border-left: none;
border-right: none;
}

pre, .mw-code {
background-color: #F0F0F2;
}
.vectorTabs li.selected span {
border: none;
}

.vectorTabs li.selected {
background-image: none;
background-color: white;
}

.vectorTabs li a {
color: white;
}

.vectorTabs li a:visited {
color: #F0F0F2;
}

.vectorTabs li.new a, .vectorTabs li.new a:visited {
color: #5E7EFF;
}

#p-logo {
padding-top: 0.5em;
}

#mw-panel .portal .body li a {
color: white;
}

#mw-panel .portal .body li a:visited {
color: #E0E1E4;
}

a.new, #p-personal a.new {
color: #F05C6C;
}

a {
color: white;
}

a:visited {
color: #F0F0F2;
}

.mw-parser-output a.external {
color: #D15E54;
}

#content a {
color: #5E7EFF;
text-decoration: underline;
}

#content a:visited {
color: #192369;
}

#content a.new {
color: #D15E54;
}

#footer ul li,
#mw-panel .portal h3,
.vectorMenu h3 span {
color: #F0F0F2;
}

.vector-menu a[href]:link, .vector-menu a[href]:visited {
color: white;
transition: color 150ms ease, opacity 150ms ease;
}

.vector-menu a[href]:hover {
opacity: 0.7;
}

.vector-menu a[href]:active {
opacity: 0.9;
}

MediaWiki:Common.css

2023-07-11T11:15:51Z

SleeplessByte: Fix CSS menu, maybe

/* CSS placed here will be applied to all skins */
body {
background-image: linear-gradient(245deg, rgba(89,149,255,1) 0%, rgba(104,88,255,1) 100%);
background-color: rgb(104,88,255);
}

.vectorTabs, #mw-head .vectorMenu h3 {
background-image: none;
}

#mw-page-base {
background: transparent;
}

.mw-body, .parsoid-body {
background-color: white;
}
.mw-body {
border-color: rgb(104,88,255);
}

.vectorTabs ul li {
background: #293590;
}

.vectorTabs span {
background-image: none;
border-left: none;
border-right: none;
}

pre, .mw-code {
background-color: #F0F0F2;
}
.vectorTabs li.selected span {
border: none;
}

.vectorTabs li.selected {
background-image: none;
background-color: white;
}

.vectorTabs li a {
color: white;
}

.vectorTabs li a:visited {
color: #F0F0F2;
}

.vectorTabs li.new a, .vectorTabs li.new a:visited {
color: #5E7EFF;
}

#p-logo {
padding-top: 0.5em;
}

#mw-panel .portal .body li a {
color: white;
}

#mw-panel .portal .body li a:visited {
color: #E0E1E4;
}

a.new, #p-personal a.new {
color: #F05C6C;
}

a {
color: white;
}

a:visited {
color: #F0F0F2;
}

.mw-parser-output a.external {
color: #D15E54;
}

#content a {
color: #5E7EFF;
text-decoration: underline;
}

#content a:visited {
color: #192369;
}

#content a.new {
color: #D15E54;
}

#footer ul li,
#mw-panel .portal h3,
.vectorMenu h3 span {
color: #F0F0F2;
}

#p-navigation a[href]:link, #p-navigation a[href]:visited {
color: white;
transition: color 150ms ease, opacity 150ms ease;
}

#p-navigation a[href]:hover {
opacity: 0.7;
}

#p-navigation a[href]:active {
opacity: 0.9;
}

User:SleeplessByte

2023-07-10T21:48:29Z

SleeplessByte: Created page with "Derk-Jan Karrenbeld == External links == * [https://derk-jan.com Bits and Pieces (Personal Blog)] * [https://github.com/SleeplessByte SleeplessByte on GitHub] * [https://gitlab.com/SleeplessByte SleeplessByte on GitLab] * [https://twitter.com/SleeplessByte @SleeplessByte on Twitter] * [https://exercism.org/profiles/SleeplessByte SleeplessByte on Exercism] * [https://steamcommunity.com/id/derk-jan/ Looking For Amy on Steam] * [https://www..."

[[File:Profile-1920.jpg|thumb|Derk-Jan Karrenbeld]]

== External links ==

* [https://derk-jan.com Bits and Pieces (Personal Blog)]
* [https://github.com/SleeplessByte SleeplessByte on GitHub]
* [https://gitlab.com/SleeplessByte SleeplessByte on GitLab]
* [https://twitter.com/SleeplessByte @SleeplessByte on Twitter]
* [https://exercism.org/profiles/SleeplessByte SleeplessByte on Exercism]
* [https://steamcommunity.com/id/derk-jan/ Looking For Amy on Steam]
* [https://www.linkedin.com/in/derkjankarrenbeld/ Derk-Jan Karrenbeld on LinkedIn]

File:Profile-1920.jpg

2023-07-10T21:48:09Z

SleeplessByte:

Derk-Jan Karrenbeld

Booking wages

2023-07-10T20:55:06Z

SleeplessByte: /* Reconciliation (after payment) */

== Wage slips ==

# Exact
# Financieel > Boekingen en grootboekrekeningen > Boekingen > Aanmaken
# Inkoop > '''80 - Salarisstroken'''
# Add file (wage slip)
# Beschijving: Salaris periode {YEAR}-{MONTH}-M / {YEAR}-{MONTH}-M {NAME}, Periode: M
# Totaalbedrag: '''Totaal netto'''
# Line items:
## '''Brutto > Salaris''' as '''Loon (4000)''',
## '''Loonheffing''' as '''af te dragen loonheffing (1700)'''
## '''Netto > Onkostenvergoeding''' as '''Kosten nettovergoedingen'''

== Wage taxes ==
# Exact
# Financieel > Boekingen en grootboekrekeningen > Boekingen > Aanmaken
# Inkoop > '''80 - Salarisstroken'''
# Add file (LH)
# Beschrijving: {LHnummer en volgnummer}, Periode: D
## '''Loonbelasting / premie volksverzekering''' as '''Af te dragen loonheffing (1700)'''
## '''Premies''' as '''Kosten sociale premies werkgever (4041)'''

== Pension ==

# Exact
# Financieel > Boekingen en grootboekrekeningen > Boekingen > Aanmaken
# Inkoop > '''80 - Salarisstroken'''
# Add file (JP)
# Beschrijving: Pensioen {YEAR}-{MONTH}-M, Periode: IN
# Totaalbedrag: '''Amount under pensions on JP'''
# Line items:
## For each person who received pension payment: '''Pensioenen (4011)'''

== Reconciliation (after payment) ==
# Financieel > Boekingen en grootboekrekeningen > Overzichten > Kaarten
# Search for '''Af te dragen loonheffing (1700)'''
# Mark the wage slips together with the IRS slip for wage taxes. The difference at the bottom (e.g. 0.16 EUR) needs to be added to the wage tax document.
## '''Af te dragen loonheffing (1700)''' 0.16 EUR
## '''Rekenverschillen (9200)''' -0.16 EUR
## Repeat process; difference should now be 0
# Afletteren; difference should be 0