Handling Security Exploits & Incidents
How critical issues are identified and fixed
The majority of the code in Orbit undergoes reviews and is scrutinised for security exploits multiple times a year, where critical issues are fixed and promptly deployed to vulnerable installations.
The fixes are also backported to solutions that may run older versions of Orbit. Orbit Online does however try to keep all customer solutions up to date with the latest stable version of Orbit at all times.
Orbit relies on a considerable number of supporting services to perform its functions. Similar to how Orbit itself is monitored for issues. Orbit Online employees monitor the supporting services for any vulnerabilities that have been published by their maintainers or security researchers.
To facilitate a short reaction time when exploits are published, all Orbit servers support being updated through a provisioning tool. This tool can disable services, update existing ones, or reconfigure them to mitigate any issues.
At Orbit Online the guiding metric for evaluating, mitigating, and finally fixing the root cause of security incidents is severity. During the processing of a security incident, the discovery of new information may change the severity and cause the process to accelerate or decelerate.
If the incident is reported by a user of the system, both the authenticity of the user and the veracity of the claim are confirmed first. This is done to prevent the process itself from being the cause for a security incident (e.g., causing a denial of service by claiming there is a critical, but hard to reproduce security vulnerability. And, requesting a shutdown of the system while it is being resolved).
Once the reported symptom or symptoms have been verified, a severity (high, medium, low) is assigned to the incident. This is done to make better-informed decisions later in the process. When determining the severity, the requests of the customer and their unique circumstances are, of course, taken into account.
Using the initial assessment of severity, a mitigation strategy is formulated. Here, multiple factors are taken into account:
- Is mitigation without first finding the root cause possible or at all effective?
- Will the mitigation cause interruption of service?
- Could mistakes made during the implementation of the mitigation cause further issues?
- Does this issue affect multiple customers?
While the mitigation strategy is formulated, a search for the root cause of the issue is put into motion. The customer is informed of either the mitigation strategy that will be implemented, depending on the estimated completion of this search. Or said strategy is discarded in favour of fixing the root cause instead.
Once the root cause has been identified, other potential symptoms are extrapolated and evaluated as well. This may result in an escalation of the incident severity. A plan for fixing the root cause is then developed. Based on the estimate of the implementation and severity of the incident, an alternative mitigation strategy may be applied while the root cause is being fixed.
Using the root cause analysis, the actual impact of the issue is determined by way of audit logs and other forensic tools. Future potential impact assessment is performed as well. When all of the above information has been gathered, the results of the analysis are sent to the customer.
Depending on the severity of the issue and instructions from the customer, the root cause will be either hotfixed or added to the bug list of the next patch release of Orbit.