Security Development and Operations - codemagic-ci-cd/company-handbook GitHub Wiki

Codemagic’s Security Development Lifecycle (SDLC) is a security assurance process focused on developing and operating secure software. The SDLC provides detailed, measurable security requirements for engineers at Codemagic to reduce the number and severity of vulnerabilities in our product and services. All Engineers must follow SDLC requirements, and we continuously update the SDLC to reflect the changing threat landscape, industry best practices, and regulatory standards for compliance.

How does Codemagic test software for common vulnerabilities?

Codemagic defines and publishes a list of approved tools and packages for our developers to use. Our developers use the latest versions of approved tools to take advantage of new security features.

Additionally, Codemagic follows best practices to reduce vulnerabilities in its tools, infrastructure, systems, and processes (System Hardening):

  • Prefers single-function systems over multipurpose ones.
  • Removes unnecessary software and services:
    • This includes disabling unneeded ports and blocking all incoming traffic by default.
    • Initial hardening of new servers is performed using Ansible scripts.
  • Applies patches and critical fixes within 90 days of discovery.
  • Implements various other protective measures.

Before code can be merged into a release branch, the SDLC requires manual code review by a separate reviewer. Code reviewers check for coding errors and verify that code changes meet SDLC and design requirements, pass functional and security tests, and perform reliably. They also review associated documentation, configs, and dependencies to ensure code changes are documented appropriately and will not cause unintended side effects. If a reviewer finds problems during code review, they can ask the submitter to resubmit the code with suggested changes and additional testing. Code reviewers may also decide to block changes entirely for code that does not meet requirements. Once the code has been deemed satisfactory by the reviewer, the reviewer provides approval, which is required before code can proceed to the next deployment phase.

How does Codemagic manage open-source software?

Codemagic has adopted a high-level strategy for managing open-source security designed to:

  • Understand which open-source components are being used in our products and services.
  • Track where and how those components are used.
  • Determine whether those components have any vulnerabilities.
  • Respond properly when vulnerabilities are discovered that affect those components.

Codemagic’s SDLC

Codemagic SDLC consists of seven components including five core phases and two supporting security activities. The five core phases are:

  • requirements
  • design
  • implementation
  • verification
  • and release.

Each of these phases contains mandatory checks and approvals to ensure all security and privacy requirements and best practices are properly addressed. The two supporting security activities, training and response, are conducted before and after the core phases respectively to ensure they're properly implemented, and the product remains secure after deployment.

Training

All Codemagic employees are required to complete general security awareness training and specific training appropriate to their role. Initial security awareness training is provided to new employees upon hire.

All engineers must also participate in role-specific training to keep them informed on security basics and recent trends in secure development.

Requirements

Codemagic product team defines security and privacy requirements based on factors such as the type of data the product will handle, known threats, best practices, regulations and industry requirements, and lessons learned from previous incidents.

Design

The design starts after the security, privacy, and functional requirements have been defined. As a part of the design process, threat models are created to help identify, categorize, and rate potential threats according to risk. The threat modelling begins by defining the different components of the product and how they interact with each other in key functional scenarios, what ports and protocols are used. The threat model is used to identify and prioritize threats for mitigation that are added to the product's security requirements.

Codemagic engineers are responsible to:

  • Communicate about the security design.
  • Document the security design, guidelines and policies, and details of all changes.
  • Analyze security designs for potential security issues using a proven methodology.
  • Suggest and manage mitigation for security and critical issues, system impairments, customer impacts, rollback plans, etc.

Before any changes are released, all threat models are reviewed for accuracy and completeness, including mitigation for unacceptable risks.

Implementation

Implementation begins with developers writing code according to the plan they created in the previous two phases. Codemagic provides developers with a suite of secure development tools to effectively implement all the security, privacy, and function requirements. These tools include:

  • Secure development environments. These environments are logically segregated but similar to the production environment. Customer data and other production data must not be used in development activities.
  • Task management, knowledge and collaboration tools to document, record responsible parties, track changes, and priority.
  • Source code management tools to track and review code and configuration changes.

During this phase, developers should ensure that the latest security patches are applied to all development tools, libraries, and frameworks they are using. This helps in preventing known vulnerabilities from being introduced into the codebase.

Verification

Before any written code can be released, several checks and approvals are required to verify that the code conforms to SDLC, meets design requirements, and is free of coding errors. SDLC requires that manual reviews are conducted by a reviewer separate from the engineer who developed the code.

Various automated checks are also required and are built into the commit pipeline to analyze code changes. These automated checks fall into the following categories:

  • Conflict (or merge) analysis. Analysis to confirm that new changes were made on top of the latest versions and do not conflict with other changes.
  • Static code analysis. Analysis of source code for quality, reliability, and security.
  • Unit, Integration, and System testing. Dynamic analysis for security and error detection.

If the manual reviewer or automated tools find any issues in the code, the submitter will be notified, and they're required to make the necessary changes before submitting it for review again.

Based on the complexity of changes or new features additional testing can be required. Testing requirements should vary based on the complexity of the release (e.g. smoke testing for routine maintenance, acceptance testing for major releases).

As part of the security testing process, it's important to verify that all software components, including third-party libraries, are up-to-date with the latest patches. Automated tools can be used to scan for known vulnerabilities and check if patches are missing. Additionally, penetration and security tests are regularly conducted by external providers.

Release

Any changes to the production environment require approval and the use of production deployment pipelines.

Before deploying the software to production, ensure that all servers and environments are fully patched. This includes the application itself, as well as the operating system and any other software running on the servers.

Codemagic engineers should monitor the success of the deployment. Failures that require new development and testing must also obtain a new release approval.

Successful deployments are automatically reported to the #cm-deployments Slack channel and should be manually recorded in the task management tool by changing the task status to close.

In order to quickly respond to customer or internal incidents that require updates to production code or configuration:

  • The incident should be classified and communicated in an internal Slack channel.
  • The changes or mitigation actions must be approved by the CTO, or in the case he is not available by Senior Engineers.

Codemagic’s CTO is responsible for periodically performing code/config reviews for all instances in the production environment.

Response

All components, services, and core infrastructure are logged and monitored after release to identify potential problems and security incidents. In addition to system logging, we monitor application-level data. Our key principles for security monitoring and alerting are:

  • Robustness: signals and logic to detect unusual behaviors.
  • Accuracy: meaningful alerts to avoid distractions from noise.
  • Speed: ability to catch attackers quickly enough to stop them.

When an issue is detected, we need the ability to mitigate the risk at scale. For that, we use cloud-based tools to automatically apply countermeasures and provide engineers with tools to apply approved mitigation actions quickly across the environment. Using proven cloud-based solutions is a key pillar of our monitoring and response strategy.

We regularly monitor for new patches and vulnerabilities, and apply updates promptly. This includes scheduled maintenance windows for applying patches and emergency patching for critical vulnerabilities.