How to NOT integrate security tools in DevSecOps : Top 3 Mistakes Across Process, Technology, And People

Kuldeep Kumar

Introduction

One of the most important activities in achieving DevSecOps is integration of security tools in the CI CD pipeline to catch vulnerabilities early and get them addressed early as well.

I would like to share my experience on the top 3 mistakes done during this activity across the three pillars (Process, Technology and People) of this activity.

Process

  • Ambiguous Security gates or blocks
    • I am not a huge supporter of security gates or blocks but if you must do it, try to find the solution for at least below questions before implementing them in CI CD Pipeline.
      • What will be the process to unblock the builds on failure ?
      • What will be the turnaround time ?
      • Who can unblock those builds ?
      • What if a developer claims vulnerabilities blocking builds to be False Positives ? Who will review them and How fast will they be reviewed ?
  • Unclear Vulnerability Management Process & Workflow
    • Create and document a process around how a vulnerability discovered by these tools will be managed.
    • Try to have all the vulnerabilities flow into a single place as much as possible instead of multiple toolings. Multiple tooling reduces the speed of developers and discourages the use.
    • Even better, Try to bring these vulnerabilities into the backlog of the development teams.
  • Not following a gradual Rollout process
    • Start with a Proof of concept of the tooling with few dev teams participating in this. If a team volunteers for the POC it is much better as they will be more interested and happy to provide feedback.
    • Learn from the POC mistakes and try to address them.
    • Start rolling out in phases to a group of teams instead of enabling it across the organization at once.

Technology

  • Going with a barrage of tools
    • Start with integrating one tool (Maximum of two if you have a big product security team to support the rollout). Never go all out trying to push all the security tools in one go.
  • Not Selecting tools by False Positives Ratios
    • Choose tool with minimum rate of false positives findings first
    • If a tool has a high rate of false positives try to tune it before rolling it out to everyone during a POC with few teams.
    • Tools with a high rate of false positives will overwhelm developers and may not lead to adoption.
    • My suggestion is to go first with a SCA Tool ( or a secrets scanning tool if you have a really lean product security team)
  • Tools with incorrect feedback loop
    • Many tools (or the way they are configured) require developers to check the security tool’s platform or email or slack for the results generated by the tool instead of the workflow in which build and these tests run.
    • These are good as a secondary medium but the intention should be to show the results in the software or pipeline itself where those tests run.
    • If this is not possible only then opt for other mediums along with bringing the vulnerabilities into the developer workflow for triage and remediation.

People

  • No Training or Education
    • Providing training without understanding their needs is as good as no training in most of the cases.
    • Also, It is important to train development teams not only on the tool but also around the processes pertaining to the tool.
  • No time for adoption
    • Provide them time to let them get used to the tooling and the process.
    • Do not even think about blocking PR or builds unless people get familiar with the tooling and process.
  • Choosing wrong stakeholder for getting this done
    • Generally these kinds of integrations are managed and controlled by a team which has admin rights on the CI CD pipeline. These are known by different names in different companies with names like  engineering / developer / development + experience / productivity.
    • Many times they would not allow you to make changes there as it may break their babies (which nobody likes) hence you would end up relying on them and their limited bandwidth for security.
    • This leads to a slow down in which case security should work in collaboration with these teams for the integration by dividing up responsibilities as needed to ensure reliability and efficiency both.
    • Follow Trust but Verify Principle.

Additional Resources on DevSecOps

Conclusion

Although the idea of integrating security tools into the CI CD pipeline is a lucrative one, it must be done carefully and considerably otherwise it can lead to other problems. 

Do share your experiences, questions or feedback on my LinkedIn handle.

Related Posts