DevSecOps Keys to Success

To understand the current and future state of DevSecOps, we gathered insights from 29 IT professionals in 27 companies. We asked them, "What do you consider to be the most important elements of a successful DevSecOps implementation?" Here's what they told us:

Automation

• Include the automation of security into the process. Automate as many steps as possible. Have DevOps teams engage the security side throughout the value stream, looking for hold points, templatizing security aspects, standardizing the process, automate so those things just happen as part of the process. Bring everyone together onto one team.
• Mobile grew out of DevOps and security was always important to the process. Netflix just does DevOps as part of engineering. Eliminate as many manual steps as possible so security becomes a first-class citizen in DevOps workflows. Add security to the mix. You can’t create a manual workflow, or it will get bypassed. Security has to be tightly coupled with Jenkins, JIRA, and other frequently used DevOps tools.
• Empower customers to get more automated when it comes to vulnerability management. There are four components to this: 1) people, more collaboration around shared goals around security; 2) processes move from waterfall of CD/CD/CI infusing security early in the process; 3) select tools and technology to increase the velocity of deployment in a DevOps model, remediate as you find errors or vulnerabilities, automate at speed; 4) guiding principles to follow around security and methodology. Create a measurement and monitoring piece.
• Integrate security into the CI/CD lifecycle. Bake security into the entire automation process. Bake in security automation like scanning and scanning different environments enables DevOps teams to create a certain level of hygiene. Invest in small cycles to set up profiles and a baseline to automate and detect hygiene risks.
• The most important elements of a successful DevSecOps implementation are automation and collaboration. 1) With DevSecOps, the goal is to embed security early on into every phase of the development/deployment lifecycle. By designing a strategy with automation in mind, security is no longer an afterthought; instead, it becomes part of the process from the beginning. This ensures security is ingrained at the speed and agility of DevOps without slowing business outcomes. 2) Similar to DevOps where there is close alignment between developers and technology operations engineers, collaboration is crucial in DevSecOps. Rather than considering security to be “someone else’s job,” developers, technology operations and security teams all work together on a common goal. By collaborating around shared goals, DevSecOps teams make informed decisions in a workflow where there is the biggest context around how changes will impact production and the least business impact to take corrective action.
• The key to successful DevSecOps is the automation of security controls inside the DevOps pipeline. Ten principles to follow for DevSecOps implementation: 1) Principle of least privilege for all services that process (read, write, or update) data. 2) Enforcing tight access security for API endpoints. 3) Running SAST (static application security testing) tools as part of the nightly build process and running DAST (dynamic application security testing) tools to identify security defects in running containers. 4) Scanning any pre-built container images for known security vulnerabilities as they are pulled into the build pipeline. 5) Automated tests for security capabilities wired into the acceptance test process. These automated tests include input validation as well as authentication and authorization enforcement. 6) Isolation of containers from one another, avoiding any dependencies and keeping them entirely stateless to eliminate high-value targets for attackers. 7) Automated security updates, such as patches for known vulnerabilities, by means of the DevOps pipeline with an audit log. 8) Reduce the attack surface by using a secure API gateway that enforces fine-grained and scope-grained access to sensitive API endpoints. 9) Automated service configuration management, allowing for compliance with security policies and the elimination of manual errors. 10) Continuous monitoring, audit, and remediation of security defects across the application lifecycle. Also, firewalls should continue to defend in-depth by isolating services. Intrusion detection is a lot harder using containers, so looking at network behavior helps detect abnormal traffic patterns. If possible, security tooling should be a gate to deployments (applies to SAST and DAST). However, all this automated flow should still be validated by external pen tests to make sure automation covers all aspects. Additionally, Incident Response plans should be created and practiced for all new environments to ensure they have the capability to preserve evidence to aid in investigations and staff knows how to execute the plan, either themselves or who to outsource it to.

Shift Left

• Start testing at every programming unit, every unit of code that is being created. Test a few times a day with every unit of code and then the process goes into the build. Test on the entire build level since you’ve tested at the unit level. See the results of the test and remediate. When it comes to deploying, you have the assurance you detected and remediated all vulnerabilities. Test in large volumes. Year over year we see vulnerabilities like SQL injection in 40% of applications, in 2018 it was the same as 2006. We are not getting better. We are not winning the war. While we are building more secure applications, it still takes more than 100 days to fix vulnerabilities. Vulnerabilities deemed to be unimportant take 200 days to remediate.
• The fundamental premise is we’re moving some sort of requirement left. Moving a requirement earlier in the project and creation stage. The code people write. The infrastructure that supports the code. How to support the infrastructure and how to move security to the design phase. PCI is an example of moving design requirements left. From the infrastructure design, you are shifting left. The same thing happens from a code pipeline POV. If we’re going to do code scanning, are we going to define the parameters for secure code in the beginning? You accomplish by upgrading the IDE to use a parameterized query.
• The most important element of a successful DevSecOps implementation is to shift security left in the engineering lifecycle. Although ARB review meetings before the start of a project or implementation include InfoSec teams, security is not implemented early in the lifecycle, often leaving too much to handle in the end before a release. Security should be imbibed as a design mentality instead of being an afterthought.
• It’s really about where security is plumbed in the process. The most successful DevSecOps implementations happen when developers do not have to think about security. This isn’t because they aren’t doing it, but because they don’t notice it or see it as unusual. You want to build security into the happy path for getting code out and work completed. If the best way to get something done includes all the security steps, why would your team bypass it?
• Know that security activities are aligned with the SDLC. If it doesn’t occur quality suffers, and costs go in remediating problems later. You have a higher risk of being breached.
• Given the complexity of today's applications and the adoption of new technologies, organizations need to treat security as an integral part of software delivery, rather than leaving it to the end of the development phase. As we know, DevOps encourages teams to test early and often, and since security is an integral part of testing activities, organizations are shifting security left, testing for security as early as possible, and continuing to test for security vulnerabilities as software flows through the DevOps pipeline and out into production.

Collaboration

• Integrating security experts into the design process so security is “baked in” to applications. Educating developers that security is a business-critical characteristic of applications.
• Thinking about it through the lens of security takes a shift in mindset and methodology. First around software lifecycle on-prem or SaaS. If you don’t make DevSecOps part of the lifecycle from planning design, code, and rollout perspective you will miss opportunities to expose vulnerabilities that could expose a business to risk. Think about the security, operations, and development teams need to do. Coordination becomes important to deliver secure, tested software as needed. Software engineers don’t need to be security experts but they do need to understand secure coding. Many organizations come along after the fact and look to apply secure coding as opposed to doing it upfront. Also: 1) Minimize the introduction of vulnerabilities. Take the extra time to make sure you are not introducing vulnerabilities that will lead to a breach. Do with secure coding and evaluate the libraries you are going to use to make sure they are not vulnerable. 2) When you think about this with development tools and methods secure coding and testing is very important. Highly recommend they do penetration testing as well to see how people are able to break in using common exploits and vulnerabilities.
• A successful DevSecOps implementation starts with strong alignment between development and security teams, based on shared objectives and an agreement to collaborate throughout the design and operation of their new shared practice, as DevSecOps fundamentally is about breaking down barriers between parts of the organization. In conjunction with this cultural shift comes a focus on cross-education/traditional security experts taking an active role in engaging with developers to spread security-focused thinking further “left” in the process, and developers giving security specialists insight into the environment and architecture where software is being created and operated so that security practices can integrate smoothly into development processes rather than be jarring roadblocks. Strong reporting and insights are required, based on agreed-upon shared objectives and metrics that have meaning across both involved teams, so that improvements to processes can be genuinely data-driven and free from subjectivity. All this collaboration and education needs to be supported organizationally, meaning developers are allowed time to implement new security protocols, security professionals are incentivized to actively participate in the development process - even to the point of contributing user stories, and the metrics tracked by the combined DevSecOps team are aligned with business goals and translate into business value.
• Two critical components for a successful DevSecOps implementation are collaboration and automation. First, collaboration; each team has their own unique needs, strengths, and constraints. Collaborating enables the group to leverage their respective strengths to work around constraints and achieve business goals. Second, automation: the speed and scale made possible by DevOps practices and cloud-native platforms can be completely wrecked by manual processes. A well-functioning DevSecOps implementation uses automation to perform the routine security tasks, identify risks and validate compliance.
• Organizations that successfully adopt DevOps position themselves well to mature to DevSecOps. For them, DevSecOps becomes a bridge between building fast and secure software, so that choice no longer needs to be made. It’s a way to unite security and engineering cultures and helps ensure that security becomes an integral part of application design. It’s important to remember that DevOps and DevSecOps are not job titles or roles. They are significant shifts in thinking and processes. When companies don’t embrace them, they affect their ability to fuel innovation. They fall behind.

Other

• Participation, adoption and streamlining — but most importantly, coverage.
• DevSecOps is the evolution of DevOps asking the same questions about security that we asked about operations when DevOps began. People treat security like operations. Evolve from a security culture of saying “no.” Don’t take on risk to generate business value. The dev team has less motivation to develop a security mindset. There’s no way to add on security at the end when it goes to production. Incorporate security into development. Security team members educate developers on security and help ensure security is integrated throughout the process. SecOps is a resource group for doing good work.
• Hold security in the highest regard. Continuous pressure from product development to churn out new features and security slows testing. We are delivering new features at a rapid clip but there is no compromise on the security aspects of what’s delivered. Understand the flow with DevOps processes, automation, and tools with a security component. Understand how to react to things just like a test bug found by QA. There’s another vector of alerts to react to. Tools will only get you so far. Alerts that come to engineers are useful, not a waste. Information that comes out is actionable. Pay attention to it, do not dismiss it.
• Empower data to defend itself. By moving data security into the data tier, it’s more protected and secure. Have the capability of guaranteeing data has never been tampered with. Prove when data was entered and if it was tampered with.
• A successful DevSecOps implementation – building security into the CI/CD pipeline – requires an appreciation for the need to secure “micro-perimeters” around all containers and services within the containerized environment and throughout the entirety of the build-ship-run application lifecycle. This understanding of the real challenge of securing both external and internal container environment traffic (and cultural buy-in among developers acknowledging that security is essential at each stage of the development process) are key factors shaping what a DevSecOps implementation can achieve.
• The most important element of a successful DevSecOps is the “secure-first.” Just because you need to deploy and distribute at a fast pace, you still need to be mindful of safety and security.
• There are several key elements for a successful implementation: 1) A clear vision of the importance of security in the context of rapid DevOps release cycles. 2) Clear process metrics that help you track where you can improve; clear role-based metrics to drive the right behavior. 3) Support from senior executives on the importance of DevSecOps. 4) Starting small and growing organically in order to avoid major disruption with business continuity. 5) Ongoing emphasis on security monitoring.
• Start with visibility. It’s easy to create K8s clusters, but we also need to get the clusters registered. Register them with us so we know where they are and what’s running on them. Then set a centralized set of security policies. Are the different clusters meeting our standards? Includes cluster access, security policies, image scanning to determine if they have known vulnerabilities. What’s running, who has access, is anything running with vulnerabilities. Implementing access clusters. Controls around creating new clusters.
• The most important thing from a DevSecOps perspective is holistic security: security that spans the entire stack. Moreover, successful DevSecOps teams are enablers of the business – they can speed time to market by allowing developers to write code quickly. When DevSecOps provide tooling (not just docs and slides) that solves/prevents security problems, then developers can focus on business-value features without adding security, operations, and compliance risk.
• Transparency and minimal performance impact. Ideally, the app developer should not be aware of the security that’s been implemented.

Here's who provided their insights:

• Anne Baker, V.P. of Product Management and Marketing, Adaptiva
• Steven Aiello, Solutions Principal, AHEAD
• Gadi Naor, Co-founder and CTO, Alcide
• Mike Stahnke, VP of Platform, CircleCI
• Brian Nash, Director of Product Marketing, and Brian Dawson, DevOps Evangelist, CloudBees
• Michael Rose, Vice President of Engineering, Cybera
• Doug Dooley, COO, Data Theorem
• OJ Ngo, CTO and Co-Founder, DH2i
• Kris Lahiri, Co-founder, Egnyte
• Brian Platz, Co-founder and Co-chairman, Fluree
• Javed Shah, Director of Product Management for Cloud and DevOps, ForgeRock
• Malcolm Isaacs, Senior Solutions Manager, Application Delivery Management, Micro Focus
• Gary Duan, CTO, NeuVector
• Yogesh Badwe, Director of Information Security, Okta
• Franklin Mosley, Senior Application Security Engineer/Evangelist, PagerDuty
• David Strauss, CTO and Co-founder, Pantheon
• Jeff Keyes, Director of Product Marketing, Plutora
• Vishnu Nallani, VP & Head of Innovation, Qentelli
• Sheng Liang, Co-founder and CEO, and Shannon Williams, Co-founder and VP Sales & Marketing, Rancher Labs
• Gene Yoo, CEO, Resecurity
• Altaz Valani, Research Director, SecurityCompass
• Jim Hansen, V.P. Products, SolarWinds
• Colby Dyess, Director of Cloud Marketing, Tufin
• Tim Hinrichs, CTO and co-founder, Styra
• Joseph Feiman, CSO, WhiteHat Security
• Andrei Bezdedeanu, VP of Engineering, ZeroNorth
• Tim Reilly, COO and CFO, Zettaset