Articles

Epic Failures in DevSecOps: DJ Schleen

Sep 4, 2018 12:28:21 PM By Mark Miller DJ Schleen - Epic Failures v01

Editor's Note

This is the overview and outline for a chapter in our upcoming book, "Epic Failures in DevSecOps". Each chapter is a unique voice, telling us a story about an epic failure that has been encountered as part of a personal DevOps/DevSecOps transformation.

Would you like to proofread a chapter and give us feedback? If so, look at the bottom right of this page and confirm you'd like to be a proofreader. (If you don't see the box, leave a comment.) This will put you on the "keep me up to date on the project" list, and we'll reach out when ready for your help. There are going to be eight chapters.Keep an eye out for the others. -- Mark Miller, Executive Editor 

Chapter Overview: 
"The Problem with Success" by DJ Schleen

I remember the day like it was yesterday. After months of planning we finally added a security control into our automated build pipeline. This particular integration was the first of many we had planned to roll out over the next few years. Static code analysis was ready to check our source code for security vulnerabilities. Little did we know that we just set up the security team to experience years of frustration.

The Story

Traditional security teams haven't been very technical, but here we were building infrastructure and configuring software; much to the developers dismay. They looked at the security tool we were proposing and didn't take it very seriously. It was just  another mandate from above in their eyes and the oroduct owners weren't very impressed either.  They weren’t very flexible in allocating precious development time to remediate security issues. This was going to be a cautious roll out. We didn't know was the beginning of our transition from DevOps to DevSecOps. 

We decided to start with the iOS team and a small installation. We were successful - but at an extremely small scale. Discuss who we did mobile, pull requests, jenkins plugins to call into XCode Server to build and deploy to TestFlight. Business owner still wanted to manually approve. No trust in automation.

Introduced the tool to a non-DevSecOps affiliate… large code bases, 48 hours to scan. Left them with the ball to figure out false positives.

Database issues… kept having to reboot or restart sql…

Gave an internal team access to the system - within a week they onboarded 500 applications. Everything stopped

From one engine to thirty - lets waste some more compute time

The Outcome

We had availability issues… in a bad way.

  1. Teams removed the control from their pipelines in order to get product out
  2. Nobody trusted the security team, or the tool
  3. 48 hours to scan clogs a scan queue

With one team we implemented the control and they had no idea what to do with the results. We had no plan. We iterated and constantly improved. At a point where things are well oiled.

Lessons Learned

Track security drag

Know that you need to iterate. Take the risk. If you have 10% success with 100% of the effort you put in, realize that you’ll have 0% success if you have 0% effort.

Always have a plan. You can’t just implement a software security tool and enforce controls without “watching” traffic for awhile.

Discuss “Just pressing the scan button - develop a program

Prepare for success. What happens when you start with 9 applications and end up with 500+? How do you support over 1000?

Put some health checks in...

About DJ Schleen

DJ is a DevSecOps pioneer and currently works as a DevSecOps Evangelist and Security Architect at a large healthcare organization. He provides DevSecOps thought leadership throughout their journey of cultural revolution and digital transformation. DJ specializes in automating security controls in DevSecOps environments and is an ethical hacker as well – doing significant R&D work in Moving Target Defense, Mobile Security, System Exploitation, and Penetration Testing.

As an expert in Application Lifecycle Management (ALM) and ITIL, DJ has worked to streamline development pipelines for many Fortune 100 organizations by focusing on people,  process, and the right technology .  He is an active speaker, blogger, instructor and author in the growing DevSecOps community where he encourages organizations to deeply integrate a culture of security into their core values and product development journey..

Want to be a proofreader for the book? Fill in your email address in the box on the right. (If you don't see the box, leave a comment). We'll get back to you within the next three weeks with a chapter for your review.