How Security, Reliability, and Design Teams Can Get Other Teams To Do Work for Them - The Objective Expert Model

Example: A Security Team That Partnered With Engineering

The best example I’ve seen was the security team at New Relic. 

• The security team recorded vulnerabilities in Jira. They were assigned to teams. The security engineer categorized the ticket by severity. More severe vulnerabilities needed to be done faster than less severe vulnerabilities (according to a service level agreement).
• The security team emailed all managers, directors, and VPs a security report each week. 
• The report included a list of all the teams in engineering. Next to each team was a count of overdue security vulnerabilities. You could click on the count to see the vulnerabilities. 

Security team leaders told everyone they were responsible for acting on this information. They would escalate urgent issues. Everything else depended on engineering and product prioritization. 

Their weekly report became an easy way for leaders to measure themselves. It also became an easy way to measure each other. Local teams had the autonomy to prioritize security concerns against their own roadmap. And it gave leaders a way to see how their part of the organization was doing. 

This social engineering resulted in leaders taking ownership of security issues. 

Handling Security Vulnerabilities as a Front-Line Manager

When I was a front-line manager, I would get the report each week. I would make sure my team didn’t have any overdue vulnerabilities. If we did, that meant something had broken down in our team process. Usually, it meant we hadn’t prioritized the work to complete it ahead of the security team SLA. I would dig into the issue and see what happened. 

Notice that the security team was a passive participant at this point. They did all the analysis to determine the severity of incidents. Their SLAs communicated their expectations. It was clear what a reasonable time was to address vulnerabilities. But they mostly left managing prioritization to the teams. 

Handling Security Vulnerabilities as a Director or VP

When I became a manager of managers, I interacted with the report in a new way. I was now responsible for my organization’s security vulnerabilities. The report listed the teams in my organization. And it showed how each of them was doing. 

When one of my teams had a lot of overdue vulnerabilities, I would talk with the engineering manager. I wanted to make sure they had made a conscious decision about prioritization. If they hadn’t made a decision about it, it was often a sign that their team needed help. They might be underwater, or they might have poor processes. I would talk with the manager and make sure we put together a plan to improve things.

Thus the security team not only outsourced prioritization but also outsourced accountability. They created some pressure and visibility around vulnerabilities. But they didn’t have to muck around in every team to get the work done. It was a very nice contract between teams and worked quite well.

Site Reliability Engineering Teams Can Use the Objective Expert Model

New Relic went through tremendous growth during my time there. We confronted scaling and reliability every day. Monitoring companies receive insane levels of data from their customers. So the demands on our systems were high. 

One of the best things we did was to create a repository for incident information. Nowadays, you can buy incident management tools from many vendors including Blameless, Jeli, Fire Hydrant, Incident.io, etc. But at the time, we didn’t have such options, so we built a tool we called Upboard. (Blameless was built with the lessons from Upboard).

Initially, we used it to store all of the incidents we had, and the follow-up items. But over time, we developed a team-by-team metric for reliability. 

Although there was a lot of discussion about the metrics we used, they helped people make the case for doing reliability work. And they helped the entire organization focus more effectively on pain points. It also helped us realize what parts of our efforts were making a difference. 

Upboard also had a weekly report. It showed (on a team by team, and org by org basis), how teams were doing with reliability. The report transferred the responsibility for reliability to the teams that could do something about it. The reliability team then became the team that helped teams improve. They acted as Consultants. 

This made for a healthy relationship. I always appreciated my interactions with the reliability part of the org. They were experts that had wonderful advice on how to make things better for my teams. Their reports showed me the experience I was offering customers.

Design Teams Can Also Use the Objective Expert Model

Companies often neglect usability. It can be frustrating to see customers offer experiences that people can’t navigate. Design is often iterative, but companies fail to iterate.

Ideally, your designers are working with local teams to prioritize usability work. When that fails, consider the Objective Expert Model. It gives more leverage for design teams.

Here’s how it can work:

1. I assume your designers embed on product engineering teams. They identify issues that need attention. 
2. The designers would grade the issues by some sort of severity. This would correspond to an SLA for addressing it.
3. You produce a weekly report: “Top product usability issues”. 
4. Product managers then use the report as input into their prioritization decisions.

See later in this post for how to put this in place. 

Non-Specialist Teams Can Use This Model for Projects

Platform teams can steal this approach for non-specialist teams. If you have something you need others to adopt, you can make it visible who has done the work to adopt it. 

For example, let’s say your team is responsible for permissions, and you have created a way to do Role-Based Access Controls (RBAC). Your work has to be integrated into the code of every product team in the organization. 

There are many ways to solve that problem: 

• You can do the work across all the engineering teams. (I’m a fan of this approach)
• You can create documentation on what needs to be done to integrate and work through product management to get the work prioritized on each team. 
• You can create documentation on what needs to be done to integrate and create tickets for each team to request them to do the work (I don’t recommend this approach)
• If you have the right prioritization, you can have a Program Manager coordinate all the parties involved. 

Another approach you can take is similar to the approach we described for specialist teams:

1. Make sure the prioritization is agreed upon. (Perhaps a deadline or stack ranking of priorities).
2. Tell every product manager it is up to them to decide and ask them to surface risks of not achieving it on time.
3. Automate a report that shows progress by the team. 
4. Share that report. 

What’s nice about this approach is you make it easy for the product managers to do their own prioritization. They might have other urgent work scheduled. You don’t have to know about their local details. You then focus your team on making it as easy as possible for every team to take part.

This again is a way to outsource the prioritization to the teams involved. 

When To Use the Objective Expert Model

The Object Expert Model is one of the most scalable coordination models. One team can produce a report that affects every other team in engineering. The main consideration is the cost required to produce the report. Coordination costs are low once you have set it up. The process and SLA combine to make it lightweight. 

If your team wants other teams to prioritize work from you, use this model. 

you can take this too far. It isn’t a model that every team should be using. Think about what it would look like if every team tried this model. Your organization would generate too many prioritization artifacts. It would be unwieldy. So if you see more than a couple of teams using this model, think twice before adding another one.

When Using This Model

The general approach is:

1. Make this part of a larger partnership effort, and consider the human incentives
2. Get buy-in and establish service level agreements or standards.
3. Send out a weekly report to managers, directors, and VPs. Ideally automated.
4. Be patient and look for opportunities for reinforcement.

Make This Part of a Larger Partnership Effort, and Consider the Human Incentives

You won’t be effective if you do this as a mechanical change. Rebecca Campbell: “Security [at New Relic] wasn’t successful because they emailed me a report. They were successful with me and my team because they met with me when I first started, explained how things worked and checked in with me regularly. They were happy when I asked questions or reached out to them and delightful to work with. They saw themselves as enablers rather than blockers.”

She’s right. The security team used the Objective Expert Model. But they weren’t just sending a report. They were doing many things to make their team approachable. And they aimed to make it easy to do the right thing with security. 

Think about how your team interacts with the rest of the organization. How would other teams welcome your report? You are successful if people view your report as helpful. I looked forward to the reliability and security team reports. They helped me create a better product. 

I have an example of a project that got it right. The Role-Based Access Controls (RBAC) project was initially a “disaster project”. Everything seemed to be going wrong. But the team turned that around, mostly because of how they approached other teams. It became known as a model project. 

Rebecca Campbell said: “The thing that was most successful about RBAC was the roadshow we did prior to asking teams to do anything. We met with every team, had their PM lead a portion of the meeting (very important to demonstrate priority and buy-in), shared sample code, and told engineers what was needed and where to get help.” Pay attention to what will make people think your team is helpful. 

You’ll likely need to do some work with your team to prepare them for how to interact with the rest of the organization. I interacted with several security team members at New Relic. They all told me the same thing. If I asked, “do I have to work on this vulnerability”, they all said, “this is our assessment of the level of risk, and when it should be worked on. But it’s up to you to prioritize it”. They spoke with one voice. This requires a lot of work within your team. 

Get Buy-In and Establish Service Level Agreements or Standards

After you’ve thought about the human side, you’ll need to get an agreement that the work is important. You then can codify this as a set of standards of a Service Level Agreement (SLA).

Start out by working with leadership. Socialize the idea so that the SLA seems like a natural next step. Talk through the problems you’re seeing, and share the SLAs for feedback. 

It’s important to make your SLAs easy to consume. When you approach teams with work, you need to be using the SLAs to simplify the decision-space for them. They’re going to ignore your report unless it’s easy to consume.

A few considerations as you develop these SLAs:

• Some things are urgent. They may need a different process. For example, security teams may need to do this for remote code exploits that are live in the wild.
• Balance the timelines with the capabilities of the company. You can start with something achievable, ratchet up responsiveness over time.
• See below for an example to start from. You can vary the description and SLA turnaround, according to the needs of your product. This is a design example.