Agile In IT Support and IT Operation Teams

Introduction

A few colleagues and I had a discussion and we tried to capture our experience in the article below. During our engagement we devoted some time to understanding IT Operations and IT support teams, to discover their world, and how different their world was from the IT Product Development team.

These are a few of the questions we asked during our transformation drive:

• How do they measure their success? 
• What element of Agility can we bring with those teams?
• How do they think and what is the way they operate?
• How can we converge these two worlds for maximizing Value?

What we have observed is a conflicting goal—the IT Support team measures themselves by bringing Stability (less turbulence), and the product development team measures themselves by producing frequent releases! More releases, more chances of chaos! 

We were trying to bring harmony by involving both parties on the same page.

IT Support teams manage their program by maintaining a visual management board (Kanban board), which they measure based on the ticket inflow and outflow. The flow measurement time is critical to those teams. Another one they manage is system stability—MTTR (mean time to recovery, repair, respond, or resolve).

There are many challenges the IT service and IT support team experience. Most of these challenges are not only concentrated in the IT Support and IT operation areas. These are originated upstream of the organizations: IT Product Development, IT Business areas, IT Operation, and IT Infrastructure. 

Challenges:

• Silos organizations: IT product development, IT infrastructure support, and IT operations are totally different units. They have their own goal and own rules. They have their own separate kingdom! They do not collaborate with each other, nor are they ready to cooperate with each other. Services are fragmented among multiple departments.
• Long cycle time to serve the customer request. There are many layers within the organization. With horizontal and vertical layers, it takes time to traverse through the long path. Manual lengthy release process, manual deployment, manual approval, manual steps, tests, etc.
• Measurement: IT operation measurement data and IT product development measurement data are different. IT operation measures themselves based on stability. IT product development measures themselves based on the speed of delivery. IT operation does not advocate too much turbulence, but IT product development would like to deploy as many releases as feasible.
• Process: IT operation and IT infra support teams are run by a heavyweight process to reinforce stability. They tend to follow a sequential, approval-based process. IT product development minimizes process and tries rapid product development with minimizing process overhead. With the traditional process, the overall time to market and respond time slows down.
• Skills: IT operation and IT infra support team members are slow in adopting new automated tools. Most of the time, they rely on manual steps. Skills and competencies are not modernized for automation.
• Goal: Conflicting goals in each unit mean there is no easy way to align the whole team; team members are distracted in multiple works. It takes time to figure out which is the appropriate-optimized way to work out the ticket faster.
• Mindset: Most of the team members are operating in the traditional organization and traditional management style within IT operation and IT infra service. These 2 organizations have completely different cultures compared to IT product development.
• Raising tickets, assigning to each department, tracking, and the follow-up creates a lot of hands-off and waste in the system. Trust deficiency is the issue most of the time. Conflicts are common among IT operations and the IT product development team.
• Team ownership is missed most of the time; activities lay on the departmental border and teams then later blaming each other when such tasks aren't completed causes lot of confusion and trust issues.
• Old, outdated infrastructure, tools, applications, and processes are very fragile to operate or deploy. Many defects and IT operation teams are loaded with L1, L2, and L3 issues.
• Traditional command-and-control type organizations lead to the risk-averse climate where mistakes are punished politically. Team members want to be in their comfort zone to avoid making any mistake.
• Never taking into account the bigger pictures while architecting an application causes waste in the overall IT operation. Teams must take into account how easy or difficult it will be to operate or maintain the application.
• Too many specialist team members cause bloat in team size and silos operation (missing cross-functional team members, missing autonomous empowered teams). Outdated skills, a legacy fixed mindset, traditional leadership style, and anti-agile behavior all spell waste. 
• Once Applications are developed and deployed, the IT product development team’s ownership and commitment are minimized. Now it is up to the IT infra and IT operation teams to figure out all the disruptions happening with the application. Why are IT Product Development teams not accountable for the application’s failure?
• IT operation and IT support teams tend to have many team members with specialized skills. Every role is justified and, in less time, there are many team members added. 
• Poor customer feedback from the survey. Long delivery cycle time. Less feedback collection after service offering, customer collaboration is minimal, collaboration among teams is not so great. 
• If anything goes wrong, they punish the person behind it. The system is filled with fear to take any steps; team members are not psychologically safe to take any steps not written in the document or written in the approval.
• There is no production-like environment with the development team, so, more errors appear when features are deployed in production for the first time. It causes a delay in the release of the features into the market.
• Collect data “As-is” about the systems 
  • Release frequency 
  • How many changes are deployed
  • Cycle time 
  • Rework time 
  • Approval process time, if any 
  • Build failure %, deployment failure % 
  • Number critical bugs discovered monthly/weekly 
  • Recovery time from failure 
  • Service Availability
  • Infra uses %

All this information will reflect if inefficiencies are present.

In the above figure, a typical organization composition has been demonstrated. The IT product development organization is serving numerous business units. They are aligned with the business to produce new features or complying with new regularity policies, etc.

The IT infra service and IT operation organizations sometimes work together based on the volume of the work these units have to deal with.

IT infra service and IT operations are serving business continuity and ensuring all the hosted applications are up and running all the time. These organizations are the backbone of the business. These applications and systems are directly accessed by millions of customers day and night.

All these organizations orchestrate together to present a better customer experience.

The Agility of any organization is measured by the rapid speed of the offering organizations are providing at any given period of time. All it depends on the agility of the people, systems, structure, process, and culture.

What Can We Work Out?

We must focus mainly on: 

1. People
2. Structure
3. Process
4. Leadership 
5. Technology

As the above picture shows, all these areas have to be looked upon to ensure these areas are enabling the transformation to happen smoothly. With the help of the leadership team, Agile coaches have to work together to strengthen all these areas for better value generation. As transformation is a journey by taking various actions, we change the organizational culture.

The above picture depicts what could be the ideal high-performance team when we have all the team members are coming from diverse expertise groups and form a squad with a common mission and purpose. There will not be any silos in this model. This Squad will continuously improve to rapidly generate value to the end customer. They will cross-collaborate with other squads to do the dependency management. Such formation and maturity develop only after working with several iterations, experiments, and learning from the mistakes. The squad will have one backlog, one common metric to measure themselves to claim success.

Short Term Goal

• Give training on systems thinking tools. Coach on applying these tools. How do we visualize the whole so that we align all the teams? Minimize frictions and maximize collaboration for a common purpose. What is the common purpose of concentrating on customer-centricity?
• Make tools team centralized. Standardize those tools from central Tools CoE teams.
• Maximize automation with a 90% automation target. Look for the opportunity to identify manual steps and promote to automate those steps (automate testing, automated deployment, etc). Real-time monitoring and automatically detect, alert, and resolve issues.
• Encourage security; reliability audit frequently to look at the resiliency of the IT infra and IT operations
• Coach for maximizing collaboration among IT product teams, IT infra team, and IT operations teams (mindset, structure, process, and leadership side)
• Increase the customer centricity mindset. Looks for an opportunity where this attribute is missing. Catch people in behaviors that are not aligning with this approach.
• Inject continues improvement culture. Track the monthly actions taken by the teams to improve the overall systems. From Doing to “Being” Agile.
• Drive Value Stream Mapping and find flow time (Find the bottleneck and optimize the flow)
• Look into the defect trend and find a pattern. Take action to mitigate such defects (issues due to?)
• Find data of meantime to detection, meantime to recover, etc. Process what all this data tells. Let us go deeper beyond these data. What do the various cultures, mindsets, people, processes say about the current state of agile? What can we improve?
• Look for fail-fast culture and mindset. Leadership style needs to encourage to experiment and learn from it. Coach people to perform such steps more. 
• Create one team concept: “You build it, you run it”—everyone is responsible. Minimize hands-off. Create a virtual team from IT Development + IT support + IT Operation. Create a cross-functional, self-organizing, self-driven, autonomous team. Invite all the team members in various iteration events, collaborate as much as possible, align for the common goal.
• Create visual management information radiators in all places. Do a GEMBA walk with the leadership team and inspect the data. Give the award to the best such teams who are demonstrating improvement.
• Apply Kanban in all the teams in IT Operation and IT Service, watch out for WIP limit, measure lead time and cycle time. Manage the batch size.
• Maintain proper Value, Flow, and Quality
• Coach for pull-based teamwork. Look for hero culture and minimize this mindset. Create a culture for collaboration.
• Build a Squad of Transformation evangelists who will drive the culture transformation.
• Conduct lean coffee sessions to understand the challenges, impediments, and things that are working. 
• Communicate, Communicate and Communicate. Drive Open Space Agility to solve the complex organizational problem. Involve all employees to solve the problem. Townhall, Hackathon, blogs, etc approaches are needed to bring changes.

Long Term Goal

• Organizational structural change to create one team concept. Few % capacity reserve for core development, % capacity reserve for support activities. Have the mindset that everyone is capable of doing most of the work. Reskill people to that level.
• Look for organization architectural strategy. Look for eliminating monolithic architecture and more microservice architecture. Bring data and evidence to demonstrate that due to poor architecture team members are spending a lot of time in IT operations. Architect for better operation. Architect for security. Architect system for resiliency.
• Reward and recognize people with rapid value delivery by bringing all the changes which enable organization and teams to do that. (Change the people’s thought, change the process steps, change the tools and infra to achieve this)
• Coaching to adopt a One team mindset. Coaching to take ownership mindset. Coaching for overcoming silos mindset.
• Maintain optimum team loading (Most of the time we have seen support team members are working for many applications, chances are there to make many mistakes in these scenarios)
• Maximize automation wherever feasible, build expertise for automation, train team members. Procure automated tools wherever feasible. Minimize human dependency and manual work wherever feasible. Create a long-term plan on automation.
• Look for the opportunity to improve team ownership, team commitment, with less management (work smart). Improve those through the Kaizen approach.
• Involve the support team to know more about the upcoming dev features that are coming so that the features that are in production no longer surprise them. The Product Management team should frequently sync up with the IT support team to know more about each other’s world better.
• What works today will not help us to stay relevant, so what new concepts, policies, and process innovations should we should try? Speed and Quality both have to be achieved.
• Auto monitoring, auto rollback, auto mail trigger, auto-deployment (Reliability, Security, and stability) is the key.
• Come out with Metrics & Measurement which helps the teams to reflect the real-time system inflow and outflow status. Educate people to use the data in various governance meetings.
• Identify and coach the people who are not aligned with the Modern Product Development process and are more comfortable with the traditional product development process. IT operation (to prevent frequent changes to production systems, not enable them) lets us change those approaches.
• Create a Learning organization by encouraging people to share knowledge, award them, and recognize the team people who are performing remarkable work. Let us add goals into each individual’s target to achieve these learning and sharing attributes.
• Build Capability (People capability, Process Capability…… so on). Measure the maturity of the capability building.
• Push for infra as code automation. We may get rid of application-specific engineering and production support team members.
• New Agile role creation e.g., Site reliability engineers, Automation architects, etc., outdated some of the traditional roles (e.g., Incident manager, Service level agreement manager) update HR policy if required to accommodate those changes.
• Release on Demand: the system is ready to release at any given point of time by the people and it will just work fine.

Coaching Strategy

• Conduct the Team Agility Maturity assessment, Enterprise Agility Maturity. Identify the gaps in knowledge and skills (Start with High priority teams/High-Value teams).
• Creating the right ecosystem for success—Conduct Value Stream—Create Roadmap for Transformation—Establish Governance to measure progress.
• Run a discovery workshop to explore the business problem—Current customer/stakeholder experience, plan to improve the same (Process design), MVP creation, risk, issue, and dependencies, etc.
• Come out with a plan & proposal for training, mentoring, and coaching strategies based on the gap analysis done.
• Train and coach teams on CALMS (Culture, Automation, Lean-Flow, Measurement, Share).
• Look for Team Dysfunction Behaviour and coach to improve those.
• Reformulate some of the team metrics. Define team success criteria.
• Reformulate some of the team’s roles and responsibilities. Reskill the impacted people due to Transformation and Change.
• Coach to establish a cross-skilled self-driven team. Help them to build their backlog.
• Train and coach teams to look into the Automation Strategy.
• Train and coach team to look for Architectural strategy (legacy applications, modernization of those applications, etc.).
• Train and Coach teams to look for Testing Strategy (Agile Testing Quadrant).
• Coach the team into applying Scrum and Kanban in a different situation at IT Operation and IT service context
• Look into the IT Service Strategy, Service Design, and other phases where Agility can apply.
• Make Leaders as Lean-Agile Leaders so that they redesign the whole system in a more Agile way. Leadership to drive all the changes to create new Culture.
• Coaching for continuous delivery and on-demand release (Work on the cultural change these changes to happen)
• Teach Design thinking techniques so that team can design services and customer experiences, create new service concepts, identify customer opportunities, etc.

Work with the Leadership team to achieve

• Increased frequency, quality, and security of product innovation.
• Decreased deployment risk with increased learning cycles.
• Accelerated solution time-to-market.
• Improved solution quality and reduced lead time for fixes.
• Reduced severity and frequency of failures and defects.
• Improved Mean Time to Recover (MTTR) from production incidents.

In transformation, most of the steps are emergent. It works well when all the team members exhibit a growth mindset trait which means we discover better ways of working as and when we progress.

Transformations could be a Bottom-up and Top-down approach, the transformation could also be incremental with many small pilots or transformation could also be a big bang approach. It is the context that drives the decision making what works best to the scenarios.