Change management
Change management (CM) is the end-to-end process that governs the lifecycle of a change, from the initial request for change to the final deployment and communication of the change.
Using CM to govern changes in your Splunk environment can provide the following benefits:
- A documented, defined process to authorize and deploy changes in your Splunk environment
- A common and visible method to address changes
- Awareness of changes
- An ability to respond to issues by providing a record of change to support platform health and troubleshooting
As you consider which changes in your Splunk implementation warrant CM oversight, you can use the Change Management Framework to define the steps, activities, and the roles and responsibilities involved in making changes. Because every organization is unique, you might find that how you implement CM varies from these examples, but you can use the framework as a starting point.
Be aware of the differences between change management and change control. Change management encompasses the entire change lifecycle. Change control is a procedure within change management that focuses on specific steps and activities to ensure that releases and deployments do not conflict with production components. Critical and time-sensitive platform changes, such as security patches, often lead directly to change control. As a result, change control has a higher level of risk that organizations typically accept.
The Change Management Framework
The Change Management Framework helps you design and implement a change management process in your organization. The framework design allows for structure and flexibility so you can build a new CM process or build on an existing CM process.
The CM Framework consists of a number of different tasks to complete when building out your full change management process:
- Define a scope
- Define a process
- Develop a responsibility assignment (RACI) matrix
- Create CM pathways
You can use the Change Management Framework Template to guide you in completing these tasks.
Define a scope
You should have a common understanding of what type of changes are in the scope of your CM process. Here are some examples of common changes that are in- or out-of-scope of Splunk-related change management processes. You should use these examples as a starting point for conversations with the rest of your organization to determine what is appropriate for your organization.
In-scope
The scope of the CM process covers all production systems and platforms in your organization.
The primary functional components covered in the CM process can include the following examples:
- Changes handled through the formal software development lifecycle
- Installing, modifying, removing, or relocating computing hardware equipment
- Installing, patching, upgrading, or removing software products, including operating systems, access methods, commercial off-the-shelf (COTS) packages, internally developed packages, and utilities
- Changes to system configuration, including moves, adds, changes, and deletions
- Promoting application changes to production, integrating new application systems, and removing obsolete elements
- Changes to system configuration, including moves, adds, changes, and deletions
- Creating, deleting, or revising job schedules, back-up schedules, or other regularly scheduled jobs managed by the company's IT organization
In-scope changes are typically classified into several categories: Standard, Normal, and Emergency.
- Standard changes are low risk, easily repeatable, and follow existing documentation. These changes can often follow a streamlined path through CM based on predefined approvals. A Splunk-specific standard change may include something like a minor modification to a server class or app, pushed out from a Deployment Server.
- Normal changes would be most any other change that does not fall under the predefined Standard category. These changes have higher risk for impact and likely merit CM review processes. A Splunk-specific normal change may include something like a more customized app deployment directly to an Enterprise Search Head Cluster.
- Emergency changes are completed for urgent break/fix scenarios, and are expedited through a normal CM process.
Out-of-scope
Routine operations and the administration of your Splunk environment are outside the scope and governance of the CM process. Identify and document these activities as out-of-scope to maintain operational efficiency.
Out-of-scope operations can include the following examples:
- Contingency or disaster recovery
- Changes to non-production elements or resources, such as changes in test environment
- Changes under the governance of large-scale projects or initiatives that have a separate CM process
- Changes made within the daily administrative process
Out-of-scope daily administrative tasks can include the following examples:
- Password changes or resets
- User adds or deletes
- Adding, deleting, or revising security groups
- Rebooting machines when there is no change to the configuration of the system
- File permission changes
Define a process
After you have decided what types of changes are in- and out-of-scope for your organization, you'll need to start outlining what the process for handling these changes looks like. You should ensure that all changes, regardless of the type of change, follow a standardized process so changes can be handled quickly and efficiently.
The CM process is normally grouped into two distinct phases: the approval phase and the run phase.
You can use the CM pathways to select which CM process steps and activities you want to implement for a given change. Think of the CM process as the checklist for defining each change pathway.
Approval phase
In the approval phase you'll complete steps to decide whether the change should go ahead, and perform authorization activities prior to the change being implemented.
When defining your approval phase process, first review each step and its purpose. The screenshot below shows example Approval phase steps in the Change Management Framework Template that you can use as a starting point. Each step has associated activities that you'll define later while considering who will be responsible for each activity in the process.
Run phase
In the run phase you'll complete steps to build and deploy the change. In the same way you worked with the Approval phase steps, first review each Run step and its purpose. The screenshot below shows example steps in the Change Management Framework Template that you can use as a starting point. Each step has associated activities that you'll define later while considering who will be responsible for each activity in the process.
Develop a responsibility assignment (RACI) matrix
The Responsible, Accountable, Consulted, and Informed (RACI) matrix defines how different people complete activities in a change management (CM) process.
- Responsible: The individuals who do the work to complete the activity. At least one takes on the responsible role, although others can assist in the work required.
- Accountable: The individual who is accountable for the outcome. The accountable role ensures all prerequisites are met and delegates the work to the individual in the Responsible role. You should assign only one Accountable role for each activity.
- Consulted: The individuals whose opinions are sought. Individuals in the Consulted role are typically subject matter experts engaged in two-way communication.
- Informed: The individuals who are kept up to date about the task. Individuals in the Informed role are alerted to the completion of the activity, and generally engage in one-way communication.
A role and an individually identified person are two different things. A role has an associated set of tasks that can be performed by many people. An individually identified person can perform many roles. For example, an organization may have 10 people who can perform the role of system administrator, and one person who can perform the role of system administrator can also perform the role of developer. You should assign only one participation type for every activity per role in the CM pathway.
You can build out your RACI matrix using the Change Management Framework Template. Here is a screenshot of the template you can build on:
For more information, see Managing data based on role.
Define your change management pathways
A change management (CM) pathway shows the steps and activities from the CM process and combines these with the roles from the RACI matrix. You should define a number of default CM pathways to use for different scenarios. For example, changing a source type definition can have a significant impact on data ingestion, which can impact many business processes, so multiple parties need to analyze and approve this type of change. In contrast, a change to a dashboard panel doesn't require the same level of analysis or approval because it doesn't affect business processes.
CM pathways should provide the flexibility to address various change scenarios and improve overall efficiency. You should aim to define several CM pathways to accommodate all change categories in your organization. You should also consider defining a CM pathway for situations when a user submits a change request, but they are unsure of the change category or the request is out of the CM scope.
You can build out your CM pathway using the Change Management Framework Template. Here is a screenshot of the template you can build on:
Keep the quantity of CM pathways to a manageable number. Start with a smaller set and expand as you learn how your organization addresses change.
Do the following items to define your CM pathways:
- Use the CM process to identify the process steps for each change category and select the appropriate activities within each process step.
- Use the CM RACI matrix to define the roles and select the responsibilities for the participants for each change category.
- Identify your deployment requirements, for example Dev-Test-Prod promotion, and segregation of duties.
Timing can be critical to a business situation, so it's important to maintain a flexible CM framework. You need to be comfortable with temporarily altering an existing CM pathway to fit a business need instead of defining a new CM pathway. For example, to expedite a change request, you may need to streamline the approval phase to get to the run phase.