Compliance Automated Standard Solution (COMPASS), Part 7: Compliance-to-Policy for IT Operation Policies Using Auditree

(Note: A list of links for all articles in this series can be found at the conclusion of this article.)

In Part 4 of this multi-part series on continuous compliance, we presented designs for Compliance Policy Administration Centers (CPAC) which are typically part of larger platforms known in the industry under various names such as Cloud-Native Application Protection Platform (CNAPP), Cloud Security Posture Management (CSPM), Cloud Workload Protection Platforms (CWPP), or Cloud Infrastructure Entitlement Management (CIEM), bundled into those platforms to facilitate the management of the compliance artifacts connecting the Regulatory Policies expressed programmatically as Compliance-as-Code with technical policies implemented as Policy-as-Code. The separation of Compliance-as-Code and Policy-as-Code is purposeful, as different personas (see Part 1) need to independently manage their respective responsibilities according to their expertise; e.g., compliance controls and parameters selection, crosswalks mapping across regulations for compliance and auditor experts, or runtime evidence collectors and checks implementation for code developers or security focals.

The CPAC enables users to deploy and run their policy checks according to different Regulatory Policies on different Policy Validation Points (PVPs) and, depending upon the level of generality or specialty of the inspected systems, the CPAC also performs specific normalization and aggregation transformations. We introduced three different designs for CPAC: two for handling specialized PVPs with their dedicated policy languages and native interfaces, and one for orchestrating diverse PVPs across heterogeneous IT stack levels and cloud service configurations.

In this blog, we present an example implementation of a CPAC whose PVP specializes in automating complex IT operational procedures, for instance as required by regulations (e.g., SOC2, DORA) on managing a Cloud infrastructure and Financial Services. Our CPAC design was introduced in section 3 of COMPASS Part 4. We showed how we flexibly allow the extension of our CPAC through plugins to any specific PVP, while here we focus on a specific PVP, namely Auditree. We describe how our CPAC can bridge compliance artifacts, produced using the NIST OSCAL Compliance-as-Code standard-based Agile Authoring methodology, to policy assessment results produced using Auditree. This bridging is the key enabler of end-to-end continuous compliance, through the linking of controls to technical policies and rules to evidence and assessment results. The final result is the aggregated regulatory compliance posture for the whole environment.

Using Auditree to Manage Operational Policies 

When it comes to operational processes, implementing clear policies can improve availability, streamline the audits, and build trust with the customers. These policies define the exact systems and processes (e.g., asset management, backup, business continuity, software development life cycle, vendor management, vulnerability management, etc.) that need to be in place in order to be compliant with a variety of regulatory programs. Best practices suggest keeping the policy regulation agnostic and reusing them and their evidence and assessments across the programs to foster scalability and ease the introduction of new programs. 

We assume the compliance artifacts containing the controls of the applicable regulatory program(s), the ensuing policies and rules applicable to the operational processes, and the mapping to the associated assessment checks have been authored and approved using our open-source Trestle-based compliance Agile Authoring supply chain, and registered with the CPAC for production usage. Now the challenge is how to deal with the runtime policy assessment checks execution and integration and aggregation of their assessment results back with the compliance artifacts.

Implementing such operational policy assessment checks is typically done using imperative programmatic languages, commonly Python and JavaScript, given the complexity of the validation compared to the much simpler configuration policies that benefit from being implemented using declarative programmatic languages such as those presented in Part 6. Auditree employs a structured Python project format to fetch the current states from target systems, compares them against the desired states, and outputs the assessment results in a proprietary format. These results are automated using a GitOps approach. In our scenario, the policies are maintained in a Git repository, and the execution runtime is triggered by Git events, such as commits and Pull Requests, or by a scheduler, such as cron jobs.

Using OSCAL Agile Authoring to Manage Compliance-as-Code

As described in Part 3, our Trestle-based Agile Authoring tool provides a mechanism for various personas to author and manage their compliance artifacts in Git repos using an interface of their choice and the underlying tooling and automation converts them to the machine-readable OSCAL format. 

Let's take an example where security focals need to check and validate compliance of GitHub service with NIST 800-53 and/or ISO27001 regulations. For this, they first either import the OSCAL format of these regulations (if already available) in Agile Authoring Git repos for use, or they can use the source format such as xlsx, csv, etc., and use Agile Authoring automation to convert them to OSCAL Catalog format. 

Next, they need to identify which controls from these regulations they need their GitHub service to be compliant with. As the control requirements are specified at a high level, the security focals need to identify technology-specific rules for the GitHub service for those controls. For this, the security focals need to create a Component Definition and specify the mapping for these rules to controls in the regulations. For example, if the security focal selects account management-related controls (AC-2 in NIST 800-53, or A.9.2 in ISO27001), they need to “translate” the control high-level requirements into one or more technology-specific rules to express compliance in terms of requirements that can be implemented and validated against the technology or processes. 

The third row in Figure 1 shows the rule for members in the GitHub org and the corresponding regulation control it represents. Security focal can also define parameters specific to the rule. For example, if we need to apply the rule to specific organization names, we define that argument as a rule parameter. 

Similarly, the operational team implementing the checks for validating these rules needs to specify the PVP (Auditree in this case) that will be used and the check IDs of the corresponding rules validation scripts, as shown in Figure 2 below. The semantics of the check IDs depend on the PVP. We will explain the details of the case of Auditree in the next section. These two mappings from control to rule, and rule to check help in building the posture at control level when the check results become available.

These operational compliance rules and their associated checks IDs can be captured by the team in one or multiple CSV spreadsheet(s) (as shown above) and the Agile Authoring automation will convert them to machine-readable OSCAL format.

The checks in the case of the operational policy use-case considered here are not simply configuration parameter values setting. They invoke a complex logic for which an imperative policy like Auditree check (Python code) is required. In the above example, first, the members of all the organizations specified need to be fetched. Then for each organization, their count needs to be verified whether the number is above a threshold. If even one organization does not meet this threshold, the rule will fail. This kind of check is more cumbersome to be specified in a declarative policy.

After the Catalog and Component Definition authoring, the next step is to deploy the policy checks for the specific PVP(s) and perform the assessment scan. Once the individual results are generated, the normalized aggregated posture at the regulation level can be calculated. The next sections exemplify these last steps that bridge the compliance domain to the policy domain for Auditree PVP via our Compliance-to-Policy tool.

Compliance-to-Policy for End-To-End Compliance Automation Enablement

To bridge the Compliance-as-Code supply chain to the Policy-as-Code execution at runtime to validate the compliance status of the environment, and back to the Compliance-as-Code for aggregation at the control posture (remember, PVPs are typically, and recommended to be control agnostic) requires first retrieving the applicable policies and their parameter values from the Compliance-as-Code artifacts (e.g., OSCAL Component Definition) as input for the PVP evaluation, and second, using the PVP results output to aggregate the overall posture. We achieve this using our Compliance-to-Policy (C2P) pluggable tool. Below, we will show the steps needed to implement a compliance-to-policy plugin for Auditree in our C2P tool.

For creating a C2P plugin for Auditree, we first need to know its interface. Auditree uses a structured Python project consisting of "fetchers" and "checks" directories, control.json, and auditree.json (see the official Auditree demonstration project). 

The "fetchers" and "checks" directories contain Python codes to be executed by the Auditree framework. The "fetchers" code is responsible for fetching the current states from the target systems and the "checks" code is responsible for comparing them against the desired states described in the rule parameter. For example, in the demonstration project, fetch_github.py retrieves members from a GitHub organization, where the names of the GitHub org are provided by a parameter (parameters are managed in the auditree.json file, which is described hereafter). The fetched data is automatically stored on a local disk or another GitHub repository as evidenced by the Auditree Framework. The test_github.py uses the fetched results to check the rule; i.e., that the member list of each organization is not empty or below a threshold (the desired state). The controls.json file contains the list of all the checks as Python code module path and their associated accreditations (used to execute the checks). The auditree.json file contains the global configuration properties such as the evidence locker repo URL, and also the parameter values for each rule/check pair. The output results of these checks are stored by Auditree in the evidence locker in a proprietary format as shown below (compliance-to-policy/plugins_public/tests/data/auditree /check_results.json):

{
 "demo_examples.checks.test_github.GitHubOrgs": {
   "accreditations": [
     "demo.custom.accred"
   ],
   "checks": {
     "test_members_is_not_empty_0_nasa": {
       "failures": {},
       "failures_count": 0,
       "status": "pass",
       "successes": {},
       "successes_count": 0,
       "timestamp": 1717294709.394064,
       "warnings": {},
       "warnings_count": 0
     },
     "test_members_is_not_empty_1_esa": {
       "failures": {},
       "failures_count": 0,
       "status": "pass",
       "successes": {},
       "successes_count": 0,
       "timestamp": 1717294709.394502,
       "warnings": {},
       "warnings_count": 0
     }
   },
   "evidence": [],
   "reports": {
     "reports/github/members.md": ""
   }
 },
"demo_examples.checks.test_github.GitHubAPIVersionsCheck": {
   "accreditations": [
     "demo.custom.accred"
   ],
   "checks": {
     "test_supported_versions": {
       "failures": {},
       "failures_count": 0,
       "status": "pass",
       "successes": {},
       "successes_count": 0,
       "timestamp": 1717294709.257631,
       "warnings": {},
       "warnings_count": 0
     }
   },
   "evidence": [
     {
       "commit_sha": "f131629de4207886db21e5f94cf68ea88d00da52",
       "description": "Supported GitHub API versions",
       "last_update": "2024-06-02T02:15:10.557205",
       "locker_url": null,
       "path": "raw/github/api_versions.json",
       "ttl": 86400
     }
   ],
   "reports": {
     "reports/github/api_versions.md": "Supported GitHub versions report."
   }
 }, …
}

Given the way the checks are specified in the controls.json file using the full path of the test method including the module path, it makes sense to use the same as "Check_Id" in the Component Definition. The entire component definition is available in component-definition.csv and the corresponding OSCAL format in component-definition.json.

Next, we will show how to implement the C2P plugin for Auditree to bridge between the Compliance-as-Code artifacts and the Policy-as-Code artifacts for Auditree. The plugin is required to take the compliance artifacts (e.g., Component Definition) and configure the corresponding policy (Auditree fetchers and checks in this case). It is also expected to deploy and execute policy and collect results corresponding to the rules/checks (rule_github_org_member and rule_github_api_version) from Auditree check output (check_results.json). In the Compliance-to-Policy (C2P) framework, this function is implemented in the method "generate_pvp_result" of the plugin interface. 

The plugin is also expected to put the parameter names and their values from the Component Definition, such as "org.gh.orgs" in the auditree.json file. 

If one needs to set the parameter ID to an arbitrary name, an additional column can be added in Component Definition to map the parameter ID to the JSON path in auditree.json and the plugin should substitute the value in auditree.json with that parameter value. The code for this functionality is implemented in the method "generate_pvp_policy" (see a reference implementation). 

The plugin is initialized with two parameters: 1) the file path of the auditree.json template, and 2) the output file path for the generated auditree.json file. C2P takes the OSCAL Component Definition and generates an auditree.json file in the output path as shown in sample usage for compliance-to-policy. Then, a user or a GitOps process triggers the execution of the Auditree tool with this auditree.json file. which outputs the check results (check_results.json). C2P takes the check_results.json, aggregates each test result, and outputs Compliance Assessment Results as shown in sample usage for result-to-compliance.

Demo of the End-to-End Workflow for Auditree With C2P Plugin

Below we provide steps for executing the end-to-end workflow on your local machine.

Prerequisites

• Python version 11.x or earlier
• Install Python packages for Auditree and Compliance-to-Policy
  • auditree-framework 2.0.0
    • pip install auditree-framework==2.0.0
  • auditree-arboretum 0.17.1
    • pip install auditree-arboretum==0.17.1
  • compliance-to-policy 0.4.0
    • pip install compliance-to-policy==0.4.0

Demonstration Steps

1. Clone the auditree-framework and go to the demo directory (see also the Quick Start guide). 

git clone --depth 1 --branch v2.0.0 https://github.com/ComplianceAsCode/auditree-framework.git

2. Clone compliance-to-policy repository and install C2P.

git clone --depth 1 --branch v0.4.0 https://github.com/oscal-compass/compliance-to-policy.git

3. Prepare OSCAL Component Definition:

sed 's/nasa/oscal-compass/g' ./compliance-to-policy/plugins_public/tests/data/auditree/component-definition.json > ./component-definition.json

4. Generate auditree.json by C2P:

python ./compliance-to-policy/samples_public/auditree/compliance_to_policy.py -i ./auditree-framework/demo/auditree_demo.json -c ./component-definition.json -o ./auditree-framework/demo/auditree.json

5. Execute Auditree.

• cd auditree-framework/demo
• compliance --fetch --evidence local -C auditree.json -v

  • You'll see the path to the local evidence locker directory in the log.

    Shell

     

    $ compliance --check demo.arboretum.accred,demo.custom.accred --evidence local -C auditree.json -v
    INFO: Using locker found in /var/folders/yx/1mv5rdh53xd93bphsc459ht00000gn/T/compliance...
    ...

    
    

  • Please set the environment variable "LOCKER_PATH" to the directory path displayed in the log (for example in the above case, export LOCKER_PATH=/var/folders/yx/1mv5rdh53xd93bphsc459ht00000gn/T/compliance.)

• compliance --check demo.arboretum.accred,demo.custom.accred --evidence local -C auditree.json -v
• cd ../..

6. Generate OSCAL Assessment Result by C2P (please replace the path to check_results.json with your local evidence locker directory shown in Auditree command output above).

python ./compliance-to-policy/samples_public/auditree/result_to_compliance.py -i ${LOCKER_PATH}/check_results.json -c ./component-definition.json > assessment-results.json

7. OSCAL Assessment Results is not a human-readable format. You can use a quick viewer in the C2P tool to display it as markdown as shown in Figure 4.

c2p tools viewer -ar assessment-results.json -cdef ./component-definition.json > assessment-results.md

Conclusion

In this blog, we provided details on implementing the functionality of a Compliance and Policy Administration Center for IT operations compliance that requires imperative policies and with an example C2P plugin implementation for Auditree. We showed how the regulatory controls can be linked to the policy assessments via a plugin implemented in our Compliance-to-Policy tool, and demonstrated its end-to-end flow with an example for Auditree. 

Coming Next

The next post will provide details on implementing a CPAC for the Ansible Playbook with the Ansible Automation Platform (AAP) as PVP. We will also provide details of the plugin implementation and showcase the artifacts used with an example.

Learn More

If you would like to use our C2P tool, see the compliance-to-policy GitHub project (linked earlier in the article). For our open-source Trestle SDK see compliance-trestle to learn about various Trestle CLIs and their usage. For more details on the markdown formats and commands for authoring various compliance artifacts see this tutorial from Trestle.

Below are the links to our other articles in this series:

• Compliance Automated Standard Solution (COMPASS), Part 1: Personas and Roles
• Compliance Automated Standard Solution (COMPASS), Part 2: Trestle SDK
• Compliance Automated Standard Solution (COMPASS), Part 3: Artifacts and Personas 
• Compliance Automated Standard Solution (COMPASS), Part 4: Topologies of Compliance Policy Administration Centers
• Compliance Automated Standard Solution (COMPASS), Part 5: A Lack of Network Boundaries Invites a Lack of Compliance
• Compliance Automated Standard Solution (COMPASS), Part 6: Compliance to Policy for Multiple Kubernetes Clusters