» Testing

Sentinel has a built-in test framework to validate a policy behaves as expected.

With an ever-increasing amount of automation surrounding technology, the guardrails provided by policies are a critical piece towards ensuring expected behavior. As a reliance on correct policy increases, it is important to test and verify policies.

Testing is a necessary step to fully realize policy as code. Just as good software is well tested, a good set of policies should be equally well tested.

» Test Folder Structure

Policies are tested by asserting that rules are the expected values given a pre-configured input. Tests are run by executing the test command.

Sentinel is opinionated about the folder structure required for tests. This opinionated structure allows testing to be as simple as running sentinel test with no arguments. Additionally, it becomes simple to test in a CI or add new policies.

The structure Sentinel expects is test//*.json where is the name of your policy file without the file extension. Within that folder is a list of JSON files. Each JSON file represents a single test case. Therefore, each policy can have multiple tests associated with it.

» Test Case Format

Each JSON file within the test folder for a policy is a single test case.

The JSON file is the same configuration format as the apply configuration file. The format lets you define mock data, imports to use, and more. This mock data is the key piece in being able to test policies: you craft a specific scenario and assert your policy behaves as you expect.

Test cases also use the special key test within the configuration file to assert the boolean value of rules. If the test key is omitted, the policy is expected to pass (the same as asserting that the main rule is true). If the test key is specified, only the rules specified in the map will be asserted. This means if you omit main, then the final policy result is not asserted.

Example with assertions:

{
    "global": {
        "day": "monday",
        "hour": 7
    },

    "test": {
        "main": false,
        "is_open_hours": false,
        "is_weekday": true
    }
}

The configuration above specifies some mock global data, and asserts the result of some rules. This is the same configuration used in the example section below.

» Example

Lets use the following file as an example. Save this file to a directory and name it policy.sentinel. It can be named anything with the sentinel extention, but by naming it policy.sentinel your output should match the example output on this page.

is_weekday = rule { day not in ["saturday", "sunday"] }
is_open_hours = rule { hour > 8 and hour < 17 }
main = rule { is_open_hours and is_weekday }

» A Passing Test

Next, let's define a single test case. Relative to where you saved the policy, create a file at the path test/policy/good.json.

{
    "global": {
        "day": "monday",
        "hour": 14
    }
}

Now run sentinel test:

$ sentinel test
PASS - policy.sentinel
  PASS - test/policy/good.json

This passed because the policy passed. We didn't assert any specific rules. By not specifying any assertions, test expects the policy itself to fully pass.

» A Failing Test

Define another test case to fail. We want to verify our policy fails when expected, too.

Save the following as test/policy/7-am.json:

{
    "global": {
        "day": "monday",
        "hour": 7
    }
}

Now run sentinel test:

$ sentinel test
FAIL - policy.sentinel
  FAIL - test/policy/7-am.json
    expected "main" to be true, got: false

    trace:
      FALSE - policy.sentinel:3:1 - Rule "main"
        FALSE - policy.sentinel:3:15 - is_open_hours
          FALSE - policy.sentinel:2:24 - hour > 8 and hour < 17
            FALSE - policy.sentinel:2:24 - hour > 8

      FALSE - policy.sentinel:2:1 - Rule "is_open_hours"
        FALSE - policy.sentinel:2:24 - hour > 8
  PASS - test/policy/good.json

As you can see, the test fails because "main" is false. This is good because the policy should have failed since we specified an invalid hour. But, we expect main to be false and don't want our test to fail! Update 7-am.json to add test assertions:

{
    "global": {
        "day": "monday",
        "hour": 7
    },

    "test": {
        "main": false,
        "is_open_hours": false,
        "is_weekday": true
    }
}

And when we run the tests:

$ sentinel test
PASS - policy.sentinel
  PASS - test/policy/7-am.json
  PASS - test/policy/good.json

The test passes. We asserted that we expect the main rule to be false, the is_open_hours rule to be false, and the is_weekday rule to be true. By asserting some rules are true, we can verify that our policy is failing for reasons we expect.

» Test Automation

The sentinel test command was designed to be automation friendly. We encourage you to enable a CI such as Travis CI on your policy repositories to continuously run tests. An example .travis.yml configuration file is shown below:

language: bash
script: sentinel test