How to add a Validation Operator - V2 (Batch Kwargs) API

Note

This guide is only relevant if you are using the V2 (Batch Kwargs) API. For the V3 (Batch Request) API, please refer to the Checkpoints (>=0.13.12) tab in the guide on How to create a new Checkpoint.

This guide will help you add a new instance of a Validation Operator. Validation Operators give you the ability to encode business logic around validation, such as validating multiple batches of data together, differentiating between warnings and errors, and kicking off actions based on the results of validation.

As a general rule, Validation Operators should be invoked from within Checkpoints. Separating out the configuration for Validation Operators and Checkpoints can help make Operator code reusable.

Prerequisites: This how-to guide assumes you have already:

Steps

The snippet below shows a portion of your great_expectations.yml configuration after you perform the following steps. The steps will explain each line in this snippet.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
validation_operators:
  action_list_operator:
    class_name: ActionListValidationOperator
    action_list:
    - name: store_validation_result
      action:
        class_name: StoreValidationResultAction
    - name: store_evaluation_params
      action:
        class_name: StoreEvaluationParametersAction
    - name: update_data_docs
      action:
        class_name: UpdateDataDocsAction
  # Next Validation Operator was added manually
  my_second_validation_operator:
    class_name: ActionListValidationOperator
    action_list:
    - name: store_validation_result
      action:
        class_name: StoreValidationResultAction
  1. Find the validation_operators section in your great_expectations config file.

    Open your project’s great_expectations.yml configuration file and navigate to the validation_operators section (line 1 in the snippet). This section contains the Validation Operator instance action_list_operator that was automatically created by the great_expectations init CLI command.


  2. Add a new Validation Operator block.

    Add a new block after the existing Validation Operator instances. The name of the block is the name you are giving to the new Validation Operator instance (line 15 in the snippet). These names must be unique within a project.


  3. Pick a class_name.

    Add a class_name attribute in the new block you added in the previous step (line 16 in the snippet). The value is the name of the class that implements the Validation Operator that you are adding. This can be one of the classes that are included in Great Expectations or a class that you implemented. This example adds another instance of great_expectations.validation_operators.validation_operators.ActionListValidationOperator.

    Note:

    • If you are adding a custom Validation Operator, you will have to add a module_name attribute in addition to class_name. You will find more details about custom Validation Operators in this guide.

  4. Configure additional fields.

    Consult the reference documentation of the class that implements the Validation Operator you are adding for additional properties (required or optional) that are specific to that class. The snippet above configured one such property specific to the ActionListValidationOperator class.


  5. Test your configuration.

    Test that your new Validation Operator is configured correctly:

    1. Open the configuration file of a Checkpoint you created earlier and replace the value of validation_operator_name with the value from Step 2 above. The details of Checkpoint configuration can be found in this guide.

    2. Run the Checkpoint and verify that no errors are thrown. You can run the Checkpoint from the CLI as explained here or from Python, as explained here.

Additional notes

Two Validation Operator classes are currently shipped with Great Expectations:

Additional resources