How to add a Validation Operator¶
This guide will help you add a new instance of a Validation Operator. Validation Operators give you the ability to encode business logic around validation, such as validating multiple batches of data together, differentiating between warnings and errors, and kicking off actions based on the results of validation.
As a general rule, Validation Operators should be invoked from within Checkpoints. Separating out the configuration for Validation Operators and Checkpoints can help make Operator code reusable.
Prerequisites: This how-to guide assumes you have already:
Set up a working deployment of Great Expectations.
Created at least one Expectation Suite.
Created at least one Checkpoint. You will need it in order to test that your new Validation Operator is working.
The snippet below shows a portion of your
great_expectations.yml configuration after you perform the following steps. The steps will explain each line in this snippet.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
validation_operators: action_list_operator: class_name: ActionListValidationOperator action_list: - name: store_validation_result action: class_name: StoreValidationResultAction - name: store_evaluation_params action: class_name: StoreEvaluationParametersAction - name: update_data_docs action: class_name: UpdateDataDocsAction # Next Validation Operator was added manually my_second_validation_operator: class_name: ActionListValidationOperator action_list: - name: store_validation_result action: class_name: StoreValidationResultAction
Find the validation_operators section in your great_expectations config file.
Open your project’s
great_expectations.ymlconfiguration file and navigate to the
validation_operatorssection (line 1 in the snippet). This section contains the Validation Operator instance
action_list_operatorthat was automatically created by the
great_expectations initCLI command.
Add a new Validation Operator block.
Add a new block after the existing Validation Operator instances. The name of the block is the name you are giving to the new Validation Operator instance (line 15 in the snippet). These names must be unique within a project.
Pick a class_name.
class_nameattribute in the new block you added in the previous step (line 16 in the snippet). The value is the name of the class that implements the Validation Operator that you are adding. This can be one of the classes that are included in Great Expectations or a class that you implemented. This example adds another instance of
If you are adding a custom Validation Operator, you will have to add a
module_nameattribute in addition to
class_name. You will find more details about custom Validation Operators in this guide.
Configure additional fields.
Consult the reference documentation of the class that implements the Validation Operator you are adding for additional properties (required or optional) that are specific to that class. The snippet above configured one such property specific to the ActionListValidationOperator class.
Test your configuration.
Test that your new Validation Operator is configured correctly:
Open the configuration file of a Checkpoint you created earlier and replace the value of
validation_operator_namewith the value from Step 2 above. The details of Checkpoint configuration can be found in this guide.
Two Validation Operator classes are currently shipped with Great Expectations:
ActionListValidationOperatorinvokes a configurable list of actions on every Validation Result validation result. Firing a Slack notification and updating Data Docs are examples of these actions.
WarningAndFailureExpectationSuitesValidationOperatorextends the class above and allows to group Expectation Suites into two groups - critical and warning.