Skip to main content
Version: 0.18.21

Manage Expectations

An Expectation is a verifiable assertion about your data. They make implicit assumptions about your data explicit, and they provide a flexible, declarative language for describing expected behavior. They can help you better understand your data and help you improve data quality. An Expectation Suite contains multiple Expectations.

Custom SQL Query Expectations

To create custom SQL query Expectations, you'll need to use the GX API. See Create a Custom Query Expectation.

To learn more about Expectations, see Expectation.

Prerequisites

Available Expectations

The following table lists the available GX Cloud Expectations.

Data Quality IssueExpectationDescription
Cardinalityexpect_column_values_to_be_uniqueExpect each column value to be unique.
Cardinalityexpect_compound_columns_to_be_uniqueExpect the compound columns to be unique.
Cardinalityexpect_select_column_values_to_be_unique_within_recordExpect the values for each record to be unique across the columns listed. Note that records can be duplicated.
Cardinalityexpect_column_proportion_of_unique_values_to_be_betweenExpect the proportion of unique values to be between a minimum value and a maximum value.
Cardinalityexpect_column_unique_value_count_to_be_betweenExpect the number of unique values to be between a minimum value and a maximum value.
Data Integrityexpect_column_pair_values_to_be_equalExpect the values in column A to be the same as column B.
Data Integrityexpect_multicolumn_sum_to_equalExpect that the sum of row values in a specified column list is the same for each row, and equal to a specified sum total.
Distributionexpect_column_pair_values_A_to_be_greater_than_BExpect the values in column A to be greater than column B.
Distributionexpect_column_values_to_be_betweenExpect the column entries to be between a minimum value and a maximum value (inclusive).
Distributionexpect_column_z_scores_to_be_less_thanExpect the Z-scores of a column's values to be less than a given threshold.
Distributionexpect_column_stdev_to_be_betweenExpect the column standard deviation to be between a minimum value and a maximum value.
Distributionexpect_column_sum_to_be_betweenExpect the column sum to be between a minimum value and a maximum value.
Missingnessexpect_column_values_to_be_nullExpect the column values to be null.
Missingnessexpect_column_values_to_not_be_nullExpect the column values to not be null.
Numerical Dataexpect_column_max_to_be_betweenExpect the column maximum to be between a minimum and a maximum value.
Numerical Dataexpect_column_mean_to_be_betweenExpect the column mean to be between a minimum and a maximum value (inclusive).
Numerical Dataexpect_column_median_to_be_betweenExpect the column median to be between a minimum and a maximum value.
Numerical Dataexpect_column_min_to_be_betweenExpect the column minimum to be between a minimum value and a maximum value.
Pattern matchingexpect_column_value_length_to_equalExpect the column entries to be strings with length between a minimum value and a maximum value (inclusive).
Pattern matchingexpect_column_value_length_to_be_betweenExpect the column entries to be strings with length between a minimum value and a maximum value (inclusive).
Pattern matchingexpect_column_values_to_match_like_patternExpect the column entries to be strings that match a given like pattern expression.
Pattern matchingexpect_column_values_to_match_like_pattern_listExpect the column entries to be strings that match any of a provided list of like pattern expressions.
Pattern matchingexpect_column_values_to_match_regexExpect the column entries to be strings that match a given regular expression.
Pattern matchingexpect_column_values_to_match_regex_listExpect the column entries to be strings that can be matched to either any of or all of a list of regular expressions.
Pattern matchingexpect_column_values_to_not_match_like_patternExpect the column entries to be strings that do NOT match a given like pattern expression.
Pattern matchingexpect_column_values_to_not_match_like_pattern_listExpect the column entries to be strings that do NOT match any of a provided list of like pattern expressions.
Pattern matchingexpect_column_values_to_not_match_regexExpect the column entries to be strings that do NOT match a given regular expression.
Pattern matchingexpect_column_values_to_not_match_regex_listExpect the column entries to be strings that do not match any of a list of regular expressions. Matches can be anywhere in the string.
Schemaexpect_column_to_existChecks for the existence of a specified column within a table.
Schemaexpect_column_values_to_be_in_type_listExpect a column to contain values from a specified type list.
Schemaexpect_column_values_to_be_of_typeExpect a column to contain values of a specified data type.
Schemaexpect_table_column_count_to_be_betweenExpect the number of columns in a table to be between two values.
Schemaexpect_table_column_count_to_equalExpect the number of columns in a table to equal a value.
Schemaexpect_table_columns_to_match_ordered_listExpect the columns in a table to exactly match a specified list.
Schemaexpect_table_columns_to_match_setExpect the columns in a table to match an unordered set.
Setsexpect_column_values_to_be_in_setExpect each column value to be in a given set.
Setsexpect_column_values_to_not_be_in_setExpect column entries to not be in the set.
Setsexpect_column_distinct_values_to_be_in_setExpect the set of distinct column values to be contained by a given set.
Setsexpect_column_distinct_values_to_contain_setExpect the set of distinct column values to contain a given set.
Setsexpect_column_distinct_values_to_equal_setExpect the set of distinct column values to equal a given set.
Setsexpect_column_most_common_value_to_be_in_setExpect the most common value to be within the designated value set.
Volumeexpect_table_row_count_to_be_betweenExpect the number of rows to be between two values.
Volumeexpect_table_row_count_to_equalExpect the number of rows to equal a value.
Volumeexpect_table_row_count_to_equal_other_tableExpect the number of rows to equal the number in another table within the same database.

Add an Expectation

  1. In GX Cloud, click Data Assets.

  2. In the Data Assets list, click the Data Asset name.

  3. Click the Expectations tab.

  4. Click New Expectation.

  5. Select an Expectation type. See Available Expectations.

  6. If you are adding your first expectation on this data asset, you may be able to select a time-based Batch interval for that asset.

  • A batch is a feature of the data asset, and allows you to validate your data incrementally. A batch interval can only be defined once per data asset.

  • In order to be able to select a batch interval, the data asset must have at least one DATE or DATETIME column.

  • Select the Entire table tab to provide all Data Asset records to your Expectations and validations, or select the Yearly/Monthly/Daily tab to use subsets of Data Asset records for your Expectations and validations.

  • Select Yearly to partition Data Asset records by year, select Monthly to partition Data Asset records by year and month, or select Daily to partition Data Asset records by year, month, and day.

  • Batch column - Select a name column from a prefilled list of DATE and DATETIME columns containing the date and time data.

  1. Complete the mandatory and optional fields for the Expectation. A recurring validation schedule will be applied automatically to your Expectation, based on the settings of your Expectation Suite.

  2. Click Save or click Save & Add More and then repeat steps 4 and 5 to add additional Expectations.

  3. Optional. Run a Validation. See Run a Validation.

Edit an Expectation

  1. In GX Cloud, click Data Assets.

  2. In the Data Assets list, click the Data Asset name.

  3. Click the Expectations tab.

  4. Click Edit Expectations for the Expectation that you want to edit.

  5. Edit the Expectation configuration.

  6. Click Save.

View Expectation history

View the Expectation history to determine when an Expectation was changed and who made the change.

  1. In GX Cloud, click Expectation Suites.

  2. In the Expectation Suites list, click the Expectation Suite name.

  3. Click the Change Log tab.

  4. Optional. Select an Expectation in the Columns pane to view the change history for a specific Expectation.

    The date, time, and email address of the users who created, edited, or deleted the Expectation appears below the Expectation name. Strikethrough text indicates an Expectation was deleted.

Delete an Expectation

  1. In GX Cloud, click Data Assets.

  2. In the Data Assets list, click the Data Asset name.

  3. Click the Expectations tab.

  4. Click Delete Expectation for the Expectation you want to delete.

  5. Click Yes, delete Expectation.