How to create a new Expectation Suite without a sample Batch

This guide shows how to create an Expectation Suite without a sample Batch.

Here are some reasons why you may wish to do this but please contact us if there are other use cases we have not yet considered:

  1. You don’t have a sample.

  2. You don’t currently have access to the data to make a sample.

  3. You know exactly how you want your Expectations to be configured.

  4. You want to create Expectations parametrically (you can also do this in interactive mode).

  5. You don’t want to spend the time to validate against a sample.

Steps

Show Docs for V2 (Batch Kwargs) API

Prerequisites: This how-to guide assumes you have already:

  1. Load your Data Context

    This code is very similar to the boilerplate you see after creating an Expectation Suite using the CLI, with the only difference being that the Expectation Suite is created not loaded from the Data Context:

    import great_expectations as gx
    
    context = gx.data_context.DataContext()
    suite = context.create_expectation_suite(
        "my_suite_name", overwrite_existing=True # Configure these parameters for your needs
    )
    

    This block just creates an empty Expectation Suite object. Next up, start creating your Expectations.

  2. Create Expectation Configurations in your notebook

    You are adding Expectation configurations to the suite. Since there is no sample Batch of data, no validation happens during this process. To illustrate how to do this, consider a hypothetical example. Suppose that you have a table with the columns account_id, user_id, transaction_id, transaction_type, and transaction_amt_usd. Then the following code snipped adds an expectation that the columns of the actual table will appear in the order specified above:

    # Create an Expectation
    expectation_configuration = ExpectationConfiguration(
        # Name of expectation type being added
        expectation_type="expect_table_columns_to_match_ordered_list",
        # These are the arguments of the expectation
        # The keys allowed in the dictionary are Parameters and
        # Keyword Arguments of this Expectation Type
        kwargs={
            "column_list": [
                "account_id", "user_id", "transaction_id", "transaction_type", "transaction_amt_usd"
            ]
        },
        # This is how you can optionally add a comment about this expectation.
        # It will be rendered in Data Docs.
        # See this guide for details:
        # `How to add comments to Expectations and display them in Data Docs`.
        meta={
            "notes": {
                "format": "markdown",
                "content": "Some clever comment about this expectation. **Markdown** `Supported`"
            }
        }
    )
    # Add the Expectation to the suite
    suite.add_expectation(expectation_configuration=expectation_configuration)
    

    Here are a few more example expectations for this dataset:

    expectation_configuration = ExpectationConfiguration(
        expectation_type="expect_column_values_to_be_in_set",
        kwargs={
            "column": "transaction_type",
            "value_set": ["purchase", "refund", "upgrade"]
        },
        # Note optional comments omitted
    )
    suite.add_expectation(expectation_configuration=expectation_configuration)
    
    
    
    expectation_configuration = ExpectationConfiguration(
        expectation_type="expect_column_values_to_not_be_null",
        kwargs={
            "column": "account_id",
            "mostly": 1.0,
        },
        meta={
            "notes": {
                "format": "markdown",
                "content": "Some clever comment about this expectation. **Markdown** `Supported`"
            }
        }
    )
    suite.add_expectation(expectation_configuration=expectation_configuration)
    
    
    
    expectation_configuration = ExpectationConfiguration(
        expectation_type="expect_column_values_to_not_be_null",
        kwargs={
            "column": "user_id",
            "mostly": 0.75,
        },
        meta={
            "notes": {
                "format": "markdown",
                "content": "Some clever comment about this expectation. **Markdown** `Supported`"
            }
        }
    )
    suite.add_expectation(expectation_configuration=expectation_configuration)
    

    You can see all the available Expectations in the Glossary of Expectations.

  3. Save your Expectation Suite

    Run this in a cell in your notebook:

    context.save_expectation_suite(suite, expectation_suite_name)
    

    This will create a JSON file with your Expectation Suite in the Store you have configured, which you can then load and use for Validation.

Show Docs for V3 (Batch Request) API

Prerequisites: This how-to guide assumes you have already:

  1. Use the CLI to generate a helper notebook

    From the command line, run:

    great_expectations --v3-api suite new
    
  2. Create Expectation Configurations in the helper notebook

    You are adding Expectation configurations to the suite. Since there is no sample Batch of data, no validation happens during this process. To illustrate how to do this, consider a hypothetical example. Suppose that you have a table with the columns account_id, user_id, transaction_id, transaction_type, and transaction_amt_usd. Then the following code snipped adds an expectation that the columns of the actual table will appear in the order specified above:

    # Create an Expectation
    expectation_configuration = ExpectationConfiguration(
        # Name of expectation type being added
        expectation_type="expect_table_columns_to_match_ordered_list",
        # These are the arguments of the expectation
        # The keys allowed in the dictionary are Parameters and
        # Keyword Arguments of this Expectation Type
        kwargs={
            "column_list": [
                "account_id", "user_id", "transaction_id", "transaction_type", "transaction_amt_usd"
            ]
        },
        # This is how you can optionally add a comment about this expectation.
        # It will be rendered in Data Docs.
        # See this guide for details:
        # `How to add comments to Expectations and display them in Data Docs`.
        meta={
            "notes": {
                "format": "markdown",
                "content": "Some clever comment about this expectation. **Markdown** `Supported`"
            }
        }
    )
    # Add the Expectation to the suite
    suite.add_expectation(expectation_configuration=expectation_configuration)
    

    Here are a few more example expectations for this dataset:

    expectation_configuration = ExpectationConfiguration(
        expectation_type="expect_column_values_to_be_in_set",
        kwargs={
            "column": "transaction_type",
            "value_set": ["purchase", "refund", "upgrade"]
        },
        # Note optional comments omitted
    )
    suite.add_expectation(expectation_configuration=expectation_configuration)
    
    
    
    expectation_configuration = ExpectationConfiguration(
        expectation_type="expect_column_values_to_not_be_null",
        kwargs={
            "column": "account_id",
            "mostly": 1.0,
        },
        meta={
            "notes": {
                "format": "markdown",
                "content": "Some clever comment about this expectation. **Markdown** `Supported`"
            }
        }
    )
    suite.add_expectation(expectation_configuration=expectation_configuration)
    
    
    
    expectation_configuration = ExpectationConfiguration(
        expectation_type="expect_column_values_to_not_be_null",
        kwargs={
            "column": "user_id",
            "mostly": 0.75,
        },
        meta={
            "notes": {
                "format": "markdown",
                "content": "Some clever comment about this expectation. **Markdown** `Supported`"
            }
        }
    )
    suite.add_expectation(expectation_configuration=expectation_configuration)
    

    You can see all the available Expectations in the Glossary of Expectations.

  3. Save your Expectation Suite

    Run the final cell in the helper notebook to save your Expectation Suite.

    This will create a JSON file with your Expectation Suite in the Store you have configured, which you can then load and use for Validation.