How to configure a Validation Result store on a filesystem

By default, Validation results are stored in the uncommitted/validations/ directory. Since Validations may include examples of data (which could be sensitive or regulated) they should not be committed to a source control system. This guide will help you configure a new storage location for Validations on your filesystem.

Prerequisites: This how-to guide assumes that you have already:

  • Configured a Data Context.

  • Configured an Expectation Suite.

  • Configured a Checkpoint.

  • Determined a new storage location where you would like to store Validations. This can either be a local path, or a path to a secure network filesystem.

Steps

Show Docs for V2 (Batch Kwargs) API

  1. Configure a new folder on your filesystem where Validation results will be stored.

    Create a new folder where you would like to store your Validation results, and move your existing Validation results over to the new location. In our case, the name of the Validation result is npi_validations and the path to our new storage location is shared_validations/.

    # in the great_expectations/ folder
    mkdir shared_validations
    mv uncommitted/validations/npi_validations/ uncommitted/shared_validations/
    
  2. Identify your Data Context Validations Store

    In your great_expectations.yml, look for the following lines. The configuration tells Great Expectations to look for Validations in a store called validations_store. The base_directory for validations_store is set to uncommitted/validations/ by default.

    validations_store_name: validations_store
    
    stores:
        validations_store:
            class_name: ValidationsStore
            store_backend:
                class_name: TupleFilesystemStoreBackend
                base_directory: uncommitted/validations/
    
  3. Update your configuration file to include a new store for Validation results on your filesystem

    In the example below, Validations Store is being set to shared_validations_filesystem_store, but it can be any name you like. Also, the base_directory is being set to uncommitted/shared_validations/, but it can be set to any path accessible by Great Expectations.

    validations_store_name: shared_validations_filesystem_store
    
    stores:
        shared_validations_filesystem_store:
            class_name: ValidationsStore
            store_backend:
                class_name: TupleFilesystemStoreBackend
                base_directory: uncommitted/shared_validations/
    
  4. Confirm that the location has been updated by running great_expectations store list.

    Notice the output contains two Validation stores: the original validations_store and the shared_validations_filesystem_store we just configured. This is ok, since Great Expectations will look for Validations in the uncommitted/shared_validations/ folder as long as we set the validations_store_name variable to shared_validations_filesystem_store. The config for validations_store can be removed if you would like.

    great_expectations store list
    
    - name: validations_store
    class_name: ValidationsStore
    store_backend:
        class_name: TupleFilesystemStoreBackend
        base_directory: uncommitted/validations/
    
    - name: shared_validations_filesystem_store
    class_name: ValidationsStore
    store_backend:
        class_name: TupleFilesystemStoreBackend
        base_directory: uncommitted/shared_validations/
    

Show Docs for V3 (Batch Request) API

  1. Configure a new folder on your filesystem where Validation results will be stored.

    Create a new folder where you would like to store your Validation results, and move your existing Validation results over to the new location. In our case, the name of the Validation result is npi_validations and the path to our new storage location is shared_validations/.

    # in the great_expectations/ folder
    mkdir shared_validations
    mv uncommitted/validations/npi_validations/ uncommitted/shared_validations/
    
  2. Identify your Data Context Validations Store

    In your great_expectations.yml, look for the following lines. The configuration tells Great Expectations to look for Validations in a store called validations_store. The base_directory for validations_store is set to uncommitted/validations/ by default.

    validations_store_name: validations_store
    
    stores:
        validations_store:
            class_name: ValidationsStore
            store_backend:
                class_name: TupleFilesystemStoreBackend
                base_directory: uncommitted/validations/
    
  3. Update your configuration file to include a new store for Validation results on your filesystem

    In the example below, Validations Store is being set to shared_validations_filesystem_store, but it can be any name you like. Also, the base_directory is being set to uncommitted/shared_validations/, but it can be set to any path accessible by Great Expectations.

    validations_store_name: shared_validations_filesystem_store
    
    stores:
        shared_validations_filesystem_store:
            class_name: ValidationsStore
            store_backend:
                class_name: TupleFilesystemStoreBackend
                base_directory: uncommitted/shared_validations/
    
  4. Confirm that the location has been updated by running great_expectations --v3-api store list.

    Notice the output contains two Validation stores: the original validations_store and the shared_validations_filesystem_store we just configured. This is ok, since Great Expectations will look for Validations in the uncommitted/shared_validations/ folder as long as we set the validations_store_name variable to shared_validations_filesystem_store. The config for validations_store can be removed if you would like.

    great_expectations --v3-api store list
    
    - name: validations_store
    class_name: ValidationsStore
    store_backend:
        class_name: TupleFilesystemStoreBackend
        base_directory: uncommitted/validations/
    
    - name: shared_validations_filesystem_store
    class_name: ValidationsStore
    store_backend:
        class_name: TupleFilesystemStoreBackend
        base_directory: uncommitted/shared_validations/
    
  1. Confirm that the Validations store has been correctly configured

    Run a Checkpoint to store results in the new Validations store on in your new location then visualize the results by re-building Data Docs.

If it would be useful to you, please comment with a +1 and feel free to add any suggestions or questions below. Also, please reach out to us on Slack if you would like to learn more, or have any questions.