great_expectations.core

Subpackages

Package Contents

Classes

ExpectationConfiguration(expectation_type: str, kwargs: dict, meta: Optional[dict] = None, success_on_last_run: Optional[bool] = None, ge_cloud_id: Optional[str] = None, expectation_context: Optional[ExpectationContext] = None, rendered_content: Optional[List[RenderedAtomicContent]] = None)

ExpectationConfiguration defines the parameters and name of a specific expectation.

ExpectationConfigurationSchema()

ExpectationSuite(expectation_suite_name: str, data_context: Optional[AbstractDataContext] = None, expectations: Optional[List[Union[dict, ExpectationConfiguration]]] = None, evaluation_parameters: Optional[dict] = None, data_asset_type: Optional[str] = None, execution_engine_type: Optional[Type[ExecutionEngine]] = None, meta: Optional[dict] = None, ge_cloud_id: Optional[str] = None)

This ExpectationSuite object has create, read, update, and delete functionality for its expectations:

ExpectationSuiteSchema()

ExpectationSuiteValidationResult(success: Optional[bool] = None, results: Optional[list] = None, evaluation_parameters: Optional[dict] = None, statistics: Optional[dict] = None, meta: Optional[ExpectationSuiteValidationResultMeta] = None, ge_cloud_id: Optional[UUID] = None)

A convenience class for migrating away from untyped dictionaries to stronger typed objects.

ExpectationSuiteValidationResultSchema()

ExpectationValidationResult(success: Optional[bool] = None, expectation_config: Optional[‘ExpectationConfiguration’] = None, result: Optional[dict] = None, meta: Optional[dict] = None, exception_info: Optional[dict] = None, rendered_content: Optional[RenderedAtomicContent] = None, **kwargs: dict)

A convenience class for migrating away from untyped dictionaries to stronger typed objects.

ExpectationValidationResultSchema()

IDDict()

dict() -> new empty dictionary

RunIdentifier(run_name: Optional[str] = None, run_time: Optional[Union[datetime.datetime, str]] = None)

A RunIdentifier identifies a run (collection of validations) by run_name and run_time.

RunIdentifierSchema()

Functions

get_metric_kwargs_id(metric_name, metric_kwargs)

class great_expectations.core.ExpectationConfiguration(expectation_type: str, kwargs: dict, meta: Optional[dict] = None, success_on_last_run: Optional[bool] = None, ge_cloud_id: Optional[str] = None, expectation_context: Optional[ExpectationContext] = None, rendered_content: Optional[List[RenderedAtomicContent]] = None)

Bases: great_expectations.types.SerializableDictDot

ExpectationConfiguration defines the parameters and name of a specific expectation.

kwarg_lookup_dict
runtime_kwargs = ['result_format', 'include_config', 'catch_exceptions']
process_evaluation_parameters(self, evaluation_parameters, interactive_evaluation: bool = True, data_context: Optional[DataContext] = None)
get_raw_configuration(self)
patch(self, op: str, path: str, value: Any)
Parameters
  • op – A jsonpatch operation. One of ‘add’, ‘replace’, or ‘remove’

  • path – A jsonpatch path for the patch operation

  • value – The value to patch

Returns

The patched ExpectationConfiguration object

property ge_cloud_id(self)
property expectation_context(self)
property expectation_type(self)
property kwargs(self)
property rendered_content(self)
_get_default_custom_kwargs(self)
get_domain_kwargs(self)
get_success_kwargs(self)
get_runtime_kwargs(self, runtime_configuration: Optional[dict] = None)
applies_to_same_domain(self, other_expectation_configuration: ExpectationConfiguration)
isEquivalentTo(self, other: Union[dict, ExpectationConfiguration], match_type: str = 'success')

ExpectationConfiguration equivalence does not include meta, and relies on equivalence of kwargs.

__eq__(self, other)

ExpectationConfiguration equality does include meta, but ignores instance identity.

__ne__(self, other)

Return self!=value.

__repr__(self)

Return repr(self).

__str__(self)

Return str(self).

to_json_dict(self)

# TODO: <Alex>2/4/2022</Alex> A reference implementation can be provided, once circular import dependencies, caused by relative locations of the “great_expectations/types/__init__.py” and “great_expectations/core/util.py” modules are resolved.

get_evaluation_parameter_dependencies(self)
static _update_dependencies_with_expectation_suite_urn(dependencies: dict, urn: ParseResults)
_get_expectation_impl(self)
validate(self, validator: Validator, runtime_configuration=None)
metrics_validate(self, metrics: Dict, runtime_configuration: Optional[dict] = None, execution_engine: Optional[ExecutionEngine] = None, **kwargs: dict)
get_domain_type(self)

Return “domain_type” of this expectation.

render(self)

Renders content using the atomic prescriptive renderer for this Expectation Configuration to ExpectationConfiguration.rendered_content.

class great_expectations.core.ExpectationConfigurationSchema

Bases: marshmallow.Schema

expectation_type
kwargs
meta
ge_cloud_id
expectation_context
rendered_content
REMOVE_KEYS_IF_NONE = ['ge_cloud_id', 'expectation_context', 'rendered_content']
clean_null_attrs(self, data: dict, **kwargs: dict)

Removes the attributes in ExpectationConfigurationSchema.REMOVE_KEYS_IF_NONE during serialization if their values are None.

make_expectation_configuration(self, data: dict, **kwargs)
class great_expectations.core.ExpectationSuite(expectation_suite_name: str, data_context: Optional[AbstractDataContext] = None, expectations: Optional[List[Union[dict, ExpectationConfiguration]]] = None, evaluation_parameters: Optional[dict] = None, data_asset_type: Optional[str] = None, execution_engine_type: Optional[Type[ExecutionEngine]] = None, meta: Optional[dict] = None, ge_cloud_id: Optional[str] = None)

Bases: great_expectations.types.SerializableDictDot

This ExpectationSuite object has create, read, update, and delete functionality for its expectations: -create: self.add_expectation() -read: self.find_expectation_indexes() -update: self.add_expectation() or self.patch_expectation() -delete: self.remove_expectation()

property name(self)
add_citation(self, comment: str, batch_request: Optional[Union[str, Dict[str, Union[str, Dict[str, Any]]]]] = None, batch_definition: Optional[dict] = None, batch_spec: Optional[dict] = None, batch_kwargs: Optional[dict] = None, batch_markers: Optional[dict] = None, batch_parameters: Optional[dict] = None, profiler_config: Optional[dict] = None, citation_date: Optional[Union[str, datetime.datetime]] = None)
isEquivalentTo(self, other)

ExpectationSuite equivalence relies only on expectations and evaluation parameters. It does not include: - data_asset_name - expectation_suite_name - meta - data_asset_type

__eq__(self, other)

ExpectationSuite equality ignores instance identity, relying only on properties.

__ne__(self, other)

Return self!=value.

__repr__(self)

Return repr(self).

__str__(self)

Return str(self).

__deepcopy__(self, memo: dict)
to_json_dict(self)

# TODO: <Alex>2/4/2022</Alex> A reference implementation can be provided, once circular import dependencies, caused by relative locations of the “great_expectations/types/__init__.py” and “great_expectations/core/util.py” modules are resolved.

get_evaluation_parameter_dependencies(self)
get_citations(self, sort: bool = True, require_batch_kwargs: bool = False, require_batch_request: bool = False, require_profiler_config: bool = False)
static _filter_citations(citations: List[Dict[str, Any]], filter_key)
static _sort_citations(citations: List[Dict[str, Any]])
append_expectation(self, expectation_config)

Appends an expectation.

Parameters

expectation_config (ExpectationConfiguration) – The expectation to be added to the list.

Notes

May want to add type-checking in the future.

remove_expectation(self, expectation_configuration: Optional[ExpectationConfiguration] = None, match_type: str = 'domain', remove_multiple_matches: bool = False, ge_cloud_id: Optional[Union[str, uuid.UUID]] = None)
Parameters
  • expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against for the removal of expectations.

  • match_type

    This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters

    that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters

  • remove_multiple_matches – If True, will remove multiple matching expectations. If False, will raise a ValueError.

Returns: The list of deleted ExpectationConfigurations

Raises
  • No match

  • More than 1 match, if remove_multiple_matches = False

remove_all_expectations_of_type(self, expectation_types: Union[List[str], str])
find_expectation_indexes(self, expectation_configuration: Optional[ExpectationConfiguration] = None, match_type: str = 'domain', ge_cloud_id: Optional[str] = None)

Find indexes of Expectations matching the given ExpectationConfiguration on the given match_type. If a ge_cloud_id is provided, match_type is ignored and only indexes of Expectations with matching ge_cloud_id are returned.

Parameters
  • expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the index of any matching Expectation Configurations on the suite.

  • match_type

    This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters

    that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters

  • ge_cloud_id – Great Expectations Cloud id

Returns: A list of indexes of matching ExpectationConfiguration

Raises

InvalidExpectationConfigurationError

find_expectations(self, expectation_configuration: Optional[ExpectationConfiguration] = None, match_type: str = 'domain', ge_cloud_id: Optional[str] = None)

Find Expectations matching the given ExpectationConfiguration on the given match_type. If a ge_cloud_id is provided, match_type is ignored and only Expectations with matching ge_cloud_id are returned.

Parameters
  • expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the index of any matching Expectation Configurations on the suite.

  • match_type

    This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters

    that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters

  • ge_cloud_id – Great Expectations Cloud id

Returns: A list of matching ExpectationConfigurations

replace_expectation(self, new_expectation_configuration: Union[ExpectationConfiguration, dict], existing_expectation_configuration: Optional[ExpectationConfiguration] = None, match_type: str = 'domain', ge_cloud_id: Optional[str] = None)

Find Expectations matching the given ExpectationConfiguration on the given match_type. If a ge_cloud_id is provided, match_type is ignored and only Expectations with matching ge_cloud_id are returned.

Parameters
  • expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the index of any matching Expectation Configurations on the suite.

  • match_type

    This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters

    that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters

  • ge_cloud_id – Great Expectations Cloud id

Returns: A list of matching ExpectationConfigurations

patch_expectation(self, expectation_configuration: ExpectationConfiguration, op: str, path: str, value: Any, match_type: str)
Parameters
  • expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the expectation to patch.

  • op – A jsonpatch operation (one of ‘add’,’update’, or ‘remove’) (see http://jsonpatch.com/)

  • path – A jsonpatch path for the patch operation (see http://jsonpatch.com/)

  • value – The value to patch (see http://jsonpatch.com/)

  • match_type – The match type to use for find_expectation_index()

Returns: The patched ExpectationConfiguration

Raises
  • No match

  • More than 1 match

_add_expectation(self, expectation_configuration: ExpectationConfiguration, send_usage_event: bool = True, match_type: str = 'domain', overwrite_existing: bool = True)

This is a private method for adding expectations that allows for usage_events to be suppressed when Expectations are added through internal processing (ie. while building profilers, rendering or validation). It takes in send_usage_event boolean.

Parameters
  • expectation_configuration – The ExpectationConfiguration to add or update

  • send_usage_event – Whether to send a usage_statistics event. When called through ExpectationSuite class’ public add_expectation() method, this is set to True.

  • match_type – The criteria used to determine whether the Suite already has an ExpectationConfiguration and so whether we should add or replace.

  • overwrite_existing – If the expectation already exists, this will overwrite if True and raise an error if False.

Returns

The ExpectationConfiguration to add or replace.

Raises
  • More than one match

  • One match if overwrite_existing = False

send_usage_event(self, success: bool)
add_expectation_configurations(self, expectation_configurations: List[ExpectationConfiguration], send_usage_event: bool = True, match_type: str = 'domain', overwrite_existing: bool = True)
Parameters
  • expectation_configurations – The List of candidate new/modifed “ExpectationConfiguration” objects for Suite.

  • send_usage_event – Whether to send a usage_statistics event. When called through ExpectationSuite class’ public add_expectation() method, this is set to True.

  • match_type – The criteria used to determine whether the Suite already has an “ExpectationConfiguration” object, matching the specified criteria, and thus whether we should add or replace (i.e., “upsert”).

  • overwrite_existing – If “ExpectationConfiguration” already exists, this will cause it to be overwritten if True and raise an error if False.

Returns

The List of “ExpectationConfiguration” objects attempted to be added or replaced (can differ from the list of “ExpectationConfiguration” objects in “self.expectations” at the completion of this method’s execution).

Raises
  • More than one match

  • One match if overwrite_existing = False

add_expectation(self, expectation_configuration: ExpectationConfiguration, send_usage_event: bool = True, match_type: str = 'domain', overwrite_existing: bool = True)
Parameters
  • expectation_configuration – The ExpectationConfiguration to add or update

  • send_usage_event – Whether to send a usage_statistics event. When called through ExpectationSuite class’ public add_expectation() method, this is set to True.

  • match_type – The criteria used to determine whether the Suite already has an ExpectationConfiguration and so whether we should add or replace.

  • overwrite_existing – If the expectation already exists, this will overwrite if True and raise an error if False.

Returns

The ExpectationConfiguration to add or replace.

Raises
  • More than one match

  • One match if overwrite_existing = False

show_expectations_by_domain_type(self)

Displays “ExpectationConfiguration” list, grouped by “domain_type”, in predetermined designated order.

show_expectations_by_expectation_type(self, expectation_configurations: Optional[List[ExpectationConfiguration]] = None)

Displays “ExpectationConfiguration” list, grouped by “expectation_type”, in predetermined designated order.

get_grouped_and_ordered_expectations_by_domain_type(self)

Returns “ExpectationConfiguration” list in predetermined order by passing appropriate methods for retrieving “ExpectationConfiguration” lists by corresponding “domain_type” (with “table” first; then “column”, and so on).

get_grouped_and_ordered_expectations_by_expectation_type(self)

Returns “ExpectationConfiguration” list, grouped by “expectation_type”, in predetermined designated order.

get_table_expectations(self)

Return a list of table expectations.

get_column_expectations(self)

Return a list of column map expectations.

get_column_pair_expectations(self)

Return a list of column_pair map expectations.

get_multicolumn_expectations(self)

Return a list of multicolumn map expectations.

get_grouped_and_ordered_expectations_by_column(self, expectation_type_filter: Optional[str] = None)
static _get_expectations_by_domain_using_accessor_method(domain_type: str, accessor_method: Callable)
render(self)

Renders content using the atomic prescriptive renderer for each expectation configuration associated with this ExpectationSuite to ExpectationConfiguration.rendered_content.

class great_expectations.core.ExpectationSuiteSchema

Bases: marshmallow.Schema

expectation_suite_name
ge_cloud_id
expectations
evaluation_parameters
data_asset_type
meta
clean_empty(self, data)
prepare_dump(self, data, **kwargs)
great_expectations.core.expectationSuiteSchema
class great_expectations.core.ExpectationSuiteValidationResult(success: Optional[bool] = None, results: Optional[list] = None, evaluation_parameters: Optional[dict] = None, statistics: Optional[dict] = None, meta: Optional[ExpectationSuiteValidationResultMeta] = None, ge_cloud_id: Optional[UUID] = None)

Bases: great_expectations.types.SerializableDictDot

A convenience class for migrating away from untyped dictionaries to stronger typed objects.

Can be instantiated with arguments:

my_A = MyClassA(

foo=”a string”, bar=1,

)

Can be instantiated from a dictionary:

my_A = MyClassA(
**{

“foo”: “a string”, “bar”: 1,

}

)

Can be accessed using both dictionary and dot notation

my_A.foo == “a string” my_A.bar == 1

my_A[“foo”] == “a string” my_A[“bar”] == 1

Pairs nicely with @dataclass:

@dataclass() class MyClassA(DictDot):

foo: str bar: int

Can be made immutable:

@dataclass(frozen=True) class MyClassA(DictDot):

foo: str bar: int

For more examples of usage, please see test_dataclass_serializable_dot_dict_pattern.py in the tests folder.

__eq__(self, other)

ExpectationSuiteValidationResult equality ignores instance identity, relying only on properties.

__repr__(self)

Return repr(self).

__str__(self)

Return str(self).

to_json_dict(self)

# TODO: <Alex>2/4/2022</Alex> A reference implementation can be provided, once circular import dependencies, caused by relative locations of the “great_expectations/types/__init__.py” and “great_expectations/core/util.py” modules are resolved.

get_metric(self, metric_name, **kwargs)
get_failed_validation_results(self)
class great_expectations.core.ExpectationSuiteValidationResultSchema

Bases: marshmallow.Schema

success
results
evaluation_parameters
statistics
meta
ge_cloud_id
checkpoint_name
prepare_dump(self, data, **kwargs)
make_expectation_suite_validation_result(self, data, **kwargs)
class great_expectations.core.ExpectationValidationResult(success: Optional[bool] = None, expectation_config: Optional['ExpectationConfiguration'] = None, result: Optional[dict] = None, meta: Optional[dict] = None, exception_info: Optional[dict] = None, rendered_content: Optional[RenderedAtomicContent] = None, **kwargs: dict)

Bases: great_expectations.types.SerializableDictDot

A convenience class for migrating away from untyped dictionaries to stronger typed objects.

Can be instantiated with arguments:

my_A = MyClassA(

foo=”a string”, bar=1,

)

Can be instantiated from a dictionary:

my_A = MyClassA(
**{

“foo”: “a string”, “bar”: 1,

}

)

Can be accessed using both dictionary and dot notation

my_A.foo == “a string” my_A.bar == 1

my_A[“foo”] == “a string” my_A[“bar”] == 1

Pairs nicely with @dataclass:

@dataclass() class MyClassA(DictDot):

foo: str bar: int

Can be made immutable:

@dataclass(frozen=True) class MyClassA(DictDot):

foo: str bar: int

For more examples of usage, please see test_dataclass_serializable_dot_dict_pattern.py in the tests folder.

__eq__(self, other)

ExpectationValidationResult equality ignores instance identity, relying only on properties.

__ne__(self, other)

Return self!=value.

__repr__(self)

# TODO: <Alex>5/9/2022</Alex> This implementation is non-ideal (it was agreed to employ it for development expediency). A better approach would consist of “__str__()” calling “__repr__()”, while all output options are handled through state variables.

__str__(self)

# TODO: <Alex>5/9/2022</Alex> This implementation is non-ideal (it was agreed to employ it for development expediency). A better approach would consist of “__str__()” calling “__repr__()”, while all output options are handled through state variables.

render(self)

Renders content using the: - atomic prescriptive renderer for the expectation configuration associated with this

ExpectationValidationResult to self.expectation_config.rendered_content

  • atomic diagnostic renderer for the expectation configuration associated with this ExpectationValidationResult to self.rendered_content.

static validate_result_dict(result)
to_json_dict(self)

# TODO: <Alex>2/4/2022</Alex> A reference implementation can be provided, once circular import dependencies, caused by relative locations of the “great_expectations/types/__init__.py” and “great_expectations/core/util.py” modules are resolved.

get_metric(self, metric_name, **kwargs)
class great_expectations.core.ExpectationValidationResultSchema

Bases: marshmallow.Schema

success
expectation_config
result
meta
exception_info
rendered_content
REMOVE_KEYS_IF_NONE = ['rendered_content']
convert_result_to_serializable(self, data, **kwargs)
clean_null_attrs(self, data: dict, **kwargs: dict)

Removes the attributes in ExpectationValidationResultSchema.REMOVE_KEYS_IF_NONE during serialization if their values are None.

make_expectation_validation_result(self, data, **kwargs)
great_expectations.core.expectationSuiteValidationResultSchema
great_expectations.core.expectationValidationResultSchema
great_expectations.core.get_metric_kwargs_id(metric_name, metric_kwargs)
class great_expectations.core.IDDict

Bases: dict

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s

(key, value) pairs

dict(iterable) -> new dictionary initialized as if via:

d = {} for k, v in iterable:

d[k] = v

dict(**kwargs) -> new dictionary initialized with the name=value pairs

in the keyword argument list. For example: dict(one=1, two=2)

_id_ignore_keys :Set[str]
to_id(self, id_keys=None, id_ignore_keys=None)
__hash__(self)

Overrides the default implementation

class great_expectations.core.RunIdentifier(run_name: Optional[str] = None, run_time: Optional[Union[datetime.datetime, str]] = None)

Bases: great_expectations.core.data_context_key.DataContextKey

A RunIdentifier identifies a run (collection of validations) by run_name and run_time.

property run_name(self)
property run_time(self)
to_tuple(self)
to_fixed_length_tuple(self)
__repr__(self)

Return repr(self).

__str__(self)

Return str(self).

to_json_dict(self)
classmethod from_tuple(cls, tuple_)
classmethod from_fixed_length_tuple(cls, tuple_)
class great_expectations.core.RunIdentifierSchema

Bases: marshmallow.Schema

run_name
run_time
make_run_identifier(self, data, **kwargs)
great_expectations.core.ge_urn