great_expectations.execution_engine.execution_engine

Module Contents

Classes

NoOpDict()

BatchData(execution_engine)

MetricFunctionTypes()

Generic enumeration.

MetricDomainTypes()

Generic enumeration.

DataConnectorStorageDataReferenceResolver()

ExecutionEngine(name=None, caching=True, batch_spec_defaults=None, batch_data_dict=None, validator=None)

Helper class that provides a standard way to create an ABC using

MetricPartialFunctionTypes()

Generic enumeration.

great_expectations.execution_engine.execution_engine.logger
great_expectations.execution_engine.execution_engine.yaml
great_expectations.execution_engine.execution_engine.default_flow_style = False
class great_expectations.execution_engine.execution_engine.NoOpDict
__getitem__(self, item)
__setitem__(self, key, value)
update(self, value)
class great_expectations.execution_engine.execution_engine.BatchData(execution_engine)
property execution_engine(self)
head(self, *args, **kwargs)
class great_expectations.execution_engine.execution_engine.MetricFunctionTypes

Bases: enum.Enum

Generic enumeration.

Derive from this class to define new enumerations.

VALUE = value
MAP_VALUES = value
WINDOW_VALUES = value
AGGREGATE_VALUE = value
class great_expectations.execution_engine.execution_engine.MetricDomainTypes

Bases: enum.Enum

Generic enumeration.

Derive from this class to define new enumerations.

COLUMN = column
COLUMN_PAIR = column_pair
MULTICOLUMN = multicolumn
TABLE = table
class great_expectations.execution_engine.execution_engine.DataConnectorStorageDataReferenceResolver
DATA_CONNECTOR_NAME_TO_STORAGE_NAME_MAP :Dict[str, str]
STORAGE_NAME_EXECUTION_ENGINE_NAME_PATH_RESOLVERS :Dict[Tuple[str, str], Callable]
static resolve_data_reference(data_connector_name: str, execution_engine_name: str, template_arguments: dict)

Resolve file path for a (data_connector_name, execution_engine_name) combination.

class great_expectations.execution_engine.execution_engine.ExecutionEngine(name=None, caching=True, batch_spec_defaults=None, batch_data_dict=None, validator=None)

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

recognized_batch_spec_defaults
configure_validator(self, validator)

Optionally configure the validator as appropriate for the execution engine.

property active_batch_data_id(self)

The batch id for the default batch data.

When an execution engine is asked to process a compute domain that does not include a specific batch_id, then the data associated with the active_batch_data_id will be used as the default.

property active_batch_data(self)

The data from the currently-active batch.

property loaded_batch_data_dict(self)

The current dictionary of batches.

property loaded_batch_data_ids(self)
property config(self)
property dialect(self)
get_batch_data(self, batch_spec: BatchSpec)

Interprets batch_data and returns the appropriate data.

This method is primarily useful for utility cases (e.g. testing) where data is being fetched without a DataConnector and metadata like batch_markers is unwanted

Note: this method is currently a thin wrapper for get_batch_data_and_markers. It simply suppresses the batch_markers.

abstract get_batch_data_and_markers(self, batch_spec)
load_batch_data(self, batch_id: str, batch_data: Any)

Loads the specified batch_data into the execution engine

_load_batch_data_from_dict(self, batch_data_dict)

Loads all data in batch_data_dict into load_batch_data

resolve_metrics(self, metrics_to_resolve: Iterable[MetricConfiguration], metrics: Optional[Dict[Tuple[str, str, str], MetricConfiguration]] = None, runtime_configuration: Optional[dict] = None)

resolve_metrics is the main entrypoint for an execution engine. The execution engine will compute the value of the provided metrics.

Parameters
  • metrics_to_resolve – the metrics to evaluate

  • metrics – already-computed metrics currently available to the engine

  • runtime_configuration – runtime configuration information

Returns

a dictionary with the values for the metrics that have just been resolved.

Return type

resolved_metrics (Dict)

abstract resolve_metric_bundle(self, metric_fn_bundle)

Resolve a bundle of metrics with the same compute domain as part of a single trip to the compute engine.

abstract get_domain_records(self, domain_kwargs: dict)

get_domain_records computes the full-access data (dataframe or selectable) for computing metrics based on the given domain_kwargs and specific engine semantics.

Returns

data corresponding to the compute domain

abstract get_compute_domain(self, domain_kwargs: dict, domain_type: Union[str, MetricDomainTypes])

get_compute_domain computes the optimal domain_kwargs for computing metrics based on the given domain_kwargs and specific engine semantics.

Returns

  1. data corresponding to the compute domain;

  2. a modified copy of domain_kwargs describing the domain of the data returned in (1);

  3. a dictionary describing the access instructions for data elements included in the compute domain

    (e.g. specific column name).

In general, the union of the compute_domain_kwargs and accessor_domain_kwargs will be the same as the domain_kwargs provided to this method.

Return type

A tuple consisting of three elements

add_column_row_condition(self, domain_kwargs, column_name=None, filter_null=True, filter_nan=False)

EXPERIMENTAL

Add a row condition for handling null filter.

Parameters
  • domain_kwargs – the domain kwargs to use as the base and to which to add the condition

  • column_name – if provided, use this name to add the condition; otherwise, will use “column” key from table_domain_kwargs

  • filter_null – if true, add a filter for null values

  • filter_nan – if true, add a filter for nan values

resolve_data_reference(self, data_connector_name: str, template_arguments: dict)

Resolve file path for a (data_connector_name, execution_engine_name) combination.

class great_expectations.execution_engine.execution_engine.MetricPartialFunctionTypes

Bases: enum.Enum

Generic enumeration.

Derive from this class to define new enumerations.

MAP_FN = map_fn
MAP_SERIES = map_series
MAP_CONDITION_FN = map_condition_fn
MAP_CONDITION_SERIES = map_condition_series
WINDOW_FN = window_fn
WINDOW_CONDITION_FN = window_condition_fn
AGGREGATE_FN = aggregate_fn
property metric_suffix(self)