great_expectations.datasource.batch_kwargs_generator.table_batch_kwargs_generator

Module Contents

Classes

AssetConfigurationSchema(*, only: types.StrSequenceOrSet = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: typing.Dict = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: typing.Union[bool, types.StrSequenceOrSet] = False, unknown: str = None)

Base schema class with which to define custom schemas.

AssetConfiguration(table, schema=None)

TableBatchKwargsGenerator(name=’default’, datasource=None, assets=None)

Provide access to already materialized tables or views in a database.

great_expectations.datasource.batch_kwargs_generator.table_batch_kwargs_generator.logger
great_expectations.datasource.batch_kwargs_generator.table_batch_kwargs_generator.sqlalchemy
class great_expectations.datasource.batch_kwargs_generator.table_batch_kwargs_generator.AssetConfigurationSchema(*, only: types.StrSequenceOrSet = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: typing.Dict = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: typing.Union[bool, types.StrSequenceOrSet] = False, unknown: str = None)

Bases: great_expectations.marshmallow__shade.Schema

Base schema class with which to define custom schemas.

Example usage:

import datetime as dt
from dataclasses import dataclass

from great_expectations.marshmallow__shade import Schema, fields


@dataclass
class Album:
    title: str
    release_date: dt.date


class AlbumSchema(Schema):
    title = fields.Str()
    release_date = fields.Date()


album = Album("Beggars Banquet", dt.date(1968, 12, 6))
schema = AlbumSchema()
data = schema.dump(album)
data  # {'release_date': '1968-12-06', 'title': 'Beggars Banquet'}
Parameters
  • only – Whitelist of the declared fields to select when instantiating the Schema. If None, all fields are used. Nested fields can be represented with dot delimiters.

  • exclude – Blacklist of the declared fields to exclude when instantiating the Schema. If a field appears in both only and exclude, it is not used. Nested fields can be represented with dot delimiters.

  • many – Should be set to True if obj is a collection so that the object will be serialized to a list.

  • context – Optional context passed to fields.Method and fields.Function fields.

  • load_only – Fields to skip during serialization (write-only fields)

  • dump_only – Fields to skip during deserialization (read-only fields)

  • partial – Whether to ignore missing fields and not require any fields declared. Propagates down to Nested fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.

  • unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE.

Changed in version 3.0.0: prefix parameter removed.

Changed in version 2.0.0: __validators__, __preprocessors__, and __data_handlers__ are removed in favor of marshmallow.decorators.validates_schema, marshmallow.decorators.pre_load and marshmallow.decorators.post_dump. __accessor__ and __error_handler__ are deprecated. Implement the handle_error and get_attribute methods instead.

table
schema
make_asset_configuration(self, data, **kwargs)
class great_expectations.datasource.batch_kwargs_generator.table_batch_kwargs_generator.AssetConfiguration(table, schema=None)
property table(self)
property schema(self)
great_expectations.datasource.batch_kwargs_generator.table_batch_kwargs_generator.assetConfigurationSchema
class great_expectations.datasource.batch_kwargs_generator.table_batch_kwargs_generator.TableBatchKwargsGenerator(name='default', datasource=None, assets=None)

Bases: great_expectations.datasource.batch_kwargs_generator.batch_kwargs_generator.BatchKwargsGenerator

Provide access to already materialized tables or views in a database.

TableBatchKwargsGenerator can be used to define specific data asset names that take and substitute parameters, for example to support referring to the same data asset but with different schemas depending on provided batch_kwargs.

The python template language is used to substitute table name portions. For example, consider the following configurations:

my_generator:
  class_name: TableBatchKwargsGenerator
  assets:
    my_table:
      schema: $schema
      table: my_table

In that case, the asset my_datasource/my_generator/my_asset will refer to a table called my_table in a schema defined in batch_kwargs.

recognized_batch_parameters
_get_iterator(self, data_asset_name, query_parameters=None, limit=None, offset=None, partition_id=None)
get_available_data_asset_names(self)

Return the list of asset names known by this batch kwargs generator.

Returns

A list of available names

_build_batch_kwargs(self, batch_parameters)
get_available_partition_ids(self, generator_asset=None, data_asset_name=None)

Applies the current _partitioner to the batches available on data_asset_name and returns a list of valid partition_id strings that can be used to identify batches of data.

Parameters

data_asset_name – the data asset whose partitions should be returned.

Returns

A list of partition_id strings