great_expectations.datasource.batch_kwargs_generator.manual_batch_kwargs_generator

Module Contents

Classes

ManualBatchKwargsGenerator(name=’default’, datasource=None, assets=None)

ManualBatchKwargsGenerator returns manually-configured batch_kwargs for named data assets. It provides a

great_expectations.datasource.batch_kwargs_generator.manual_batch_kwargs_generator.logger
class great_expectations.datasource.batch_kwargs_generator.manual_batch_kwargs_generator.ManualBatchKwargsGenerator(name='default', datasource=None, assets=None)

Bases: great_expectations.datasource.batch_kwargs_generator.batch_kwargs_generator.BatchKwargsGenerator

ManualBatchKwargsGenerator returns manually-configured batch_kwargs for named data assets. It provides a convenient way to capture complete batch definitions without requiring the configuration of a more fully-featured batch kwargs generator.

A fully configured ManualBatchKwargsGenerator in yml might look like the following:

my_datasource:
  class_name: PandasDatasource
  batch_kwargs_generators:
    my_generator:
      class_name: ManualBatchKwargsGenerator
      assets:
        asset1:
          - partition_id: 1
            path: /data/file_1.csv
            reader_options:
              sep: ;
          - partition_id: 2
            path: /data/file_2.csv
            reader_options:
              header: 0
        logs:
          path: data/log.csv
recognized_batch_parameters
property assets(self)
get_available_data_asset_names(self)

Return the list of asset names known by this batch kwargs generator.

Returns

A list of available names

_get_data_asset_config(self, data_asset_name)
_get_iterator(self, data_asset_name, **kwargs)
get_available_partition_ids(self, generator_asset=None, data_asset_name=None)

Applies the current _partitioner to the batches available on data_asset_name and returns a list of valid partition_id strings that can be used to identify batches of data.

Parameters

data_asset_name – the data asset whose partitions should be returned.

Returns

A list of partition_id strings

_build_batch_kwargs(self, batch_parameters)

Build batch kwargs from a partition id.