Data Drift Policy Configuration

Data Drift calculates the percentage of change in certain metrics when the underlying data changes. You can create data drift rules to validate the data change against a tolerance threshold for each type of metric. You can configure alerts for data drift rules, to get notified when the drift goes beyond a threshold.

  1. Profile an asset at least once to configure a Data Drift policy.
  2. An asset can have only one Data Drift policy.
  3. Data Drift policies cannot be manually executed. They are automatically executed after every profile that is performed on the asset.

Data Drift Rules

The below table describes the type of drift rules you can generate on different metrics:

COMPLETENESSSum of null values
DISTINCT_VALUESExamines the table to ensure that all values are different from one another.
MEANAverage value of a column
MINMinimum value of a column
MAXIMUMMaximum value of a column
SUMSum of the values of a column
STANDARD DEVIATIONAmount of variation or dispersion of values
TOP_10_VALUESTop 10 repeated values in a column

Add Data Drift Policy

To add a Data Drift policy, perform the following:

  1. Click Discover from the left navigation menu.

  2. Search for an asset and click on it. The asset details page for the selected asset is displayed.

  3. Click the Add Data Drift Policy link. The Create Data Drift Policy window is displayed. create data drift

  4. Provide the following information required to create the data drift policy:

    Property NameDescription
    Policy NameName of the data drift policy.
    DescriptionNote describing the data drift policy.
    Rule DefinitionSelect a metric and provide a threshold to apply for the rule definition. If the percentage value exceeds the threshold, then an alert will be raised.
    Alert ConfigurationsEnable the alert configuration toggle to receive notifications on failure, success, or when an error occurs while running the data drift policy. You can receive notifications via email, Slack, and Webhook.
  5. Click the Enable Policy toggle button.

  6. Click Save.

Edit Data Drift Policy

Once a Data Drift policy is added, the Edit Data Drift Policy link is available on the asset details page. edit data drift

  1. Click Edit Data Drift Policy. The Edit Data Drift Policy window is displayed. Edit data drift policy window
  2. Make your changes.
  3. Click Save Policy.