Data Drift calculates the percentage of change in certain metrics when the underlying data changes. You can create data drift rules to validate the data change against a tolerance threshold for each type of metric. You can configure alerts for data drift rules, to get notified when the drift goes beyond a threshold.
- Profile an asset at least once to configure a Data Drift policy.
- An asset can have only one Data Drift policy.
- Data Drift policies cannot be manually executed. They are automatically executed after every profile that is performed on the asset.
Data Drift Rules
The below table describes the type of drift rules you can generate on different metrics:
|COMPLETENESS||Sum of null values|
|DISTINCT_VALUES||Examines the table to ensure that all values are different from one another.|
|MEAN||Average value of a column|
|MIN||Minimum value of a column|
|MAXIMUM||Maximum value of a column|
|SUM||Sum of the values of a column|
|STANDARD DEVIATION||Amount of variation or dispersion of values|
|TOP_10_VALUES||Top 10 repeated values in a column|
Add Data Drift Policy
To add a Data Drift policy, perform the following:
Click Discover from the left navigation menu.
Search for an asset and click on it. The asset details page for the selected asset is displayed.
Click the Add Data Drift Policy link. The Create Data Drift Policy window is displayed.
Provide the following information required to create the data drift policy:
Property Name Description Policy Name Name of the data drift policy. Description Note describing the data drift policy. Rule Definition Select a metric and provide a threshold to apply for the rule definition. If the percentage value exceeds the threshold, then an alert will be raised. Alert Configurations Enable the alert configuration toggle to receive notifications on failure, success, or when an error occurs while running the data drift policy. You can receive notifications via email, Slack, and Webhook.
Click the Enable Policy toggle button.
Edit Data Drift Policy
Once a Data Drift policy is added, the Edit Data Drift Policy link is available on the asset details page.
- Click Edit Data Drift Policy. The Edit Data Drift Policy window is displayed.
- Make your changes.
- Click Save Policy.