Admin Guide: Data Verification

Overview

This guide is for administrators who manage verification rules. The Rules Configuration tab is only visible to users with the Admin role.

Access: Monitor > Data Verification > Rules Configuration

URL: `/monitor/data-verification/rules`

Rules Configuration

Viewing Rules

• Rules are listed alphabetically by display name

• Each rule card shows: Display Name, Layer badge, Description, Applies To, and Thresholds

• Use the search box to filter rules by name, description, layer, or device type

• Toggle the enable/disable switch to activate or deactivate a rule without deleting it

 Creating a New Rule

1. Click + New Rule

2. Fill in:

• Rule ID: Uppercase with underscores (e.g., `MY_CUSTOM_RULE`). Cannot be changed after creation.

• Display Name: Human-readable name shown in the UI

• Description: What the rule checks

• Layer: Category — `config`, `presence`, `health`, or `sensibility`

• Applies To: Which device types this rule targets. Leave empty for all devices.

• Thresholds: JSON configuration for the rule's parameters

3. Click Create Rule

Important: Creating a rule in the UI only adds the configuration to DynamoDB. For the rule to actually execute, a corresponding Python rule class must be registered in the backend with `@register_rule("YOUR_RULE_ID")`. Rules without a backend handler are silently skipped during verification.

Editing a Rule

1. Click the pencil icon on a rule card

2. Modify any field (except Rule ID)

3. Click Save Changes

Changes take effect immediately on the next verification run. No deployment needed.

Deleting a Rule

1. Click the trash icon on a rule card

2. Confirm deletion by clicking Yes

Deleted rules are permanently removed from DynamoDB.

Enabling/Disabling Rules

Toggle the switch on the right side of each rule card. Disabled rules are skipped during verification. This is useful for temporarily turning off a noisy rule without deleting it.

Rule Layers

Verification Rules

Presence Rules

These rules check if the device is sending data.

1. Data Freshness

Checks how long since the device last sent data to Timestream.

When a device has no data or stale data, the rule also checks IoT Core to determine the root cause:

• Decoder failure: IoT Core received data but it didn't reach Timestream

• Device issue: IoT Core also has no recent uplink — likely battery, hardware, or connectivity problem

• Never connected: Device has never sent data to IoT Core

Health Rules

These rules check device hardware health.

1. Battery Low

Checks battery level. Automatically detects whether the sensor reports battery as voltage (Synetica, Vutility, Laird) or percentage (Milesight, Browan).

Detection: If battery value >= 10, it's percentage. If < 10, it's voltage.

Devices without battery data (e.g., mains-powered) pass silently.

2. Signal Weak

Checks GatewayRssi signal strength.

Note: RSSI measures signal strength. Poor RSSI with good SNR indicates distance;
good RSSI with poor SNR indicates environmental noise.

Sensibility Rules

These rules check if the data values make sense.

1. Constant Reading Detection

Detects when consumption measures (USGallon, PulseCount1/2/3) stop changing across 20 consecutive readings.

What it catches:

• Pulse counter disconnected from meter

• Sensor not reading flow

• Meter wire disconnected

Applies to: WM, PWM, PGM, PEM devices types

Example issue: "PulseCount1 not changing: 20 consecutive identical readings (value=15.0)"

2. Consumption Check

Checks if daily or monthly consumption exceeds physically plausible limits.

Daily limits (24h delta):

Monthly limits: maxDaily x 30

What it catches:

• Wrong pulse scalar configuration (e.g., scalar of 1 instead of 0.001 causing billions of gallons)

• Sustained misconfiguration that may not be visible in a single day

Example: The NZMN sensor accumulated 382 billion gallons in 30 days. The 24h delta was only 6,526 gal (under the daily limit), but the 30d delta of 382 billion far exceeds the 15 million monthly limit.

3. Negative Consumption

Fails if the cumulative meter reading decreases between readings.

What it catches:

• Meter rollover

• Misconfiguration causing negative values

• Data corruption

Example issue: "Consumption decreased by 5,000.00 in 24h — meter reading went from 80,000.00 down to 75,000.00"

4. Spike Detection

Warns if a single reading-to-reading jump exceeds 10x the rolling average delta.

What it catches:

• Sudden pulse scalar change mid-operation

• Meter malfunction causing a burst of readings

• Intermittent wiring issues

Example: If a sensor normally changes by ~50 gal per reading, but suddenly jumps by 5,000 gal, that's 100x the average — flagged as a spike.

Configuration

• `multiplier`: How many times the average constitutes a spike (default: 10x)

• `minSamples`: Minimum readings needed to calculate average (default: 5)

5. Zero Flow Detection

Warns if daily consumption is zero — the meter reading hasn't changed in 24 hours.

What it catches:

• Pulse counter disconnected from meter

• No actual flow through the meter

• Meter wire disconnected

Applies to: PWM, WM, PGM, PEM device types

Config Rules

These rules check device configuration in DynamoDB.

1. Pulse Scalar Presence

Checks that pulse devices (PWM, PGM, PEM) have a `PulseScalarValue`(Pulse Rate) configured.

2. Device Type Presence

Warns if a device has no `DeviceType` set, which prevents proper classification and rule targeting.

3. LWM No Pulse Inputs

Fails if an LWM (LoRaWAN Water Meter) device has no pulse counter inputs (_P1/_P2/_P3) configured in the device table. LWM base devices without pulse counters cannot collect meter data.

Adding a Custom Rule

To add a new verification rule:

1. Click + New Rule

2. Enter a Rule ID in uppercase with underscores (e.g., HIGH_HUMIDITY). This cannot be changed later.

3. Enter a Display Name (e.g., "High Humidity Alert")

4. Enter a Description of what the rule checks

5. Select a Layer: config, presence, health, or sensibility

6. Set Applies To: Select which device types this rule targets, or leave empty to apply to all devices

7. Configure Thresholds as JSON. Example for a threshold-based rule:

   { "warnAbove": 80, "failAbove": 95, "measure": "Humidity" }

8. Click Create Rule

Note: Custom rules created via the UI require a matching backend evaluator to execute. Contact the engineering team to add the rule logic for new rule types. Existing rule types (threshold checks, field presence checks, consumption checks) can be reused by setting the appropriate Type field.

Audit Trail

All rule modifications are tracked:

• CreatedAt / CreatedBy: Set when a rule is created (via UI or seed script)

• UpdatedAt / UpdatedBy: Set automatically on every update via the UI

The user's email is extracted from the Cognito login token.