vmalert

Rule: KafkaConsumerLagGrowingok

Expr

kafka:csg:lag > 100000 and on (job, cls, consumergroup) delta(kafka:csg:lag[30m]) > 0

For

1800 seconds

Labels

category=kafka level=1 severity=WARN

Annotations

description:

Consumer group lag is above 100k records and still growing after 30 minutes ({{ $value }} records). /ui/d/kafka-consumer?var-cls={{ $labels.cls }}&var-group={{ $labels.consumergroup }}

summary:

WARN KafkaConsumerLagGrowing {{ $labels.consumergroup }}

Debug

false

Group

18216149610123607395

Warning: some of updates have "Series fetched" equal to 0.
It might be that either this data is missing in the datasource or there is a typo in rule's expression. For example, foo{label="bar"} > 0 could never trigger because foo{label="bar"} metric doesn't exist.
Rule's expressions without time series selector, like expr: 42 or expr: time() aren't fetching time series from datasource, so they could have "Series fetched" equal to 0 and this won't be a problem.
See more details about this detection here.

Last 20/20 updates:

Updated at	Duration	Execution timestamp	cURL
2026-10-16T07:35:09-07:00	0.001s	2026-10-16T07:34:30-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A34%3A30-07%3A00'
2026-10-16T07:34:59-07:00	0.000s	2026-10-16T07:34:20-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A34%3A20-07%3A00'
2026-10-16T07:34:49-07:00	0.001s	2026-10-16T07:34:10-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A34%3A10-07%3A00'
2026-10-16T07:34:39-07:00	0.001s	2026-10-16T07:34:00-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A34%3A00-07%3A00'
2026-10-16T07:34:29-07:00	0.001s	2026-10-16T07:33:50-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A33%3A50-07%3A00'
2026-10-16T07:34:19-07:00	0.000s	2026-10-16T07:33:40-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A33%3A40-07%3A00'
2026-10-16T07:34:09-07:00	0.001s	2026-10-16T07:33:30-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A33%3A30-07%3A00'
2026-10-16T07:33:59-07:00	0.000s	2026-10-16T07:33:20-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A33%3A20-07%3A00'
2026-10-16T07:33:49-07:00	0.001s	2026-10-16T07:33:10-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A33%3A10-07%3A00'
2026-10-16T07:33:39-07:00	0.001s	2026-10-16T07:33:00-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A33%3A00-07%3A00'
2026-10-16T07:33:29-07:00	0.000s	2026-10-16T07:32:50-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A32%3A50-07%3A00'
2026-10-16T07:33:19-07:00	0.000s	2026-10-16T07:32:40-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A32%3A40-07%3A00'
2026-10-16T07:33:09-07:00	0.001s	2026-10-16T07:32:30-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A32%3A30-07%3A00'
2026-10-16T07:32:59-07:00	0.000s	2026-10-16T07:32:20-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A32%3A20-07%3A00'
2026-10-16T07:32:49-07:00	0.000s	2026-10-16T07:32:10-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A32%3A10-07%3A00'
2026-10-16T07:32:39-07:00	0.000s	2026-10-16T07:32:00-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A32%3A00-07%3A00'
2026-10-16T07:32:29-07:00	0.000s	2026-10-16T07:31:50-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A31%3A50-07%3A00'
2026-10-16T07:32:19-07:00	0.001s	2026-10-16T07:31:40-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A31%3A40-07%3A00'
2026-10-16T07:32:09-07:00	0.001s	2026-10-16T07:31:30-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A31%3A30-07%3A00'
2026-10-16T07:31:59-07:00	0.001s	2026-10-16T07:31:20-07:00	curl -X POST -H 'Content-Type: application/json' 'http://127.0.0.1:8428/api/v1/query?query=kafka%3Acsg%3Alag+%3E+100000+and+on+%28job%2C+cls%2C+consumergroup%29+delta%28kafka%3Acsg%3Alag%5B30m%5D%29+%3E+0&step=300s&time=2026-10-16T07%3A31%3A20-07%3A00'