Skip to content

Observability — the Prometheus surface

Everything TasksWorkerPlugin.get_metrics() emits, and the one rule that matters before you touch any of it:

These names are a public interface. internal/infrastructure selects five of them by name in config/alerts/fastpluggy-tasks.rules.yml. Renaming or dropping a gauge does not break a build, does not fail a test, and does not turn anything red — the alert's selector simply matches nothing, and a rule over an empty selector reports health=ok forever (infra#219). Deprecate, emit both for a release, then drop. Never rename in place.

There is no bespoke /metrics route. The plugin exposes a MetricsProvider capability; the central fastpluggy-metrics aggregator collects it and serves everything under /plugin-pkg/metrics/api/prometheus, where each series is prefixed — so fastpluggy_broker_beats here is scraped as fp_tasks_worker_fastpluggy_broker_beats and that is the name an alert rule must use.

The gauges, by what they can actually see

Broker / topic counts — fastpluggy_broker_topics, ..._workers, ..._locks, ..._topic_{queued,running,dead_letter,subscribers}, ..._messages_{queued,running}_total. Volume, not health: a wedged topic and a saturated one are identical in every one of them.

fastpluggy_broker_beats — live beat-role workers. 0 means no scheduler anywhere.

fastpluggy_schedule_overdue_seconds — worst lateness across enabled schedules. ⚠ Read the section below before trusting this one.

fastpluggy_broker_oldest_running_claim_seconds{topic} — age of the oldest in-flight claim. The discriminator the counts cannot give you: a healthy topic's resets on every completion, a wedged one's only grows. Emitted only while a topic has a running claim, so absence means "nothing in flight", not "broken".

fastpluggy_broker_orphans_{reclaimed,dead_lettered}_total, ..._orphan_reclaim_skipped_total — reaper activity. The skipped counter rising means the reaper is failing closed and protecting nothing. ⚠ It also ticks once at every container start, before the first heartbeat exists, so increase(...[1h]) > 0 fires on a plain deploy — threshold on a sustained rate.

fastpluggy_scheduled_task_since_success_seconds{task} + ..._interval_seconds{task} (0.3.300,

23) — seconds since the schedule last produced a successful report, and the period it is meant

to run at. See below.

Task telemetry — fastpluggy_tasks_{submitted,completed,failed}_total and their _per_topic variants, ..._running. Process-local counters, reported by whichever process serves the metrics endpoint (in a FastPluggy app, also the one hosting the beat), so they reset on restart.

The trigger end vs the execution end

fastpluggy_schedule_overdue_seconds is computed from ScheduledTaskDB.expected_next_run, which derives from last_attempt — and tasks/scheduler.py stamps last_attempt before it looks the function up and submits the message:

if should_run:
    sched.last_attempt = now_utc     # "we tried"
    db.commit()
    ...                              # then: registry lookup, submit, (maybe dropped)

So it answers "is the beat beating?", never "is the task running?". A task whose message is discarded on every tick — an orphaned tw_locks row (#18), a starved worker, a wedged topic — keeps pushing its own expectation forward and holds this gauge at zero.

That is not theoretical: brain lost four scheduled tasks for 13 days (152 consecutive skipped runs of one of them) with this gauge, broker_beats and the topic counts all green. It surfaced two layers downstream, as Instagram captures arriving without media.

The 0.3.300 pair measures the other end, and the interval is exported next to the age on purpose so one fleet rule needs no per-task threshold:

fastpluggy_scheduled_task_since_success_seconds
  / fastpluggy_scheduled_task_interval_seconds > 3

Deliberately blind to the cause, which is why it is the first rule to reach for: it fires for an orphaned lock, a task failing on every run, a wedged topic, a starved worker and a dead beat alike. Deployed as FastpluggyScheduleNotProducing; it caught an unrelated second outage within minutes.

Two limits, both intentional and both tested (tests/scheduler/test_schedule_health.py):

  • a schedule that has never succeeded emits no series at all — otherwise every fresh deploy pages while the task waits its first turn. "Registered but never ran once" is therefore not covered here;
  • success is the report status, never the returned payload. A task may legitimately return {"skipped": "healthy"} and still have run — keying on the payload would flag the healthy no-op as a dead schedule.

Cron schedules have no interval, so the period is derived from two successive croniter occurrences; dropping them would recreate the blind spot one level down.

Downstream consumers

rule (internal/infrastructure:config/alerts/fastpluggy-tasks.rules.yml) reads
FastpluggyBeatDead schedule_overdue_seconds + broker_beats
FastpluggyScheduleOverdue schedule_overdue_seconds + broker_beats
FastpluggyTopicWedged broker_oldest_running_claim_seconds
FastpluggyOrphanReclaimFailClosed broker_orphan_reclaim_skipped_total
FastpluggyScheduleNotProducing scheduled_task_since_success_seconds ÷ ..._interval_seconds

A consumer app below 0.3.300 emits neither of the last two series, so that rule is inert for it — not passing. Raising the app's floor is what turns the alert on; there is no other signal that would tell you it is off.