Prometheus

If there are more than 3 WAL segments (older than 2 hours), something is wrong, check the logs (journalctl -u prometheus on Debian).

Lots of options in /etc/default/prometheus. E.g. --web.enable-admin-api, --storage.tsdb.retention.time and --storage.tsdb.retention.size.

Delete timeseries through admin-api:

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=apt_upgrades_pending&match[]={instance="rorschach.hjp.at:9100"}'

(May be best option if you get weird error messages about a specific timeseries.)

Speaking of retention.time, the default is 15 days.