Increasing number of go routines when postgresql cannot be accessed

[KaczDev]

What's wrong?

I have noticed that at one of our production servers there way too many go routines reported in comparison to other instances.

When I checked the logs I found that there are issues with connectin to the PostgreSQL database using prometheus.exporter.postgres.

I have reproduced the issue with an Alloy instance that has configured a postgres exporter to a database that it cannot connect to (pg_hba rejected or simply the db doesn't exist).

Steps to reproduce

Create a prometheus.exporter.postgres component and configure the URL to a non-existing postgresql URL.

System information

No response

Software version

Grafana Alloy v.1.4.2

Configuration

logging {
    level  = "info"
    format = "logfmt"
}

prometheus.exporter.postgres "POSTGRESQL_METRICS" {
    data_source_names = ["postgresql://postgres:postgres@postgres:5432/postgres?sslmode=disable"]
    autodiscovery {
        enabled = true
    }
}

prometheus.scrape "SCRAPE_POSTGRESQL_METRICS" {
    targets    = prometheus.exporter.postgres.POSTGRESQL_METRICS.targets
    forward_to = [prometheus.remote_write.PROMETHEUS_REMOTE_WRITE.receiver]
    // increased the number of intervals to reproduce the issue faster.
    scrape_interval = "5s"
    scrape_timeout="4s"
}


prometheus.exporter.self "SELF_REPORT" {}

prometheus.scrape "SELF_SCRAPER" {
    targets    = prometheus.exporter.self.SELF_REPORT.targets
    forward_to = [prometheus.remote_write.PROMETHEUS_REMOTE_WRITE.receiver]
    scrape_interval = "15s"
}

prometheus.remote_write "PROMETHEUS_REMOTE_WRITE" {
    endpoint {
        url = format("http://%s:%s/api/v1/push", env("MIMIR_URL"), env("MIMIR_PORT")) 
    }
}

Logs

ts=2024-10-18T15:36:51.273854616Z level=info msg="Established new database connection" component_path=/ component_id=prometheus.exporter.postgres.POSTGRESQL_METRICS fingerprint=postgres:5432
ts=2024-10-18T15:36:51.348885966Z level=error msg="Error opening connection to database" component_path=/ component_id=prometheus.exporter.postgres.POSTGRESQL_METRICS err="error querying postgresql version: dial tcp: lookup postgres on 127.0.0.11:53: server misbehaving"
ts=2024-10-18T15:36:52.34972657Z level=error msg="Error opening connection to database" component_path=/ component_id=prometheus.exporter.postgres.POSTGRESQL_METRICS dsn="postgresql://postgres:PASSWORD_REMOVED@postgres:5432/postgres?sslmode=disable" err="dial tcp: lookup postgres on 127.0.0.11:53: server misbehaving"

[dehaansa]

This appears to be an issue that was resolved in newer releases of the postgres prometheus exporter. Will take a look at updating the exporter version that is bundled with Alloy. Thanks for the detailed bug report!

[dehaansa]

After some testing, it appears that while other connection leaks have been fixed, this is still an issue in v0.15.0 of the prometheus exporter. Looking into fixing the issue upstream.

[dehaansa]

@KaczDev this should be resolved -- however as we've cut the release candidate for the current release, this will likely not hit until v1.6 in December. Thanks for reporting it!

[KaczDev]

Thanks a lot, @dehaansa!