Revamp Prometheus Merics + add Internal Metrics #3

[syepes]

Description

Revamped the Prometheus Merics and added some internal metrics

• EVENTS_INGESTED - Events ingested
• EVENTS_INGESTED_SIZE - Events ingested size bytes
• STORAGE_SIZE - Storage size bytes
• STAGING_FILES - Active Staging files
• QUERY_EXECUTE_TIME - Query execution time
• ALERTS_STATES - Alert states
• REQUEST_RESPONSE_TIME - Underlying Storage latency (@trueleo )

Example:

# HELP parseable_alerts_states Alerts States
# TYPE parseable_alerts_states counter
parseable_alerts_states{name="Status Alert",state="Resolved",stream="load"} 1
parseable_alerts_states{name="Status Alert",state="SetToFiring",stream="load"} 1
# HELP parseable_events_ingested Events ingested
# TYPE parseable_events_ingested counter
parseable_events_ingested{format="json",stream="load"} 16482
# HELP parseable_events_ingested_size Events ingested size bytes
# TYPE parseable_events_ingested_size gauge
parseable_events_ingested_size{format="json",stream="load"} 484
# HELP parseable_staging_files Active Staging files
# TYPE parseable_staging_files gauge
parseable_staging_files{stream="load"} 0
# HELP parseable_storage_size Storage size bytes
# TYPE parseable_storage_size gauge
parseable_storage_size{format="parquet",stream="load"} 2062413

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

[nitisht]

Great stuff @syepes . I am testing this now, and we'll merge it soon.

[nitisht]

I see that the number reported for json and parquet stored is different than the metrics reported by the stats API.

I was expecting the numbers to be same - would you take a look please

[syepes]

Nice catch, yes I'll have a look at this later today.
I'll also be adding more metrics

[syepes]

@trueleo Don't you think it would be best to keep all the metric definitions and registrations (REQUEST_RESPONSE_TIME, REQUEST_RESPONSE_TIME) under the /metrics/mod.rs file?
We can thing simply import the necessary metric into each component...

[trueleo]

@trueleo Don't you think it would be best to keep all the metric definitions and registrations (REQUEST_RESPONSE_TIME, REQUEST_RESPONSE_TIME) under the /metrics/mod.rs file? We can thing simply import the necessary metric into each component...

@syepes Now that i think of it maybe instead of registering them through ObjectStorageProvider. I can define some other trait and use that as interface for registering.

[syepes]

@trueleo By the way, I have had a look at why the values of EVENTS_INGESTED_SIZE and STORAGE_SIZE are different from those of the stats API, but I can not figure out why. Could you have a look?

[trueleo]

@syepes Sure

[nitisht]

Looking great now, just tested. Merging. Thank you again for your hardwork @syepes @trueleo