Skip to content

Conversation

@trueleo
Copy link
Contributor

@trueleo trueleo commented Nov 3, 2022

Description

Adds stats implementation based on atomic u64 counters. On each event ingestion counter is updated and on every sync parquet file size is added to compressed counter. On call to metadata api for stats it produces a json serializable Stats instance which is copy of metadata stats counter. Stats are synced on every s3 sync cycle and is stored inside parseable.json file.

Changes:

  • Stats module with StatsCounter and Stats type
  • Define stats handler for Actix
  • Update events and metadata method for stats
  • Add put_stats requirement for storage trait and provide s3 implementation

This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

@trueleo trueleo changed the title Stats Introduce Stats API Nov 3, 2022
@trueleo trueleo requested a review from nitisht November 3, 2022 12:40
Adds stats implementation based on atomic u64 counters. On each event ingestion
counter is updated and on every sync parquet file size is added to compressed
counter. On call to metadata api for stats it produces a json serializable
`Stats` instance which is copy of metadata stats counter. Stats are synced on
every s3 sync cycle and is stored inside parseable.json file.
@nitisht nitisht merged commit 0fa77b8 into parseablehq:main Nov 4, 2022
@trueleo trueleo deleted the stats branch November 19, 2022 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants