Skip to content

Conversation

@Eshanatnight
Copy link
Contributor

Fixes #656

Description

Implement Ingest Server

Changes

  1. Implemented Ingest Server
  2. Ingest Server creates and handles it's own manifest files
  3. Ingest Server creates a metadata file for the Query Server
  4. Logstreams will not be created automatically when sending the request to the Ingest API for the first time Only in Distributed Setup
    • User needs to create a logstream first from the console (Possible changes Needed in console)
  5. Disallow Users to run Server in Distributed Setup with local storage

This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

add DELETE endpoint for stream on IngestServer
add QuriedStats struct
add stats endpoint for logstream on Ingest Server
add guard to create stream automatically during ingest on Ingest Server
fix: api path /logstream on ingest server
make get_ingestor_info func pub
add logstream api endpoint to put
fix: func dir_with_stream to use the proper stream metadata filename
fix: update stream metadata file name
make stream_json_path give the correct file path when in different server modes
@Eshanatnight Eshanatnight force-pushed the impl-ingest-server branch 3 times, most recently from 21acd48 to b5176aa Compare March 6, 2024 09:14
Add Guard to disallow the user to run the server in distributed setup with local storage
If Streams Existed in the data store Server was considering it as stale data
On Startup of a new Ingest Server, Streams were not being synced up properly
@nitisht nitisht merged commit b663831 into parseablehq:distributed Mar 11, 2024
@Eshanatnight Eshanatnight deleted the impl-ingest-server branch March 11, 2024 05:50
Eshanatnight added a commit to Eshanatnight/parseable that referenced this pull request Apr 1, 2024
- add DELETE endpoint for stream on IngestServer
- add QuriedStats struct
- add stats endpoint for logstream on Ingest Server
- add guard to create stream automatically during ingest on Ingest Server
- api path /logstream on ingest server
- make get_ingestor_info func pub
- add logstream api endpoint to put
- func dir_with_stream to use the proper stream metadata filename
- update stream metadata file name
- make stream_json_path give the correct file path when in different server modes
- Add Guard to disallow the user to run the server in distributed setup with local storage
- Other Ingest Servers were not starting if streams existed
- If Streams Existed in the data store Server was considering it as stale data
- On Startup of a new Ingest Server, Streams were not being synced up properly
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants