Distributed rebased on main #730
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…#656) This PR contains the ingester side changes
- add DELETE endpoint for stream on IngestServer - add QuriedStats struct - add stats endpoint for logstream on Ingest Server - add guard to create stream automatically during ingest on Ingest Server - api path /logstream on ingest server - make get_ingestor_info func pub - add logstream api endpoint to put - func dir_with_stream to use the proper stream metadata filename - update stream metadata file name - make stream_json_path give the correct file path when in different server modes - Add Guard to disallow the user to run the server in distributed setup with local storage - Other Ingest Servers were not starting if streams existed - If Streams Existed in the data store Server was considering it as stale data - On Startup of a new Ingest Server, Streams were not being synced up properly
* feat: add endpoint on Query Server to fetch the cluster info
on hitting {url}/api/v1/cluster/info
it would return a json response with the information on the cluster
Return JSON looks like
```
[
{
domain_name: "<domain_name>:<port>",
reachable: bool,
error: string error message,
status: string
},
.
.
.
]
* feat: add auth for /cluster/info endpoint
1. Add Action ListCluster
2. Authorize /cluster/info for ListCluster
* fix: get_objects fetch the proper files
When in S3 Mode, get_objects was fetching all the files
that contained the pattern "ingester".
add + update about api
1. Add About API for Ingest Servers
2. Update About API Response
```
{
"version": current_version,
"uiVersion": ui_version,
"commit": commit,
"deploymentId": deployment_id,
"updateAvailable": update_available,
"latestVersion": latest_release,
"llmActive": is_llm_active,
"llmProvider": llm_provider,
"oidcActive": is_oidc_active,
"license": "AGPL-3.0-only",
"mode": mode,
"staging": staging,
"cache": cache_details,
"grpcPort": grpc_port,
"store": {
"type": CONFIG.get_storage_mode_string(),
"path": store_endpoint
}
}
```
* feat: Impl Query Server Add: Schema API for Ingestion Server RM: Redundant API Endpoints feat: Last min data is Queryable Add: Unit Tests * fix: func flatten_objects_for_count Fix function flatten_objects_for_count Update Tests * fix: base_path_without_preciding_slah function * fix: Session Context not in sync 1. Session Context was not being synced at the time of schema update 2. Make struct TableScanVisitor pub at crate level
…eablehq#707) * disable standalone when in distributed mode impl standalone/distributed check add migration function - Eshan Chatterjee update StorageMetadata struct add StandaloneWithDistributed error in MetadataError impl ToString for Mode * fix: double print of error message * fix: logic to handle server mode if EnvChange None If the server was restarted with the same staging and in standalone mode the app was not blocking that case * Refactor determine_environment function
fix: bug in response of stats api
* fix: bug in v3 migration function * fix: stats response FIX: When Migrated from Standalone Mode stats updated CHORE: Remove Dead Code
1. add: new endpoint /cluster/metrics 2. add: prometheus metrics parser 3. chore: code cleanup
fix: first_event_at value in GET cluster/info API
* move: store validation to each servers init function * feat: validate ingest server creds on start up On a new Ingest Server Startup server validates the credentials with other Ingest Servers
* fix: query fail when an Ingest Server is down * add delete ingester endpoint 1. Add delete_ingester_meta func in ObjectStorage Trait 2. Update Permission Actions 3. Update PostError 4. Add delete Ingeter Endpoint -> `api/v1/cluster/ingester_ip%3Aingester_port` * fix: handle edge cases while parsing ingester indentifiers
* update the root path for the metadata files only the .parseable.json and the ingest metadata files * migration of metadata files to seperate directorie
Eshanatnight
force-pushed
the
distributed-rebased-on-main
branch
from
6de55c1 to
c37fd58
Compare
nikhilsinhaparseable
approved these changes
Apr 1, 2024
Contributor
nikhilsinhaparseable
left a comment
nikhilsinhaparseable
left a comment
There was a problem hiding this comment.
validated all the subsystems in distributed and standalone mode, approved to be merged
nikhilsinhaparseable
pushed a commit
to nikhilsinhaparseable/parseable
that referenced
this pull request
Apr 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
All the distributed code rebased on current main.
This PR has: