Index configuration update

docs/configuration/index-config.md

@@ -12,8 +12,9 @@ In addition to the `index_id`, the index configuration lets you define five item
 - The **doc mapping**: it defines how a document and the fields it contains are stored and indexed for a given index.
 - The **indexing settings**: it defines the timestamp field used for sharding, and some more advanced parameters like the merge policy.
 - The **search settings**: it defines the default search fields `default_search_fields`, a list of fields that Quickwit will search into if the user query does not explicitly target a field.
+- The **retention policy**: it defines how long Quickwit should keep the indexed data. If not specified, the data is stored forever.
 
-Configuration is set at index creation and cannot be modified with the current version of Quickwit.
+Configuration is generally set at index creation and cannot be modified, except for some specific attributes like the search settings and retention policy, which can be changed using the [update endpoint](../reference/rest-api.md) or the [CLI](../reference/cli.md).
 
 ## Config file format
 

docs/reference/cli.md

@@ -116,7 +116,7 @@ quickwit run
 | Option | Description | Default |
 |-----------------|-------------|--------:|
 | `--config` | Config file location | `config/quickwit.yaml` |
-| `--service` | Services (indexer,searcher,janitor,metastore or control-plane) to run. If unspecified, all the supported services are started. |  |
+| `--service` | Services (`indexer`, `searcher`, `metastore`, `control-plane`, or `janitor`) to run. If unspecified, all the supported services are started. |  |
 
 *Examples*
 
@@ -141,7 +141,7 @@ curl "http://127.0.0.1:7280/api/v1/wikipedia/search?query=barack+obama"
 ```
 
 ## index
-Manages indexes: creates, deletes, ingests, searches, describes...
+Manages indexes: creates, updates, deletes, ingests, searches, describes...
 
 ### index create
 
@@ -180,6 +180,51 @@ quickwit index create --endpoint=http://127.0.0.1:7280 --index-config wikipedia_
 
 ```
 
+### index update
+
+`quickwit index update [args]`
+#### index update search-settings
+
+Updates default search settings.  
+`quickwit index update search-settings [args]`
+
+*Synopsis*
+
+```bash
+quickwit index update search-settings
+    --index <index>
+    --default-search-fields <default-search-fields>
+```
+
+*Options*
+
+| Option | Description |
+|-----------------|-------------|
+| `--index` | ID of the target index |
+| `--default-search-fields` | List of fields that Quickwit will search into if the user query does not explicitly target a field. Space-separated list, e.g. "field1 field2". If no value is provided, existing defaults are removed and queries without target field will fail. |
+#### index update retention-policy
+
+Configure or disable the retention policy.  
+`quickwit index update retention-policy [args]`
+
+*Synopsis*
+
+```bash
+quickwit index update retention-policy
+    --index <index>
+    [--period <period>]
+    [--schedule <schedule>]
+    [--disable]
+```
+
+*Options*
+
+| Option | Description |
+|-----------------|-------------|
+| `--index` | ID of the target index |
+| `--period` | Duration after which splits are dropped. Expressed in a human-readable way (`1 day`, `2 hours`, `1 week`, ...) |
+| `--schedule` | Frequency at which the retention policy is evaluated and applied. Expressed as a cron expression (0 0 * * * *) or human-readable form (hourly, daily, weekly, ...). |
+| `--disable` | Disable the retention policy. Old indexed data will not be cleaned up anymore. |
 ### index clear
 
 Clears an index: deletes all splits and resets checkpoint.  
@@ -661,8 +706,8 @@ quickwit split list
 | Option | Description |
 |-----------------|-------------|
 | `--index` | Target index ID |
-| `--offset` | Number of splits to skip |
-| `--limit` | Maximum number of splits to retrieve |
+| `--offset` | Number of splits to skip. |
+| `--limit` | Maximum number of splits to retrieve. |
 | `--states` | Selects the splits whose states are included in this comma-separated list of states. Possible values are `staged`, `published`, and `marked`. |
 | `--create-date` | Selects the splits whose creation dates are before this date. |
 | `--start-date` | Selects the splits that contain documents after this date (time-series indexes only). |

docs/reference/rest-api.md

@@ -223,13 +223,13 @@ The response is a JSON object, and the content type is `application/json; charse
 POST api/v1/indexes
 ```
 
-Create an index by posting an `IndexConfig` payload. The API accepts JSON with `content-type: application/json`) and YAML `content-type: application/yaml`.
+Create an index by posting an `IndexConfig` payload. The API accepts JSON with `content-type: application/json` and YAML `content-type: application/yaml`.
 
 #### POST payload
 
-| Variable              | Type               | Description                                                                                                           | Default value                         |
-|-----------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------|
-| `version`          | `String`           | Config format version, use the same as your Quickwit version. (mandatory)                                             |                                       |
+| Variable            | Type               | Description                                                                                                           | Default value                         |
+|---------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+| `version`           | `String`           | Config format version, use the same as your Quickwit version. (mandatory)                                             |                                       |
 | `index_id`          | `String`           | Index ID, see its [validation rules](../configuration/index-config.md#index-id) on identifiers. (mandatory)           |                                       |
 | `index_uri`         | `String`           | Defines where the index files are stored. This parameter expects a [storage URI](../configuration/storage-config.md#storage-uris).           | `{default_index_root_uri}/{index_id}` |
 | `doc_mapping`       | `DocMapping`       | Doc mapping object as specified in the [index config docs](../configuration/index-config.md#doc-mapping) (mandatory)  |                                       |
@@ -301,8 +301,62 @@ curl -XPOST http://0.0.0.0:8080/api/v1/indexes --data @index_config.json -H "Con
 
 The response is the index metadata of the created index, and the content type is `application/json; charset=UTF-8.`
 
-| Field                | Description                               |         Type          |
-|----------------------|-------------------------------------------|:---------------------:|
+| Field                | Description                             |         Type          |
+|----------------------|-----------------------------------------|:---------------------:|
+| `index_config`     | The posted index config.                  |     `IndexConfig`     |
+| `checkpoint`       | Map of checkpoints by source.             |   `IndexCheckpoint`   |
+| `create_timestamp` | Index creation timestamp                  |       `number`        |
+| `sources`          | List of the index sources configurations. | `Array<SourceConfig>` |
+
+
+### Update an index (search settings and retention policy only)
+
+```
+PUT api/v1/indexes/<index id>
+```
+
+Updates the search settings and retention policy of an index. This endpoint follows PUT semantics (not PATCH), which means that all the updatable fields of the index configuration are replaced by the values specified in this request. In particular, omitting an optional field like retention_policy will delete the associated configuration. Unlike the create endpoint, this API only accepts JSON payloads.
+
+#### PUT payload
+
+| Variable            | Type               | Description                                                                                                           | Default value                         |
+|---------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+| `search_settings`   | `SearchSettings`   | Search settings object as specified in the [index config docs](../configuration/index-config.md#search-settings).     |                                       |
+| `retention`         | `Retention`        | Retention policy object as specified in the [index config docs](../configuration/index-config.md#retention-policy).   |                                       |
+
+
+**Payload Example**
+
+curl -XPUT http://0.0.0.0:8080/api/v1/indexes --data @index_update.json -H "Content-Type: application/json"
+
+```json title="index_update.json
+{
+    "search_settings": {
+        "default_search_fields": ["body"]
+    },
+    "retention": {
+        "period": "3 days",
+        "schedule": "@daily"
+    }
+}
+```
+
+:::warning
+Calling the update endpoint with the following payload will remove the current retention policy.
+```json
+{
+    "search_settings": {
+        "default_search_fields": ["body"]
+    }
+}
+```
+
+#### Response
+
+The response is the index metadata of the updated index, and the content type is `application/json; charset=UTF-8.`
+
+| Field                | Description                             |         Type          |
+|----------------------|-----------------------------------------|:---------------------:|
 | `index_config`     | The posted index config.                  |     `IndexConfig`     |
 | `checkpoint`       | Map of checkpoints by source.             |   `IndexCheckpoint`   |
 | `create_timestamp` | Index creation timestamp                  |       `number`        |

quickwit/quickwit-cli/src/generate_markdown.rs

@@ -43,11 +43,13 @@ fn markdown_for_subcommand(
     subcommand: &Command,
     command_group: Vec<String>,
     doc_extensions: &toml::Value,
+    level: usize,
 ) {
     let subcommand_name = subcommand.get_name();
 
     let command_name = format!("{} {}", command_group.join(" "), subcommand_name);
-    println!("### {command_name}\n");
+    let header_level = "#".repeat(level);
+    println!("{header_level} {command_name}\n");
 
     let subcommand_ext: Option<&Value> = {
         let mut val_opt: Option<&Value> = doc_extensions.get(command_group[0].to_string());
@@ -195,20 +197,20 @@ fn generate_markdown_from_clap(command: &Command) {
             continue;
         }
 
-        let excluded_doc_commands = ["merge"];
+        let excluded_doc_commands = ["merge", "local-search"];
         for subcommand in command
             .get_subcommands()
             .filter(|subcommand| !excluded_doc_commands.contains(&subcommand.get_name()))
         {
             let commands = vec![command.get_name().to_string()];
-            markdown_for_subcommand(subcommand, commands, &doc_extensions);
+            markdown_for_subcommand(subcommand, commands, &doc_extensions, 3);
 
             for subsubcommand in subcommand.get_subcommands() {
                 let commands = vec![
                     command.get_name().to_string(),
                     subcommand.get_name().to_string(),
                 ];
-                markdown_for_subcommand(subsubcommand, commands, &doc_extensions);
+                markdown_for_subcommand(subsubcommand, commands, &doc_extensions, 4);
             }
         }
     }

quickwit/quickwit-cli/src/index.rs → quickwit/quickwit-cli/src/index/mod.rs

@@ -54,13 +54,16 @@ use tabled::{Table, Tabled};
 use thousands::Separable;
 use tracing::{debug, Level};
 
+use self::update::{build_index_update_command, IndexUpdateCliCommand};
 use crate::checklist::GREEN_COLOR;
 use crate::stats::{mean, percentile, std_deviation};
 use crate::{client_args, make_table, prompt_confirmation, ClientArgs, THROUGHPUT_WINDOW_SIZE};
 
+pub mod update;
+
 pub fn build_index_command() -> Command {
     Command::new("index")
-        .about("Manages indexes: creates, deletes, ingests, searches, describes...")
+        .about("Manages indexes: creates, updates, deletes, ingests, searches, describes...")
         .args(client_args())
         .subcommand(
             Command::new("create")
@@ -74,9 +77,12 @@ pub fn build_index_command() -> Command {
                         .required(false),
                 ])
             )
+        .subcommand(
+            build_index_update_command().display_order(2)
+        )
         .subcommand(
             Command::new("clear")
-                .display_order(2)
+                .display_order(3)
                 .alias("clr")
                 .about("Clears an index: deletes all splits and resets checkpoint.")
                 .long_about("Deletes all its splits and resets its checkpoint. This operation is destructive and cannot be undone, proceed with caution.")
@@ -88,7 +94,7 @@ pub fn build_index_command() -> Command {
             )
         .subcommand(
             Command::new("delete")
-                .display_order(3)
+                .display_order(4)
                 .alias("del")
                 .about("Deletes an index.")
                 .long_about("Deletes an index. This operation is destructive and cannot be undone, proceed with caution.")
@@ -102,7 +108,7 @@ pub fn build_index_command() -> Command {
             )
         .subcommand(
             Command::new("describe")
-                .display_order(4)
+                .display_order(5)
                 .about("Displays descriptive statistics of an index.")
                 .long_about("Displays descriptive statistics of an index. Displayed statistics are: number of published splits, number of documents, splits min/max timestamps, size of splits.")
                 .args(&[
@@ -113,12 +119,12 @@ pub fn build_index_command() -> Command {
         .subcommand(
             Command::new("list")
                 .alias("ls")
-                .display_order(5)
+                .display_order(6)
                 .about("List indexes.")
             )
         .subcommand(
             Command::new("ingest")
-                .display_order(6)
+                .display_order(7)
                 .about("Ingest NDJSON documents with the ingest API.")
                 .long_about("Reads NDJSON documents from a file or streamed from stdin and sends them into ingest API.")
                 .args(&[
@@ -148,14 +154,14 @@ pub fn build_index_command() -> Command {
                         .conflicts_with("wait"),
                     Arg::new("commit-timeout")
                         .long("commit-timeout")
-                        .help("Duration of the commit timeout operation.")
+                        .help("Timeout for ingest operations that require waiting for the final commit (`--wait` or `--force`). This is different from the `commit_timeout_secs` indexing setting, which sets the maximum time before commiting splits after their creation.")
                         .required(false)
                         .global(true),
                 ])
             )
         .subcommand(
             Command::new("search")
-                .display_order(7)
+                .display_order(8)
                 .about("Searches an index.")
                 .args(&[
                     arg!(--index <INDEX> "ID of the target index")
@@ -256,6 +262,7 @@ pub enum IndexCliCommand {
     Ingest(IngestDocsArgs),
     List(ListIndexesArgs),
     Search(SearchIndexArgs),
+    Update(IndexUpdateCliCommand),
 }
 
 impl IndexCliCommand {
@@ -278,6 +285,7 @@ impl IndexCliCommand {
             "ingest" => Self::parse_ingest_args(submatches),
             "list" => Self::parse_list_args(submatches),
             "search" => Self::parse_search_args(submatches),
+            "update" => Ok(Self::Update(IndexUpdateCliCommand::parse_args(submatches)?)),
             _ => bail!("unknown index subcommand `{subcommand}`"),
         }
     }
@@ -438,6 +446,7 @@ impl IndexCliCommand {
             Self::Ingest(args) => ingest_docs_cli(args).await,
             Self::List(args) => list_index_cli(args).await,
             Self::Search(args) => search_index_cli(args).await,
+            Self::Update(args) => args.execute().await,
         }
     }
 }