Add sparse_embedding_format param for sparse_encoding/sparse_tokenize model (#10939)

* add sparse_embedding_format param

Signed-off-by: zhichaog <[email protected]>

* Doc review

Signed-off-by: Fanit Kolchina <[email protected]>

---------

Signed-off-by: zhichaog <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Co-authored-by: Fanit Kolchina <[email protected]>

Author: zhichao-aws

_ml-commons-plugin/pretrained-models.md

@@ -286,9 +286,9 @@ The response contains text embeddings for the provided sentence:
 }
 ```
 
-### Sparse encoding model
+### Sparse encoding model or sparse tokenizer
 
-For a sparse encoding model, send the following request:
+For a sparse encoding model or sparse tokenizer, send the following request:
 
 ```json
 POST /_plugins/_ml/_predict/sparse_encoding/cleMb4kBJ1eYAeTMFFg4
@@ -298,7 +298,7 @@ POST /_plugins/_ml/_predict/sparse_encoding/cleMb4kBJ1eYAeTMFFg4
 ```
 {% include copy-curl.html %}
 
-The response contains the tokens and weights:
+The response contains the extracted tokens and their corresponding weights:
 
 ```json
 {
@@ -323,6 +323,42 @@ The response contains the tokens and weights:
 }
 ```
 
+The preceding example uses the default `lexical` output format, which returns strings as keys. You can control the output key format by setting `parameters.sparse_embedding_format` to `lexical` (returns string tokens) or `token_id` (returns integer token IDs). The following example sets the `sparse_embedding_format` to `token_id`:
+
+```json
+POST /_plugins/_ml/_predict/sparse_encoding/cleMb4kBJ1eYAeTMFFg4
+{
+  "text_docs": ["hello world"],
+  "parameters": {
+    "sparse_embedding_format": "token_id"
+  }
+}
+```
+{% include copy-curl.html %}
+
+The response contains the token IDs and their corresponding token weights:
+
+```json
+{
+  "inference_results": [
+    {
+      "output": [
+        {
+          "dataAsMap": {
+            "response": [
+              {
+                "2088": 3.4208686,
+                "7592": 6.9377565
+              }
+            ]
+          }
+        }
+      ]
+    }
+  ]
+}
+```
+
 ### Cross-encoder model
 
 For a cross-encoder model, send the following request: