ti_opencti: support filtering of indicators and deduplication mechanism

packages/ti_opencti/_dev/build/docs/README.md

@@ -15,7 +15,7 @@ Each event in the Indicator data stream collected by the OpenCTI integration is
 
 This integration requires Filebeat version 8.16.0, or later.
 
-It has been updated for OpenCTI version 5.12.24 and requires that version or later.
+It has been updated for OpenCTI version 6.1.0 and requires that version or later.
 
 ## Setup
 
@@ -25,6 +25,98 @@ When adding the OpenCTI integration, you will need to provide a base URL for the
 
 The simplest authentication method to use is an API key (bearer token). You can find a value for the API key on your profile page in the OpenCTI user interface. Advanced integration settings can be used to configure various OAuth2-based authentication arrangements, and to enter SSL settings for mTLS authentication and for other purposes. For information on setting up the OpenCTI side of an authentication strategy, please refer to [OpenCTI's authentication documentation](https://docs.opencti.io/latest/deployment/authentication/).
 
+### Filtering
+
+The OpenCTI integration supports advanced filtering capabilities to help you control which indicators are ingested. This allows you to focus on specific types of indicators, confidence levels, authors, or time ranges that are most relevant to your security operations.
+
+#### Available Filters
+
+The following filters can be configured when setting up the integration (Note: The integration automatically filters for entity type 'Indicator' only):
+
+- **Pattern Types**: Filter indicators by pattern type (e.g., 'stix'). The values are customizable in OpenCTI, and any custom pattern types defined in your OpenCTI instance are supported (if an observable is associated).
+
+- **Indicator Types**: Filter indicators by type. Values are customizable in OpenCTI. Common defaults include: 'malicious-activity', 'attribution', 'benign', 'anomalous-activity', 'compromised', 'unknown'. Custom types defined in your OpenCTI instance are also supported.
+
+- **Revoked Status**: Filter by revoked status. Set to 'true' to get only revoked indicators, 'false' for only active (non-revoked) indicators, or leave empty to get all indicators regardless of revoked status.
+
+- **Valid From (Start Date)**: Filter indicators with valid_from date after this date. Use ISO 8601 format (e.g., '2024-01-01T00:00:00Z') or relative date expressions (e.g., 'now-30d', 'now-7d').
+
+- **Valid Until (End Date)**: Filter indicators with valid_until date before this date. Use ISO 8601 format (e.g., '2024-12-31T23:59:59Z') or relative date expressions (e.g., 'now+30d', 'now+7d').
+
+- **Label IDs**: Filter by label IDs. Enter the UUIDs of the labels to filter indicators that have these labels applied. **Important: You must use label IDs (UUIDs), not label names.** You can find label IDs in the OpenCTI interface by navigating to Settings > Taxonomies > Labels, or via the API.
+
+- **Minimum Confidence Level**: Filter indicators with confidence level greater than or equal to a specified value (0-100).
+
+- **Author IDs**: Filter by author IDs (createdBy relationship). Enter the UUIDs of the authors to filter indicators created by them. **Important: You must use author IDs (UUIDs), not author names.** You can find author IDs in the OpenCTI interface by clicking on an entity and checking its details, or via the API.
+
+- **Creator IDs**: Filter by technical creator IDs. Enter the UUIDs of the internal users who created the indicators in OpenCTI.
+
+- **Created After**: Filter indicators created after a specific date. Use ISO 8601 format (e.g., '2024-01-01T00:00:00Z') or relative date expressions (e.g., 'now-30d', 'now-7d', 'now-24h').
+
+- **Modified After**: Filter indicators modified after a specific date. Use ISO 8601 format (e.g., '2024-01-01T00:00:00Z') or relative date expressions (e.g., 'now-30d', 'now-7d', 'now-24h').
+
+- **Marking Definition IDs**: Filter by marking definitions (e.g., TLP levels). Enter the UUIDs of the marking definitions. **Important: You must use marking definition IDs (UUIDs), not names.** Common TLP marking IDs:
+  - TLP:CLEAR (TLP:WHITE): `marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da`
+  - TLP:GREEN: `marking-definition--bab4a63c-aed9-4cf5-a766-dfca5abac2bb`
+  - TLP:AMBER: `marking-definition--55d920b0-5e8b-4f79-9ee9-91f868d9b421`
+  - TLP:RED: `marking-definition--e828b379-4e03-4974-9ac4-e53a884c97c1`
+
+#### Filter Examples
+
+Here are some practical examples of filter configurations:
+
+1. **High-confidence indicators only**: Set `Minimum Confidence Level` to 75 to ingest only indicators with high confidence.
+
+2. **Active threat indicators**: Set `Indicator Types` to ['malicious-activity', 'compromised'] and `Revoked Status` to 'false' to focus on active, non-revoked threats.
+
+3. **Currently valid indicators**: Set `Valid From (Start Date)` to 'now-365d' and `Valid Until (End Date)` to 'now+30d' to get indicators that are currently within their validity period.
+
+4. **Recent indicators**: Set `Created After` to 'now-7d' to collect only indicators created in the last 7 days.
+
+5. **Specific pattern types**: Set `Pattern Types` to ['stix'] to collect only STIX pattern indicators, or include your custom pattern types defined in OpenCTI.
+
+6. **Specific campaign tracking**: Use `Label IDs` filter with specific campaign label UUIDs (e.g., ['550e8400-e29b-41d4-a716-446655440000']) to track indicators related to particular threat campaigns.
+
+7. **Indicators from specific sources**: Use `Author IDs` with the UUIDs of specific threat intelligence sources (e.g., ['123e4567-e89b-12d3-a456-426655440000']) to filter indicators from trusted sources.
+
+8. **Recently modified high-value indicators**: Combine `Modified After` set to 'now-24h', `Minimum Confidence Level` to 80, and `Revoked Status` to 'false' to get recently updated, high-confidence active indicators.
+
+9. **TLP-restricted indicators**: Use `Marking Definition IDs` with TLP:CLEAR and TLP:GREEN UUIDs to only ingest indicators that are safe to share broadly: ['marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da', 'marking-definition--bab4a63c-aed9-4cf5-a766-dfca5abac2bb'].
+
+All filters work together using AND logic at the top level. Within each multi-value filter (like pattern types or label IDs), OR logic is applied between values.
+
+#### High Availability and Deduplication
+
+The OpenCTI integration supports running on multiple Elastic Agents for high availability. When multiple agents fetch the same indicators:
+
+- **Automatic Deduplication**: The integration uses a fingerprint-based document ID to prevent duplicates. Each indicator gets a consistent ID based on its `standard_id` and `modified` timestamp.
+- **No Manual Configuration Needed**: Deduplication works automatically - just deploy the integration to multiple agents.
+- **Update Handling**: When an indicator is updated in OpenCTI, the new version replaces the old one in Elasticsearch.
+
+#### Best Practices for HA Setup
+
+1. **Stagger Execution Times**: To avoid all agents hitting OpenCTI simultaneously, consider offsetting their schedules slightly (e.g., Agent 1 at :00, Agent 2 at :02).
+2. **Use the Same Configuration**: Ensure all agents use identical filter settings to fetch the same dataset.
+3. **Monitor Performance**: Check OpenCTI server load when multiple agents are polling.
+
+### Finding IDs in OpenCTI
+
+Since several filters require UUIDs rather than names, here are ways to find these IDs:
+
+1. **Label IDs**:
+   - In OpenCTI UI: Navigate to Settings → Taxonomies → Labels. Click on a label to see its ID in the URL or details.
+   - Via API: Query the `labels` endpoint to list all labels with their IDs.
+
+2. **Author IDs**:
+   - In OpenCTI UI: Click on any entity that has an author, then click on the author name to see its details including the ID.
+   - Via API: Query the `identities` endpoint to list all identities (organizations, individuals) with their IDs.
+
+3. **Creator IDs**:
+   - In OpenCTI UI: Navigate to Settings → Security → Users to see user IDs.
+   - Via API: Query the `users` endpoint (requires appropriate permissions).
+
+For more information about OpenCTI's filtering system, refer to the [OpenCTI filters documentation](https://docs.opencti.io/latest/reference/filters/).
+
 ## Logs
 
 ### Indicator

packages/ti_opencti/_dev/deploy/docker/files/config.yml

@@ -4,7 +4,7 @@ rules:
     request_headers:
       Authorization: "Bearer test_api_key"
       Content-Type: application/json
-    request_body: '/"variables":\{"after":null,"first":3,"orderBy":"modified","orderMode":"asc"\}/'
+    request_body: '/"variables":\{"after":null,"filters":\{"filterGroups":\[\],"filters":\[\{"key":"entity_type","operator":"eq","values":\["Indicator"\]\},\{"key":"revoked","operator":"eq","values":\["false"\]\}\],"mode":"and"\},"first":3,"orderBy":"modified","orderMode":"asc"\}/'
     responses:
       - status_code: 200
         body: |-
@@ -17,7 +17,7 @@ rules:
                       "id": "d019b01c-b637-4eb2-af53-6d527be3193d",
                       "standard_id": "indicator--cde0a6e1-c622-52c4-b857-e9aeac56131b",
                       "is_inferred": false,
-                      "revoked": true,
+                      "revoked": false,
                       "confidence": 15,
                       "lang": "en",
                       "created": "2018-02-05T08:04:53.000Z",
@@ -78,7 +78,7 @@ rules:
                       "id": "82ecce19-82e6-43e4-9565-70488293b7ec",
                       "standard_id": "indicator--c721b81a-6c3c-55ab-9bf3-038642e8add8",
                       "is_inferred": false,
-                      "revoked": true,
+                      "revoked": false,
                       "confidence": 15,
                       "lang": "en",
                       "created": "2020-08-03T06:40:58.000Z",
@@ -139,7 +139,7 @@ rules:
                       "id": "c7253531-0832-4839-accd-b2b3ee950465",
                       "standard_id": "indicator--bc622a61-52e0-5785-91d3-c7371f07f15d",
                       "is_inferred": false,
-                      "revoked": true,
+                      "revoked": false,
                       "confidence": 15,
                       "lang": "en",
                       "created": "2018-02-05T08:04:53.000Z",
@@ -209,7 +209,7 @@ rules:
     request_headers:
       Authorization: "Bearer test_api_key"
       Content-Type: application/json
-    request_body: '/"variables":\{"after":"WzE2NzM5MzQ4MjU3MTYsImluZGljYXRvci0tYmM2MjJhNjEtNTJlMC01Nzg1LTkxZDMtYzczNzFmMDdmMTVkIl0=","first":3,"orderBy":"modified","orderMode":"asc"\}/'
+    request_body: '/"variables":\{"after":"WzE2NzM5MzQ4MjU3MTYsImluZGljYXRvci0tYmM2MjJhNjEtNTJlMC01Nzg1LTkxZDMtYzczNzFmMDdmMTVkIl0=","filters":\{"filterGroups":\[\],"filters":\[\{"key":"entity_type","operator":"eq","values":\["Indicator"\]\},\{"key":"revoked","operator":"eq","values":\["false"\]\},\{"key":"updated_at","operator":"gt","values":\["2023-01-17T05:53:45.716Z"\]\}\],"mode":"and"\},"first":3,"orderBy":"modified","orderMode":"asc"\}/'
     responses:
       - status_code: 200
         body: |-
@@ -222,7 +222,7 @@ rules:
                       "id": "e572d399-18e7-4f76-939b-b8b30c2a79b3",
                       "standard_id": "indicator--65e96880-3652-5aa7-b625-b7b0f7103a37",
                       "is_inferred": false,
-                      "revoked": true,
+                      "revoked": false,
                       "confidence": 15,
                       "lang": "en",
                       "created": "2018-02-05T08:04:53.000Z",
@@ -283,7 +283,7 @@ rules:
                       "id": "f123bd58-30c9-43c8-b3e9-28cc0740afca",
                       "standard_id": "indicator--62337ac9-3e17-50db-8c67-4a499cbca0a4",
                       "is_inferred": false,
-                      "revoked": true,
+                      "revoked": false,
                       "confidence": 15,
                       "lang": "en",
                       "created": "2018-02-05T08:04:53.000Z",
@@ -344,7 +344,7 @@ rules:
                       "id": "225c9e43-8b63-482b-a6af-897ab7f0d289",
                       "standard_id": "indicator--5f95a26a-f9f0-5f8d-9d37-c912539462e3",
                       "is_inferred": false,
-                      "revoked": true,
+                      "revoked": false,
                       "confidence": 15,
                       "lang": "en",
                       "created": "2018-02-05T08:04:53.000Z",
@@ -414,7 +414,7 @@ rules:
     request_headers:
       Authorization: "Bearer test_api_key"
       Content-Type: application/json
-    request_body: '/"variables":\{"after":"WzE2NzM5MzQ4NDA4MTAsImluZGljYXRvci0tNWY5NWEyNmEtZjlmMC01ZjhkLTlkMzctYzkxMjUzOTQ2MmUzIl0=","first":3,"orderBy":"modified","orderMode":"asc"\}/'
+    request_body: '/"variables":\{"after":"WzE2NzM5MzQ4NDA4MTAsImluZGljYXRvci0tNWY5NWEyNmEtZjlmMC01ZjhkLTlkMzctYzkxMjUzOTQ2MmUzIl0=","filters":\{"filterGroups":\[\],"filters":\[\{"key":"entity_type","operator":"eq","values":\["Indicator"\]\},\{"key":"revoked","operator":"eq","values":\["false"\]\},\{"key":"updated_at","operator":"gt","values":\["2023-01-17T05:54:00.81Z"\]\}\],"mode":"and"\},"first":3,"orderBy":"modified","orderMode":"asc"\}/'
     responses:
       - status_code: 200
         body: |-
@@ -427,7 +427,7 @@ rules:
                       "id": "2b3fe8d7-29d9-403b-9454-2be96af85d7b",
                       "standard_id": "indicator--52fd9e7a-2ea9-5e9a-9aa0-9bea534c5158",
                       "is_inferred": false,
-                      "revoked": true,
+                      "revoked": false,
                       "confidence": 15,
                       "lang": "en",
                       "created": "2020-08-03T06:40:55.000Z",
@@ -488,7 +488,7 @@ rules:
                       "id": "f18778b9-f692-490d-9971-72012ea86520",
                       "standard_id": "indicator--363accc6-ab9b-5776-bb80-21abe4a049ce",
                       "is_inferred": false,
-                      "revoked": true,
+                      "revoked": false,
                       "confidence": 15,
                       "lang": "en",
                       "created": "2018-02-05T08:04:53.000Z",
@@ -552,7 +552,7 @@ rules:
                       "id": "32b60cc0-4b03-4b28-8436-0979ae34712e",
                       "standard_id": "indicator--35e958f3-978e-56a9-9801-121c92a7142e",
                       "is_inferred": false,
-                      "revoked": true,
+                      "revoked": false,
                       "confidence": 15,
                       "lang": "en",
                       "created": "2018-02-05T08:04:53.000Z",

packages/ti_opencti/changelog.yml

@@ -1,4 +1,18 @@
 # newer versions go on top
+- version: "2.10.0"
+  changes:
+    - description: Add comprehensive filtering support for indicators including pattern types, confidence levels, labels, dates, authors, creators, and marking definitions.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/15876
+    - description: Implement deduplication mechanism using fingerprint processor to prevent duplicate indicators.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/15876
+    - description: Add state management to track last modified timestamp and prevent re-fetching already processed indicators.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/15876
+    - description: Update OpenCTI logo for better visual consistency.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/15876
 - version: "2.9.0"
   changes:
     - description: Avoid adding documents with errors to the transform destination index.

packages/ti_opencti/data_stream/indicator/_dev/test/system/test-default-config.yml

@@ -9,5 +9,6 @@ data_stream:
     page_size: 3
     preserve_original_event: true
     enable_request_tracer: true
+    revoked: "false"
 assert:
   hit_count: 9