Third-party vendors provide semantically distinct entities—such as events, incidents, and alerts—using varied JSON structures and key names. To normalize inter-vendor and payload heterogeneity for streamlined processing and operational support, raw JSON keys containing relevant event- or incident-related information are reassigned to D3 data model or user-defined field names. This key reassignment process is known as field mapping.
IFM and EFM in the Data Pipeline
Field Mapping Types
There are two field mapping types: Incident Field Mapping (IFM) for incident-level data and Event Field Mapping (EFM) for event-level data. Both occur in the post-ingestion processing pipeline, which varies slightly based on the integration fetch command used.
Fetch commands are of two main types:
Fetch Incident – Applies both incident field mappings and event field mappings.
Fetch Event –Applies only event field mappings.
Fetch Incident - Post-Ingestion Process Flow (IFM and EFM)
flow diagram
The array data representing a D3 event is nested within the array data representing a D3 incident.
incident-event payload
Fetch Event - Post-Ingestion Process Flow (EFM)
flow diagram
event payload
Source Field as Individual Mappings
A source field is a relative JSONPath expression used in EFM and IFM to reference a specific field within the raw JSON payload. The source field begins at the array defined by the Main JSON Path (IFM) or Event Root Path (EFM), and traverses deeper into the JSON structure to reach the desired key.
D3’s system integrations already define the default Main JSON Path, Event Root Path and field mappings. If a data source sends its raw data in a different structure, but the data should still fit those paths and field mappings, you do not need to rebuild the mappings. Use an event formatter to reshape the raw data so that it fits the existing paths and field mappings before D3 processes it. Event formatters are available on webhook data sources and scheduled fetches; on a webhook data source, choose it under Additional Settings > Data Formatter. See Event Formatter.
Source Resolution
Each event or incident source defines how data for a vendor-specific entity type should be interpreted and normalized. In D3, source is implemented as an integration-specific, named configuration that links to an EFM or IFM set. To apply the correct set, D3 must resolve each object within the array defined by the Main JSON Path or Event Root Path to a specific source.
The Event Root Path points to an array, with each object evaluated against configured EFM sources. The Main JSON Path also points to an array, with each object evaluated against configured IFM sources and potentially containing a nested array referenced by the Event Root Path. A source's applicability is determined by a search string, which may be built-in or user-defined.
Example - Event Sources and Search Strings (Zendesk)
objective – Understand the relationship between event sources and search strings.
The Zendesk integration includes five built-in event sources. Each is a mapping set, listed under Field Mapping Sets in the Event Field Mapping window of the Fetch Event command:
Add an incident source, within a site, using the following search string:
{$..discriminator}=value_for_demo_incident_source
Observe the alignment between the search string and the payload's source discriminator.
Search Strings
Each non-default source involves a search string that determines its applicability during source resolution. A search string evaluates data within the Main JSON Path or Event Root Path to determine whether a specific key exists and whether its value matches the configured value. When a key-value pair in the payload matches a configured search string, the corresponding EFM or IFM set is used to normalize the third-party data.
The search string must adhere to one of the following formats:
format 1
{<Descriminator JSONPath>}=<Discriminator's literal value without quotes>
Testing EFM – Search Strings, Source Field, Data Ingestion (Zendesk)
test 1
Navigate to the Fetch Event command of the Zendesk integration.
Under External Access, turn on Accept events from outside D3, then select Webhook keys.
Go to Data Ingestion > Event Intake, select New > Webhook, then choose the Zendesk integration and a site. Under Webhook Keys, select Generate a Key, enter a key name, and select Generate. Note the selected site.
Copy the Request URL, the Request Header Key (d3key) and the Request Header Value to Postman. Optionally, select a preprocessing playbook under Additional Settings.
Under Event Field Mapping, select Setup Event Field Mapping.
Note the Event Root Path.
WHAT IT MEANS *
At runtime, the $.results expression directs the system to iterate over each object in the top-level "results" array.
If this array is not at the top level, the Event Root Path must include either the full path of parent objects or (cautiously) use recursive descent to locate it.
Thesource field (step 8) operates relative to each object in this scope.
Add a mapping set called Field Mappings Demo Event Source 1. Open the Field Mapping Sets menu (…) and select Add Mapping Set, then enter the name. Under Condition, select … to open the Source Field JSON Picker. The picker lists the fields of a sample payload: select + and add the payload from step 5 as custom sample data, then select discriminator and Save. Enter value_for_demo_event_source_1 as the value and select the check mark. The set’s condition reads:
{$.discriminator}=value_for_demo_event_source_1
This instructs the system to evaluate the discriminator key in the payload, and apply this event source when its value equals the string "value_for_demo_event_source_1".
Select Field Mappings Demo Event Source 1, then select Add Mapping. Under Source fields from, select … and pick layer1 > layer2 > layer3 > target. Under Destination fields in D3, search for Description and select it, then select Save.
RELATED RESOURCE *
Learn more about JSONPath expressions for source field here.
Send the POST request prepared in step 4 and step 5.
Verify that an event was created: in the sidebar, go to Events, open the view menu and select All Events.
Note the EventID of the new event. Its Data Source is Zendesk and its Event Source is the mapping set that matched, Field Mappings Demo Event Source 1.
If the list shows more than one site, select the site from step 3 in the site selector.
Select the EventID from step 12 to open the event’s details.
Verify that the Unique Event Key is the id from the payload in step 5 (1).
Verify that the value from the "target" payload field appears in Description, the destination field configured in step 9).
Add a mapping set called Field Mappings Demo Event Source 2 as in step 8: pick discriminator in the Source Field JSON Picker and enter {{2$}} as the value. The set’s condition reads:
{$.discriminator}={{2$}}
This instructs the system to evaluate the discriminator key in the payload, and apply this event source when its value matchesany string that ends with "2".
Add a mapping from layer1 > layer2 > layer3 > target to Description for Field Mappings Demo Event Source 2, as in step 9.
Send the POST request.
Verify that an event was created: go to Events and select All Events. Its Event Source is Field Mappings Demo Event Source 2.
Note the EventID of the new event.
If the list shows more than one site, select the site from step 3 in the site selector.
Select the EventID from step 22 to open the event’s details.
Verify that the Unique Event Key is the id from the payload in step 17 (2).
Verify that the value from the "target" payload field appears in Description, the destination field configured in step 19.
Navigate to the Fetch Incident command of the Microsoft Sentinel integration.
Under External Access, turn on Accept events from outside D3, then select Webhook keys.
Go to Data Ingestion > Incident Intake, select New > Webhook, then choose the Microsoft Sentinel integration and a site. Under Webhook Keys, select Generate a Key, enter a key name, and select Generate. Note the selected site.
Copy the Request URL, the Request Header Key (d3key) and the Request Header Value to Postman. Optionally, select a preprocessing playbook under Additional Settings.
Under Incident Field Mapping, select Setup Incident Field Mapping.
Select the site from step 3, then select Add Source. Enter an Incident Source Name (for example, Field Mappings Demo Incident Source) and the following Search String, then select Save:
{.discriminator}={{ipsum$}}
With the new source selected, select each of the following system fields, enter its Source Field, and select Save.
Verify that the IFM reflects the updated source fields.
Under Event Field Mapping, select Setup Event Field Mapping.
Set the Event Root Path to $.alertDetails.
Add a mapping set called Field Mappings Demo Event Source as in the Zendesk test: add the alertDetails part of the payload as custom sample data in the Source Field JSON Picker, pick Type, and enter demo_type as the value. The set’s condition reads:
{$.Type}=demo_type
This instructs the system to evaluate the $.value.alertDetails.Type key in the payload, and apply this event source when its value equals the string "demo_type".
Select Add Mapping four times to map Tactics to Tactics, Type to Event Type, SystemAlertId to Unique Event Key, and Placeholder to Description.
PROGRESS SUMMARY
The value array serves as the entry point for the IFM.
Each unique object within it (based on incidentNumber) contains data used to generate a D3 incident.
Three IFMs are configured.
The alertDetails array serves as the entry point for the EFM.
Each unique object within it (based on SystemAlertId) contains data used to generate a D3 event.
Four EFMs are configured.
The Description of the D3 event is extracted from the path $.value.alertDetails.Placeholder.
Verify that a D3 incident was created: go to Incidents, select All, and note the Incident No. of the new incident from Microsoft Sentinel.
If the list shows more than one site, select the site from step 3 in the site selector.
Select the incident’s title, then select View Full Details.
Verify that the incident title is the title from the payload, as configured in step 9.
Verify that the incident’s Description also reflects the value configured in step 9.
In Events Summary, expand the event.
Verify that the four values configured in step 13are displayed.
Order Priority
When search strings from different sources simultaneously match different key-value pairs in the ingested JSON payload, only one non-default source is applied. For event mapping sets, the set higher in the Field Mapping Sets list is applied; to change the order, open the Field Mapping Sets menu (…), select Reorder, drag the sets, and select Save. For incident sources, the source with the lower Order Priority value is applied.
The (Default Event Source) mappings apply regardless of Order Priority.
Only one non-default event source applies per post-ingestion pipeline execution.
Search strings from multiple event sources may match fields within the same payload.
For event mapping sets, the set higher in the Field Mapping Sets list takes precedence.
For incident sources, the source with the lower Order Priority value takes precedence, even when it was created later.
In the example below, two incident sources match the same payload and set the title from different fields. With Order Priority 1 and 2, the title came from the first source; after the first source was changed to 3, the title came from the second.
Testing Order Priority (Zendesk)
objective – Understand the effect of order priority when search strings from different sources simultaneously match different key-value pairs in the ingested JSON payload.
Add the following two mapping sets, in this order, as in the Zendesk test above: Field Mappings Demo Priority 1 with condition discriminator1 = value1, then Field Mappings Demo Priority 2 with condition discriminator2 = value2.
Add the following field mappings. The (Default Event Source) maps the id to Unique Event Key by default.
Field Mappings Demo Priority 1: target1 to Description.
Field Mappings Demo Priority 2: target2 to Description.
The two sets in the Field Mapping Sets list, Field Mappings Demo Priority 1 above Field Mappings Demo Priority 2:
Send the POST request.
Verify that an event was created: go to Events and select All Events.
Note the EventID of the new event.
Check its Event Source: Field Mappings Demo Priority 1.
If the list shows more than one site, select the site in which the key was generated.
Select the EventID observed in step 6 to open the event’s details.
Verify that the Unique Event Key matches the id specified in step 2, and that Description shows Demo Event Source 1 matched: both the (Default Event Source) and Field Mappings Demo Priority 1 mappings were applied.
Field Mappings Demo Priority 1 was applied because it is higher in the Field Mapping Sets list than Field Mappings Demo Priority 2.