Agent Management

Agents are services that run on an external server. Agents act as a liaison between the server and the D3 SOAR Platform to ensure security and stability for data ingestion.

The Agent Management module allows you to quickly manage, deploy, update and monitor Agents in the D3 SOAR Platform. Once installed, the Agents will be listed on the left side with details such as Agent Name, Description, Version, Site and Status. You can click on any Agent from the list to view its status, version, service URL, location, host, IP and Mac Address in the right-side panel.

Proxy Agent Overview

D3 Proxy Agent is used to assist the communication (1) between D3 Web Server and certain integrations sitting in a local environment for the SaaS solution or (2) between D3 Web Server and certain integrations sitting in a separate environment disconnected from the environment (via Internet) where Web Server is sitting for the On-Premises solution.

Figure 1: D3 SOAR Proxy Agent Architecture Diagram for SaaS Implementation

Figure 2: D3 SOAR Proxy Agent Architecture Diagram for On-Premise Implementation

D3 SOAR’s proxy agent needs to be installed in the local environment to initiate the connection to the D3 SOAR web server. After a connection has been established, the D3 SOAR web server will be able to send a request to the proxy agent, then pass the request to integrations installed on a separate environment. The integrations will send responses back to the D3 SOAR web server via the proxy agent.

D3 Proxy Agent was previously hosted in a Windows container. From v15 onwards, it is hosted in a Linux container. D3 is supporting the docker installed on Ubuntu or Redhat.

Proxy Agent Version Compatibility Matrix

Below is a table indicating the recommended version numbers for the agent and executor to be installed with each release version of D3 SOAR, starting from version 16.0.18.3. It is important to avoid installing incompatible versions of the agent and executor when updating your instance of D3 SOAR, as it may lead to issues. Please refer to the table for the appropriate versions:

Proxy Agent Installation on Linux

Environment Requirement

Hardware (Can be virtual)

Operating System

Firewall Ports

Please Note:

1. The server for D3 Proxy Agent must be able to communicate with D3 VSOC servers via TCP ports 443.

2. D3 Proxy Agent must be installed on servers that have direct access to all your internal resources, such as firewalls, SIEM, Exchange, etc.

3. If Web Proxy is enabled in your network environment, Web Proxy IP needs to be provided for docker-compose configuration during the installation.(See Using D3 Proxy Agent with a Web Proxy Server)

Install Docker

On Ubuntu

Follow the instructions from the documentation to install docker using the apt package manager: https://docs.docker.com/engine/install/ubuntu/

Alternatively, you can install from a package with these instructions:

Link to distributions: https://download.docker.com/linux/ubuntu/dists/

Download and install four .deb packages from <version>/pool/stable/amd64:

1. docker-ce-cli

2. docker-ce

3. containerd.io

4. docker-compose-plugin

Command to install on ubuntu: sudo dpkg -i /path/to/<replace_with_package>.deb

Command to check if docker service is running: sudo systemctl status docker

On Redhat

Follow the instructions from the documentation to install docker using the yum package manager: https://docs.docker.com/engine/install/rhel/

Alternatively, you can install from a package with these instructions:

Link to package distributions: https://download.docker.com/linux/rhel/

Download and install four .rpm packages from <version>/s390x/stable/packages:

1. docker-ce-cli

2. docker-ce

3. containerd.io

4. docker-compose-plugin

Command to install on redhat: sudo yum install /path/to/<replace_with_package>.rpm

Command to start docker server: sudo systemctl start docker

Authenticate to Google Container Registry

The D3 Proxy Agent image hosted on Google Container Registry is private and requires authentication to pull it.

D3 Security will provide a json file containing credentials to authenticate, which can be done with the following command in your Proxy Agent server:

sudo docker login -u _json_key --password-stdin https://gcr.io < service_account_d3soar.json

Reader Note

Please contact the D3 account representative to obtain the JSON file.

Run docker-compose to Start Containers

On the Proxy Agent, we will need to run both the d3executor and the d3agent containers. A docker-compose file is provided to run both the d3executor and the d3agent.

Create a new file named docker-compose.yml and paste the following content into the file.

version: "3.3"
services:
  d3agent:
    image: "gcr.io/nimble-cortex-285618/d3prod/d3agent:<Agent Version Number>"
    restart: always
    environment:
      - REMOTE_SERVER_URL=<VSOC Url>
      - PROXY_IDENTITY=<VSOC Agent Token>
      - PYTHON_REMOTE_URL=http://<Docker host ip>:9092/
      - PYTHON_REMOTE_GUID=7e036a86dbee40d9913c3794e779eae4
      - SERVICE_DISPLAY_NAME=<Proxy Agent Display Name>
  d3executor:
    image: "gcr.io/nimble-cortex-285618/d3prod/d3executor:<Executor Version number>"
    ports:
      - "9092:9090"
    restart: always
    environment:
      - EXECUTOR_GUID=7e036a86dbee40d9913c3794e779eae4

Reader Note

Please refer to the Proxy Agent Version Compatibility Matrix to determine the appropriate version numbers for your compatible agent and executor and replace <Agent Version Number> and <Executor Version number> accordingly. 

For instance, if you are using D3 SOAR release version 16.0.18.3, the value for <Agent Version Number> should be 16.0.18.0 and for <Executor Version Number> it should be 16.0.18.0.

Please note that the <VSOC Url>, <VSOC Agent Token>, <Docker host ip> and <Proxy Agent Display Name> need to be replaced with specific values. The port for docker host needs to be aligned for service d3agent and service d3executor, e.g. 9092.

VSOC Agent Token can be generated from the D3 SOAR application.

Open a terminal and browse to the same directory as where the docker-compose.yml file is located and run the following command:

sudo docker-compose up -d

Firewall Whitelisting for Docker installation and D3 Proxy Agent Container Image

If all internet traffic is blocked by default, then certain Domains/IPs will need to be whitelisted in order to be able to download the docker package as well as the images hosted on GCR.

1. Sites that require whitelisting

1. Docker (for downloading docker package):

   1. download.docker.com

2. GCR (for downloading docker images from GCR):

   1. gcr.io

   2. storage.googleapis.com

2. Example of Checking IP address

Use nslookup to check IP address of the domain that needs to be accessed

We can see that download.docker.com has a few IP addresses that it uses. We can allow traffic to this domain download.docker.com or to an IP address list that includes all the resolved public IP (the same domain may resolve to different public IP from different regions, if you use IP address for the whitelist, you need to run nslookup to get these IP from your server.)

Using D3 Proxy Agent with a Web Proxy Server

To route D3 Proxy Agent’s outbound internet connections to a web proxy server, make the following configuration changes.

Open docker-compose.yml. Add the following to the Environment section:

HTTP_PROXY=<Web Proxy Server:Port>

HTTPS_PROXY=<Web Proxy Server:Port>

NO_PROXY=localhost,127.0.0.1,<Docker host ip>,<Other local integration host address,...>

docker-compose.yml

version: "3.3"
services:
  d3agent:
    image: "gcr.io/nimble-cortex-285618/d3prod/d3agent:15.3.72.0"
    restart: always
    environment:
      - REMOTE_SERVER_URL=<VSOC Url>
      - PROXY_IDENTITY=<VSOC Agent Token>
      - PYTHON_REMOTE_URL=http://<Docker host ip>:9092/
      - PYTHON_REMOTE_GUID=7e036a86dbee40d9913c3794e779eae4
      - SERVICE_DISPLAY_NAME=<Proxy Agent Display Name>
      - HTTP_PROXY=<Web Proxy Server IP:Port>
      - HTTPS_PROXY=<Web Proxy Server IP:Port>
      - NO_PROXY=localhost,127.0.0.1,<Docker host ip>,<Other local integration host address,...>
  d3executor:
    image: "gcr.io/nimble-cortex-285618/d3prod/d3executor:15.1.39.1"
    ports:
      - "9092:9090"
    restart: always
    environment:
      - EXECUTOR_GUID=7e036a86dbee40d9913c3794e779eae4
      - HTTP_PROXY=<Web Proxy Server IP:Port>
      - HTTPS_PROXY=<Web Proxy Server IP:Port>
      - NO_PROXY=localhost,127.0.0.1,<Docker host ip>,<Other local integration host address,...>

HTTP_PROXY and HTTPS_PROXY are going to be the web proxy server HTTP and HTTPS connections respectively. You will need to add more addresses for the private IP, DNS name, and all other local integrations you want to connect to (i.e. skipping the proxy) to the NO_PROXY.

Proxy Agent Installation on Windows

Environment Requirement

Server Hardware Requirements

Operating System

Firewall Ports

Please Note:

1. The server for D3 Proxy Agent must be able to communicate with D3 VSOC servers via TCP ports 443.

2. D3 Proxy Agent must be installed on servers that have direct access to all your internal resources, such as firewalls, SIEM, Exchange, etc.

3. If Web Proxy is enabled in your network environment, Web Proxy IP needs to be configured in the D3PlaybookAgent.exe.config during the installation.(See How to Configure Internet Proxy Settings)

Installing New Agents

The Agent installation process is done via an Installation Wizard. The installation package can be downloaded directly from D3 SOAR. The steps below will outline the process of installing new Agents.

To install a new Agent:

1. Download Proxy Agent Package from within VSOC GUI

3. Save the Security Token Key
   Generate Security Token, copy and paste it in a txt file. This token will be used during the installation process.

2. Unzip the Installation Package

Reader Note

The Zip file size should be 170MB or larger. If it is around 3MB, please see the Troubleshooting section.

Make sure this zip file was NOT blocked by Windows. Right click on the file and check the block state.  If it is blocked, Unblock it first before you unzip. Otherwise, even if you can unzip it, but some unzipped files are still locked, you may NOT be able to install it successfully. The installation will not be able to proceed and hang there.

4. Right click and run D3PlaybookAgentInstaller.exe as administrator

5. Input VSOC Server URL, Security Token, Agent Service Name, Agent Name. Description and Location details are optional.

Click Next and wait for Installation to complete.

Please see the Troubleshooting section if the install does not complete.

6. Check the Proxy Agent Service Status in Windows Services. Make sure the proxy agent is running properly.

Check the logs to see connection status:
C:\Program Files (x86)\D3security\D3PlaybookAgent\log\D3PlaybookAgent.log

How to Configure Internet Proxy Settings

In some environments, the D3 Proxy Agent can NOT access the Internet directly, it has to access the Internet via a Proxy server.

Edit the D3PlaybookAgent.exe.config file, add Proxy setting as follows:

C:\Program Files (x86)\D3Security\D3PlaybookAgent14\D3PlaybookAgent.exe.config

Add proxy configuration between <system.net>..... </system.net> shown below.

Reader Note

This example set the proxy server IP 192.168.1.10, port 3128

All local IP addresses 192.168.xxx.xxx are bypassed.

Proxy Agent High Availability

Proxy Agent supports High Availability by installing multiple Proxy Agents in different folders. Each agent must use a unique Secure Token.

Example

1. Install the first Agent in \D3PlaybookAgent01 folder

2. Each Agent needs a unique Security Token. Please generate the token in VSOC page.

3. Enter a unique Security Token, Agent Service Name and Agent Name for this Agent

4. Install the 2nd Agent by right clicking D3playbookAgentInstaller.exe and run as administrator. Choose Install a new agent.

5. Use a unique folder name, for example \D3PlaybookAgent02, Keep "Install Python" checked and leave the folder as it is. (That will configure the Python path for the 2nd Proxy Agent)

6. Use a Unique Security Token, Agent Service Name and Agent Name

Complete the installation.

7. After 2nd agent installation finished, you will see two folders

8. Check the service status in Windows Task Manager. Make sure the services are running.

9. Check the logs at C:\Program Files (x86)\D3security\D3PlaybookAgent01\log, State:Connected

10. Check the logs at C:\Program Files (x86)\D3security\D3PlaybookAgent02\log, State:Connected

11. From the VSOC Agent Management interface, both of the Proxy Agents Status should be Connected. Installation has finished and is successful.

Proxy Agent Windows Server Hardening

How Proxy Agent Works

In short, the Proxy Agent does NOT need incoming TCP connections. It always initiates TCP connections to VSOC server and RestfulAPI targets. So we can block all incoming connections to the Windows server where D3 Proxy Agent is running on. Just run the powershell scripts to disable all the default enabled incoming firewall rules on Windows Server.

Troubleshooting

Package download is NOT successful

Agent Key

Download PlaybookAgent from VSOC GUI. The file size of D3PlaybookAgentInstaller14.0.xxx.x.zip should be at least 170MB or larger which includes the Python package. If the download does NOT include the Python package, its size may be around 3MB and requires troubleshooting.

On DB Server, in SSMS (SQL Server Management Studio), right click the database, run new query

1. Run the following command to check if the Proxy Agent key exists or not.

select * from VSOC_Settings_Default where keyname like 'Cyber.PythonInstallationFolderForPlaybookAgent'

2. If the PlaybookAgent key does not exist, run the insert command to add it to the database.

insert into VSOC_Settings_Default (keyName, keyValue) values ('Cyber.PythonInstallationFolderForPlaybookAgent', 'C:\Python\Python38')

3. Run the update command to update the key value.

update VSOC_Settings_Default set keyValue = 'C:\Python\Python38' where keyname like 'Cyber.PythonInstallationFolderForPlaybookAgent'

Reader Note

C:\Python\Python38 is an example here.

It is the Python path on your VSOC server. Change it to your Python path where Python.exe resides.

4. After updating the key into the Database, stop D3TaskManager services, stop App Pool, then restart the services and App Pool again.

5. Delete Agent folder under C:\D3App\D3Temp\Agent or C:\D3App\Virtual_Dir\D3Temp\Agent
   See examples below. Your D3App folder may be on other disks like D:\ or E:\ as well.

No File Downloaded

When attempting to download the Agent, sometimes no file is downloaded. In this case, check your D3Temp directory properties. Make sure users have full control so the Agent software can be written into this folder.

1. Set users with full control.

2. Go to VSOC to download the agent again. The correct zipped installation package size should be at least 170MB or larger.

Reader Note

If user VSOC was updated to a new version, you need to download and update your Proxy agent accordingly.
The Proxy Agent version should match the VSOC version.

3. Download new Proxy Agent and install

Reader Note

You can install multiple Proxy Agents, each with a different Security Token.

Installing Agent Not Completing

If the installation stalls and does not complete, the root cause is that the Playbook Installation package was blocked by Windows when downloading from VSOC servers.

1. Go to the .zip file and view Properties

2. Check the Unblock box before unzipping the file. Even if the file is able to unzip successfully without this permission, some files will still be blocked by Windows. Ensure that this is checked before proceeding.

3. Run the installation again

Agent Configuration Scenarios

Overview

On-Premises Integration Example: IBM QRadar Custom Command

Scenario 1

☑ Allow command to be run on agent
Connected agent selected in the Connection
Result Command was run on the Agent Python Executor completely and result was returned successfully

Scenario 2

☑ Allow command to be run on agent
Disconnected agent selected in the Connection
Result The execution of the command was stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all

Scenario 3

☑ Allow command to be run on agent
NO agent selected in the Connection
Result Command was run on the VSOC Remote Python Executor first. Rest API request would fail since it cannot connect to integration inside VSOC environment

Scenario 4

☐ Allow command to be run on agent
Connected agent selected in the Connection
Result Command was run on the VSOC Remote Python Executor. Rest API request was redirected to run on the Agent Result was returned successfully

Scenario 5

☐ Allow command to be run on agent
Disconnected agent selected in the Connection
Result The execution of the command was stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all

Scenario 6

☐ Allow command to be run on agent
NO agent selected in the Connection
Result Command was run on the VSOC Remote Python Executor first. Rest API request would fail since it cannot connect to integration inside VSOC environment

On Cloud Integration Example: VirusTotal V3 Custom Command

Scenario 1

☑ Allow command to be run on agent
Connected agent selected in the Connection
Result Command was run on the Agent Python Executor completely and result was returned successfully

Scenario 2

☑ Allow command to be run on agent
Disconnected agent selected in the Connection
Result The execution of the command was stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all

Scenario 3

☑ Allow command to be run on agent
NO agent selected in the Connection
Result Command was run on the VSOC Remote Python Executor completely and result was returned successfully

Scenario 4

☐ Allow command to be run on agent
Connected agent selected in the Connection
Result Command was run on the VSOC Remote Python Executor. Rest API request was redirected to run on the Agent Result was returned successfully

Scenario 5

☐ Allow command to be run on agent
Disconnected agent selected in the Connection
Result The execution of the command was stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all

Scenario 6

☐ Allow command to be run on agent
NO agent selected in the Connection
Result Command was run on the VSOC Remote Python Executor completely and result was returned successfully

Network Configuration for D3 Application in Restricted Client-Side Environments

To ensure the proper functioning of the D3 application in a client-side network with default outbound traffic restrictions, please perform the following steps:

1. Ping the D3 application URL (e.g., smartsoar.d3securityonline.com) to determine the required inbound IPs.

2. Whitelist the identified inbound IPs on your firewall's outbound rules. This step will allow the necessary communication between the D3 application and the external network.

Please note that these steps should be followed in the following scenarios:

• When the D3 proxy agent is installed in a client-side network that blocks outbound traffic to the internet by default.

• When client users are accessing the D3 application within a client-side network that denies outbound traffic to the internet by default.