Agent Management
Agents are services that run on an external server. Agents act as a liaison between the server and the D3 SOAR Platform to ensure security and stability for data ingestion.
The Agent Management module allows you to quickly manage, deploy, update and monitor Agents in the D3 SOAR Platform. Once installed, the Agents will be listed on the left side with details such as Agent Name, Description, Version, Site and Status. You can click on any Agent from the list to view its status, version, service URL, location, host, IP and Mac Address in the right-side panel.
Proxy Agent Overview
D3 Proxy Agent is used to assist the communication (1) between D3 Web Server and certain integrations sitting in a local environment for the SaaS solution or (2) between D3 Web Server and certain integrations sitting in a separate environment disconnected from the environment (via Internet) where Web Server is sitting for the On-Premises solution.
Figure 1: D3 SOAR Proxy Agent Architecture Diagram for SaaS Implementation
Figure 2: D3 SOAR Proxy Agent Architecture Diagram for On-Premise Implementation
D3 SOAR’s proxy agent needs to be installed in the local environment to initiate the connection to the D3 SOAR web server. After a connection has been established, the D3 SOAR web server will be able to send a request to the proxy agent, then pass the request to integrations installed on a separate environment. The integrations will send responses back to the D3 SOAR web server via the proxy agent.
D3 Proxy Agent was previously hosted in a Windows container. From v15 onwards, it is hosted in a Linux container. D3 is supporting the docker installed on Ubuntu or Redhat.
Proxy Agent Version Compatibility
Alert
It is important to avoid installing incompatible versions of the agent and executor when updating your instance of D3 SOAR, as it may lead to issues. Ensure you update the agent (applicable to both SaaS and on-premise deployments) and executor (only for on-premise deployments) versions to the matching D3 SOAR release version. This release version can be found on the sign-in page of D3 SOAR.
Proxy Agent Installation on Linux
Environment Requirement
Hardware (Can be virtual)
Component | Minimum | Recommended |
---|---|---|
CPU | 4 CPU Cores | 8 CPU Cores |
Memory | 8GB RAM | 16GB RAM |
Host Local Storage | 30GB SSD | 100GB SSD (Raid 1 or 10) |
Operating System
Version | License |
---|---|
Linux Ubuntu / Redhat | - |
Firewall Ports
Service | Protocol | Port | Direction | Firewall / Proxy |
---|---|---|---|---|
D3 SOAR Proxy Agent | TCP | 443 | Outbound to Internet | Allow |
Please Note:
The server for D3 Proxy Agent must be able to communicate with D3 VSOC servers via TCP ports 443.
D3 Proxy Agent must be installed on servers that have direct access to all your internal resources, such as firewalls, SIEM, Exchange, etc.
If Web Proxy is enabled in your network environment, Web Proxy IP needs to be provided for docker-compose configuration during the installation.(See Using D3 Proxy Agent with a Web Proxy Server)
Install Docker
On Ubuntu
Follow the instructions from the documentation to install docker using the apt package manager: https://docs.docker.com/engine/install/ubuntu/
Alternatively, you can install from a package with these instructions:
Link to distributions: https://download.docker.com/linux/ubuntu/dists/
Download and install four .deb packages from <version>/pool/stable/amd64:
docker-ce-cli
docker-ce
containerd.io
docker-compose-plugin
Command to install on ubuntu: sudo dpkg -i /path/to/<replace_with_package>.deb
Command to check if docker service is running: sudo systemctl status docker
On Redhat
Follow the instructions from the documentation to install docker using the yum package manager: https://docs.docker.com/engine/install/rhel/
Alternatively, you can install from a package with these instructions:
Link to package distributions: https://download.docker.com/linux/rhel/
Download and install four .rpm packages from <version>/s390x/stable/packages:
docker-ce-cli
docker-ce
containerd.io
docker-compose-plugin
Command to install on redhat: sudo yum install /path/to/<replace_with_package>.rpm
Command to start docker server: sudo systemctl start docker
Authenticate to Google Container Registry
The D3 Proxy Agent image hosted on Google Container Registry is private and requires authentication to pull it.
D3 Security will provide a json file containing credentials to authenticate, which can be done with the following command in your Proxy Agent server:
sudo docker login -u _json_key --password-stdin https://gcr.io < service_account_d3soar.json
Reader Note
Please contact the D3 account representative to obtain the JSON file.
Run docker-compose to Start Containers
On the Proxy Agent, we will need to run both the d3executor and the d3agent containers. A docker-compose file is provided to run both the d3executor and the d3agent.
Create a new file named docker-compose.yml
and paste the following content into the file.
version: "3.3"
services:
d3agent:
image: "gcr.io/nimble-cortex-285618/d3prod/d3agent:<Agent Version Number>"
restart: always
environment:
- REMOTE_SERVER_URL=<VSOC Url>
- PROXY_IDENTITY=<VSOC Agent Token>
- PYTHON_REMOTE_URL=http://<Docker host ip>:9092/
- PYTHON_REMOTE_GUID=7e036a86dbee40d9913c3794e779eae4
- SERVICE_DISPLAY_NAME=<Proxy Agent Display Name>
d3executor:
image: "gcr.io/nimble-cortex-285618/d3prod/d3executor:<Executor Version Number>"
ports:
- "9092:9090"
restart: always
environment:
- EXECUTOR_GUID=7e036a86dbee40d9913c3794e779eae4
Reader Note
Ensure that the version numbers for your compatible agent and executor match the version number of your VSOC instance. Replace <Agent Version Number>
and <Executor Version Number>
with the corresponding version numbers.
For example, if you are using D3 SOAR release version 16.1.80, set the <Agent Version Number>
to 16.1.80 and the <Executor Version Number>
to 16.1.80.
Please note that the <VSOC Url>
, <VSOC Agent Token>
, <Docker host ip>
and <Proxy Agent Display Name>
need to be replaced with specific values. The port for docker host needs to be aligned for service d3agent and service d3executor, e.g. 9092.
VSOC Agent Token can be generated from the D3 SOAR application.
Open a terminal and browse to the same directory as where the docker-compose.yml
file is located and run the following command:
sudo docker-compose up -d
Firewall Whitelisting for Docker installation and D3 Proxy Agent Container Image
If all internet traffic is blocked by default, then certain Domains/IPs will need to be whitelisted in order to be able to download the docker package as well as the images hosted on GCR.
1. Sites that require whitelisting
Docker (for downloading docker package):
download.docker.com
GCR (for downloading docker images from GCR):
gcr.io
storage.googleapis.com
2. Example of Checking IP address
Use nslookup to check IP address of the domain that needs to be accessed
We can see that download.docker.com
has a few IP addresses that it uses. We can allow traffic to this domain download.docker.com
or to an IP address list that includes all the resolved public IP (the same domain may resolve to different public IP from different regions, if you use IP address for the whitelist, you need to run nslookup to get these IP from your server.)
Using D3 Proxy Agent with a Web Proxy Server
To route D3 Proxy Agent’s outbound internet connections to a web proxy server, make the following configuration changes.
Open docker-compose.yml
. Add the following to the Environment section:
HTTP_PROXY=<Web Proxy Server:Port>
HTTPS_PROXY=<Web Proxy Server:Port>
NO_PROXY=localhost,127.0.0.1,<Docker host ip>,<Other local integration host address,...>
docker-compose.yml
version: "3.3"
services:
d3agent:
image: "gcr.io/nimble-cortex-285618/d3prod/d3agent:15.3.72.0"
restart: always
environment:
- REMOTE_SERVER_URL=<VSOC Url>
- PROXY_IDENTITY=<VSOC Agent Token>
- PYTHON_REMOTE_URL=http://<Docker host ip>:9092/
- PYTHON_REMOTE_GUID=7e036a86dbee40d9913c3794e779eae4
- SERVICE_DISPLAY_NAME=<Proxy Agent Display Name>
- HTTP_PROXY=<Web Proxy Server IP:Port>
- HTTPS_PROXY=<Web Proxy Server IP:Port>
- NO_PROXY=localhost,127.0.0.1,<Docker host ip>,<Other local integration host address,...>
d3executor:
image: "gcr.io/nimble-cortex-285618/d3prod/d3executor:15.1.39.1"
ports:
- "9092:9090"
restart: always
environment:
- EXECUTOR_GUID=7e036a86dbee40d9913c3794e779eae4
- HTTP_PROXY=<Web Proxy Server IP:Port>
- HTTPS_PROXY=<Web Proxy Server IP:Port>
- NO_PROXY=localhost,127.0.0.1,<Docker host ip>,<Other local integration host address,...>
HTTP_PROXY
and HTTPS_PROXY
are going to be the web proxy server HTTP and HTTPS connections respectively. You will need to add more addresses for the private IP, DNS name, and all other local integrations you want to connect to (i.e. skipping the proxy) to the NO_PROXY
.
Proxy Agent Installation on Windows
Environment Requirement
Server Hardware Requirements
Component | Minimum |
---|---|
Virtual Machine | 1* virtual machine is required |
Platform | VMware vSphere v6.7 + is recommended |
CPU | 4-core, Intel Xeon 3.0 GHz or higher is required |
RAM | 16GB or higher |
Disk space | 200GB or higher, solid-state drive-based storage is required |
Operating System
Version | License |
---|---|
Windows 2019 | Windows Server 2019 Standard |
Firewall Ports
Service | Protocol | Port | Direction | Firewall / Proxy |
---|---|---|---|---|
D3 SOAR Proxy Agent | TCP | 443 | Outbound to Internet | Allow |
Please Note:
The server for D3 Proxy Agent must be able to communicate with D3 VSOC servers via TCP ports 443.
D3 Proxy Agent must be installed on servers that have direct access to all your internal resources, such as firewalls, SIEM, Exchange, etc.
If Web Proxy is enabled in your network environment, Web Proxy IP needs to be configured in the D3PlaybookAgent.exe.config during the installation.(See How to Configure Internet Proxy Settings)
Installing New Agents
The Agent installation process is done via an Installation Wizard. The installation package can be downloaded directly from D3 SOAR. The steps below will outline the process of installing new Agents.
To install a new Agent:
Download Proxy Agent Package from within VSOC GUI
Save the Security Token Key
Generate Security Token, copy and paste it in a txt file. This token will be used during the installation process.
Unzip the Installation Package
Reader Note
The Zip file size should be 170MB or larger. If it is around 3MB, please see the Troubleshooting section.
Make sure this zip file was NOT blocked by Windows. Right click on the file and check the block state. If it is blocked, Unblock it first before you unzip. Otherwise, even if you can unzip it, but some unzipped files are still locked, you may NOT be able to install it successfully. The installation will not be able to proceed and hang there.
Right click and run D3PlaybookAgentInstaller.exe as administrator
Input VSOC Server URL, Security Token, Agent Service Name, Agent Name. Description and Location details are optional.
Click Next and wait for Installation to complete.
Please see the Troubleshooting section if the install does not complete.
Check the Proxy Agent Service Status in Windows Services. Make sure the proxy agent is running properly.
Check the logs to see connection status:C:\Program Files (x86)\D3security\D3PlaybookAgent\log\D3PlaybookAgent.log
How to Configure Internet Proxy Settings
In some environments, the D3 Proxy Agent can NOT access the Internet directly, it has to access the Internet via a Proxy server.
Edit the D3PlaybookAgent.exe.config file, add Proxy setting as follows:
C:\Program Files (x86)\D3Security\D3PlaybookAgent14\D3PlaybookAgent.exe.config
Add proxy configuration between <system.net>
..... </system.net>
shown below.
Reader Note
This example set the proxy server IP 192.168.1.10, port 3128
All local IP addresses 192.168.xxx.xxx are bypassed.
Proxy Agent High Availability
Proxy Agent supports High Availability by installing multiple Proxy Agents in different folders. Each agent must use a unique Secure Token.
Example
Install the first Agent in
\D3PlaybookAgent01
folder
Each Agent needs a unique Security Token. Please generate the token in VSOC page.
Enter a unique Security Token, Agent Service Name and Agent Name for this Agent
Install the 2nd Agent by right clicking
D3playbookAgentInstaller.exe
and run as administrator. Choose Install a new agent.
Use a unique folder name, for example
\D3PlaybookAgent02
, Keep "Install Python" checked and leave the folder as it is. (That will configure the Python path for the 2nd Proxy Agent)
Use a Unique Security Token, Agent Service Name and Agent Name
Complete the installation.
After 2nd agent installation finished, you will see two folders
Check the service status in Windows Task Manager. Make sure the services are running.
Check the logs at
C:\Program Files (x86)\D3security\D3PlaybookAgent01\log
, State:Connected
Check the logs at
C:\Program Files (x86)\D3security\D3PlaybookAgent02\log
, State:Connected
From the VSOC Agent Management interface, both of the Proxy Agents Status should be Connected. Installation has finished and is successful.
Proxy Agent Windows Server Hardening
How Proxy Agent Works
In short, the Proxy Agent does NOT need incoming TCP connections. It always initiates TCP connections to VSOC server and RestfulAPI targets. So we can block all incoming connections to the Windows server where D3 Proxy Agent is running on. Just run the powershell scripts to disable all the default enabled incoming firewall rules on Windows Server.
Troubleshooting
Package download is NOT successful
Agent Key
Download PlaybookAgent from VSOC GUI. The file size of D3PlaybookAgentInstaller14.0.xxx.x.zip should be at least 170MB or larger which includes the Python package. If the download does NOT include the Python package, its size may be around 3MB and requires troubleshooting.
On DB Server, in SSMS (SQL Server Management Studio), right click the database, run new query
Run the following command to check if the Proxy Agent key exists or not.
select * from VSOC_Settings_Default where keyname like 'Cyber.PythonInstallationFolderForPlaybookAgent'
If the PlaybookAgent key does not exist, run the insert command to add it to the database.
insert into VSOC_Settings_Default (keyName, keyValue) values ('Cyber.PythonInstallationFolderForPlaybookAgent', 'C:\Python\Python38')
Run the update command to update the key value.
update VSOC_Settings_Default set keyValue = 'C:\Python\Python38' where keyname like 'Cyber.PythonInstallationFolderForPlaybookAgent'
Reader Note
C:\Python\Python38
is an example here.
It is the Python path on your VSOC server. Change it to your Python path where Python.exe resides.
After updating the key into the Database, stop D3TaskManager services, stop App Pool, then restart the services and App Pool again.
Delete Agent folder under
C:\D3App\D3Temp\Agent
orC:\D3App\Virtual_Dir\D3Temp\Agent
See examples below. Your D3App folder may be on other disks like D:\ or E:\ as well.
No File Downloaded
When attempting to download the Agent, sometimes no file is downloaded. In this case, check your D3Temp directory properties. Make sure users have full control so the Agent software can be written into this folder.
Set users with full control.
Go to VSOC to download the agent again. The correct zipped installation package size should be at least 170MB or larger.
Reader Note
If user VSOC was updated to a new version, you need to download and update your Proxy agent accordingly.
The Proxy Agent version should match the VSOC version.
Download new Proxy Agent and install
Reader Note
You can install multiple Proxy Agents, each with a different Security Token.
Installing Agent Not Completing
If the installation stalls and does not complete, the root cause is that the Playbook Installation package was blocked by Windows when downloading from VSOC servers.
Go to the .zip file and view Properties
Check the Unblock box before unzipping the file. Even if the file is able to unzip successfully without this permission, some files will still be blocked by Windows. Ensure that this is checked before proceeding.
Run the installation again
Agent Configuration Scenarios
Overview
Scenario | Configuration | Status | On-Prem Integration | On Cloud Integration |
1 | Allow command to be run on agent | ☑ | Command will be run on the Agent Python Executor completely | Command will be run on the Agent Python Executor completely |
Agent selected in the Connection | Connected | |||
2 | Allow command to be run on agent | ☑ | The execution of the command is stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all | The execution of the command is stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all |
Agent selected in the Connection | Disconnected | |||
3 | Allow command to be run on agent | ☑ | Command will be run on the VSOC Remote Python Executor first. Rest API request will fail since it cannot connect to integration inside VSOC environment | Command will be run on the VSOC Remote Python Executor completely |
Agent selected in the Connection | None | |||
4 | Allow command to be run on agent | ☐ | Command will be run on the VSOC Remote Python Executor. Rest API request will be redirected to run on the Agent | Command will be run on the VSOC Remote Python Executor. Rest API request will be redirected to run on the Agent |
Agent selected in the Connection | Connected | |||
5 | Allow command to be run on agent | ☐ | The execution of the command is stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all | The execution of the command is stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all |
Agent selected in the Connection | Disconnected | |||
6 | Allow command to be run on agent | ☐ | Command will be run on the VSOC Remote Python Executor first. Rest API request will fail since cannot connect to integration inside VSOC environment | Command will be run on the VSOC Remote Python Executor completely |
Agent selected in the Connection | None |
On-Premises Integration Example: IBM QRadar Custom Command
Scenario 1
☑ Allow command to be run on agent | |
Connected agent selected in the Connection | |
Result Command was run on the Agent Python Executor completely and result was returned successfully |
Scenario 2
☑ Allow command to be run on agent | |
Disconnected agent selected in the Connection | |
Result The execution of the command was stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all |
Scenario 3
☑ Allow command to be run on agent | |
NO agent selected in the Connection | |
Result Command was run on the VSOC Remote Python Executor first. Rest API request would fail since it cannot connect to integration inside VSOC environment |
Scenario 4
☐ Allow command to be run on agent | |
Connected agent selected in the Connection | |
Result Command was run on the VSOC Remote Python Executor. Rest API request was redirected to run on the Agent Result was returned successfully |
Scenario 5
☐ Allow command to be run on agent | |
Disconnected agent selected in the Connection | |
Result The execution of the command was stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all |
Scenario 6
☐ Allow command to be run on agent | |
NO agent selected in the Connection | |
Result Command was run on the VSOC Remote Python Executor first. Rest API request would fail since it cannot connect to integration inside VSOC environment |
On Cloud Integration Example: VirusTotal V3 Custom Command
Scenario 1
☑ Allow command to be run on agent | |
Connected agent selected in the Connection | |
Result Command was run on the Agent Python Executor completely and result was returned successfully |
Scenario 2
☑ Allow command to be run on agent | |
Disconnected agent selected in the Connection | |
Result The execution of the command was stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all |
Scenario 3
☑ Allow command to be run on agent | |
NO agent selected in the Connection | |
Result Command was run on the VSOC Remote Python Executor completely and result was returned successfully |
Scenario 4
☐ Allow command to be run on agent | |
Connected agent selected in the Connection | |
Result Command was run on the VSOC Remote Python Executor. Rest API request was redirected to run on the Agent Result was returned successfully |
Scenario 5
☐ Allow command to be run on agent | |
Disconnected agent selected in the Connection | |
Result The execution of the command was stopped by the initial Agent Health Check. (Error Message "Proxy is invalid or closed.") Cannot run the command at all |
Scenario 6
☐ Allow command to be run on agent | |
NO agent selected in the Connection | |
Result Command was run on the VSOC Remote Python Executor completely and result was returned successfully |
Network Configuration for D3 Application in Restricted Client-Side Environments
To ensure the proper functioning of the D3 application in a client-side network with default outbound traffic restrictions, please perform the following steps:
Ping the D3 application URL (e.g., smartsoar.d3securityonline.com) to determine the required inbound IPs.
Whitelist the identified inbound IPs on your firewall's outbound rules. This step will allow the necessary communication between the D3 application and the external network.
Please note that these steps should be followed in the following scenarios:
When the D3 proxy agent is installed in a client-side network that blocks outbound traffic to the internet by default.
When client users are accessing the D3 application within a client-side network that denies outbound traffic to the internet by default.