‎Install Agent on Linux

LAST UPDATED: MAY 5, 2025

PREREQUISITE

Completion of the Linux Environment Setup steps.

Configuring and Running D3 Agent and D3 Executor

On the Proxy Agent, we will need to run both the d3executor and the d3agent containers.

Create and edit a docker-compose.yml file and include the following content:

YAML

services:
  d3agent:
    image: "d3soar.azurecr.io/d3prod/d3agent:<D3 vSOC version>"
    restart: always
    environment:
      - REMOTE_SERVER_URL=<D3 vSOC URL>
      - PROXY_IDENTITY=<D3 vSOC Agent Security Token>
      - PYTHON_REMOTE_URL=http://<Docker Host IP>:<Docker Host Port>/
      - PYTHON_REMOTE_GUID=<Executor Identifier>
      - SERVICE_DISPLAY_NAME=<Proxy Agent Display Name>
      - DOCKER_GID=<Docker Group ID>
      - DOCKER_PORT=<Docker Host Port>
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  
  d3executor:
    image: "d3soar.azurecr.io/d3prod/d3executor:<D3 vSOC version>"
    ports:
      - "<Docker Host Port>:9090"
    restart: always
    environment:
      - EXECUTOR_GUID=<Executor Identifier>

Step-by-Step Instructions to Modify the docker-compose.yml Template in Vim

Procedure

Navigate into a directory to create the docker-compose.yml (we are using the same directory created in the Authenticating to Azure Container Registry):
BASH
```
cd DemoD3AgentDirectory
```
Create the docker-compose.yml file using the text editor of your preference, such as nano, vim, or vi. For the purposes of this demonstration, let us use vim:
BASH
```
vim docker-compose.yml
```
This will open and create the file in Normal Mode (Vim's default mode). When a file is opened in Vim, it will appear as follows:
Press i to enter Insert Mode (the user will now be able to type or paste text in Vim).
Copy and paste the template into Vim.
1. Copy the template above from your browser or other text editor.
2. Navigate back to the Vim terminal.
3. Paste the template in Insert Mode:
  - Press Ctrl + Shift + V (in most terminals).
    Note: Ensure that the indentation remains correct after pasting.
  - Important: Every directive (such as services) should have exactly two spaces of indentation. The list items (like - "90:90") should not have leading spaces before the hyphen (-). Learn more about Docker Compose.
  - (Optional) If you encounter an error like the one below, check your docker-compose.yml spacing.
Modify the template as necessary.
- Replace the < > placeholders with specific values.
- Replace the following placeholders (in Insert Mode) with appropriate values:
  - <D3 vSOC version>
  - <D3 vSOC URL>
  - <D3 vSOC Agent Security Token>
  - <Docker Host IP>
  - <Docker Host Port>
  - <Proxy Agent Display Name>
  - <Docker Group ID>

The details and meanings of the values can be found below.

Exit Insert Mode once you have replaced all the placeholders.
1. Press the Esc key to exit Insert Mode and return to Normal Mode.
Save and quit the docker-compose.ymlfile.
1. In Normal Mode, type:
  BASH
```
:wq
```
2. Press the Enter key to save and quit.
Verify that all changes were saved correctly by using the following Linux command:
BASH
```
cat docker-compose.yml
```

Quick Summary of Commands:

i – Enter Insert Mode
Esc – Exit Insert Mode
:wq – Save and quit
Ctrl + Shift + C – Copy
Ctrl + Shift + V – Paste into Vim
cat docker-compose.yml – View file content

D3 VSOC VERSION
- Replace <D3 vSOC version> with your vSOC version, which can be found on the D3 login page. For example, if you are using D3 vSOC release version 17.0.0, replace the <D3 vSOC version> placeholders within the d3agent and d3executor services to 17.0.0.
D3 VSOC URL
- This the the vSOC URL, starting with https or http, and ending with /VSOC.
D3 VSOC AGENT SECURITY TOKEN
- The D3 vSOC Agent Security Token can be generated by following these steps:

a. Navigate to the the Configuration navigational tab.

b. Click on the Agent Management menu item on the left sidebar

c. Click on the + Generate Security Token button.

d. Select your site from the dropdown menu.

e. Click on the Generate Key button.

f. Click on the Copy button, then paste the security token into the docker-compose.yml file.

DOCKER HOST IP
- The <Docker Host IP> refers to the private IP address of the Linux machine hosting Docker. Find this address by executing the following command:
  BASH
```
ip addr show
```

READER NOTE *

Ensure that the port is accessible and not being used by any other program.

DOCKER HOST PORT
- The <Docker Host Port> is the unique port number on the Docker host machine that maps to the container's internal port for the d3executor service. This enables communication between the D3 agent and the executor.
- Run the following Docker CLI command to retrieve the Docker host port:
  BASH
```
docker ps
```
  - The Docker host port is located under the PORTS column, immediately before the -> arrow.
- The <Docker Host Port> specified in the PYTHON_REMOTE_URL environment variable of the d3agent service must identically match the <Docker Host Port> specified in the ports field of the d3executor service.
- When deploying multiple D3 agents within one host, ensure each uses a unique <Docker Host Port>.
EXECUTOR IDENTIFIER
- A GUID, typically represented as a 36-character string in the format XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX, is used to ensure that the D3 agent communicates only with its intended D3 executor.
- Run the following Linux shell command to generate a new GUID:
  BASH
```
uuidgen | tr '[:lower:]' '[:upper:]'
```
- D3 recommends using uppercase for the alphabetic characters in this GUID.
- When deploying multiple D3 agents within one host, ensure each uses a distinct <Executor Identifier>.
PROXY AGENT DISPLAY NAME
- Input a descriptive name for this Agent, it displayed on vSOC.
DOCKER GROUP ID – Run the following command to obtain the <Docker Group ID>:
BASH
```
getent group docker | cut -d: -f3
```

Open a terminal and navigate to the directory where the docker-compose.yml file. Once there, execute the following command:
BASH
```
sudo docker compose up -d
```
Upon successfully bringing up the Docker Compose services, one’s terminal will display outputs similar to the following:

Options for Running the Edited docker-compose.yml File

Option 1: Running All Services

To start all services defined in the docker-compose.yml in detached mode (in the background), run the following command:

BASH

sudo docker compose up -d

You may need to use sudo because Docker commands interact with system-level resources, such as the Docker daemon or sockets, which require elevated permissions.

Option 2: Starting a Specific Service (e.g., d3agent or d3executor)

If you only need to start one specific service, run the following commands:

BASH

sudo docker compose up d3agent -d 
sudo docker compose up d3executor -d

-d: Runs the service in detached mode, preventing it from blocking the terminal.
You can replace d3agent or d3executor with other service names defined in their docker-compose.yml file. These names may vary depending on your setup or specific requirements.

Option 3: Rebuilding and Restarting a Specific Service

If you have made changes to the service’s Docker image or dependencies, and need to rebuild and restart docker-compose.yml, run the following commands:

BASH

sudo docker compose up d3agent --build -d 
sudo docker compose up d3executor --build -d

--build: Forces Docker to rebuild the service’s image before starting it.

Docker Service Logs and Status Checks

Checking Service Logs

To view the logs for a specific service, run the following commands:

BASH

sudo docker compose logs d3agent 
sudo docker compose logs d3executor

For continuous log output, run the following command:

BASH

sudo docker compose logs -f d3agent

Checking the Status of All Running Containers

To view the status of all containers managed by your docker-compose.yml, run the following command:

BASH

docker compose ps

This command shows the status of the services, indicating which ones are running, stopped, or restarting.
See Learn more about Docker Compose for more details.

Confirm the presence of your new agent.
1. Navigate to the Configuration page.
2. Click on the Agent Management module.
3. Verify that the Agent Name matches the one specified in the docker-compose.yml file.
4. Check the agent (connection) status and version. The agent version should match your D3 vSOC version.
5. (Optional) Enter a description for your agent.
6. Click on the Save button.
(Optional) Troubleshoot via Docker Compose log commands.
DOCKER COMPOSE LOGS To see all Docker Compose service logs, including those for d3executor and d3agent, execute the following command:
BASH
```
 sudo docker compose logs  
```
Ideally, only d3executor logs display—this confirms successful agent setup.

D3EXECUTOR LOGS To view a list of d3executor service logs, run the following command:
BASH
```
 sudo docker compose logs d3executor 
```
D3AGENT LOGS To view a list of d3agent service logs, run the following command:
BASH
```
 sudo docker compose logs d3agent
```

Automating D3 Agent and D3 Executor Deployment

The create_d3agent_file.sh Bash script automates the deployment of two essential containers: d3agent and d3executor. It validates system prerequisites, ensures Docker is installed and running, and authenticates with Azure Cloud Container Registry using credentials specified in a local configuration file. The script dynamically generates a Docker Compose file for the containers, prompts the user to start them, performs iterative health checks, and provides deployment success feedback or error diagnostics.

READER NOTE

Deployment automation for the D3 Agent and D3 Executor is supported beginning with D3 vSOC version 16.8.161.

Procedure

Contact us to obtain the service_account_d3soar.key file required for authentication with D3's container registry.
Place the service_account_d3soar.key file in the same directory as create_d3agent_file.sh and vars.conf.

create_d3agent_file.sh

BASH

#!/bin/bash
set -eu

# Defines exit codes
EXIT_DOCKER_INSTALL_FAILED=20
EXIT_ACR_AUTH_FAILED=30

###########
# FUNCTIONS
###########

# Prints the current date and time [YYYY-MM-DD HH:MM:SS] followed by the message.
function logdate() {
   echo "[$(date +'%F %T')] $1"
}

# Validates the provided URL to ensure it follows the HTTP/HTTPS format.
function validate_url() {
  if [[ ! $1 =~ ^https?://.+ ]]; then
    logdate "Invalid URL. Please enter a valid http/https URL."
    exit 1
  fi
}
# Validates the provided IP address to ensure it conforms to IPv4 format.
function validate_ip() {
  if [[ ! $1 =~ ^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then
    logdate "Invalid IP address format. Please enter a valid IP."
    exit 1
  fi
}

# Validates the security token to ensure it is a 32-character lowercase alphanumeric string.
function validate_token() {
  if [[ ! $1 =~ ^[a-z0-9]{32}$ ]]; then
    logdate "Invalid token. Please enter a 32-character alphanumeric lowercase string."
    exit 1
  fi
}

# Validates the provided vSOC version to ensure it matches the required numeric format with four segments.
function validate_version() {
  if [[ ! $1 =~ ^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then
    logdate "Invalid version. Please enter a version like 16.xx.xx.xx"
    exit 1
  fi
}

# Checks for available ports, starting at port 9090.
function find_available_port() {
    local port=9090
    local max_attempts=100
    local attempt=1
    while ss -tuln | grep -q ":$port\\b"; do
        if ((attempt >= max_attempts)); then
            logdate "No available ports found after $max_attempts attempts."
            exit 1
        fi
        ((port++))
        ((attempt++))
    done
    echo "$port"
}

# Checks the operational status of specified Docker containers and reports issues.
function check_container_status() {

    # Checks the agent container status
    container_agent_status=$(sudo docker ps --filter "name=$1" --format "{{.Names}} {{.Status}}" | grep -q "$1.*Up" && echo "running" || echo "not running")
    container_executor_status=$(sudo docker ps --filter "name=$2" --format "{{.Names}} {{.Status}}" | grep -q "$2.*Up" && echo "running" || echo "not running")
   
    # Sets the agent and executor status variables.
    agent_status=$container_agent_status
    executor_status=$container_executor_status
   
    # Checks whether the containers are in a "Restarting" state. If either container is in this state, the script outputs an error message and terminates.
    if sudo docker ps --filter "name=$1" --format "{{.Names}} {{.Status}}" | grep -q "$1.*Restarting"; then
        logdate "installation failed, please check $1's log"
        exit 1
    fi
    if sudo docker ps --filter "name=$2" --format "{{.Names}} {{.Status}}" | grep -q "$2.*Restarting"; then
        logdate "installation failed, please check $2's log"
        exit 1
    fi
   
    # Notifies the user of any container that is not in a running state.
    if [[ "$agent_status" == "not running" ]]; then
        logdate "$1 container is not running"
        return 1
    fi
    if [[ "$executor_status" == "not running" ]]; then
        logdate "$2 container is not running"
        return 2
    fi
    return 0
}
######################################
# INSTALL DOCKER + AUTHENTICATE TO ACR
######################################

# Ensures Docker is installed and attempts automatic installation if missing.
function install_docker() {
  if ! command -v docker >/dev/null 2>&1; then
      logdate "Docker is not installed. Installing Docker..."
     
      # Installs Docker based on the helper script
      logdate "This script will attempt to install Docker automatically by downloading and executing this helper script provided by Docker Inc: <https://get.docker.com>."
      read -p "Would you like to download and execute this script to install Docker automatically? (Y/N): " user_response
      if [[ $user_response == "y" || $user_response == "Y" ]]; then
          curl -fsSL <https://get.docker.com> -o get-docker.sh
          sudo sh ./get-docker.sh
          sudo systemctl enable docker --now
         
          # Verifies whether Docker was installed successfully.
          if ! command -v docker >/dev/null 2>&1; then
              logdate "Docker installation failed."
              exit $EXIT_DOCKER_INSTALL_FAILED
          fi
      else
          logdate "Docker not installed. Please install manually and re-run this script."
          exit $EXIT_DOCKER_INSTALL_FAILED
      fi
  fi
}

# Authenticates to ACR using a service account key, ensuring access to D3 container images.
function auth_docker_registry() {
  local docker_registry=$1
  if [[ ${docker_registry} != "ACR" ]];then
    logdate "docker registry not support, exit 1"
    exit 1
  fi
  SERVICE_ACCOUNT_NAME="d3acrpull"
  SERVICE_ACCOUNT_FILE="service_account_d3soar.key"
  SERVICE_ENDPOINT="d3soar.azurecr.io"
  
  # Uses the service account file, if found, to log in to ACR. If not found or if login fails, exit with an error code and display an appropriate message.
  if [ -f "$SERVICE_ACCOUNT_FILE" ]; then
      logdate "Logging in to Azure Container Registry using service account key..."
      LOGIN_OUTPUT=$(cat "$SERVICE_ACCOUNT_FILE" | sudo docker login -u ${SERVICE_ACCOUNT_NAME} --password-stdin ${SERVICE_ENDPOINT} 2>&1)
      logdate "$LOGIN_OUTPUT"
     
      # Checks for "Login Succeeded" in the output.
      if logdate "$LOGIN_OUTPUT" | grep -q "Login Succeeded"; then
          logdate "Generating Docker Compose file..."
      else
          logdate "Docker login failed."
          exit $EXIT_ACR_AUTH_FAILED
      fi
  else
      logdate "Service account file $SERVICE_ACCOUNT_FILE not found. Please contact your D3 representative to obtain it."
      exit $EXIT_ACR_AUTH_FAILED
  fi
}
#################################
# GENERATE docker-compose-d3.yaml
#################################

# Creates a Docker Compose YAML file for deploying d3agent and d3executor containers, configured with vSOC settings and host-specific details.
function generate_docker_compose_yaml() {
 
  # Loads the vSOC URL and security token from vars.conf configuration file.
  if [ -f "./vars.conf" ]; then
      source vars.conf
  else
      logdate "'vars.conf' doesn't exist. Create vars.conf with two line-separated key=value pairs: vsoc_url and security_token."
      exit 1
  fi
  
  # Validates that required variables are set.
  if [[ -z "${vsoc_url:-}" || -z "${security_token:-}" ]]; then
    logdate "Error: Missing required variables (vsoc_url, security_token). Verify vars.conf has these keys set."
    exit 1
  fi
  
  # Obtains the host machine's IP address.
  host_ip=$(ip -o route get "8.8.8.8" 2>/dev/null | awk '{print $7}')
  
  # Generates a unique GUID for the remote Python agent.
  python_remote_guid=$(uuidgen | tr -d '-' | tr '[:lower:]' '[:upper:]')
  
  # Finds an available Docker port.
  docker_port=$(find_available_port)
  
  # Sets the agent display name.
  agent_display_name="D3Agent - $host_ip"
  
  # Retrieves the Docker GID.
  docker_gid=$(getent group docker | cut -d: -f3)
  
  # Ensures the vSOC URL is properly formatted with a trailing slash.
  vsoc_url="${vsoc_url%/}/"
  
  # Validates the loaded configuration values.
  validate_ip "$host_ip"
  validate_token "$security_token"
  validate_url "$vsoc_url"
  
  # Constructs the full URL.
  url="${vsoc_url}api/REST/GetD3Version"
  
  # Fetches the content of the URL specified in $url.
  html=$(curl -s -k "$url")
  
  # Extracts the vSOC version number.
  vsoc_version=$(echo "$html" | grep -oP '[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+')
  
  # Verifies whether the vsoc_version value matches the expected format.
  validate_version "$vsoc_version"
 
  # Checks whether vsoc_version is empty. If so, prints an error and exits.
  if [[ -z "$vsoc_version" ]]; then
          logdate "Version information not found."
      exit 1
  fi
  
  # Generates unique container names, and sets the Docker registry.
  container_agent=d3agent_$RANDOM
  container_executor=d3executor_$RANDOM
  DOCKER_REGISTRY="d3soar.azurecr.io"
  
  # Generates the Docker Compose YAML
  cat <<EOF >docker-compose-d3.yml
services:
  d3agent:
    container_name: $container_agent
    image: "${DOCKER_REGISTRY}/d3prod/d3agent:${vsoc_version}"
    restart: always
    environment:
      - REMOTE_SERVER_URL=${vsoc_url}
      - PROXY_IDENTITY=${security_token}
      - PYTHON_REMOTE_URL=http://${host_ip}:${docker_port}/
      - PYTHON_REMOTE_GUID=${python_remote_guid}
      - SERVICE_DISPLAY_NAME=${agent_display_name}
      - DOCKER_GID=${docker_gid}
      - DOCKER_PORT=${docker_port}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      
  d3executor:
    container_name: $container_executor
    image: "${DOCKER_REGISTRY}/d3prod/d3executor:${vsoc_version}"
    ports:
      - "${docker_port}:9090"
    restart: always
    environment:
      - EXECUTOR_GUID=${python_remote_guid}
EOF
  logdate "A Docker Compose file for D3 Proxy Agent has been generated successfully: docker-compose-d3.yml"
}

# Launches the d3agent and d3executor containers using Docker Compose and verifies their operational status with health checks.
function start_container() {
  local docker_compose_cmd=$1
 
  # Prompts the user to start the containers.
  read -p "Would you like to start the containers? (Y/N): " user_response
  if [[ $user_response == "y" || $user_response == "Y" ]]; then
     
      # Attempts to bring up the Docker containers.
      sudo ${docker_compose_cmd} -f docker-compose-d3.yml up -d --remove-orphans || logdate "Failed to bring up Docker containers. Verify the contents of 'docker-compose-d3.yml' and run 'sudo docker compose -f docker-compose-d3.yml up -d' to manually start the containers."
      logdate "Checking container status..."
      counter=1
     
      # Performs up to 5 health checks on the containers. If any container fails, the loop exits early.
      while [ $counter -le 3 ]; do
         
          # Checks container statuses.
          check_container_status $container_agent $container_executor
         
          # Checks whether the agent container is running.
          if [[ $? -eq 1 ]]; then
              logdate "Unhealthy Agent!"
              break
          fi
         
          # Checks whether the executor container is running.
          if [[ $? -eq 2 ]]; then
              logdate "Unhealthy Executor!"
              break
          fi
          logdate "Health Check #$counter/3 passed"
          counter=$((counter + 1))
          sleep 5
      done
      
      # Checks final container statuses and print a success or failure message with troubleshooting suggestions.
      if [[ "$agent_status" != "running" ]] || [[ "$executor_status" != "running" ]]; then
        logdate "Installation failed. You can check the container statuses with sudo ${docker_compose_cmd} -f docker-compose-d3.yml ps' or the container logs with sudo ${docker_compose_cmd} -f docker-compose-d3.yml logs"
      else
        logdate "Installation succesful! Log in to your VSOC instance at $vsoc_url to check the Agent Status from the UI."
      fi
  else
      logdate "Verify the contents of 'docker-compose-d3.yml' and run sudo ${docker_compose_cmd} -f docker-compose-d3.yml up -d to manually start the containers."
  fi
}

# Configures Podman as a Docker alternative by installing podman-compose and setting up a compatible socket for container management.
function adapter_podman() {

  # Checks if Podman is masquerading as Docker by inspecting 'docker version' output for "podman".
  if [[ $(sudo docker version 2>&1 |grep -Eo podman) != 'podman' ]];then
      return
  fi
  local to_install=false

  # Tests whether podman-compose is installed by running its version command, setting to_install=true if it fails.
  podman-compose -v || to_install=true
 
  # Installs podman-compose if required, based on the to_install variable.
  if [ "${to_install}" = "true" ];then
      logdate "install podman-compose ... "
      sudo yum install -y podman-compose
  else
      logdate "podman-compose has been installed"
  fi
 
  # Verifies podman-compose is installed
  podman-compose -v

  # Creates a docker user if one does not exist.
  id docker || sudo useradd docker
  
  SERVICE_CONF='/usr/lib/systemd/system/podman.service'
  SOCK_FILE='/var/run/podman/podman.sock'
  
  # Creates a backup of the existing Podman service file before modification.
  sudo cp ${SERVICE_CONF} ${SERVICE_CONF}.bak

  # Overwrites or creates a new podman.service file, enabling Podman to run in the background as a managed systemd service.
  cat <<EOF >./podman.service
[Unit]
Description=Podman API Service
Requires=podman.socket
After=podman.socket
Documentation=man:podman-system-service(1)
StartLimitIntervalSec=0

[Service]
Delegate=true
Type=exec
KillMode=process
Environment=LOGGING="--log-level=info"
ExecStart=/usr/bin/podman \\$LOGGING system service --time=0
ExecStartPost=/bin/chown root:docker /run/podman/podman.sock && ln -s /run/podman/podman.sock /var/run/docker.sock
[Install]
WantedBy=default.target
EOF

  # Moves the service file into the correct systemd directory, making systemd aware of it.
  sudo cp ./podman.service ${SERVICE_CONF}

  # Instructs systemd to rescans its service directories and loads the new podman.service file.
  sudo systemctl daemon-reload

  # Instructs systemd to stop and then start the podman.service, effectively reloading the Podman systemd service.
  sudo systemctl restart podman

  # Checks the current status of the podman.service (active, inactive, failed).
  sudo systemctl status podman

  # Checks whether Podman is running, and logs a message if it is.
  if [[ "$(pgrep podman)"x != x ]];then
      logdate "podman is running"
  fi

  logdate "display sock file"

  # Ensures the Podman socket exists and verifies the Docker socket is correctly linked for compatibility.
  sudo ls -al ${SOCK_FILE} /var/run/docker.sock
}

# Orchestrates Docker/Podman setup, ACR authentication, config generation, and container startup.
function main() {
  D3_DOCKER_REGISTRY="ACR"
  
  # Installs docker.
  install_docker
 
  # Adapts podman.
  adapter_podman
  
  # Authenticates to ACR.
  auth_docker_registry ${D3_DOCKER_REGISTRY}
  logdate 'auth done'
 
  # Generates Docker Compose.
  generate_docker_compose_yaml
  
  # Determines whether to use Docker Compose or Podman Compose for container deployment.
  DOCKER_COMPOSE_CMD="docker compose"
  if [[ $(sudo docker version 2>&1 |grep -Eo podman) = 'podman' ]];then
      DOCKER_COMPOSE_CMD="podman-compose"
      logdate "use podman-compose to start container ..."
  fi

  # Starts containers with the appropriate command.
  start_container "${DOCKER_COMPOSE_CMD}"
}

main

logdate ">>>>>> DONE <<<<<<"

Set up the vars.conf configuration file as follows:

TEXT

vsoc_url=<VSOC URL>
security_token=<Agent Token Generated in the UI>

Execute the create_d3agent_file.sh script by running the following two commands:
BASH
```
chmod +x create_d3agent_file.sh
./create_d3agent_file.sh
```