‎Install Agent on Linux

LAST UPDATED: FEB 28, 2025

Configuring and Running D3 Agent and D3 Executor

On the Proxy Agent, we will need to run both the d3executor and the d3agent containers.

Create and edit a docker-compose.yml file and include the following content:

YAML

version:
services:
  d3agent:
    image: "d3soar.azurecr.io/d3prod/d3agent:<D3 vSOC version>"
    restart: always
    environment:
      - REMOTE_SERVER_URL=<D3 vSOC URL>
      - PROXY_IDENTITY=<D3 vSOC Agent Security Token>
      - PYTHON_REMOTE_URL=http://<Docker Host IP>:<Docker Host Port>/
      - PYTHON_REMOTE_GUID=7e036a86dbee40d9913c3794e779eae4
      - SERVICE_DISPLAY_NAME=<Proxy Agent Display Name>
      - DOCKER_GID=<Docker Group ID>
      - DOCKER_PORT=<Docker Host Port>
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  
  d3executor:
    image: "d3soar.azurecr.io/d3prod/d3executor:<D3 vSOC version>"
    ports:
      - "<Docker Host Port>:9090"
    restart: always
    environment:
      - EXECUTOR_GUID=7e036a86dbee40d9913c3794e779eae4

Step-by-Step Instructions to Modify the docker-compose.yml Template in Vim

Procedure

Navigate into a directory to create the docker-compose.yml (we are using the same directory created in the Authenticating to Azure Container Registry):
BASH
```
cd D3ProxyAgent 
```
Create the docker-compose.yml file using the text editor of your preference, such as nano, vim, or vi. For the purposes of this demonstration, let us use vim:
BASH
```
vim docker-compose.yml
```
This will open and create the file in Normal Mode (Vim's default mode). When a file is opened in Vim, it will appear as follows:
Press i to enter Insert Mode (the user will now be able to type or paste text in Vim).
Copy and paste the template into Vim.
1. Copy the template above from your browser or other text editor.
2. Navigate back to the Vim terminal.
3. Paste the template in Insert Mode:
  - Press Ctrl + Shift + V (in most terminals).
    Note: Ensure that the indentation remains correct after pasting.
  - Important: Every directive (such as services) should have exactly two spaces of indentation. The list items (like - "90:90") should not have leading spaces before the hyphen (-). Learn more about Docker Compose.
  - (Optional) If you encounter an error like the one below, check your docker-compose.yml spacing.
Modify the template as necessary.
- Replace the < > placeholders with specific values.
- Replace the following placeholders (in Insert Mode) with appropriate values:
  - <D3 vSOC version>
  - <D3 vSOC URL>
  - <D3 vSOC Agent Security Token>
  - <Docker Host IP>
  - <Docker Host Port>
  - <Proxy Agent Display Name>
  - <Docker Group ID>

The details and meanings of the values can be found below.

Exit Insert Mode once you have replaced all the placeholders.
1. Press the Esc key to exit Insert Mode and return to Normal Mode.
Save and quit the docker-compose.ymlfile.
1. In Normal Mode, type:
  BASH
```
:wq
```
2. Press the Enter key to save and quit.
Verify that all changes were saved correctly by using the following Linux command:
BASH
```
cat docker-compose.yml
```

Quick Summary of Commands:

i – Enter Insert Mode
Esc – Exit Insert Mode
:wq – Save and quit
Ctrl + Shift + C – Copy
Ctrl + Shift + V – Paste into Vim
cat docker-compose.yml – View file content

D3 VSOC VERSION – Replace <D3 vSOC version> with your vSOC version, which can be found on the D3 login page. For example, if you are using D3 vSOC release version 16.1.80.0, replace the <D3 vSOC version> placeholders within the d3agent and d3executor services to 16.1.80.0.
D3 VSOC URL – This the the vSOC URL, starting with https or http, and ending with /VSOC.
D3 VSOC AGENT SECURITY TOKEN – The D3 vSOC Agent Security Token can be generated by following these steps:

a. Navigate to the the Configuration navigational tab.

b. Click on the Agent Management menu item on the left sidebar

c. Click on the + Generate Security Token button.

d. Select your site from the dropdown menu.

e. Click on the Generate Key button.

f. Click on the Copy button, then paste the security token into the docker-compose.yml file.

DOCKER HOST IP – The <Docker Host IP> refers to the private IP address of the Linux machine hosting Docker. You can find this address using the command ip addr show on Linux. Ensure that the port is accessible and not being used by any other program.
DOCKER HOST PORT – The <Docker Host Port> specified in the PYTHON_REMOTE_URL environment variable of the d3agent service must match the <Docker Host Port> specified in the ports mapping of the d3executor service. This ensures proper communication between the d3agent and d3executor services.
PROXY AGENT DISPLAY NAME – Input a descriptive name for this Agent, it displayed on vSOC.
DOCKER GROUP ID – Run the following command to obtain the <Docker Group ID>:

BASH

getent group docker | cut -d: -f3

Open a terminal and navigate to the directory where the docker-compose.yml file. Once there, execute the following command:
BASH
```
sudo docker compose up -d
```
Upon successfully bringing up the Docker Compose services, your terminal will display outputs similar to the following:

Options for Running the Edited docker-compose.yml File

Option 1: Running All Services

To start all services defined in the docker-compose.yml in detached mode (in the background), run the following command:

BASH

sudo docker compose up -d

You may need to use sudo because Docker commands interact with system-level resources, such as the Docker daemon or sockets, which require elevated permissions.

Option 2: Starting a Specific Service (e.g., d3agent or d3executor)

If you only need to start one specific service, run the following commands:

BASH

sudo docker compose up d3agent -d 
sudo docker compose up d3executor -d

-d: Runs the service in detached mode, preventing it from blocking the terminal.
You can replace d3agent or d3executor with other service names defined in their docker-compose.yml file. These names may vary depending on your setup or specific requirements.

Option 3: Rebuilding and Restarting a Specific Service

If you have made changes to the service’s Docker image or dependencies, and need to rebuild and restart docker-compose.yml, run the following commands:

BASH

sudo docker compose up d3agent --build -d 
sudo docker compose up d3executor --build -d

--build: Forces Docker to rebuild the service’s image before starting it.

Docker Service Logs and Status Checks

Checking Service Logs

To view the logs for a specific service, run the following commands:

BASH

sudo docker compose logs d3agent 
sudo docker compose logs d3executor

For continuous log output, run the following command:

BASH

sudo docker compose logs -f d3agent

Checking the Status of All Running Containers

To view the status of all containers managed by your docker-compose.yml, run the following command:

BASH

docker compose ps

This command shows the status of the services, indicating which ones are running, stopped, or restarting.
See Learn more about Docker Compose for more details.

Confirm the presence of your new agent.
1. Navigate to the Configuration page.
2. Click on the Agent Management module.
3. Verify that the Agent Name matches the one specified in the docker-compose.yml file.
4. Check the agent (connection) status and version. The agent version should match your D3 vSOC version.
5. (Optional) Enter a description for your agent.
6. Click on the Save button.
(Optional) Troubleshoot via Docker Compose log commands.
DOCKER COMPOSE LOGS To view a complete list of Docker Compose service logs, both logs for the d3executor and d3agent services, run the following command:
BASH
```
 sudo docker compose logs  
```
Ideally, only d3executor logs display—this confirms successful agent setup.

D3EXECUTOR LOGS To view a list of d3executor service logs, run the following command:
BASH
```
 sudo docker compose logs d3executor 
```
D3AGENT LOGS To view a list of d3agent service logs, run the following command:
BASH
```
 sudo docker compose logs d3agent
```

Automating D3 Agent and D3 Executor Deployment

The create_d3agent_file.sh Bash script automates the deployment of two necessary containers: d3agent and d3executor. It validates prerequisites, ensures Docker is installed and running, and authenticates with Google Cloud Container Registry using a configuration file. The script dynamically generates a Docker Compose file for the containers, prompts the user to start them, performs iterative health checks, and provides deployment success feedback or error diagnostics.

READER NOTE

Deployment automation for the D3 Agent and D3 Executor is supported beginning with D3 vSOC version 16.8.161.

Procedure

Contact us to obtain the service_account_d3soar.json file required for authentication with D3's container registry.

Place the service_account_d3soar.json file in the same directory as create_d3agent_file.sh and vars.conf.

create_d3agent_file.sh

CODE

#!/bin/bash
set -eu

# Define exit codes
EXIT_DOCKER_INSTALL_FAILED=20
EXIT_ACR_AUTH_FAILED=30

###########
# FUNCTIONS
###########

# Validates the provided URL to ensure it follows the HTTP/HTTPS format.

logdate() {
   echo "[$(date +'%F %T')] $1"
}

validate_url() {
  if [[ ! $1 =~ ^https?://.+ ]]; then
    logdate "Invalid URL. Please enter a valid http/https URL."
    exit 1
  fi
}

# Validates the provided IP address to ensure it conforms to IPv4 format.
validate_ip() {
  if [[ ! $1 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    logdate "Invalid IP address format. Please enter a valid IP."
    exit 1
  fi
}

# Validates the security token to ensure it is a 32-character lowercase alphanumeric string.
validate_token() {
  if [[ ! $1 =~ ^[a-z0-9]{32}$ ]]; then
    logdate "Invalid token. Please enter a 32-character alphanumeric lowercase string."
    exit 1
  fi
}

# Validates the provided vSOC version to ensure it matches the required numeric format with four segments.
validate_version() {
  if [[ ! $1 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    logdate "Invalid version. Please enter a version like 16.xx.xx.xx"
    exit 1
  fi
}

# Checks for available ports, starting at port 9090.
find_available_port() {
    local port=9090
    local max_attempts=100
    local attempt=1

    while ss -tuln | grep -q ":$port\b"; do
        if ((attempt >= max_attempts)); then
            logdate "No available ports found after $max_attempts attempts."
            exit 1
        fi
        ((port++))
        ((attempt++))
    done
    echo "$port"
}

# Checks the operational status of specified Docker containers and reports issues.
check_container_status() {
    # Checks the agent container status
    container_agent_status=$(sudo docker ps --filter "name=$1" --format "{{.Names}} {{.Status}}" | grep -q "$1.*Up" && echo "running" || echo "not running")
    container_executor_status=$(sudo docker ps --filter "name=$2" --format "{{.Names}} {{.Status}}" | grep -q "$2.*Up" && echo "running" || echo "not running")

    # Sets the agent and executor status variables.
    agent_status=$container_agent_status
    executor_status=$container_executor_status

    # Checks whether the containers are in a "Restarting" state. If either container is in this state, the script outputs an error message and terminates.
    if sudo docker ps --filter "name=$1" --format "{{.Names}} {{.Status}}" | grep -q "$1.*Restarting"; then
        logdate "installation failed, please check $1's log"
        exit 1
    fi
    if sudo docker ps --filter "name=$2" --format "{{.Names}} {{.Status}}" | grep -q "$2.*Restarting"; then
        logdate "installation failed, please check $2's log"
        exit 1
    fi

    # Notifies the user of any container that is not in a running state.
    if [[ "$agent_status" == "not running" ]]; then
        logdate "$1 container is not running"
        return 1
    fi
    if [[ "$executor_status" == "not running" ]]; then
        logdate "$2 container is not running"
        return 2
    fi
    return 0
}

######################################
# INSTALL DOCKER + AUTHENTICATE TO ACR
######################################

# Ensures Docker is installed and attempts automatic installation if missing.
function install_docker() {
  if ! command -v docker >/dev/null 2>&1; then
      logdate "Docker is not installed. Installing Docker..."

      # Installs Docker based on the helper script
      logdate "This script will attempt to install Docker automatically by downloading and executing this helper script provided by Docker Inc: https://get.docker.com."
      read -p "Would you like to download and execute this script to install Docker automatically? (Y/N): " user_response

      if [[ $user_response == "y" || $user_response == "Y" ]]; then
          curl -fsSL https://get.docker.com -o get-docker.sh
          sudo sh ./get-docker.sh
          sudo systemctl enable docker --now

          # Verifies whether Docker was installed successfully.
          if ! command -v docker >/dev/null 2>&1; then
              logdate "Docker installation failed."
              exit $EXIT_DOCKER_INSTALL_FAILED
          fi
      else
          logdate "Docker not installed. Please install manually and re-run this script."
          exit $EXIT_DOCKER_INSTALL_FAILED
      fi
  fi
}

function auth_docker_registry() {
  local docker_registry=$1
  if [[ ${docker_registry} != "ACR" ]];then
    logdate "docker registry not support, exit 1"
    exit 1
  fi

  SERVICE_ACCOUNT_NAME="d3acrpull"
  SERVICE_ACCOUNT_FILE="service_account_d3soar.key"
  SERVICE_ENDPOINT="d3soar.azurecr.io"

  # Uses the service account file, if found, to log in to ACR. If not found or if login fails, exit with an error code and display an appropriate message.
  if [ -f "$SERVICE_ACCOUNT_FILE" ]; then
      logdate "Logging in to Azure Container Registry using service account key..."
      LOGIN_OUTPUT=$(cat "$SERVICE_ACCOUNT_FILE" | sudo docker login -u ${SERVICE_ACCOUNT_NAME} --password-stdin ${SERVICE_ENDPOINT} 2>&1)
      logdate "$LOGIN_OUTPUT"

      # Checks for "Login Succeeded" in the output.
      if logdate "$LOGIN_OUTPUT" | grep -q "Login Succeeded"; then
          logdate "Generating Docker Compose file..."
      else
          logdate "Docker login failed."
          exit $EXIT_ACR_AUTH_FAILED
      fi
  else
      logdate "Service account file $SERVICE_ACCOUNT_FILE not found. Please contact your D3 representative to obtain it."
      exit $EXIT_ACR_AUTH_FAILED
  fi
}


#################################
# GENERATE docker-compose-d3.yaml
#################################
function generate_docker_compose_yaml() {
  # Loads the vSOC URL and security token from vars.conf configuration file.
  if [ -f "./vars.conf" ]; then
      source vars.conf
  else
      logdate "'vars.conf' doesn't exist. Create vars.conf with two line-separated key=value pairs: vsoc_url and security_token."
      exit 1
  fi

  # Validates that required variables are set.
  if [[ -z "${vsoc_url:-}" || -z "${security_token:-}" ]]; then
    logdate "Error: Missing required variables (vsoc_url, security_token). Verify vars.conf has these keys set."
    exit 1
  fi

  # Obtains the host machine's IP address.
  host_ip=$(ip -o route get "8.8.8.8" 2>/dev/null | awk '{print $7}')

  # Generates a unique GUID for the remote Python agent.
  python_remote_guid=$(uuidgen | tr -d '-' | tr '[:lower:]' '[:upper:]')

  # Finds an available Docker port.
  docker_port=$(find_available_port)

  # Sets the agent display name.
  agent_display_name="D3Agent - $host_ip"

  # Retrieves the Docker GID.
  docker_gid=$(getent group docker | cut -d: -f3)

  # Ensures the vSOC URL is properly formatted with a trailing slash.
  vsoc_url="${vsoc_url%/}/"

  # Validates the loaded configuration values.
  validate_ip "$host_ip"
  validate_token "$security_token"
  validate_url "$vsoc_url"

  # Constructs the full URL.
  url="${vsoc_url}api/REST/GetD3Version"

  # Fetches the content of the URL specified in $url.
  html=$(curl -s -k "$url")

  # Extracts the vSOC version number.
  vsoc_version=$(echo "$html" | grep -oP '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+')

  # Verifies whether the vsoc_version value matches the expected format.
  validate_version "$vsoc_version"

  # Checks whether vsoc_version is empty. If so, prints an error and exits.
  if [[ -z "$vsoc_version" ]]; then
          logdate "Version information not found."
      exit 1
  fi

  # Generate unique names for the containers.
  container_agent=d3agent_$RANDOM
  container_executor=d3executor_$RANDOM

  DOCKER_REGISTRY="d3soar.azurecr.io"

  # Generates the Docker Compose YAML
  cat <<EOF >docker-compose-d3.yml
services:
  d3agent:
    container_name: $container_agent
    image: "${DOCKER_REGISTRY}/d3prod/d3agent:${vsoc_version}"
    restart: always
    environment:
      - REMOTE_SERVER_URL=${vsoc_url}
      - PROXY_IDENTITY=${security_token}
      - PYTHON_REMOTE_URL=http://${host_ip}:${docker_port}/
      - PYTHON_REMOTE_GUID=${python_remote_guid}
      - SERVICE_DISPLAY_NAME=${agent_display_name}
      - DOCKER_GID=${docker_gid}
      - DOCKER_PORT=${docker_port}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  d3executor:
    container_name: $container_executor
    image: "${DOCKER_REGISTRY}/d3prod/d3executor:${vsoc_version}"
    ports:
      - "${docker_port}:9090"
    restart: always
    environment:
      - EXECUTOR_GUID=${python_remote_guid}
EOF

  logdate "A Docker Compose file for D3 Proxy Agent has been generated successfully: docker-compose-d3.yml"

}

function start_container() {
  local docker_compose_cmd=$1

  # Prompts the user to start the containers.
  read -p "Would you like to start the containers? (Y/N): " user_response

  if [[ $user_response == "y" || $user_response == "Y" ]]; then
      # Attempts to bring up the Docker containers.
      sudo ${docker_compose_cmd} -f docker-compose-d3.yml up -d --remove-orphans || logdate "Failed to bring up Docker containers. Verify the contents of 'docker-compose-d3.yml' and run 'sudo docker compose -f docker-compose-d3.yml up -d' to manually start the containers."

      logdate "Checking container status..."
      counter=1

      # Performs up to 5 health checks on the containers. If any container fails, the loop exits early.
      while [ $counter -le 3 ]; do
          # Checks container statuses.
          check_container_status $container_agent $container_executor

          # Checks whether the agent container is running.
          if [[ $? -eq 1 ]]; then
              logdate "Unhealthy Agent!"
              break
          fi

          # Checks whether the executor container is running.
          if [[ $? -eq 2 ]]; then
              logdate "Unhealthy Executor!"
              break
          fi

          logdate "Health Check #$counter/3 passed"
          counter=$((counter + 1))

          sleep 5
      done

      # Checks final container statuses and print a success or failure message with troubleshooting suggestions.
      if [[ "$agent_status" != "running" ]] || [[ "$executor_status" != "running" ]]; then
        logdate "Installation failed. You can check the container statuses with sudo ${docker_compose_cmd} -f docker-compose-d3.yml ps' or the container logs with sudo ${docker_compose_cmd} -f docker-compose-d3.yml logs"
      else
        logdate "Installation succesful! Log in to your VSOC instance at $vsoc_url to check the Agent Status from the UI."
      fi

  else
      logdate "Verify the contents of 'docker-compose-d3.yml' and run sudo ${docker_compose_cmd} -f docker-compose-d3.yml up -d to manually start the containers."
  fi
}

function adapter_podman() {
  if [[ $(sudo docker version 2>&1 |grep -Eo podman) != 'podman' ]];then
      return
  fi

  local to_install=false
  podman-compose -v || to_install=true
  if [ "${to_install}" = "true" ];then
      logdate "install podman-compose ... "
      sudo yum install -y podman-compose
  else
      logdate "podman-compose has been installed"
  fi

  # if not installed, will exit directly
  podman-compose -v

  id docker || sudo useradd docker
  SERVICE_CONF='/usr/lib/systemd/system/podman.service'
  SOCK_FILE='/var/run/podman/podman.sock'
  sudo cp ${SERVICE_CONF} ${SERVICE_CONF}.bak
  cat <<EOF >./podman.service
[Unit]
Description=Podman API Service
Requires=podman.socket
After=podman.socket
Documentation=man:podman-system-service(1)
StartLimitIntervalSec=0

[Service]
Delegate=true
Type=exec
KillMode=process
Environment=LOGGING="--log-level=info"
ExecStart=/usr/bin/podman \$LOGGING system service --time=0
ExecStopPost=/bin/chown root:docker /run/podman/podman.sock && ln -s /var/run/docker.sock

[Install]
WantedBy=default.target
EOF
  sudo cp ./podman.service ${SERVICE_CONF}
  sudo systemctl daemon-reload
  sudo systemctl restart podman
  sudo systemctl status podman
  if [[ "$(pgrep podman)"x != x ]];then
      logdate "podman is running"
  fi
  logdate "display sock file"
  sudo ls -al ${SOCK_FILE} /var/run/docker.sock
}

function main() {
  D3_DOCKER_REGISTRY="ACR"
  # install docker
  install_docker
  # adapt podman
  adapter_podman
  # auth to acr
  auth_docker_registry ${D3_DOCKER_REGISTRY}
  logdate 'auth done'
  # generate docker-compose config
  generate_docker_compose_yaml

  # start d3agent/d3executor container
  DOCKER_COMPOSE_CMD="docker compose"
  if [[ $(sudo docker version 2>&1 |grep -Eo podman) = 'podman' ]];then
      DOCKER_COMPOSE_CMD="podman-compose"
      logdate "use podman-compose to start container ..."
  fi
  start_container "${DOCKER_COMPOSE_CMD}"
}

main

logdate ">>>>>> DONE <<<<<<"

Set up the vars.conf configuration file as follows:

TEXT

vsoc_url=<VSOC URL>
security_token=<Agent Token Generated in the UI>

Execute the create_d3agent_file.sh script by running the following two commands:
BASH
```
chmod +x create_d3agent_file.sh
./create_d3agent_file.sh
```