Configuring Custom CA Certificate

LAST UPDATED: OCT 22, 2024

Requirement

Background

A client has an internal HTTPS service that employs a self-signed HTTPS certificate. An internal Python3 library is used to send requests to this service. The d3executor is responsible for executing Python commands that send these requests. When the d3executor sends requests to the service, it will attempt to verify the server's identity to ensure the connection is secure.

This verification process involves checking the server's HTTPS certificate against a trusted certificate authority (CA). Because the d3executor lacks access to such a CA certificate, and that HTTPS certificate verification is programmatically enabled by default, the client may encounter errors and failed requests.

Objectives

The objectives are twofold:

1. Enabling the client to install an internal Python package, and

2. Ensuring the client can successfully send requests to the service using their Python library without encountering errors.

D3 Solutions

Option 1: Rebuild d3executor Using a Customized Dockerfile

Create a Customized Dockerfile

1. Save the below content as a Dockerfile.

   DOCKER

   # Use the base image
   FROM gcr.io/nimble-cortex-285618/d3prod/d3executor:16.8.112.0
   
   USER root
   
   # Copy the custom CA certificate to standard directories
   COPY ca.crt /etc/ssl/certs/ca.crt
   COPY ca.crt /usr/lib/ssl/certs/ca.crt
   # Set environment variable for SSL cert file
   ENV SSL_CERT_FILE=/etc/ssl/certs/ca.crt
   # Update the CA certificates to include the custom CA certificate
   RUN update-ca-certificates
   
   # Change back to DockerUser
   USER DockerUser
   # Install Python packages from an internal repository
   # Replace with your own repository host, URL and any additional packages
   RUN pip install --trusted-host <Hostname> --index-url <Repository URL> <Packages>
   
   # Append custom CA certificate to certifi CA bundle
   RUN cat /etc/ssl/certs/ca.crt >> $(python -m certifi)
   
   # Entry point to run the application
   ENTRYPOINT ["python", "remote_server.py"]
   

READER NOTE

You must replace <Packages> with your own package(s). If you include two or more packages, each package must be separated by a whitespace.

--trusted-host <Hostname> --index-url <Repository URL> are optional parameters used to install packages from your own repository.

For example, the command may look something like the following:

RUN pip install --trusted-host https://pypi.org --index-url https://pypi.org/simple art arts

2. Rename your CA certificate as ca.crt.

3. Place the ca.crt file in the same folder as this Dockerfile.

   

Build a Docker Image

1. Assign the image a new name such as d3executor:16.8.112.0_customized.

2. Run the following docker command:

   BASH

   docker build -t d3executor:16.8.112.0_customized .

Edit the docker-compose File

1. Replace the old image name with the new image name.

   

2. Ensure that your docker-compose file is structured similarly to the sample below:

   YAML

   version: "3.3"
   services: 
     d3agent: 
       image: "gcr.io/nimble-cortex-285618/d3prod/d3agent:16.8.112.0"
       restart: always 
       environment:
         - REMOTE_SERVER_URL=http://192.168.1.190/VSOCMainAppDev169
         - PROXY_IDENTITY=8012be4477724b3f915d04100d76633c
         - PYTHON_REMOTE_URL=http://192.168.1.212:9090
         - PYTHON_REMOTE_GUID=7e036a86dbee40d9913c3794e779eae4 
         - SERVICE_DISPLAY_NAME=TestAgent
     d3executor: 
       # image: "gcr.io/nimble-cortex-285618/d3prod/d3executor:16.8.112.0" 
       image: "d3executor:16.8.112.0_customized" 
       ports: 
         - "9090:9090" 
       restart: always
       # Optional: You can configure some DNS servers to allow d3executor to access your internal domains
       # dns:
       #   - 192.168.1.76
       #   - 192.168.1.251
       environment: 
         - EXECUTOR_GUID=7e036a86dbee40d9913c3794e779eae4

Update Existing Docker Container

1. Run the following command to replace the existing d3executor container with the new d3executor:16.8.112.0_customized container.

   BASH

   sudo docker compose up -d

2. Ensure that your Docker containers are active and running.

   

Tested Coverage for Python Libraries

The following packages have been tested to confirm they can use the configured ca.crt directly without requiring additional settings when sending requests:

• requests (relies on certifi CA bundle)

• urllib3

• http.client

• httplib2

• aiohttp

• httpx

• Libraries that rely on certifi CA bundle

• Libraries that rely on Linux system CA certifications

Option 2: Skip the SSL Certification Verification

1. Use the verify=False option when writing custom Python commands with the requests library to bypass SSL certificate verification. See the example below.

   PY

   import requests
   # Define the URL for demonstration purposes
   url = 'https://example.com'
   # GET request without verifying SSL certificate
   response_get = requests.get(url, verify=False)

Instead of the get() method, you can also use any HTTP request methods from the requests library, such as post(), patch(), put(), and delete().