Continuum-proxy

The continuum-proxy is a service that should be deployed by users of the Continuum API. The proxy does two things:

1. verify the Continuum deployment at https://api.ai.confidential.cloud/. The verification process is described in the attestation section.
2. transparently encrypt user prompts and decrypt responses from the Continuum API.

The proxy is published as a Docker container for Linux (amd64 and arm64 architectures) on the GitHub registry.

Example

The following command starts the proxy and exposes it on host port 8080:

docker run -p 8080:8080 ghcr.io/edgelesssys/continuum/continuum-proxy:latest

To see all available CLI flags use:

docker run ghcr.io/edgelesssys/continuum/continuum-proxy:latest --help

Supply-chain security best practices recommend to pin containers by hash to ensure you are using the intended version.

Extract a static binary

If you want to run the proxy as binary, you may extract it from the container image. Depending on your architecture (arm64 or amd64), insert the <arch> variable below and you can get a static Linux binary by extracting it from the container like so:

containerID=$(docker create --platform linux/\<arch\> ghcr.io/edgelesssys/continuum/continuum-proxy:latest)
docker cp -L "${containerID}":/bin/continuum-proxy ./continuum-proxy
docker rm "${containerID}"

Outbound network traffic

When running the proxy in an environment with restricted firewall settings, you might need to whitelist the following domains:

• weu.service.attest.azure.net: for attestation verification
• kdsintf.amd.com: for verifying the certificate chain of the signed CPU attestation report
• attestation.ai.confidential.cloud: for communication and verification of the attestation service
• cdn.confidential.cloud: for fetching the latest manifest
• api.ai.confidential.cloud: for sending encrypted LLM requests

Manifest management

Whenever continuum-proxy verifies a Continuum deployment, it relies on a manifest to decide whether a cluster should be trusted or not. Details on this process can be found in section Manifest.

The manifest can either be controlled manually or automatically.

Manually

You can write a manifest manually, save it to disk and provide the file path to the continuum-proxy via its --manifestPath CLI flag.

warning

This isn't recommended for production because updates to the Continuum API are continuously rolled out. Each update comes with a new manifest, invalidating the current manifest and thereby stopping successful validation through the proxy. The manifest therefore needs to be manually updated for each Continuum API update.

Automatically

By default the proxy fetches a manifest from a CDN controlled by Edgeless Systems. Whenever validation of the cluster fails, the proxy will try to fetch a new manifest from the CDN and retry validation. This allows the proxy to continue working without manual intervention, even if the deployment changes.

To ensure auditability of the enforced manifests over time, changes to the manifest are logged to the local file system. These logs serve as a transparency log, recording which manifest was used at what point in time to verify the Continuum deployment.

The proxy writes a file called log.txt. For each manifest that's enforced by the proxy, log.txt contains a new line with the timestamp at which enforcement began and the filename of the manifest that was enforced. log.txt and the corresponding manifests are stored in a folder continuum-manifest. The CLI flag --workspace can be used to control where the folder continuum-manifest is stored.

You should mount the workspace to the docker host to ensure this transparency log isn't lost when the container is removed:

docker run -p 8080:8080 -v proxy-logs:/app/continuum-proxy ghcr.io/edgelesssys/continuum/continuum-proxy:latest --workspace /app/continuum-proxy

TLS

Use the flags --tlsCertPath and --tlsKeyPath to supply a valid TLS certificate. This makes continuum-proxy serve traffic via HTTPS. If the flags aren't set, the proxy falls back to HTTP.

Authorization

The Continuum API only allows requests that include an API key. Use the flag --apiKey to supply a valid API key to the continuum-proxy. This makes continuum-proxy overwrite the Authorization header on all requests. Note that this also overwrites Authorization headers that are sent by clients. If the flag isn't set, clients need to set the header in their requests.

API

continuum-proxy forwards requests to the Continuum API. It offers the API of the workload deployed in Continuum. Currently, this is vLLM. A description of the API served by vLLM can be found in the vLLM docs. The encryption required to protect prompts from the client to the workload is handled by the continuum-proxy.

Upgrades

It may happen that an upgrade to the Continuum API ships a new manifest that's incompatible with your current version of continuum-proxy. You will be able to recognize situations like this because the updated manifest can not be unmarshalled anymore. In those cases you will have to update the container.

In the future we will provide documentation on how to implement automatic updates.

Helm chart

You can use the continuum-proxy helm chart for easy deployment to Kubernetes.

Prerequisites

• Kubernetes 1.16+
• Helm 3+
• (Optional) Persistent Volume for workspace
• (Optional) ConfigMap for manifest file
• (Optional) TLS secret for certificates

Installation

helm repo add edgeless https://helm.edgeless.systems/stable
helm repo update

helm install continuum-proxy edgeless/continuum-proxy

Configuration

API Key

The API key should be stored in a Kubernetes secret. You can create it using:

kubectl create secret generic continuum-api-key --from-literal=apiKey=your-api-key

Persistent volumes

To ensure that the application’s data is persisted beyond the lifetime of the current deployment, you need to configure a Persistent Volume. This is particularly important when relying on automatic manifest updates as described in the manifest management section.

Where previously the container data was just mounted to the host, a Persistent volume is required in Kubernetes. Without a Persistent Volume, the transparency log will be lost if the pod is deleted or restarted, hindering the ability to track and verify historical deployment data.

Create the PersistentVolumeClaim:

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: continuum-proxy-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

Then, configure these values for your chart:

config:
  workspace:
    enabled: true
    volumeClaimName: "continuum-proxy-pvc"

Manifest File Configuration

While manually managing manifests isn't recommended (see Manifest management), you can pass in the manifest via a ConfigMap:

Create the ConfigMap from your manifest file:

kubectl create configmap continuum-proxy-config --from-file=manifest.toml=/path/to/your/manifest.toml

Then, configure these values for your chart:

config:
  manifest:
    enabled: true
    configMapName: "continuum-proxy-config"
    fileName: "manifest.toml"
    mountPath: "/etc/config/manifest.toml"

TLS Configuration

To enable TLS for communication between your application and the continuum-proxy locally in your cluster, provide the TLS certificate and key through a Kubernetes secret:

You can create the TLS secret manually or use cert-manager to manage it:

kubectl create secret tls continuum-proxy-tls \
  --cert=<path-to-cert> --key=<path-to-key>

Then, configure these values for your chart:

config:
  tls:
    enabled: true
    secretName: "continuum-proxy-tls"

Accessing the proxy

Once the deployment is complete, you can configure your application to access the OpenAI API through the continuum-proxy service’s domain.

By default, the continuum-proxy can be accessed at the following URL:

http://continuum-proxy-continuum-proxy.default.svc.cluster.local:8080/v1

This URL is constructed as follows:

http://{helm-release}-continuum-proxy.{namespace}.svc.cluster.local:{port}/v1

• {helm-release}: The name of your Helm release.
• {namespace}: The Kubernetes namespace where the continuum-proxy is deployed.
• {port}: The port configured for the continuum-proxy service (default is 8080).

If you have configured a custom DNS entry in your Kubernetes cluster, you will need to adjust the URL accordingly. Replace the default service domain with your custom domain, ensuring that your application can correctly resolve and communicate with the continuum-proxy service.

Uninstallation

To uninstall the chart:

helm uninstall continuum-proxy