Troubleshooting - Azure-Samples/cognitive-services-containers-samples GitHub Wiki

Having trouble? Hopefully the following will help:

• Using Cognitive Service containers in a corporate network
  • Running with a custom DNS server
  • Running behind a Proxy
    • Proxy Settings As Environmental Variables
    • Proxy Settings As Runtime Arguments
• Pod Status Stuck On "ImagePullBackoff"
  • Registry Secret Does Not Exist
  • Registry Secret is in the Wrong Namespace
  • Deployment Doesn’t Reference the Correct Secret

In a corporate network, proxies and custom DNS servers can interfere with normal operation of containers.

When Docker chooses an IP address range for a container, it may choose a range that includes your custom DNS server. This means requests to that server will be trapped in the container's local network, preventing outbound HTTP traffic. In this case, Docker recommends Changing which network address ranges Docker chooses.

When run behind a proxy, the containers may fail to connect with the billing endpoint. To avoid this, you can provide your proxy information in the form of environmental variables or runtime arguments when creating a container:

docker run --rm -it -p 5000:5000 -e HTTP_PROXY=http://myproxy:8080 mcr.microsoft.com/azure-cognitive-services/sentiment:latest Eula=accept ApiKey=<apikey> Billing=<endpoint>

Or, if your proxy requires credentials, they can be provided with HTTP_PROXY_CREDS

docker run --rm -it -p 5000:5000 -e HTTP_PROXY=http://myproxy:8080 -e HTTP_PROXY_CREDS=username:password mcr.microsoft.com/azure-cognitive-services/sentiment:latest Eula=accept ApiKey=<apikey> Billing=<endpoint>

docker run --rm -it -p 5000:5000 mcr.microsoft.com/azure-cognitive-services/sentiment:latest Eula=accept ApiKey=<apikey> Billing=<endpoint>  HTTP_PROXY=http://myproxy:8080 HTTP_PROXY_CREDS=username:password

If kubectl get pods shows a pod with status 'ImagePullBackoff', it is likely due to the cluster lacking necessary credentials to authenticate with the container image registry.

To verify this, run kubectl logs <pod> and look for a message referencing failed failed authorization or authentication with the registry.

There are multiple possible causes and solutions:

If your registry credentials are stored in an Azure KeyVault, you can use to this script to create your registry secrets. Otherwise, use the following command to create your secrets

kubectl create secret docker-registry $SecretName --docker-server=$Registry --docker-email=$Email --docker-username=$id --docker-password=$token --namespace $Namespace

(Note, for best security practice it is recommended to execute this such that your tokens are not persisted on disk or output to the console)

Make sure this $SecretName is mirrored in your [deployment values)(https://github.com/Azure-Samples/cognitive-services-containers-samples/blob/master/Kubernetes/language-understanding/helm-demo/values.yaml) file.

Registry secrets must be created in the same namespace that they are used. To check if your registry secrets are in the expected namespace, use:

kubectl get secrets --namespace <namespace>

The name of your registry secret must be the same as the name defined in your values.yaml file.

To check the name of your existing secret(s), run

kubectl get secrets --namespace <namespace>