logo
Splunk Team

how to craft a dockerfile for an image with nvidia driver/cuda (support for tensorflow-gpu) and python3 with pip?

  • Last Update :
  • Techknowledgy :

you can use this :

FROM nvidia / driver: 418.40 .04 - ubuntu18 .04
RUN apt - get - y update\ &&
   apt - get install - y software - properties - common\ &&
   apt - get - y update\ &&
   add - apt - repository universe
RUN apt - get - y update
RUN apt - get - y install python3
RUN apt - get - y install python3 - pip

Suggestion : 2

This guide provides a detailed overview about containers and step-by-step instructions for pulling and running a container and customizing and extending containers. ,In the next sections, you will use these image names for running containers. Later in the document there is also a section on creating your own containers or customizing and extending existing containers. ,If the system serves as the primary resource for both development and computing, then it is possible to setup a desktop-like environment on it via containerized desktop. The instructions and Dockerfile for this can be found here. ,If you were actually developing this model, you would be iterating by making changes to the files on the host and running the training script which executes inside the container.

docker images
REPOSITORY TAG IMAGE ID
mxnet-dec latest 65a48ellda96
<none>
   <none> bfc4512ca5f2
      nvcr.io/nvidian_general/adlr_pytorch resumes al34a09668a8
      <none>
         <none> 0f4ab6d62241
            <none>
               <none> 97274da5c898
                  nvcr.io/nvidian_sas/games-libcchem cuda10 3dcl3f8347f9
                  nvidia/cuda latest 614dcdafa05c
                  ubuntu latest d355ed3537e9
                  deeper_photo latest 9326345l4d5a
                  nvcr.io/nvidia/caffe 19.03 b7b62dacdebl
                  nvidia/cuda 10.0-devel-centos7 6e3e5b71176e
                  nvcr.io/nvidia/tensorflow 19.03 56f2980ble37
                  nvidia/cuda 10.0-cudnn7-devel-ubuntu16.04 22afb0578249
                  nvidia/cuda 10.0-devel a760a0cfca82
                  mxnet/python gpu 7e7c9176319c
                  nvcr.io/nvidia_sas/chainer latest 2ea707c58bea
                  deep_photo latest ef4510510506
                  <none>
                     <none> 9124236672fe
                        nvcr.io/nvidia/cuda 10.0-cudnn7-devel-ubuntu18.04 02910409eb5d
                        nvcr.io/nvidia/tensorflow 19.05 9dda0d5c344f
bash
$ docker exec -it <CONTAINER_ID_OR_NAME> bash
Deep Learning GPU Training System™ (DIGITS)
docker run--gpus all - d--name test - digits\ -
   u $(id - u): $(id - g) - e HOME = $HOME - e USER = $USER - v $HOME: $HOME\
nvcr.io / nvidia / digits: 17.05
$ docker exec - it test - digits bash
Inside this directory, create a file called Dockerfile (capitalization is important). This is the default name that Docker looks for when creating a container. The Dockerfile should look similar to the following:
[username~] $ mkdir mydocker
   [username~] $ cd mydocker[username mydocker] $ vi Dockerfile[username mydocker] $ more Dockerfile
FROM nvcr.io / nvidia / tensorflow: 19.03

RUN apt - get update

RUN apt - get install - y octave[username mydocker] $
Docker
$ docker build - t nvcr.io / nvidian_sas / tensorflow_octave: 17.06_ with_octave
nvcr.io
[username mydocker] $ docker build - t nvcr.io / nvidian_sas / tensorflow_octave: 19.03_ with_octave.
Sending build context to Docker daemon 2.048 kB
Step 1 / 3: FROM nvcr.io / nvidia / tensorflow: 1903–-- > 56 f2980ble37
Step 2 / 3: RUN apt - get update–-- > Running in 69 cffa7bbadd
Get: 1 http: //security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
   Get: 2 http: //ppa.launchpad.net/openjdk-r/ppa/ubuntu xenial InRelease [17.5 kB]
   Get: 3 http: //archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
   Get: 4 http: //ppa.launchpad.net/openjdk-r/ppa/ubuntu xenial/main amd64 Packages [7096 B]
   Get: 5 http: //security.ubuntu.com/ubuntu xenial-security/universe Sources [42.0 kB]
   Get: 6 http: //security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [380 kB]
   Get: 7 http: //archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
   Get: 8 http: //security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.8 kB]
   Get: 9 http: //security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [178 kB]
   Get: 10 http: //security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [2931 B]
   Get: 11 http: //archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
   Get: 12 http: //archive.ubuntu.com/ubuntu xenial/universe Sources [9802 kB]
   Get: 13 http: //archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
   Get: 14 http: //archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
   Get: 15 http: //archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Docker
$ docker build - t nvcr.io / nvidian_sas / tensorflow_octave: 17.06_ with_octave

Suggestion : 3

This guide is for the latest stable version of TensorFlow. For the preview build (nightly), use the pip package named tf-nightly. Refer to these tables for older TensorFlow version requirements. For the CPU-only build use the pip package named tensorflow-cpu.,A few installation mechanisms require the URL of the TensorFlow Python package. The value you specify depends on your Python version.,Packages do not contain PTX code except for the latest supported CUDA® architecture; therefore, TensorFlow fails to load on older GPUs when CUDA_FORCE_PTX_JIT=1 is set. (See Application Compatibility for details.),Miniconda is the recommended approach for installing TensorFlow with GPU support. It creates a separate environment to avoid changing any installed software in your system. This is also the easiest way to install the required software especially for the GPU setup.

Linux

conda install - c conda - forge cudatoolkit = 11.2 cudnn = 8.1 .0
export LD_LIBRARY_PATH = $LD_LIBRARY_PATH: $CONDA_PREFIX / lib /
   python3 - m pip install tensorflow
# Verify install:
   python3 - c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

MacOS

# There is currently no official GPU support
for MacOS.
python3 - m pip install tensorflow
# Verify install:
   python3 - c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
conda install - c conda - forge cudatoolkit = 11.2 cudnn = 8.1 .0
python3 - m pip install tensorflow
# Verify install:
   python3 - c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
conda install - c conda - forge cudatoolkit = 11.2 cudnn = 8.1 .0
export LD_LIBRARY_PATH = $LD_LIBRARY_PATH: $CONDA_PREFIX / lib /
   python3 - m pip install tensorflow
# Verify install:
   python3 - c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

CPU

python3 - m pip install tensorflow
# Verify install:
   python3 - c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

Nightly

python3 - m pip install tf - nightly
# Verify install:
   python3 - c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

Suggestion : 4

Make sure you also check with the machine learning framework that you intend to use in order to know which version of cuDNN is needed. As an example, Tensorflow's NVIDIA hardware and software requirements for GPU support are listed in the Tensorflow documentation.,The following sample Dockerfile illustrates an engine on top of which machine learning frameworks such as Tensorflow and PyTorch can be used. This Dockerfile uses a deep learning library from NVIDIA called NVIDIA CUDA Deep Neural Network (cuDNN). For detailed information about compatibility between NVIDIA driver versions and CUDA, refer the cuDNN installation guide (prerequisites). ,The following sample Dockerfile uses NVIDIA's official Dockerfiles for CUDA and cuDNN images. ,When creating the Dockerfile, you must delete the Cloudera repository that is inaccessible because of the paywall by running the following:

RUN rm / etc / apt / sources.list.d /*
1._
RUN rm / etc / apt / sources.list.d /*

cuda.Dockerfile 

FROM docker.repository.cloudera.com / cdsw / engine: 10
RUN rm / etc / apt / sources.list.d
/*
RUN apt-get update && apt-get install -y --no-install-recommends \
gnupg2 curl ca-certificates && \
curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub | apt-key add - && \
echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list && \
echo "deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list && \
apt-get purge --autoremove -y curl && \
rm -rf /var/lib/apt/lists/*

ENV CUDA_VERSION 10.1.243
LABEL com.nvidia.cuda.version="${CUDA_VERSION}"

ENV CUDA_PKG_VERSION 10-1=$CUDA_VERSION-1
RUN apt-get update && apt-get install -y --no-install-recommends \
        cuda-cudart-$CUDA_PKG_VERSION && \
    ln -s cuda-10.1 /usr/local/cuda && \
    rm -rf /var/lib/apt/lists/*

RUN echo "/usr/local/cuda/lib64" >> /etc/ld.so.conf.d/cuda.conf && \
    ldconfig

RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
    echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf

ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64

RUN echo "deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list

ENV CUDNN_VERSION 7.6.5.32
LABEL com.nvidia.cudnn.version="${CUDNN_VERSION}"

RUN apt-get update && apt-get install -y --no-install-recommends \
            libcudnn7=$CUDNN_VERSION-1+cuda10.1 && \
    apt-mark hold libcudnn7 && \
    rm -rf /var/lib/apt/lists/*
cuda.Dockerfile
docker build --network host -t <company-registry>/cdsw-cuda:10 . -f cuda.Dockerfile
cuda.Dockerfile
docker build --network host -t <company-registry>/cdsw-cuda:10 . -f cuda.Dockerfile


Trending Technology


Most popular in python