详细介绍在Ubuntu 16.04下搭建CUDA7.5+Caffe深度学习环境的过程步骤。
1.安装Ubuntu 16.04
省略。不懂可以自行百度,系统安装后安装必要的更新和工具。
sudo apt update
sudo apt-get upgrade
sudo apt-get install vim
sudo apt-get install cmake
2.安装显卡驱动
进入all setting->Software Update,更换英伟达361.42驱动,重启电脑,使用nvidia-smi测试是否成功。
3.安装cuda
(1)安装必要的依赖库
ca-certificates-java
default-jre
default-jre-headless
fonts-dejavu-extra
freeglut3
freeglut3-dev
java-common
libatk-wrapper-java
libatk-wrapper-java-jni
libdrm-dev
libgl1-mesa-dev
libglu1-mesa-dev
libgnomevfs2-0
libgnomevfs2-common
libice-dev
libpthread-stubs0-dev
libsctp1
libsm-dev
libx11-dev
libx11-doc
libx11-xcb-dev
libxau-dev
libxcb-dri2-0-dev
libxcb-dri3-dev
libxcb-glx0-dev
libxcb-present-dev
libxcb-randr0-dev
libxcb-render0-dev
libxcb-shape0-dev
libxcb-sync-dev
libxcb-xfixes0-dev
libxcb1-dev
libxdamage-dev
libxdmcp-dev
libxext-dev
libxfixes-dev
libxi-dev
libxmu-dev
libxmu-headers
libxshmfence-dev
libxt-dev
libxxf86vm-dev
lksctp-tools
mesa-common-dev
openjdk-7-jre
openjdk-7-jre-headless
tzdata-java
x11proto-core-dev
x11proto-damage-dev
x11proto-dri2-dev
x11proto-fixes-dev
x11proto-gl-dev
x11proto-input-dev
x11proto-kb-dev
x11proto-xext-dev
x11proto-xf86vidmode-dev
xorg-sgml-doctools
xtrans-dev
libgles2-mesa-dev
nvidia-modprobe
build-essential
ca-certificates-java
default-jre
default-jre-headless
fonts-dejavu-extra
freeglut3
freeglut3-dev
java-common
libatk-wrapper-java
libatk-wrapper-java-jni
libdrm-dev
libgl1-mesa-dev
libglu1-mesa-dev
libgnomevfs2-0
libgnomevfs2-common
libice-dev
libpthread-stubs0-dev
libsctp1
libsm-dev
libx11-dev
libx11-doc
libx11-xcb-dev
libxau-dev
libxcb-dri2-0-dev
libxcb-dri3-dev
libxcb-glx0-dev
libxcb-present-dev
libxcb-randr0-dev
libxcb-render0-dev
libxcb-shape0-dev
libxcb-sync-dev
libxcb-xfixes0-dev
libxcb1-dev
libxdamage-dev
libxdmcp-dev
libxext-dev
libxfixes-dev
libxi-dev
libxmu-dev
libxmu-headers
libxshmfence-dev
libxt-dev
libxxf86vm-dev
lksctp-tools
mesa-common-dev
openjdk-7-jre
openjdk-7-jre-headless
tzdata-java
x11proto-core-dev
x11proto-damage-dev
x11proto-dri2-dev
x11proto-fixes-dev
x11proto-gl-dev
x11proto-input-dev
x11proto-kb-dev
x11proto-xext-dev
x11proto-xf86vidmode-dev
xorg-sgml-doctools
xtrans-dev
libgles2-mesa-dev
nvidia-modprobe
build-essential
(2)安装cuda-toolkit
① 安装cuda_7.5.18_linux.run
sudo ./cuda_7.5.18_linux.run --override
安装过程如下:
Do you accept the previously read EULA? (accept/decline/quit): accept
You are attempting to install on an unsupported configuration. Do you wish to continue? ((y)es/(n)o) [ default is no ]: y
Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 352.39? ((y)es/(n)o/(q)uit): n
Install the CUDA 7.5 Toolkit? ((y)es/(n)o/(q)uit): y
Enter Toolkit Location [ default is /usr/local/cuda-7.5 ]:
Do you want to install a symbolic link at /usr/local/cuda? ((y)es/(n)o/(q)uit): y
Install the CUDA 7.5 Samples? ((y)es/(n)o/(q)uit): y
Enter CUDA Samples Location [ default is /home/kinghorn ]: /usr/local/cuda-7.5
Installing the CUDA Toolkit in /usr/local/cuda-7.5 ...
Finished copying samples.
===========
= Summary =
===========
Driver: Not Selected
Toolkit: Installed in /usr/local/cuda-7.5
Samples: Installed in /usr/local/cuda-7.5
② 设置环境变量
vi /home/xxx/.bashrc
添加如下内容:
export PATH=/usr/local/cuda/bin:$PATH
执行如下命令使环境变量生效
source /home/xxx/.bashrc
将cuda动态库添加到动态库管理器
sudo vi /etc/ld.so.conf.d/cuda.conf
添加:
/usr/local/cuda/lib64
执行ldconfig使新加的库生效
sudo ldconfig
③ 强制使用gcc5
编辑/usr/local/cuda/include/host_config.h文件,注释掉115行
#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
改为:
//#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
(3)编译cuda例子与测试
进入到/usr/local/cuda/NVIDIA_CUDA-7.5_Samples/1_Utilities/deviceQuery目录执行:
sudo make
./deviceQuery
4.安装cudnn库
(1)解压
tar xzvf cudnn-xxx-ga.tgz
tar xzvf cudnn-xxx-ga.tgz
得到cuda文件夹里面含有的lib64和include两个文件夹
(2)拷贝到cuda安装目录
sudo cp cuda/cudnn.h /usr/local/cuda/include/
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo cp cuda/cudnn.h /usr/local/cuda/include/
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
注意:拷贝后将链接删除重新建立链接,否则,拷贝是多个多个不同名字的相同文件,链接关系参见cudnn解压后的文件夹。也可以分别拷贝每一个文��,链接文件拷贝使用cp -d命令。
5.安装opencv3.1.0
(1)安装基本必要库
sudo apt-get install build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
(2)配置opencv,生成Makefile
cd opencv-3.1.0
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
sudo apt-get install build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
(2)配置opencv,生成Makefile
cd opencv-3.1.0
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
cd opencv-3.1.0
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
在configure过程中过程中,可能会出现下面的错误:
– ICV: Downloading ippicv_linux_20151201.tgz…
在直接下载该文件的过程中,会因为超时而失败,需要收到下载,将其拷贝至opencv-3.1.0/3rdparty/ippicv/downloads/linux-8b449a536a2157bcad08a2b9f266828b目录内,重新执行配置命令。
(3)编译opencv
make -j8
make -j8
此时可能会出现另一个错误:
/usr/include/string.h: In function ‘void* __mempcpy_inline(void*, const void*, size_t)’: /usr/include/string.h:652:42: error: ‘memcpy’ was not declared in this scope return (char *) memcpy (__dest, __src, __n) + __n;
这是因为ubuntu的g++版本过高造成的,只需要在opencv-3.1.0目录下的CMakeList.txt 文件的开头加入:
set(CMAKE_CXX_FLAGS “${CMAKE_CXX_FLAGS} -D_FORCE_INLINES”)
添加之后再次进行编译链接即可。
(4)查看版本号
pkg-config --modversion opencv
(5)安装
sudo make install
6.安装caffe与配置
(1)安装必要的依赖库
sudo apt-get install build-essential
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install Python-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
pkg-config --modversion opencv
sudo make install
6.安装caffe与配置
(1)安装必要的依赖库
sudo apt-get install build-essential
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install Python-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install build-essential
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install Python-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
如果这些库都能顺利安装,会大大减少后面遇到的问题。
(2)下载caffe-master并解压得到源码包
解压:
unzip caffe-master.zip
(3)修改配置文件Make.config
cd caffe-master
cp Makefile.config.example Makefile.config
vi Makefile.config
cd caffe-master
cp Makefile.config.example Makefile.config
vi Makefile.config
将# USE_CUDNN := 1前得#注释去掉,表示使用cuDNN,如果不是使用GPU,可以将# CPU_ONLY := 1前得注释去掉。这里我使用cuDNN来加速。
(4)编译caffe
方法1:使用cmake编译
mkdir build
cd build
cmake ..
make all -j8
这种方法一般不会出现问题。
方法2:直接使用gcc编译
make -j8
错误1:
src/caffe/net.cpp:8:18: fatal error: hdf5.h: No such file or directory
参考方法
cd /usr/lib/x86_64-linux-gnu
sudo ln -s libhdf5_serial.so.10.1.0 libhdf5_serial.so
sudo ln -s libhdf5_serial_hl.so.10.0.2 libhdf5_serial_hl.so
修改Makefile.config
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial
错误2:
error -- unsupported GNU version! gcc versions later than 5.3 are not supported!
目前caffe不支持高于5.3的gcc,理论上可通过对gcc,g++降级解决,但是降级后还会引起其他兼容性问题,因此并不能解决实际问题,下面附上降级方法。解决方法在后面。
① 安装低版本gcc、g++
sudo apt-get install gcc-4.7 gcc-4.7-multilib
sudo apt-get install g++-4.7 g++-4.7-multilib
② 设置优先级
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 50
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.7 40
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 50
③ 选择版本
sudo update-alternatives --config gcc
There are 2 choices for the alternative gcc (providing /usr/bin/gcc)
Selection Path Priority Status
------------------------------------------------------------
0 /usr/bin/gcc-5 50 auto mode
* 1 /usr/bin/gcc-4.7 40 manual mode
2 /usr/bin/gcc-5 50 manual mode
sudo update-alternatives --config g++
There are 2 choices for the alternative g++ (providing /usr/bin/g++).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/bin/g++-5 50 auto mode
* 1 /usr/bin/g++-4.7 40 manual mode
2 /usr/bin/g++-5 50 manual mode
错误3:
/usr/include/string.h: In function ‘void* __mempcpy_inline(void*, const void*, size_t)’: /usr/include/string.h:652:42: error: ‘memcpy’ was not declared in this scope return (char *) memcpy (__dest, __src, __n) + __n;
NVCCFLAGS += -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)
改为:
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)
错误3:
/usr/bin/ld: cannot find -lippicv
cp opencv-3.1.0/3rdparty/ippicv/unpack/ippicv_lnx/lib/intel64/libippicv.a /usr/local/lib
再次编译即可。
至此,gcc、g++降级完成。
下面是错误2 的真正解决方法(红色字体):
sudo vi /usr/local/cuda/include/host_config.h
#if __GNUC__ > 5 || (__GNUC__ == 5 && __GNUC_MINOR__ > 3)
#error -- unsupported GNU version! gcc versions later than 5.3 are not supported!
修改为:
#if __GNUC__ > 5 || (__GNUC__ == 5 && __GNUC_MINOR__ > 4)
#error -- unsupported GNU version! gcc versions later than 5.4 are not supported!
我的gcc版本为5.4.0,可根据具体情况修改。
(5)编译caffe的python接口
make pycaffe
出错:
python/caffe/_caffe.cpp:10:31: fatal error: numpy/arrayobject.h: No such file or directory
原因是numpy路径配置错误将:
PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/lib/python2.7/dist-packages/numpy/core/include
改为:
PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/local/lib/python2.7/dist-packages/numpy/core/include
(6)测试caffe
make runtest
make runtest
这个时间有点长。
7.运行手写体例程
caffe自带手写体识别的测试例子。每一步caffe都已经写好脚本,执行几个简单命令就可以将第一个深度学习程序跑起来。
(1)获取数据(并完成数据标注)
sh data/mnist/get_mnist.sh
(2)将标签数据转换成caffe使用的LMDB数据格式
sh examples/mnist/create_mnist.sh
(3)网络求解文件修改
vi caffe-master/examples/mnist/lenet_solver.prototxt
# The train/test net protocol buffer definition
net: "examples/mnist/lenet_train_test.prototxt"
# test_iter specifies how many forward passes the test should carry out.
# In the case of MNIST, we have test batch size 100 and 100 test iterations,
# covering the full 10,000 testing images.
test_iter: 100
# Carry out testing every 500 training iterations.
test_interval: 500
# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005
# The learning rate policy
lr_policy: "inv"
gamma: 0.0001
power: 0.75
# Display every 100 iterations
display: 100
# The maximum number of iterations
max_iter: 10000
# snapshot intermediate results
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet"
# solver mode: CPU or GPU
solver_mode: GPU
sh data/mnist/get_mnist.sh
sh examples/mnist/create_mnist.sh
(3)网络求解文件修改
vi caffe-master/examples/mnist/lenet_solver.prototxt
# The train/test net protocol buffer definition
net: "examples/mnist/lenet_train_test.prototxt"
# test_iter specifies how many forward passes the test should carry out.
# In the case of MNIST, we have test batch size 100 and 100 test iterations,
# covering the full 10,000 testing images.
test_iter: 100
# Carry out testing every 500 training iterations.
test_interval: 500
# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005
# The learning rate policy
lr_policy: "inv"
gamma: 0.0001
power: 0.75
# Display every 100 iterations
display: 100
# The maximum number of iterations
max_iter: 10000
# snapshot intermediate results
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet"
# solver mode: CPU or GPU
solver_mode: GPU
vi caffe-master/examples/mnist/lenet_solver.prototxt
# The train/test net protocol buffer definition
net: "examples/mnist/lenet_train_test.prototxt"
# test_iter specifies how many forward passes the test should carry out.
# In the case of MNIST, we have test batch size 100 and 100 test iterations,
# covering the full 10,000 testing images.
test_iter: 100
# Carry out testing every 500 training iterations.
test_interval: 500
# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005
# The learning rate policy
lr_policy: "inv"
gamma: 0.0001
power: 0.75
# Display every 100 iterations
display: 100
# The maximum number of iterations
max_iter: 10000
# snapshot intermediate results
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet"
# solver mode: CPU or GPU
solver_mode: GPU
最后一行,训练过程采用CPU、GPU选择,如果不使用GPU,修改solver_mode: GPU为solver_mode: CPU即可,这里我使用GPU。
(4)执行训练脚本
sh examples/mnist/train_lenet.sh
sh examples/mnist/train_lenet.sh
大约10分钟左右,模型训练完成
I0716 14:46:01.360709 27985 solver.cpp:404] Test net output #0: accuracy = 0.9908
I0716 14:46:01.360750 27985 solver.cpp:404] Test net output #1: loss = 0.0303895 (* 1 = 0.0303895 loss)
I0716 14:46:01.360755 27985 solver.cpp:322] Optimization Done.
I0716 14:46:01.360757 27985 caffe.cpp:222] Optimization Done.
模型精度在0.99以上。
至此,Caffe+Linux深度学习环境搭建完成。
Ubuntu 15.04 下Caffe + + CUDA 7.0 安装配置指南 http://www.linuxidc.com/Linux/2016-11/137497.htm
Caffe 深度学习入门教程 http://www.linuxidc.com/Linux/2016-11/136774.htm
Ubuntu 16.04下Matlab2014a+Anaconda2+OpenCV3.1+Caffe安装 http://www.linuxidc.com/Linux/2016-07/132860.htm
Ubuntu 16.04系统下CUDA7.5配置Caffe教程 http://www.linuxidc.com/Linux/2016-07/132859.htm
Caffe在Ubuntu 14.04 64bit 下的安装 http://www.linuxidc.com/Linux/2015-07/120449.htm
深度学习框架Caffe在Ubuntu下编译安装 http://www.linuxidc.com/Linux/2016-07/133225.htm
Caffe + Ubuntu 14.04 64bit + CUDA 6.5 配置说明 http://www.linuxidc.com/Linux/2015-04/116444.htm
Ubuntu 16.04上安装Caffe http://www.linuxidc.com/Linux/2016-08/134585.htm
Caffe配置简明教程 ( Ubuntu 14.04 / CUDA 7.5 / cuDNN 5.1 / OpenCV 3.1 ) http://www.linuxidc.com/Linux/2016-09/135016.htm
Ubuntu 16.04安装 Caffe GPU版 http://www.linuxidc.com/Linux/2017-09/147111.htm
Ubuntu 16.04上安装Caffe(CPU only) http://www.linuxidc.com/Linux/2016-09/135034.htm
本文永久更新链接地址:http://www.linuxidc.com/Linux/2017-10/147612.htm