Ubuntu 18.04 development environment setup

Source code, Shell script

General

sudo apt-get update --fix-missing

gdebi replace dpkg to solve dependencies during installation

1
2
sudo apt install -y gdebi
sudo apt install -y cmake

Communication

Slack – install from “ubuntu software”

editor

vim

sudo apt install -y vim

atom

1
2
3
wget https://atom.io/download/deb -O atom.deb && \
yes | sudo gdebi atom.deb && \
rm atom.deb

image processing

image editor

1
2
sudo apt install gimp
sudo apt install pinta

screenshot

Ubuntu-18-04-screenshot-tool-flameshot-configuration
sudo apt install flameshot

browser

tor

sudo apt install -y torbrowser-launcher

chromium

sudo apt install -y chromium-browser

chrome

1
2
3
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb -O chrome.deb && \
yes | sudo gdebi chrome.deb && \
rm chrome.deb

Cloud Storage

Dropbox

1
2
3
wget https://www.dropbox.com/download?dl=packages/ubuntu/dropbox_2015.10.28_amd64.deb -O dropbox.deb && \
yes | sudo gdebi dropbox.deb && \
rm dropbox.deb

Music Player

netease(网易云)

1
2
3
4
5
6
7
wget http://d1.music.126.net/dmusic/netease-cloud-music_1.1.0_amd64_ubuntu.deb -O netease-cloud-music.deb && \
yes | sudo gdebi netease-cloud-music.deb && \
rm netease-cloud-music.deb

# https://github.com/MajeurAndroid/Adb-Remote-Screen/issues/14
# Failed to load module "canberra-gtk-module"
sudo apt install -y libcanberra-gtk-module

Ubuntu-18-04-netease-cloud-music-网易云-debug
Suspend from both command line and desktop icon
console output:
Local file: "" ("netease-cloud-music")

Solution:

  1. sudo netease-cloud-music
  2. After launching from command line or desktop icon, shutdown Ubuntu but then cancel.

Input

sudo apt install -y fcitx-bin

google pinyin

sudo apt install -y fcitx-googlepinyin

sogou pinyin

Ubuntu-18-04-fcitx-chinese-input-setup-google-and-sogou

1
2
3
wget "http://cdn2.ime.sogou.com/dl/index/1524572264/sogoupinyin_2.2.0.0108_amd64.deb?st=qC_O2p5443g1a2TJR_rSdA&e=1533163019&fn=sogoupinyin_2.2.0.0108_amd64.deb" -O sogoupinyin.deb && \
yes | sudo gdebi sogoupinyin.deb && \
rm sogoupinyin.deb

reboot -> fcitx configure -> add input

VPN

ShadowSocks

pip install git+https://github.com/shadowsocks/shadowsocks.git@master

openvpn

1
2
3
4
sudo apt install -y \
openvpn \
network-manager-openvpn \
network-manager-openvpn-gnome

Containerization and Virtualization

Virtualbox

1
2
3
wget https://download.virtualbox.org/virtualbox/5.2.16/virtualbox-5.2_5.2.16-123759~Ubuntu~bionic_amd64.deb -O virtualbox.deb && \
yes | sudo gdebi virtualbox.deb && \
rm virtualbox.deb

Docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
## SET UP THE REPOSITORY
sudo apt-get update && \
sudo apt install -y \
apt-transport-https \
ca-certificates \
curl \
software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"

## INSTALL DOCKER CE
sudo apt-get update
sudo apt-get install -y docker-ce
sudo docker run hello-world

Kubernetes

1
2
sudo snap install kubectl --classic
sudo snap install helm

Frontend

node, npm, yarn

1
2
3
4
5
6
7
8
9
10
11
sudo apt install -y \
nodejs \
npm
# node --version
# npm --version

# yarn
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list && \
sudo apt-get update && sudo apt-get install yarn
# yarn --version

Dash

1
2
3
4
5
pip install dash==0.22.0  # The core dash backend
pip install dash-renderer==0.13.0 # The dash front-end
pip install dash-html-components==0.11.0 # HTML components
pip install dash-core-components==0.26.0 # Supercharged components
pip install plotly --upgrade # Plotly graphing library used in examples

Others

1
2
3
4
5
6
7
8
9
sudo npm install -g create-react-app
sudo npm install -g hexo-cli
sudo npm install -g http-server

sudo apt install -y curl
sudo apt install -y jq

pip install gunicorn
sudo snap install postman

Nvidia driver, cuda, cuDNN

Nvidia driver

1
2
3
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update
sudo ubuntu-drivers autoinstall

Need reboot to use nvidia-smi, otherwise there would be error

1
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver

cuda[1]

1
2
3
4
sudo apt install nvidia-cuda-toolkit
sudo apt install libcupti-dev
nvcc --version
# cuda 9.1

apt would install cuda related packages into root system paths: /usr/bin, /usr/include,/usr/lib
while packages like tensorflow would look for them in /usr/local/cuda-.
so some symbolic links would be helpful

1
2
3
4
5
6
7
8
sudo mkdir -p /usr/local/cuda /usr/local/cuda/extras/CUPTI /usr/local/cuda/nvvm
sudo ln -s /usr/bin /usr/local/cuda/bin
sudo ln -s /usr/include /usr/local/cuda/include
sudo ln -s /usr/lib/x86_64-linux-gnu /usr/local/cuda/lib64
sudo ln -s /usr/local/cuda/lib64 /usr/local/cuda/lib
sudo ln -s /usr/include /usr/local/cuda/extras/CUPTI/include
sudo ln -s /usr/lib/x86_64-linux-gnu /usr/local/cuda/extras/CUPTI/lib64
sudo ln -s /usr/lib/nvidia-cuda-toolkit/libdevice /usr/local/cuda/nvvm/libdevice

cuDNN

sign in or log in to your Nvidia account here
https://developer.nvidia.com/cudnn

Download according to cuda version
https://developer.nvidia.com/rdp/cudnn-archive
Download cuDNN v7.0.5 (Dec 11, 2017), for CUDA 9.1 -> cuDNN v7.0.5 Library for Linux
sudo tar zxvf ~/Downloads/cudnn-9.1-linux-x64-v7.tgz -C /usr/local

Python, Machine Learning

Anaconda

1
2
3
4
5
6
7
wget --quiet https://repo.anaconda.com/archive/Anaconda3-5.2.0-Linux-x86_64.sh -O ~/anaconda.sh
# accept the license terms and install to ~/anaconda but have to add env path manually
# /bin/bash ~/anaconda.sh -b -p ~/anaconda
# echo 'export PATH="~/anaconda/bin:$PATH"' >> ~/.bashrc
/bin/bash ~/anaconda.sh
rm ~/anaconda.sh
source ~/.bashrc

Some python packages

1
2
3
pip install autopep8
pip install tqdm
pip install --upgrade google-api-python-client

Jupyter extension – Beakerx

require nodejs, npm, yarn
However, nodejs 10 would make yarn hangs during installation, nodejs 8 is fine[2]
[1/4] Resolving packages...

1
2
3
4
conda update -n base conda && \
conda config --env --add pinned_packages 'openjdk>8.0.121' && \
jupyter labextension install @jupyter-widgets/jupyterlab-manager && \
jupyter labextension install beakerx-jupyterlab

Tensorflow[3]

Build from source since pip install tensorflow-gpu would use default cuda 9.0 leads to error like
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

  1. git clone source code

    1
    2
    git clone https://github.com/tensorflow/tensorflow
    cd tensorflow
  2. Install bazel, the official tool to build tensorflow

    1
    2
    3
    4
    sudo apt install openjdk-8-jdk
    echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
    curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -
    sudo apt update && sudo apt install bazel
  3. Modify configuration file ./configure

    Noteworthy points:

    • GCC path: /usr/bin/gcc-6

      cuda 9.1 does not support gcc-7[4]

    • cuda version: 9.1

    • cuDNN version: 7.0.5
    • NCCL version: 1.3

      NCCL is used for multiple GPU, it is fine to use 1.3 if there is only one GPU since 1.3 would be automatically fetched, otherwise NCCL version 2 should be better but it need a bit more work to be installed beforehand

    • Cuda compute capabilities

  4. bazel build and pip install

    1
    2
    3
    bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
    bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
    pip install /tmp/tensorflow_pkg/tensorflow*.whl

Pytorch

conda install pytorch torchvision cuda91 -c pytorch

Keras

pip install keras

Xgboost[5]

Since ubuntu default version of gcc is 7, so this would happen
unsupported GNU version! gcc versions later than 6 are not supported!

Solution:

  1. -DCUDA_HOST_COMPILER=/usr/bin/gcc-6[6]
  2. export CC=/usr/local/bin/gcc-6[7]
1
2
3
4
cd ~ && \
git clone --recursive https://github.com/dmlc/xgboost && \
cd xgboost && mkdir build && cd build && cmake -DCUDA_HOST_COMPILER=/usr/bin/gcc-6 -DUSE_CUDA=ON .. && make -j && \
cd ../python-package && python setup.py install

LightGBM[8]

1
2
3
4
5
6
7
8
9
10
11
sudo apt install -y \
libboost-dev \
libboost-system-dev \
libboost-filesystem-dev

cd ~ && \
git clone --recursive https://github.com/Microsoft/LightGBM && \
cd LightGBM && mkdir build && cd build && \
cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ .. && \
make -j && \
cd ../python-package && python setup.py install --precompile

NLP

1
2
3
4
5
6
7
8
9
10
pip install -U nltk
# https://www.nltk.org/data.html
# default location: /usr/share/nltk_data
# use -d flag to specify a different location
python -m nltk.downloader all

pip install Unidecode
pip install tweepy
pip install jieba
pip install wordcloud

DevOps

Git

pre-installed
sudo apt install -y git

1
2
3
4
5
6
7
8
9
# https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup
git config --global user.name "Y** Y***"
git config --global user.email "sh***@gmail.com"
git config --global core.editor vim

# https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/
# ssh-keygen -t rsa -b 4096 -C "sh***@gmail.com"
# https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/
# copy paste ssh public key to github account (default: id_rsa.pub)

Ansible

1
2
3
4
5
sudo apt-get update && \
sudo apt-get install software-properties-common && \
sudo apt-add-repository ppa:ansible/ansible && \
sudo apt-get update && \
sudo apt-get install ansible

GCP-SDK

The google-cloud package is deprecated
The google-cloud package is no longer maintained or updated. Instead, install the google-cloud-* subpackages directly.[12]

1
2
3
4
5
6
7
8
9
10
pip install --upgrade google-cloud-storage
pip install --upgrade google-cloud-bigquery

# cli
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && \
echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
sudo apt-get update && sudo apt-get install google-cloud-sdk

gcloud init

Reference

[1] https://medium.com/@asmello/how-to-install-tensorflow-cuda-9-1-into-ubuntu-18-04-b645e769f01d
[2] https://github.com/jupyter-widgets/ipywidgets/issues/2061
[3] https://www.tensorflow.org/install/install_sources
[4] https://docs.nvidia.com/cuda/archive/9.1/cuda-installation-guide-linux/index.html
[5] https://xgboost.readthedocs.io/en/latest/build.html
[6] https://github.com/dmlc/xgboost/issues/2787
[7] https://stackoverflow.com/questions/17275348/how-to-specify-new-gcc-path-for-cmake
[8] https://lightgbm.readthedocs.io/en/latest/GPU-Tutorial.html
[9] https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup
[10] https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/
[11] https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/
[12] https://googlecloudplatform.github.io/google-cloud-python/latest/releases.html