(novice to) detailed tutorial on machine / in-depth learning with colab from scratch

During the epidemic, you can only practice at home , Simple machine learning with a notebook can barely make do with it , In depth learning it takes days and nights to train a model at random . But if we use Google Colab Of GPU resources , Any computer can play deep learning ！

What is? Colab

Quote the official introduction ：

Colaboratory It's a free Jupyter Notebook environment , You don't need to make any settings to use , And run completely in the cloud .
With the help of Colaboratory, You can write and execute code 、 Save and share analysis results , And use powerful computing resources , All of these can be used for free through the browser .

Colab It's a server , We can apply for his resources to use . Choose to use colab The biggest advantage of deep learning is its powerful GPU/TPU resources , Network training for deep learning , For parallel computing gpu The speed is comparable cpu Much faster .

You can colab The following commands are used to view colab What can be provided to us GPU To configure ：

! /opt/bin/nvidia-smi

This is all we can use for free GPU resources , If you can't meet your requirements, you can buy colab pro Or go to Taobao to rent a server .

How to use Colab

After all, it's a Google product , First you need a Google account , We should also be able to access the Internet scientifically .
Let's start with the simplest , First of all , How to create a Colab file .
Right click to create in your Google cloud docs , Pictured ：

You can't find it when you use it for the first time , stay more -> Connect more applications Just search and add in the .
Now you have created a file , Let me talk about several ways to use it .

Colab Several ways to use

image Jupyter Notebook The use of

For us Anaconda Of Jupyter Notebook More familiar with , It is easy to add code and comments , And can display the operation results in time , Very suitable for teaching presentation . Here you can be like notebook The use of colab, Even their file formats are the same , Yours notebook Files can be sent directly to the Google cloud and used colab open .
Here's an example ：

You can see the usage and notebook It's the same . Familiarity colab After the use of the simple calculation, I like to use colab To calculate , After all, old computers are slow to open any software , How about picking up some wool from Google .

How to use GPU Speed up

For deep learning, use GPU Words , We must adjust the settings first . Pictured ：


stay Execute code program -> Change the runtime type ->GPU Only in this way can we use GPU Resources for our in-depth learning .
In exchange for GPU After resources , You can use the following code to test whether you are really connected GPU resources ：

import tensorflow as tf
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
 raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))

If the output is Found GPU at: /device:GPU:0 It proves that you are connected successfully .

Here, if you can't use gpu If you accelerate, you'd better not choose , Because each account can be used every day GPU The acceleration time is limited , I used to use... For a while GPU, result colab It will be automatically disconnected for me in about twelve hours , When you want to connect again, you will be reminded that there is nothing left GPU resources , In this case, you can wait until the next day GPU Or create a new Google account and run .
Whether it's for use GPU Accelerate or not , There is an upper limit to how long the program can run , It's usually 24 Hours , But this is not a certain value , It will change at any time with the use of the server .

How to use Colab Run our program

We write our own programs or from GitHub How does a program that is copied on run normally ？
Linux The user can just press the command directly ,Colab It's the same thing .

mount Google Cloud disk
This is the first step in running our program , because Colab The operating principle of is actually to assign you a remote belt GPU The host , So its original path is not your Google cloud disk （ That is, your code file ） Path . So the first step is to mount the Google cloud disk to the remote host ：

from google.colab import drive
drive.mount('/content/gdrive')

Click the link after running , Then copy code Just type in .
After connecting to the cloud disk , In fact, other operations are similar to Linux It doesn't make much difference .

Carry out orders
colab And notebook The difference is , Its code box can not only execute code , Can also execute commands .
For command statements , We need to add before `！
For example, some common commands ：

import os
os.chdir("/content/gdrive/My Drive/3dunet/3DUnetCNN-master")

Jump to an address on your Google cloud disk

ls '/content/gdrive/My Drive/3dunet/3DUnetCNN-master'

View the contents under the path

%cd brats

Get into brasts Folder , Note that there cd Order to use % instead of ！

!pip install nipype

or

! pip install git+https://github.com/qubvel/segmentation_models

Install the required libraries

!pip install tensorflow==1.15.0

Install a version of the library . Here's a little bit of attention , We are GitHub Many of the projects I saw were written some time ago , So I used tensorflow Version may be different from colab Its own version is different . We can use the following command to view the current colab Of tensorflow edition

import tensorflow as tf

tf.__version__

What I am showing is '2.2.0', And the project may try tensorflow It was written in a little version , At this time, you need to change the version through the above command , After changing the version , Programs often need to be rerun .
And a little bit more , at present 1.15.0 Version of tensorflow It's supporting GPU Running , Many earlier versions do not support GPU, So if you lower it to, say 1.5.0 edition , Can't use GPU Of .

! python train.py

perform train.py file

In fact, these are related to our linux The next operation is the same , But now it is put into the Google cloud disk to execute .

Project operation
How to run in github On other people's projects ？

First, you should transfer other people's projects and required data sets to your own Google cloud disk , Reuse colab perform .
Note that the free space of Google cloud disk is only 15G, If your data set is large , Be careful that some images may be generated during the execution process, resulting in space overload , That will give you an error . You can buy a member , Less than ten yuan 100G, I feel it is much easier to use than the speed limit of Baidu cloud .

1. Mount Google cloud , I said just now

from google.colab import drive
drive.mount('/content/gdrive')

2. Go to the project address to be processed

import os
os.chdir("/content/gdrive/My Drive/3dclassification/3DUnetCNN-master")

3. Install the required libraries
use pip Command to install one by one , But most projects have requirements.txt file , You can directly

! pip install -r requirements.txt

Install all required libraries at once .
4. Execute project procedures

! python train.py

Training network

! python evaluate.py 

Evaluate performance

Project preparation
colab Not suitable for writing large programs , After all, are notebook file . but notebook The benefit of is intuitive , It is convenient for us to observe the output and debug , If you want to write a project , Simple projects written in one file are easy to say , But if you want to call functions in other files , First of all notebook file （.ipynb） To py File recall .

Download directly in the file as py Just send the file to your Google cloud disk .