Optimization techniques II

We discuss adaptive methods for SGD such as RMSprop and ADAM. We also talk about normalization layers and their effects on the neural network training process. Finally, we discuss a real-world example of neural nets being used in industry to make MRI scans faster and more efficient.

Please let me know if any changes need to be made before merging.

This:

• Moves PyTorch from 0.4 to 1.1 (one tiny code change)
• Moves Python from 3.6 to 3.7 (no changes to code, just env)
• Moves 1-2 requirements out of notebooks and into environment (potential nasty scipy pip install from librosa avoided!)
• Uses conda kernels so the correct environment kernel is available (all notebooks rerun to pick up proper kernel)
• Adds JuptyerLab (not required, but nice) - the interactive backend in the final notebook is still best in the classic interface. Try out built-in dark mode!

All notebooks seem to run (except noted minor issue with JupyterLab)

Hi Alf :wave:,

As indicated in my last email, I can't afford to wait for Yann's return without a big delay on my side. So here are the subtitle files:

• For English, it is the addition of the unicode. In practice:

1. The list of files not modified during this review of the unicode: practinum1 (didn't need unicode), practinum4 (the file contains blocks of 3 instead of 2 for the others), for lecture 12 (the only file I didn't translate into French)

2. The list of finished files (full English review + unicode) : lecture 6 & 9

3. The list of about clean files (partial English review + unicode) : lecture 1-3,10,11 + practinum 1-3, 7-8, 10

4. The list of not clean files (no English review + unicode): lecture 5-9,12-15 + practinum 5-6,9,11-15

• For French, these are all the subtitles (except for lecture 12 where I have huge problems understanding Mike Lewis's accent and so I preferred not to put anything than to translate badly).

I also added a disclamer for the V2 of the French translation of the website which should arrive this month. It should be my next and last PR closing the French translation work :boom:

Loïck

The following image links are broken:

• [x] Figure 2(a)
• [x] Figure 2(b)
• [x] Figure 3(a)
• [x] Figure 3(b)

See https://atcold.github.io/pytorch-Deep-Learning/en/week03/03-3/

I think the images were originally obtained from this presentation: 02 - CNN.pdf

See pages 10-11

Also, small suggestions:

• [x] Change Figure 4 to include R^7 and R^2 as in Slide 20 . This would better match the text for Figure 4.

• [x] Include Figure (4b maybe?) with that on Slide 21 to show what Padding is doing

Hi @Atcold ! I would like to know how and where should I commit markdown files in Portuguese? I recall that you have commented something with @ebetica .

Hi @JonathanSum ! Just for your info, There seems to be some inline latex broken on lecture 13-3:

The rest of the lectures I've checked seem to be fine.

This PR reorganizes the website structure, so we now have:

en/
  index.md
  about.dm
  week01/
  week02/
  ...
zh/
  index.md
  about.md
  week01/
  week02/
  ...
...

Hopefully it's less messy and easier to work with.

After this is merged, I will pull the images out into a global directory as well.

Also fixes some broken links in zh/index.md

I found an error visualizing on the github.io page the file /docs/es/week02/02-1.md.

The english version of the file appears before some parts and the layout of the spanish parts after the english parts gets a bit messed up.

A PR that fixes the timers of the sbv files that I couldn't correct in PR #660 to avoid conflicts.

I also took the opportunity to correct the few errors I caught when translating the lecture10.

I also noticed that the sbv files of the practinums of weeks 14 and 15 were missing.

I have translated the top 50% of the RNN(06-02) in Chinese.

I passed the course on deep learning.ai and I also wrote a few notebooks to help students in the coursera Tensorflow time series seq2seq notebook.

Poornima and I have compared an LSTM and RNN and visualized the gradients with respect to the input. We see that the gradients for the RNN are much smaller compared to the LSTM.

We are able to train MNIST for a large input sequence with an LSTM and failed to do so with an RNN.

Hope this is useful. If we need to make any chances, please let us know !

Added

• new truck methods for randomizing state within contraints
• new truck methods for seeing if truck is at dock or offscreen
• Training script for optimizing controller

Note

I currently have not successfully trained the controller to convergence. I have based the training off of this. On the website, they mention that the controller is hard to train. I have tried training it on the website with no success, so it seems like even their lessons are difficult to train. However, the code for training should be very similar to the code on the website. You may also alter the amount of lessons, max time steps, learning rate etc. to see if the model converges. I have been trying for over a week and have not succeeded yet.

I found that the Chinese version was basically machine translated, which caused the latex syntax to be broken. Of course, there are a lot of unreasonable translation. This PR is mainly about fixing broken latex. I also did my best to fix some of the translations that were too much bullshit.

I would question some translations in the dictionary for Russian: I've graduated this year and we haven't really translated everything. For example, it will be more understandable if I say "one-hot" in Russian as it is, rather than "унитарный код". Basically, I've never heard anyone calling it "унитарный код", to be honest...

So I guess there is a choice between being academically strict or being understood.

I think source activate is a few years old now and isn't supported anymore. https://stackoverflow.com/questions/49600611/python-anaconda-should-i-use-conda-activate-or-source-activate-in-linux

初めまして。私は皆様のおかげで深層学習の勉強に取り組み始めた学生の一人です。少しでも貢献しようと思い、リクエストをしました。修正は8-3までしか行っていません。今後も最後まで修正を続ける予定です。

主に簡単なマークダウンの記述ミスや日本語のミスを修正しました。いくつか気になった点がありましたので、報告します。これらのうちほとんどは修正をしていません。

1-3図5の部分でコメントアウトをしています。これをマークダウンファイルで見るとコメントアウトが有効に働いていることが分かるのですが、webページで見るとそうではないことが分かると思います。原因がわかる場合は教えていただきたいです。 2-3図6三層ニューラルネットワークの部分の図と説明文があっていないと思います。まず「3つのクラスがあります(C=3)」は図によればK=3です。また、d=100とd=1000の違いもあります。これは英文も同様です。 3-1「高次元の畳み込み」の数式でklとありますが、k,lが適切ではないでしょうか。これは英文も同様です。 5-1Figure5において、vsが強調表示されていない状態です。これは8-3Fig. 1: VAE vs. Classic AEでも同様です。また、これは英文も同様です。 6-1図4の直後「4)赤の障害物」は障害物が赤色というわけではないため、間違えていると思われます。英文では"4) red obstacle"となっており、こちらもおかしいです。 7-1で「eがエネルギーで、fが自由エネルギー」という記述があり、どちらも大文字でE、Fとするのが正しいと思うのですが、どうですか？eについては、英文も小文字になっています。 7-2「最尤法は、分子を大きく、分母を小さくして」の直後にある式は分母部分にもlogを付けるべきだと思います。これは英文も同様です。 8-1でwebページで見ると_heads_が強調されないまま表示されています。原因不明です。 8-3Fig. 4: Plot showing how relative entropy forces the bubbles to have variance = 1の直前の式で、V(z_i)はV(z)_iではないでしょうか。つまり、Fig. 4が正しいということです。これは英文も同様です。

長い文ですが、読んでいただけると嬉しいです。 ありがとうございます。