The Missing Semester

A missing lesson in computer education

Motivation for this course

No longer always copy and paste from documents command . No more “ Do this one by one 15 An order ”, No more “ You forgot to execute this order ”、“ You forgot to pass that Parameters ”, Don't have similar conversations anymore .

for example , A quick search of history can save a lot of time . In the following example , We showed how to get through convert command Some skills of jumping in history .

Use SSH How to keep the connection when the key is connected to the remote machine for work , And enable the terminal to reuse . No longer in order to execute only individual commands Always open many command line terminals . No longer always enter a password every time you connect . No longer because the network is disconnected or the laptop must be restarted Lose all context .

How to modify directly and easily through the command line 、 see 、 analysis 、 Draw and calculate data and files . No longer copy from log files Paste . No more manual statistics . No more drawing with spreadsheets .

Course overview and shell

You can't click a button that doesn't exist or voice input an instruction that hasn't been entered yet . In order to make full use of the power of the computer , We have to go back to the most fundamental way , Use the text interface ：Shell

Almost all platforms you have access to support some form of shell, Some even offer a variety of shell For your choice . Although there are some differences in detail between them , But its core functions are the same ： It allows you to execute programs , Input and get some semi-structured output .

In this lesson, we will use Bourne Again SHell, abbreviation “bash” . This is one of the most widely used shell, Its grammar and other shell It's all similar .

shell Is a programming environment , So it has variables 、 Conditions 、 Loops and functions . If you ask for shell Execute an instruction , But the directive is not shell Know programming keywords , Then it will consult environment variable $PATH, It will list when shell When you receive an instruction , The path to program search

Usually , The input and output streams of a program are your terminals . That is to say , Use your keyboard as input , Display as output . however , We can also redirect these streams ！

The simplest redirection is < file and > file. These two commands can redirect the input and output streams of the program to files

|、>、 and < It's through shell Executive , Instead of being executed by individual programs . echo I don't know | The existence of , They only know how to read and write from their own input and output streams

Shell Tools and scripts

majority shell Each has its own script language , Including variable 、 Control flow and its own syntax .shell Script differs from other scripting languages in ,shell Script for shell To optimize the relevant work . therefore , Create command flow （pipelines）、 Save the results to a file 、 Read input from standard input , These are all shell Native operations in scripts , This makes it easier to use than a generic scripting language .

When you pass $( CMD ) In this way CMD On this order , Its output will be replaced $( CMD ) . for example , If you execute for file in $(ls) ,shell First, I will call ls , Then traverse these returned values

<( CMD ) Will execute CMD And output the results to a temporary file , And will <( CMD ) Replace with temporary file name . This is when we want the return value to pass through the file instead of STDIN It is very useful when passing . for example , diff <(ls foo) <(ls bar) The folder will be displayed foo and bar Differences between files in .

For the file foo, foo1, foo2, foo10 and bar, rm foo? This command will delete foo1 and foo2 , and rm foo* Will be deleted except bar All other files .

Curly braces {} - When you have a series of instructions , When it contains a common substring , You can use curly braces to automatically expand these commands . This is very convenient when moving or converting files in batches .

All the classes UNIX The system contains a named find Tools for , It is shell A great tool for finding files on .
find And similar tools can use other attributes, such as file size 、 Modify the time or permission to find the file . however find The format of is a little complicated .
fd It's simpler 、 Faster 、 More friendly , It can be used as find substitute . It has many good default settings , For example, output shading 、 Regular matching is supported by default 、 Support unicode And its grammar is more intuitive .

Finding files is a useful skill , But most of the time, your goal is to check the contents of the file . One of the most common scenarios is when you want to find all the files with a certain pattern , And find their location .

To achieve this , Many kinds UNIX All systems provide grep command

grep There are many options , This also makes it a very versatile tool . Among them, I often use -C ： Get the context of the search result （Context）;-v The result will be inversely selected （Invert）, That is, output mismatched results . for instance , grep -C 5 Five lines before and after the matching result will be output . When you need to search a large number of files , Use -R Will recursively enter subdirectories and search all text files .

Therefore, there are many substitutes for it , Include ack, ag and rg. They are very easy to use , But the functions are also similar , What I use more often is ripgrep (rg) , Because it's fast , And the usage is very intuitive .

You may want to find a command you entered before . First , Press the up arrow key to display the last command you used , If you continue to press the up key, you will traverse the entire history . If we want to search history , Then you can use the pipeline to pass the output result to grep Conduct pattern search . history | grep find Will print with find Substring command .

For most shell Come on , You can use Ctrl+R Backtrack the command history . knock Ctrl+R Then you can enter the substring to match , Find the historical command line .
Press repeatedly to cycle through all search results .

Ctrl+R Can cooperate with fzf Use .fzf Is a general fuzzy search tool , It can be used with many commands . Here we can perform fuzzy search on historical commands and output the results in a pleasing format .

Another skill related to historical commands, which I like to call automatic completion based on history . This feature was originally developed by fish shell Created , It can start with the same command that you have recently used , Dynamically pair the current pair shell Command to complete . This function is in zsh Can also be used in , It can greatly improve the user experience .

You can modify shell history act , for example , If you add a space at the beginning of the command , It will not be added shell On record . This feature is used when you enter commands that contain passwords or other sensitive information . To do this, you need to .bashrc Add HISTCONTROL=ignorespace Or to .zshrc add to setopt HIST_IGNORE_SPACE.

We can use fasd and autojump These two tools are used to find the most commonly used or recently used files and directories
fasd Use command z Help us quickly switch to the most frequently visited directory . for example , If you visit /home/user/files/cool_project Catalog , So you can use it directly z cool Jump to this directory .

There are also more sophisticated tools that can be used to overview the directory structure , for example tree, broot Or a more complete file manager , for example nnn or ranger.

Editor (Vim)

Vim I spend most of my time reading 、 Browse and make a few editorial changes , Therefore, it has a variety of operation modes ：

Normal mode ： Move the cursor around the file to modify
Insertion mode ： insert text
Replace mode ： replace text
visualization （ commonly , That's ok , block ） Pattern ： Select the text block
Command mode ： Used to execute a command

In different operation modes , The meaning of keyboard tapping is also different . such as ,x Letters are inserted in insert mode x, But in normal mode Will delete the letter of the current cursor , In visual mode, the selected text block will be deleted .
Usually you will put most Time spent in normal mode and insert mode .
You can press （ Escape key ） Return to normal mode from any other mode . In normal mode , type i Go in and insert Pattern , R Go to replace mode , v Enter visual （ commonly ） Pattern , V Enter visual （ That's ok ） Pattern , （Ctrl-V, Sometimes I write ^V） Enter visual （ block ） Pattern , : Enter command mode .

Most of the time you will be in normal mode , Use the move command to navigate through the cache . stay Vim Moving inside is also called “ Noun ”, Because they point to blocks of text .
Basic mobility : hjkl （ Left , Next , On , Right ）
word ： w （ The next word ）, b （ Ci Chu ）, e （ The end of the word ）
That's ok ： 0 （ The beginning of the line ）, ^ （ The first non space character ）, $ （ At the end of the line ）
The screen ： H （ First line of screen ）, M （ In the middle of the screen ）, L （ At the bottom of the screen ）
Page turning ： Ctrl-u （ Turn up ）, Ctrl-d （ Down turn ）
file ： gg （ The file header ）, G （ End of document ）
Row number ： :{ Row number } perhaps { Row number }G ({ Row number } Is the number of rows )
miscellaneous ： % （ Find pairing , Such as brackets or /* */ Such comments are right ）
lookup ： f{ character }, t{ character }, F{ character }, T{ character }
lookup / To forward / backward In this bank { character }
, / ; For navigation matching
Search for : /{ Regular expressions }, n / N For navigation matching

Everything you need to do with a mouse , You can use the keyboard now ： Use the combination of edit command and move command to complete . This is it. Vim When the interface of starts to look like a programming language .Vim The editing command of is also called “ Verb ”, Because verbs can act on nouns .

i Enter insertion mode
But for manipulation / Edit text , You don't just want to use the backspace key to complete
O / o On top of it / Insert line below
d{ Mobile command } Delete { Mobile command }
for example , dw Delete words , d$ Delete to end of line , d0 Delete to the beginning of the line .
c{ Mobile command } change { Mobile command }
for example , cw Change words
such as d{ Mobile command } Again i
x Delete character （ Equate to dl）
s Replace character （ Equate to xi）
Visualization mode + operation
Select the text , d Delete perhaps c change
u revoke , redo
y Copy / “yank” （ Other commands such as d It's also copied ）
p Paste
More to learn : such as ~ Change the case of characters

You can combine with a count “ Noun ” and “ Verb ”, This will perform the specified operation several times .
3w Move three words forward
5j Move down the 5 That's ok
7dw Delete 7 Word

You can use modifiers to change “ Noun ” The meaning of . The modifiers are i, Express “ Inside ” perhaps “ , “, and a, Express ” Around “.
ci( Change the contents in the current brackets
ci[ Change the contents in the current square brackets
da’ Delete a single quoted string , Include single quotation marks around

Vim There are many extensions . Contrary to many outdated recommendations on the Internet , you _ No _ Need to be in Vim Use a plug-in Manager （ from Vim 8.0 Start ）. You can use the built-in plug-in management system . Just create one ~/.vim/pack/vendor/start/ Folder , Then put the plug-in here （ Such as through git clone）. Here are some of our favorite plug-ins ：
ctrlp.vim: Fuzzy file search
ack.vim: Code search
nerdtree: File browser
vim-easymotion: Magic operation

Extended data ：
vimtutor It's a Vim The tutorial comes with installation
Vim Adventures Is a learning to use Vim The game of
Vim Tips Wiki
Vim Advent Calendar There's a lot of Vim Tips
Vim Golf Yes, it is Vim As a programming language code golf
Vi/Vim Stack Exchange
Vim Screencasts
Practical Vim（ Books ）

Data collation

journalctl | grep -i intel, It will find all that contain intel( Case insensitive ) The system log of .

Regular expressions
. Except for line breaks ” Any single character ”

• Match the previous character zero or more times

• Match the previous character one or more times
  [abc] matching a, b and c Any one of
  (RX1|RX2) Anything that can match RX1 or RX2 Result
  ^ Head of line
  $ At the end of the line

Command line environment

When you are using the command line , You usually want to perform multiple tasks at the same time . Although opening a new terminal window can also achieve the goal , Using terminal multiplexers is a better way .
image tmux This kind of terminal multiplexer allows us to partition multiple terminal windows based on panels and labels , In this way, you can work with multiple shell Interact with the session .
More Than This , Terminal multiplexing allows us to separate the current terminal session and reconnect in the future .

tmux It's a highly customizable tool , You can use the relevant shortcut keys to create multiple tabs and navigate between them .
tmux We need to master the shortcut keys of , They are all similar x This combination , That is, you need to press Ctrl+b, Release and press x

bash The alias syntax in is as follows ：
alias alias_name=“command_to_alias arg1 arg2”
Be careful , = There is no space on either side , because alias It's a shell command , It accepts only one parameter .
By default shell Alias will not be saved . To keep the alias in effect , You need to put the configuration into shell In your startup file , Like .bashrc or .zshrc

Use ssh There are many ways to copy files ：
ssh+tee, The easiest way is to execute ssh command , And then through this method, using standard input cat localfile | ssh remote_server tee serverfile. recall ,tee The command writes the standard output to a file ;
scp ： When you need to copy a large number of files or directories , Use scp Orders are more convenient , Because it can easily traverse the relevant path . The grammar is as follows ：scp path/to/local_file remote_host:path/to/remote_file;
rsync Yes scp Improved , It can detect local and remote files to prevent duplicate copies . It can also provide things like symbolic connections 、 Authority management and other carefully polished functions . It can even be based on --partial Mark to achieve breakpoint continuation .rsync Grammar and scp similar ;

version control (Git)

Basics
git help : obtain git Command help information
git init: Create a new git Warehouse , The data is stored in a file called .git Under the directory of
git status: Displays the current warehouse status
git add : Add files to staging area
git commit: Create a new submission
How to write Good submission !
Why Write good submissions
git log: Show history log
git log --all --graph --decorate: Visual history （ Directed acyclic graph ）
git diff : Display the difference from the staging area file
git diff : Show the difference between two versions of a file
git checkout : to update HEAD And the current branch
Branching and merging
git branch: Show branches
git branch : Create a branch
git checkout -b : Create a branch and switch to it
amount to git branch ; git checkout
git merge : Merge into current branch
git mergetool: Use tools to handle merge conflicts
git rebase: Base a series of patches （rebase） For the new baseline
remote operation
git remote: List the remote
git remote add : Add a remote
git push :: Send the object to the remote end and update the remote reference
git branch --set-upstream-to=/: Create associations between local and remote branches
git fetch: Get the object from the far end / Indexes
git pull: amount to git fetch; git merge
git clone: Download the repository from the far end
revoke
git commit --amend: Edit submitted content or information
git reset HEAD : Recover temporary files
git checkout – : Discard changes
git restore: git2.32 Superseded after version git reset Do many undo operations
Git Advanced operations
git config: Git It's a Highly customizable Tools
git clone --depth=1: Shallow clone （shallow clone）, Complete version history information is not included
git add -p: Interactive staging
git rebase -i: Interactive variable base
git blame: See who last modified a line
git stash: Temporarily remove the changes in the working directory
git bisect: Search history by binary search
.gitignore: Appoint Files that were deliberately not tracked

Commissioning and performance analysis

The first way to debug code is often to add some print statements where you find problems , Then repeat this process until you get enough information and find the root cause of the problem .
Another way is to use logs , Instead of temporarily adding print statements . Logs have the following advantages over ordinary print statements ：

• You can write the log to a file 、socket Or even send it to a remote server instead of just standard output ;
• Logs can support severity levels （ for example INFO, DEBUG,WARN, ERROR etc. ), This allows you to filter the logs as needed ;
• For newly discovered problems , It is likely that your log already contains enough information to help you locate the problem .

For most UNIX System , You can also use the dmesg Command to read the kernel log .
If you want to add the log to the system log , You can use logger This shell Program . Most programming languages support writing logs to the system log . If you find that you need to journalctl and log show The results are filtered a lot , At this time, you can consider using their own options to filter the results first and then output . There are others like lnav Such a tool , It provides a better way to show and browse log files .

Python The debugger is pdb.
Following pair pdb The supported commands are briefly introduced ：

l(ist) - Displays the... Near the current line 11 Line or continue with the previous display ;
s(tep) - Execute the current line , And stop at the first possible place ;
n(ext) - Continue execution until the next statement of the current function or return sentence ;
b(reak) - To set breakpoints （ Based on the passed in parameters ）;
p(rint) - Evaluate the expression in the current context and print the result . Another command is pp , It USES pprint Print ;
r(eturn) - Continue execution until the current function returns ;
q(uit) - Exit debugger .

When people talk about performance analysis tools , Usually it means CPU Performance analysis tool . CPU There are two performance analysis tools ： Trace analyzer （tracing） And sampling analyzer （sampling）. Trace analyzer It will record every function call of the program , The sampling analyzer will only monitor periodically （ Usually per millisecond ） Your program and record the program stack . They use these records to generate statistics , Shows what the program spends the most time on .

Resource monitoring
The most popular tools are htop, 了 , It is top Improved version .htop You can display various statistics of the currently running process . You can also pay attention to glances , Its implementation is similar but the user interface is better . If you need to merge and measure all processes , dstat It is also a very useful tool , It can calculate the measurement data of different subsystem resources in real time , for example I/O、 The Internet 、 CPU utilization 、 Context switching and so on ;
If you want to test these tools , You can use stress Command to artificially increase the load for the system .

Metaprogramming

Version numbers have different semantics , It's in this format ： The major version number . Sub version number . Patch number . The relevant rules are ：

If the new version doesn't change API, Please increment the patch number ;
If you add API And the change is backward compatible , Please increment the minor version number ;
If you modify API But it's not backward compatible , Please increment the major version number .

test suite ： All tests are collectively referred to as .
unit testing ： A kind of “ Micro test ”, Used to test the characteristics of a package .
Integration testing ： A kind of “ Macro test ”, For a large part of the system , Test whether its different features or components work together .
regression testing ： A test that implements a specific pattern , Used to guarantee that the problem caused before bug It won't happen again .
simulation （Mocking）: Replace the function with a fake implementation 、 Module or type , Block things that are not related to testing . for example , You might “ Analog network connection ” or “ Analog hard disk ”.

A hodgepodge

Here are some software to modify key mapping ：

macOS - karabiner-elements, skhd perhaps BetterTouchTool
Linux - xmodmap perhaps Autokey
Windows - Control panel ,AutoHotkey perhaps SharpKeys
QMK - If your keyboard supports custom firmware ,QMK You can modify the key mapping directly on the hardware of the keyboard . The mapping retained in the keyboard eliminates repeated configuration on other machines .

Some of the experience of the course can be directly applied to machine learning programs . Like many scientific disciplines , In machine learning , You need to do a series of experiments , And check which data is valid , What is not effective . You can use Shell Search for these results easily and quickly , And summarize it in a reasonable way . This means that within a limited time or when a specific data set is used , Check all experimental results . By using JSON Document all relevant parameters of the experiment , Use the tools we introduced in this course , This thing can be extremely simple .