WGet: command line download tool

One 、 grammar

[email protected]:~$ wget -help
GNU Wget 1.20.3, a non-interactive network retriever.
Usage: wget [OPTION]... [URL]...

Mandatory arguments to long options are mandatory for short options too.

Startup:
  -V,  --version                   display the version of Wget and exit
  -h,  --help                      print this help
  -b,  --background                go to background after startup
  -e,  --execute=COMMAND           execute a `.wgetrc'-style command Logging and input file: -o, --output-file=FILE log messages to FILE -a, --append-output=FILE append messages to FILE -d, --debug print lots of debugging information -q, --quiet quiet (no output) -v, --verbose be verbose (this is the default) -nv, --no-verbose turn off verboseness, without being quiet --report-speed=TYPE output bandwidth as TYPE. TYPE can be bits -i, --input-file=FILE download URLs found in local or external FILE -F, --force-html treat input file as HTML -B, --base=URL resolves HTML input-file links (-i -F) relative to URL --config=FILE specify config file to use --no-config do not read any config file --rejected-log=FILE log reasons for URL rejection to FILE Download: -t, --tries=NUMBER set number of retries to NUMBER (0 unlimits) --retry-connrefused retry even if connection is refused --retry-on-http-error=ERRORS comma-separated list of HTTP errors to retry -O, --output-document=FILE write documents to FILE -nc, --no-clobber skip downloads that would download to existing files (overwriting them) --no-netrc don't try to obtain credentials from .netrc
  -c,  --continue                  resume getting a partially-downloaded file
       --start-pos=OFFSET          start downloading from zero-based position OFFSET
       --progress=TYPE             select progress gauge type
       --show-progress             display the progress bar in any verbosity mode
  -N,  --timestamping              don't re-retrieve files unless newer than local --no-if-modified-since don't use conditional if-modified-since get
                                     requests in timestamping mode
       --no-use-server-timestamps  don't set the local file's timestamp by
                                     the one on the server
  -S,  --server-response           print server response
       --spider                    don't download anything -T, --timeout=SECONDS set all timeout values to SECONDS --dns-timeout=SECS set the DNS lookup timeout to SECS --connect-timeout=SECS set the connect timeout to SECS --read-timeout=SECS set the read timeout to SECS -w, --wait=SECONDS wait SECONDS between retrievals --waitretry=SECONDS wait 1..SECONDS between retries of a retrieval --random-wait wait from 0.5*WAIT...1.5*WAIT secs between retrievals --no-proxy explicitly turn off proxy -Q, --quota=NUMBER set retrieval quota to NUMBER --bind-address=ADDRESS bind to ADDRESS (hostname or IP) on local host --limit-rate=RATE limit download rate to RATE --no-dns-cache disable caching DNS lookups --restrict-file-names=OS restrict chars in file names to ones OS allows --ignore-case ignore case when matching files/directories -4, --inet4-only connect only to IPv4 addresses -6, --inet6-only connect only to IPv6 addresses --prefer-family=FAMILY connect first to addresses of specified family, one of IPv6, IPv4, or none --user=USER set both ftp and http user to USER --password=PASS set both ftp and http password to PASS --ask-password prompt for passwords --use-askpass=COMMAND specify credential handler for requesting username and password. If no COMMAND is specified the WGET_ASKPASS or the SSH_ASKPASS environment variable is used. --no-iri turn off IRI support --local-encoding=ENC use ENC as the local encoding for IRIs --remote-encoding=ENC use ENC as the default remote encoding --unlink remove file before clobber --xattr turn on storage of metadata in extended file attributes Directories: -nd, --no-directories don't create directories
  -x,  --force-directories         force creation of directories
  -nH, --no-host-directories       don't create host directories --protocol-directories use protocol name in directories -P, --directory-prefix=PREFIX save files to PREFIX/.. --cut-dirs=NUMBER ignore NUMBER remote directory components HTTP options: --http-user=USER set http user to USER --http-password=PASS set http password to PASS --no-cache disallow server-cached data --default-page=NAME change the default page name (normally this is 'index.html'.) -E, --adjust-extension save HTML/CSS documents with proper extensions --ignore-length ignore 'Content-Length' header field --header=STRING insert STRING among the headers --compression=TYPE choose compression, one of auto, gzip and none. (default: none) --max-redirect maximum redirections allowed per page --proxy-user=USER set USER as proxy username --proxy-password=PASS set PASS as proxy password --referer=URL include 'Referer: URL' header in HTTP request --save-headers save the HTTP headers to file -U, --user-agent=AGENT identify as AGENT instead of Wget/VERSION --no-http-keep-alive disable HTTP keep-alive (persistent connections) --no-cookies don't use cookies
       --load-cookies=FILE         load cookies from FILE before session
       --save-cookies=FILE         save cookies to FILE after session
       --keep-session-cookies      load and save session (non-permanent) cookies
       --post-data=STRING          use the POST method; send STRING as the data
       --post-file=FILE            use the POST method; send contents of FILE
       --method=HTTPMethod         use method "HTTPMethod" in the request
       --body-data=STRING          send STRING as data. --method MUST be set
       --body-file=FILE            send contents of FILE. --method MUST be set
       --content-disposition       honor the Content-Disposition header when
                                     choosing local file names (EXPERIMENTAL)
       --content-on-error          output the received content on server errors
       --auth-no-challenge         send Basic HTTP authentication information
                                     without first waiting for the server's challenge HTTPS (SSL/TLS) options: --secure-protocol=PR choose secure protocol, one of auto, SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2 and PFS --https-only only follow secure HTTPS links --no-check-certificate don't validate the server's certificate --certificate=FILE client certificate file --certificate-type=TYPE client certificate type, PEM or DER --private-key=FILE private key file --private-key-type=TYPE private key type, PEM or DER --ca-certificate=FILE file with the bundle of CAs --ca-directory=DIR directory where hash list of CAs is stored --crl-file=FILE file with bundle of CRLs --pinnedpubkey=FILE/HASHES Public key (PEM/DER) file, or any number of base64 encoded sha256 hashes preceded by 'sha256//' and separated by ';', to verify peer against --random-file=FILE file with random data for seeding the SSL PRNG --ciphers=STR Set the priority string (GnuTLS) or cipher list string (OpenSSL) directly. Use with care. This option overrides --secure-protocol. The format and syntax of this string depend on the specific SSL/TLS engine. HSTS options: --no-hsts disable HSTS --hsts-file path of HSTS database (will override default) FTP options: --ftp-user=USER set ftp user to USER --ftp-password=PASS set ftp password to PASS --no-remove-listing don't remove '.listing' files
       --no-glob                   turn off FTP file name globbing
       --no-passive-ftp            disable the "passive" transfer mode
       --preserve-permissions      preserve remote file permissions
       --retr-symlinks             when recursing, get linked-to files (not dir)

FTPS options:
       --ftps-implicit                 use implicit FTPS (default port is 990)
       --ftps-resume-ssl               resume the SSL/TLS session started in the control connection when
                                         opening a data connection
       --ftps-clear-data-connection    cipher the control channel only; all the data will be in plaintext
       --ftps-fallback-to-ftp          fall back to FTP if FTPS is not supported in the target server
WARC options:
       --warc-file=FILENAME        save request/response data to a .warc.gz file
       --warc-header=STRING        insert STRING into the warcinfo record
       --warc-max-size=NUMBER      set maximum size of WARC files to NUMBER
       --warc-cdx                  write CDX index files
       --warc-dedup=FILENAME       do not store records listed in this CDX file
       --no-warc-compression       do not compress WARC files with GZIP
       --no-warc-digests           do not calculate SHA1 digests
       --no-warc-keep-log          do not store the log file in a WARC record
       --warc-tempdir=DIRECTORY    location for temporary files created by the
                                     WARC writer

Recursive download:
  -r,  --recursive                 specify recursive download
  -l,  --level=NUMBER              maximum recursion depth (inf or 0 for infinite)
       --delete-after              delete files locally after downloading them
  -k,  --convert-links             make links in downloaded HTML or CSS point to
                                     local files
       --convert-file-only         convert the file part of the URLs only (usually known as the basename)
       --backups=N                 before writing file X, rotate up to N backup files
  -K,  --backup-converted          before converting file X, back up as X.orig
  -m,  --mirror                    shortcut for -N -r -l inf --no-remove-listing
  -p,  --page-requisites           get all images, etc. needed to display HTML page
       --strict-comments           turn on strict (SGML) handling of HTML comments

Recursive accept/reject:
  -A,  --accept=LIST               comma-separated list of accepted extensions
  -R,  --reject=LIST               comma-separated list of rejected extensions
       --accept-regex=REGEX        regex matching accepted URLs
       --reject-regex=REGEX        regex matching rejected URLs
       --regex-type=TYPE           regex type (posix|pcre)
  -D,  --domains=LIST              comma-separated list of accepted domains
       --exclude-domains=LIST      comma-separated list of rejected domains
       --follow-ftp                follow FTP links from HTML documents
       --follow-tags=LIST          comma-separated list of followed HTML tags
       --ignore-tags=LIST          comma-separated list of ignored HTML tags
  -H,  --span-hosts                go to foreign hosts when recursive
  -L,  --relative                  follow relative links only
  -I,  --include-directories=LIST  list of allowed directories
       --trust-server-names        use the name specified by the redirection
                                     URL's last component -X, --exclude-directories=LIST list of excluded directories -np, --no-parent don't ascend to the parent directory

Email bug reports, questions, discussions to <[email protected]>
and/or open issues at https://savannah.gnu.org/bugs/?func=additem&group=wget.
[email protected]:~$

Two 、 Parameter description

1、Startup

Parameters	explain
-V, --version	Show Wget And the exited version
-h, --help	Print this help
-b, --background	After starting, go to the background
-e, --execute=COMMAND	perform “.wgetrc” Style commands

2、Logging and input file

Parameters	explain
-o, --output-file=FILE	Log messages to a file
-a, --append-output=FILE	Append message to file
-d, --debug	Print a lot of debugging information
-q, --quiet	quiet （ There is no output ）
-v, --verbose	detailed （ This is the default ）
-nv, --no-verbose	Close verbose , Not quiet
–report-speed=TYPE	The output bandwidth is of type . The type can be bit
-i, --input-file=FILE	Download found in local or external files URL
-F, --force-html	Treat the input file as HTML
-B, --base=URL	Analytic relative to URL Of HTML Input file link （-i -F）
–config=FILE	Specifies the configuration file to use
–no-config	No configuration files are read
–rejected-log=FILE	take URL The reason for rejection is recorded in FILE

3、Download

Parameters	explain
-t, --tries=NUMBER	Set the number of retries to NUMBER（0 Unlimited ）
–retry-connrefused	Even if the connection is rejected , Try again
–retry-on-http-error=ERRORS	Separated by commas HTTP Error list , To try again
-O, --output-document=FILE	Write the document to a file
-nc, --no-clobber	Skip downloading to existing files ( Cover them )
–no-netrc	Don't try to .netrc Obtain the credentials
-c, --continue	Continue to get some downloaded files
–start-pos=OFFSET	Download location offset from zero
–progress=TYPE	Select the type of progress meter
–show-progress	Show progress bar in any detail mode
-N, --timestamping	Do not retrieve the file again , Unless it is newer than the local file
–no-if-modified-since	Do not use conditions in timestamp mode if-modified-since Get request
–no-use-server-timestamps	Do not use the timestamp on the server to set the timestamp of the local file
-S, --server-response	Print server response
–spider	Don't download anything
-T, --timeout=SECONDS	Set all timeout values to seconds
–dns-timeout=SECS	take DNS The lookup timeout is set to SECS
–connect-timeout=SECS	Set the connection timeout to SECS
–read-timeout=SECS	Set the read timeout to SECS
-w, --wait=SECONDS	The number of seconds between waiting for retrieval
–waitretry=SECONDS	wait for 1… Wait between retries of retrieval 1… second
–random-wait	Wait for 0.5 * wait for …1.5 * Wait seconds between retrievals
–no-proxy	Explicitly close the agent
-Q, --quota=NUMBER	Set the retrieval quota to NUMBER
–bind-address=ADDRESS	Bind to the address on the local host （ Host name or IP）
–limit-rate=RATE	Limit download rate to rate
–no-dns-cache	Disable caching DNS lookup
–restrict-file-names=OS	Limit the characters in the file name to those allowed by the operating system
–ignore-case	Match file / Ignore case when Directory
-4, --inet4-only	Only connect to IPv4 Address
-6, --inet6-only	Only connect to IPv6 Address
–prefer-family=FAMILY	First connect to the address of the specified series 、IPv6、IPv4 Or none
–user=USER	take ftp and http Users are set to USER
–password=PASS	take ftp and http The password is set to PASS
–ask-password	Prompt for password
–use-askpass=COMMAND	Specify the credential handler for requesting user names and passwords . If not specified COMMAND, Then use WGET_ASKPASS or SSH_ASKPASS environment variable .
–no-iri	close IRI Support
–local-encoding=ENC	Use ENC As IRI Local code of
–remote-encoding=ENC	Use ENC As the default remote encoding
–unlink	stay clobber Delete the file before
–xattr	Open metadata storage in the extended file attribute

4、Directories

Parameters	explain
-nd, --no-directories	Do not create directory
-x, --force-directories	Forced directory creation
-nH, --no-host-directories	Do not create Host Directory
–protocol-directories	Use the protocol name in the directory
-P, --directory-prefix=PREFIX	Save the file to prefix /…
–cut-dirs=NUMBER	Ignore NUMBER Remote directory component

5、HTTP options

Parameters	explain
–http-user=USER	take http The user is set to USER
–http-password=PASS	take http Password set to PASS
–no-cache	Prohibit the server from caching data
–default-page=NAME	Change the default page name ( Usually ’index.html’).
-E, --adjust-extension	Save with the appropriate extension HTML/CSS file
–ignore-length	Ignore ’Content-Length’ Header field
–header=STRING	Insert string header
–compression=TYPE	Choose compression ,auto, gzip and none.( The default value is : nothing )
–max-redirect	The maximum number of redirects allowed per page
–proxy-user=USER	take USER Set to proxy user name
–proxy-password=PASS	take PASS Set as proxy password
–referer=URL	stay HTTP The request contains ’Referer: URL’ head
–save-headers	take HTTP Save header to file
-U, --user-agent=AGENT	Identify as agent instead of Wget/VERSION
–no-http-keep-alive	Ban HTTP Keep active （ Persistent connection ）
–no-cookies	Do not use cookies
–load-cookies=FILE	Load from file before session Cookie
–save-cookies=FILE	After the conversation Cookie Save to file
–keep-session-cookies	Load and save session （ Non permanent ）Cookie
–post-data=STRING	Use POST Method ; send out STRING As data
–post-file=FILE	Use POST Method ; Send file content
–method=HTTPMethod	Use... In requests HTTPMethod
–body-data=STRING	Send string as data .–method You have to set
–body-file=FILE	Send the contents of the file .–method You have to set
–content-disposition	Follow the content disposal header when selecting a local file name （ experimental ）
–content-on-error	Output the received content in case of server error
–auth-no-challenge	Send basic HTTP Authentication information , Without waiting for the query of the server

6、HTTPS (SSL/TLS) options

Parameters	explain
–secure-protocol=PR	Choose Security Protocol , Include auto、SSLv2、SSLv3、TLSv1、TLSv1 1、TLSv1 2 and PFS
–https-only	Only follow safe HTTPS link
–no-check-certificate	Do not verify the server's certificate
–certificate=FILE	Client certificate file
–certificate-type=TYPE	Client certificate type ：PEM or DER
–private-key=FILE	Private key file
–private-key-type=TYPE	Private key type ：PEM or DER
–ca-certificate=FILE	contain CA Bundle files
–ca-directory=DIR	Storage CA Directory of Hash list
–crl-file=FILE	contain CRL Bundle files
–pinnedpubkey=FILE/HASHES	Public key （PEM/DER） file , Or any number base64 Coded sha256 Hash , The front is "sha256//", Divided into ";", To verify equivalence
–random-file=FILE	Contains for setting SSL PRNG Random data files of seeds
–ciphers=STR	You can set the priority string directly (GnuTLS) Or password list string (OpenSSL). Please use with caution . This option will override --secure-protocol. The format and syntax of this string depend on the specific SSL/TLS engine .

7、HSTS options

Parameters	explain
–no-hsts	Ban HSTS
–hsts-file	HSTS Database path （ The default value will be overridden ）

8、FTP options

Parameters	explain
–ftp-user=USER	take ftp The user is set to USER
–ftp-password=PASS	take ftp Password set to PASS
–no-remove-listing	Don't delete ".listing" file
–no-glob	close FTP Put the file name
–no-passive-ftp	Ban " passive " transmission mode
–preserve-permissions	Reserve remote file permissions
–retr-symlinks	Recursive time , Get linked file （ No dir）

9、FTPS options

Parameters	explain
–ftps-implicit	Use implicit FTPS（ The default port is 990）
–ftps-resume-ssl	When opening the data connection, restore the SSL/TLS conversation
–ftps-clear-data-connection	Only encrypt the control channel ; All data will be displayed in clear text
–ftps-fallback-to-ftp	If the target server does not support FTPS, Back to FTP

10、WARC options

Parameters	explain
–warc-file=FILENAME	The request / The response data is saved to .warc.gz file
–warc-header=STRING	Insert the string into warcinfo On record
–warc-max-size=NUMBER	take WARC The maximum size of the file is set to NUMBER
–warc-cdx	write in CDX Index file
–warc-dedup=FILENAME	Do not store this CDX Records listed in the document
–no-warc-compression	Do not use GZIP Compress WARC file
–no-warc-digests	Don't count SHA1 Abstract
–no-warc-keep-log	Do not store log files in WARC On record
–warc-tempdir=DIRECTORY	WARC Location of temporary files created by the writer

11、Recursive download

Parameters	explain
-r, --recursive	Specifies a recursive download
-l, --level=NUMBER	Maximum recursion depth （inf or 0 It means infinite ）
–delete-after	Delete the file locally after downloading
-k, --convert-links	Make downloaded HTML or CSS Links to local files
–convert-file-only	Only convert URL Part of the document （ Usually called base name ）
–backups=N	Writing files X Before , Rotate at most N Backup files
-K, --backup-converted	Converting files X Before , Backup as X.orig
-m, --mirror	-N -r -l inf --no-remove-listing Shortcut to
-p, --page-requisites	Get display HTML All the images required by the page
–strict-comments	Enable HTML Strict annotation （SGML） Handle

12、Recursive accept/reject

Parameters	explain
-A, --accept=LIST	Comma separated list of accepted extensions
-R, --reject=LIST	Comma separated list of rejected extensions
–accept-regex=REGEX	Regular expressions match accepted URL
–reject-regex=REGEX	Regular expressions match rejected URL
–regex-type=TYPE	Regular expression type （posix|pcre）
-D, --domains=LIST	Accept a comma separated list of domains
–exclude-domains=LIST	Comma separated list of rejected domains
–follow-ftp	from HTML Access in the document FTP link
–follow-tags=LIST	Comma separated follow HTML Tag list
–ignore-tags=LIST	Comma separated are ignored HTML List of tags
-H, --span-hosts	When recursing, go to the external host
-L, --relative	Only pay attention to relative links
-I, --include-directories=LIST	List of allowed directories
–trust-server-names	Using redirection URL The name specified by the last component of
-X, --exclude-directories=LIST	Exclude directory list
-np, --no-parent	Don't go up to the parent directory

3、 ... and 、man wget

To be completed