How much memory does a TCP connection occupy?

Catalog

One 、 Memory allocation relationship

1.1 Physical memory allocation relationship

1.2TCP The relationship between the applicants

Two 、 Measured memory overhead

2.1 Client kernel parameter adjustment

2.2 Server side kernel parameter adjustment

3、 ... and 、 Service side cost calculation

3.1 slab Memory output and calculation before and after connection

3.2 TCP Object takes up memory

3.3 TCP Object memory overhead summary

Four 、 Parameter setting

4.1 linux Open file restriction configuration

4.2 TCP Memory settings

4.3 Check the connection status

4.4 other

---

         Calculate the memory occupied by the network connection , You need to know which memory to apply for , And one TCP How much memory does the connection take up .sock_inode_cache、tcp_sock、dentry、file How much space does the memory object occupy ？

One 、 Memory allocation relationship

1.1 Physical memory allocation relationship

1.2TCP The relationship between the applicants

Two 、 Measured memory overhead

2.1 Client kernel parameter adjustment

# vi /etc/sysctl.conf
# Port number 
net.ipv4.ip_local_port_range = 5000     65000
net.ipv4.tcp_tw_reuse = 0
#net.ipv4.tcp_tw_recycle        = 0
#time_wait state 
net.ipv4.tcp_max_tw_buckets = 60000
fs.file-max=210000
fs.nr_open=210000


# sysctl -p

#vim /etc/security/limits.conf

*       soft    nofile  1010485
*       hard    nofile  1011000

2.2 Server side kernel parameter adjustment

# vi /etc/sysctl.conf
net.core.rmem_max = 8388608
net.ipv4.tcp_rmem = 4096        131072  8388608

net.core.wmem_max = 8388608
net.ipv4.tcp_wmem = 4096        16384   8388608


fs.file-max=2100000
fs.nr_open= 2100000
net.core.somaxconn = 1024

# sysctl -p

#vim /etc/security/limits.conf

*       soft    nofile  1010485
*       hard    nofile  1011000

Server and client cache cleaning

echo "3" > /proc/sys/vm/drop_caches

3、 ... and 、 Service side cost calculation

3.1 slab Memory output and calculation before and after connection

[email protected]:~# cat /proc/meminfo | grep Slab
Slab:             147724 kB
[email protected]:~# 
[email protected]:~# cat /proc/meminfo | grep Slab
Slab:             180136 kB
[email protected]:~# 
[email protected]:~# cat /proc/meminfo | grep Slab
Slab:             195816 kB

180136 - 147724 = 32412 / 10000 = 3.2412K  ESTABLISH

195816 - 180136 = 15680 / 10000 = 1.568K    CLOSE_WAIT

3.2 TCP Object takes up memory

slabtop

 Active / Total Objects (% used)    : 877387 / 896605 (97.9%)
 Active / Total Slabs (% used)      : 31483 / 31483 (100.0%)
 Active / Total Caches (% used)     : 119 / 167 (71.3%)
 Active / Total Size (% used)       : 298213.95K / 308366.00K (96.7%)
 Minimum / Average / Maximum Object : 0.01K / 0.34K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
155210 155210 100%    0.02K    913      170      3652K lsm_file_cache
 99712  99712 100%    0.03K    779      128      3116K kmalloc-32
 90048  90048 100%    0.19K   4288       21     17152K dentry
 64480  64480 100%    0.12K   2015       32      8060K kernfs_node_cache
 60352  60352 100%    0.25K   3772       16     15088K filp
 56069  56069 100%    0.81K   2951       19     47216K sock_inode_cache
 50050  50050 100%    2.19K   3575       14    114400K TCP
 46949  45281  96%    0.20K   2471       19      9884K vm_area_struct
 34764  33034  95%    0.62K   2897       12     23176K inode_cache

 2.19K（TCP）+ 0.25K（flip）+0.19K（dentry）+0.81K（sock_inode_cache） = 3.44K

 TCP Object and the slab Memory comparison A little worse

 CLOSE_WAIT The state overhead is not small 1.56 k

3.3 TCP Object memory overhead summary

cache	size	object
sock_inode_cache	0.81K	struct socket_alloc
TCP	2.19K	struct tcp_sock
dentry	0.19K	struct dentry
file	0.25K	struct file

---

Four 、 Parameter setting

4.1 linux Open file restriction configuration

1） Process level , The number of open processes in a single process soft nofile And fs.nr_open

2） System level ,fs.file-max, This parameter does not limit root user

• If you set soft nofile that hard nofile Also synchronize settings
• If you set hard nofile ,fs.nr_open Also need to adjust together
• If hard nofile Set up than fs.nr_open Big , This will cause the user to be unable to log in

Set reference

# vi /etc/sysctl.conf
fs.file-max=1100000
fs.nr_open= 1100000  # Set up than hard Big 

#sysctl -p

# vi /etc/security/limits.conf
*       soft    nofile  1000000
*       hard    nofile  1000000

4.2 TCP Memory settings

Read write settings , Every TCP Establishing a connection is about 3.44K, Multiply by the number when setting the total memory

#sysctl -a 

net.core.rmem_max = 8388608
net.ipv4.tcp_rmem = 4096        131072  8388608

net.core.wmem_max = 8388608
net.ipv4.tcp_wmem = 4096        16384   8388608

4.3 Check the connection status

# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      689/systemd-resolve 
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      837/cupsd           
tcp6       0      0 ::1:631                 :::*                    LISTEN      837/cupsd 

4.4 other

• netstat -antp | grep EST |wc -l    View the individual status and count the quantity
• slabtop see slab Application of object memory , And usage

  Reference resources
https://course.0voice.com/v1/course/intro?courseId=2&agentId=0

---