Common network problems and troubleshooting methods of gbase

Network card spin down

The performance of some nodes in the cluster is poor , see nmon When you see that the sending and receiving speed is significantly lower than that of other nodes . Check it again ethtools network card , See that the network speed is not the normal Gigabit or 10 Gigabit . For example, 10 Gigabit network , Show gigabytes , Even 100 trillion . In this case, the network cable or network card is generally unstable .

Solution

You can try to restart the network card or replace other normal network ports and repair or replace the network card 、 Check the network cable .

SSH no

The performance of some nodes in the cluster is poor , see nmon When you see that the sending and receiving speed is significantly lower than that of other nodes . Check it again ethtools network card , See that the network speed is not the normal Gigabit or 10 Gigabit . For example, 10 Gigabit network , Show gigabytes , Even 100 trillion . In this case, the network cable or network card is generally unstable .

Cluster installation 、 upgrade 、 Capacity expansion 、 Node replacement and other operations , There's the Internet ssh Impassability and other problems . Such as tcp When the number of connections is not enough ,/var/log/message There is an error in the log ：TCP: time wait bucket table overflow.

This is generally the case as follows ：

• The operating system configuration prohibits root Wait for the user's remote ssh operation ;
• ssh The configuration port is not unified or the configuration file is not completely modified ;
• ssh Number of connections full .

Solution

You can adjust the operating system ssh User access policy ,ssh port 、 Number of connections, etc . Such as modification  tcp_max_tw_buckets Parameters , Configure to 60000, Guarantee tcp Enough connections .

ipv6 agreement chercksum abnormal

Use ipv6 Install cluster , After running for a period of time, the cluster is unavailable . see /var/log/messages journal , Yes UDP checksum word , Such as ：kernel:IPV6:udp checksum is 0 for ....

This is because of the use of Ipv6 Protocol deployment cluster , send out UDP Packet must be calculated when the packet checksum,tx-checksuming When the parameter is on , Calculated by the network card ;tx-checksuming When the parameter is off , Calculated by the kernel .

When the packet is checksum The value is 0x0000 when , Need to be converted to 0xffff Send again . When the receiving end receives checksum The value is 0x0000 Of UDP Packet time , The kernel discards the package .

Different nodes corosync Get token To send packets , When token Bag checksum The value is 0x0000 when , Will be discarded by the target kernel , Target end corosync Can't receive token package , The set default time has been reached 60 Seconds later ,corosync Consider that there are nodes leaving , To resynchronize , During synchronization , The cluster does not provide external services , This will cause errors in the user's table brushing operation .

The problem only exists in individual cases VMware There have been... On the virtual machine , Episodic .

Solution

have access to ethtool -K < network card name call > tx off , Such as ethtool -K em1 tx off Turn off close network card tx-checksumming Calibration parameters are solved .