Greenplum database fault analysis - semop (id=2000421076, num=11) failed: invalid argument

Service installation greenplum Database time ,master Nodes crash frequently , Colleagues check the log and report the following errors ：

2022-06-23 21:01:21.201568 CST,,,p52171,th-761145216,,,0,,,seg-1,,,,,"FATAL","XX000","no free slots in PMChildFlags array",,,,,,,0,,,"pmsignal.c",173,"Stack trace: 1 .... "

Seeing the error report above, I immediately felt incredible ,PMChildFlags array There are no free slots in the , Error code is located at AssignPostmasterChildSlot Function , For the detailed process, see PostgreSQL database PMsignal—— Back end process \Postmaster Signal communication . Even the line of code reported an error Out of slots ... should never happen.PMChildFlags Slot from PMSignalShmemSize The number can be seen in the function 2 * (MaxConnections + autovacuum_max_workers + 1 + max_worker_processes) It is much greater than MaxConnections, That is, the business connection is reaching PMChildFlags The number of slots should have been before the upper limit MaxConnections It's limited ; Besides, this is when the database is just started , There is no business connection . I can't think of my sister ...

int AssignPostmasterChildSlot(void) {
    
	int			slot = PMSignalState->next_child_flag;
	int			n;
	/* Scan for a free slot. We track the last slot assigned so as not to waste time repeatedly rescanning low-numbered slots. */
	for (n = PMSignalState->num_child_flags; n > 0; n--) {
    
		if (--slot < 0) slot = PMSignalState->num_child_flags - 1;
		if (PMSignalState->PMChildFlags[slot] == PM_CHILD_UNUSED) {
    
			PMSignalState->PMChildFlags[slot] = PM_CHILD_ASSIGNED;
			PMSignalState->next_child_flag = slot;
			return slot + 1;
		}
	}

	/* Out of slots ... should never happen, else postmaster.c messed up */
	elog(FATAL, "no free slots in PMChildFlags array");
	return 0;					/* keep compiler quiet */
}
Size PMSignalShmemSize(void){
    
	Size		size;
	size = offsetof(PMSignalData, PMChildFlags);
	size = add_size(size, mul_size(MaxLivePostmasterChildren(), sizeof(sig_atomic_t)));
	return size;
}
int MaxLivePostmasterChildren(void) {
    
	return 2 * (MaxConnections + autovacuum_max_workers + 1 + max_worker_processes);
}

Can't , Because it is related to shared memory , Directly grab relevant keywords from all logs , The following problems were found in the log of a reinstallation , And after the above error report . Search for Han Gao through Du Niang PG The lab has posted blogs on related issues , Here's an excerpt 《semctl(156532736, 0, IPC_RMID, …) failed: Invalid argument Database restart caused by 》.

[[email protected] pg_log] cat * | grep 'sem'
2022-06-23 21:05:36.272884 CST,,,p66817,th-1473353856,,,0,con2,,seg-1,,,,,"FATAL","XX000","semop(id=2000421076,num=11) failed: invalid argument"

The database log does not appear regularly. An error is reported as follows , It also causes the database to restart . system platform ：Linux x86-64 Red Hat Enterprise Linux 7 edition ：9.5

FATAL,XX000,semop(id=157450268) failed: Invalid argument
FATAL,XX000,semop(id=157843496) failed: Invalid argument
PANIC,XX000,queueing for lock while waiting on another one
terminating any other active server processes
WARNING,57P02,terminating connection because of crash of another server process,The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.,In a moment you should be able to reconnect to the database and repeat your command.
archiver process (PID 3766) exited with exit code 1
FATAL,57P03,the database system is in recovery mode
all server processes terminated; reinitializing
could not remove shared memory segment /PostgreSQL.44345806: No such file or directory
semctl(156532736, 0, IPC_RMID, ...) failed: Invalid argument
semctl(156565505, 0, IPC_RMID, ...) failed: Invalid argument
FATAL,57P03,the database system is in recovery mode
database system was interrupted; last known up at 2018-12-27 04:54:36 CST
database system was not properly shut down; automatic recovery in progress
redo starts at 5E7/7036BD30
FATAL,57P03,the database system is in recovery mode
invalid record length at 5E7/75359EC8
redo done at 5E7/75359EA0
last completed transaction was at log time 2018-12-27 05:06:26.652179+08
MultiXact member wraparound protections are now enabled
autovacuum launcher started
database system is ready to accept connections

The cause of this problem is the parameter RemoveIPC Set to yes.RemoveIPC Parameter in /etc/systemd/logind.conf Control whether to delete when the user logs off completely System V IPC object . The parameter is in systemd 212(2014-03-25) In the version, it is opened by default ,RHEL7 from 219 Version start . obviously ,RHEL7 This parameter in is off by default . When RemoveIPC = yes when ,PostgreSQL Semaphore objects used by the server are deleted at random time , Cause the server to crash , A similar log appears ：LOG: semctl(1234567890, 0, IPC_RMID, ...) failed: Invalid argument.attached Shared memory segments in state will not be cleaned up , therefore systemd Shared memory segments that are in use will not be cleaned up , but Semaphores have no process attached The concept of , So even if they are actually still in use, they will be cleaned up . Solution ：
（1） modify “/etc/systemd/logind.conf” In the document “RemoveIPC” The value of the field is “no”.
Use vim open logind.conf file .vim /etc/systemd/logind.conf
modify “RemoveIPC” The value of the field is “no”.RemoveIPC=no
（2） modify “/usr/lib/systemd/system/systemd-logind.service” In the document “RemoveIPC” The value of the field is “no”.
Use vim Command to open systemd-logind.service file .vim /usr/lib/systemd/system/systemd-logind.service
modify “RemoveIPC” The value of the field is “no”.RemoveIPC=no
（3） Reload configuration parameters .

systemctl daemon-reload
systemctl restart systemd-logind

（4） Check whether the modification is effective .

loginctl show-session | grep RemoveIPC
systemctl show systemd-logind | grep RemoveIPC

However, the modification still fails to solve the problem , Now I can only analyze it myself , from PostgreSQL Database semaphore mechanism — PGSemaphore Underlying principle From the article semop(id=2000421076,num=11) failed: invalid argument It's all in PGSemaphoreLock、PGSemaphoreLockInterruptable、PGSemaphoreUnlock and PGSemaphoreTryLock In case of any problem or error , And the platform is used SysV semaphore facilities, Its input parameters are variables of the following types , And PGSemaphoreData The memory used by type variables is in shared memory , That is to call SysV Library function semget Acquired semId And the number of acquired semaphores PGSemaphoreData In type variables .

#ifdef USE_SYSV_SEMAPHORES
typedef struct PGSemaphoreData
{
    
	int			semId;			/* semaphore set identifier */
	int			semNum;			/* semaphore number within set */
} PGSemaphoreData;
#endif

PGSemaphoreCreate Function initialization PGSemaphore Structure to indicate that the count is 1 The signal of . however PGSemaphoreCreate Functions are all in postmaster Created in the daemon , Ordinary background processes do not have permission to create , Therefore, there should be no problem if there is no error in the creation process .

void PGSemaphoreCreate(PGSemaphore sema) {
    	
	Assert(!IsUnderPostmaster); /* Can't do this in a backend, because static state is postmaster's */
	if (nextSemaNumber >= SEMAS_PER_SET) {
      /* Time to allocate another semaphore set */		
		if (numSemaSets >= maxSemaSets) elog(PANIC, "too many semaphores created");
		mySemaSets[numSemaSets] = IpcSemaphoreCreate(SEMAS_PER_SET);
		numSemaSets++;
		nextSemaNumber = 0;
	}	
	sema->semId = mySemaSets[numSemaSets - 1]; /* Assign the next free semaphore in the current set */
	sema->semNum = nextSemaNumber++;	
	IpcSemaphoreInitialize(sema->semId, sema->semNum, 1); /* Initialize it to count 1 */
}

So the ultimate problem is that our shared memory is handled by the operating system , Parameters RemoveIPC Has been ruled out , Now there are two possibilities ：1. Accidentally injured by the business script , For example, use ipcrm, I have been cleaned up by business before session The script of accidentally injures all segement Records of database processes （ You can execute through an infinite loop ipcs -a See the whole process master The time point of database semaphore loss and the time point of database crash ） 2. company linux The distribution system has special settings for specific system users （ for instance master The database process is in session-72887.scope in , stay user-1000.slice Next , and segment The database process is in user-71381.slice Next ,1000 yes ubuntu Of UID, yes 71381 yes gpadmin Of UID, For example, specific projects like to target ubuntu The account is set up specifically .

ipcs -a
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x2369b4bc 3309575    gpadmin     640        7535067136 18
------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0xd5b0c6f8 1179650    gpadmin     640        154
------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

It's quiet at night , The business is also unwilling to continue to investigate the environment for us , So I gave up this node and deployed it to other nodes . Check it out here , I feel powerless in front of the vast and complex operating system .