php fpm keeps randomly failing to reload

We get the error:

Mar 28 11:54:23 weborigin002 systemd[1]: Reloading php8.1-fpm.service - The PHP 8.1 FastCGI Process Manager...
Mar 28 11:54:23 weborigin002 systemd[1]: Reloaded php8.1-fpm.service - The PHP 8.1 FastCGI Process Manager.
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=8, "/var/run/php/8.1-site350769.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=9, "/var/run/php/8.1-site350898.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=10, "/var/run/php/8.1-site350992.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=11, "/var/run/php/8.1-site351021.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=12, "/var/run/php/8.1-site351150.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=13, "/var/run/php/8.1-site351268.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=14, "/var/run/php/8.1-site351283.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=15, "/var/run/php/8.1-site351334.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=16, "/var/run/php/8.1-site351401.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=17, "/var/run/php/8.1-site351635.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=18, "/var/run/php/8.1-site351878.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=19, "/var/run/php/8.1-site351947.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=20, "/var/run/php/8.1-site351951.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=21, "/var/run/php/8.1-site352049.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=22, "/var/run/php/8.1-site352164.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=23, "/var/run/php/8.1-site352220.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=24, "/var/run/php/8.1-site352283.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=25, "/var/run/php/8.1-site352302.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=26, "/var/run/php/8.1-site352357.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=27, "/var/run/php/8.1-site352358.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=28, "/var/run/php/8.1-site352426.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=29, "/var/run/php/8.1-site353448.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=30, "/var/run/php/8.1-site353574.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=31, "/var/run/php/8.1-site353710.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=32, "/var/run/php/8.1-site353877.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=33, "/var/run/php/8.1-site353903.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=34, "/var/run/php/8.1-site354034.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=35, "/var/run/php/8.1-site354267.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=36, "/var/run/php/8.1-site354283.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=37, "/var/run/php/8.1-site354303.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=38, "/var/run/php/8.1-site354421.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=39, "/var/run/php/8.1-site354438.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=40, "/var/run/php/8.1-site354500.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=41, "/var/run/php/8.1-site354605.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=42, "/var/run/php/8.1-site354620.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=43, "/var/run/php/8.1-site354722.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=44, "/var/run/php/8.1-site354904.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=45, "/var/run/php/8.1-site354969.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=46, "/var/run/php/8.1-site354984.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=47, "/var/run/php/8.1-site355056.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=48, "/var/run/php/8.1-site355065.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=49, "/var/run/php/8.1-site355125.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=50, "/var/run/php/8.1-site355508.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=51, "/var/run/php/8.1-site355511.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=52, "/var/run/php/8.1-site355526.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=53, "/run/php/8.1-site355633.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=54, "/var/run/php/8.1-site355830.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=55, "/var/run/php/8.1-site355831.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=56, "/var/run/php/8.1-site355837.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=57, "/run/php/8.1-site355958.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=58, "/run/php/8.1-site355979.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=59, "/run/php/php8.1-fpm.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=60, "/run/php/8.1-site356090.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] NOTICE: using inherited socket fd=61, "/run/php/8.1-site356162.sock"
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] ERROR: Another FPM instance seems to already listen on /run/php/8.1-site354034.sock
Mar 28 11:54:23 weborigin002 php-fpm8.1[1303816]: [28-Mar-2024 11:54:23] ERROR: FPM initialization failed
Mar 28 11:54:23 weborigin002 systemd[1]: php8.1-fpm.service: Main process exited, code=exited, status=78/CONFIG
Mar 28 11:54:24 weborigin002 systemd[1]: php8.1-fpm.service: Failed with result 'exit-code'.
Mar 28 11:54:24 weborigin002 systemd[1]: php8.1-fpm.service: Consumed 1d 3h 56min 17.669s CPU time.

This started happening after switching the way the control panel handles updates from the old "red-node-update" approach via php to discrete bash scripts via sudo.

I first noticed that in the PHP scripts I always slept for 2 seconds after reloading. I tried that. We still got these problems.

So, next I noticed that the red-node-update method always reloaded apache2 first, before reloading php. I think that's the real problem. So... If a request is in process, and apache2 is handling that request, then apache2 is keeping the socket opened. Then, php reloads, killing off all the php processes and then re-establishing the socket. But, when it gets to the socket apache2 has open, it fails.