Unifi Controller Update error on Pi 4

[T4rd]

I just did a routine update on my RPi 4 to update the OS and my services and my Unifi controller service had an update available. So I ran with the "sudo apt upgrade" and it seemed to install but failed to start the service again and said "See "systemctl status unifi.service" and "journalctl -xe" for details."

The journalctl -xe outputs:

-- The job identifier is 2505 and the job result is failed.
Jun 16 17:28:43 PiePi systemd[1]: unifi.service: Service RestartSec=100ms expired, scheduling restart.
Jun 16 17:28:43 PiePi systemd[1]: unifi.service: Scheduled restart job, restart counter is at 9.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Automatic restarting of the unit unifi.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Jun 16 17:28:43 PiePi systemd[1]: Stopped unifi.
-- Subject: A stop job for unit unifi.service has finished
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A stop job for unit unifi.service has finished.
--
-- The job identifier is 2585 and the job result is done.
Jun 16 17:28:43 PiePi systemd[1]: unifi.service: Found left-over process 1355 (mongod) in control group while starting unit. Ignoring.
Jun 16 17:28:43 PiePi systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 16 17:28:43 PiePi systemd[1]: Starting unifi...
-- Subject: A start job for unit unifi.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit unifi.service has begun execution.
--
-- The job identifier is 2585.
Jun 16 17:28:43 PiePi unifi[21526]: init complete...
Jun 16 17:28:43 PiePi unifi-network-service-helper[21489]: <13>Jun 16 17:28:43 unifi: init complete...
Jun 16 17:28:43 PiePi unifi[21535]: Skipping init-uos...
Jun 16 17:28:43 PiePi unifi-network-service-helper[21527]: <13>Jun 16 17:28:43 unifi: Skipping init-uos...
Jun 16 17:28:43 PiePi systemd[1]: unifi.service: Found left-over process 1355 (mongod) in control group while starting unit. Ignoring.
Jun 16 17:28:43 PiePi systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 16 17:28:43 PiePi unifi[21547]: health-check max retry count: 100
Jun 16 17:28:43 PiePi unifi-network-service-helper[21537]: <13>Jun 16 17:28:43 unifi: health-check max retry count: 100
Jun 16 17:28:52 PiePi java[21536]: WARNING: An illegal reflective access operation has occurred
Jun 16 17:28:52 PiePi java[21536]: WARNING: Illegal reflective access by org.apache.catalina.loader.WebappClassLoaderBase (file:/usr/lib/unifi/lib/tomc
Jun 16 17:28:52 PiePi java[21536]: WARNING: Please consider reporting this to the maintainers of org.apache.catalina.loader.WebappClassLoaderBase
Jun 16 17:28:52 PiePi java[21536]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Jun 16 17:28:52 PiePi java[21536]: WARNING: All illegal access operations will be denied in a future release
lines 3260-3300/3300 (END)

The "systemctl status unifi.service" outputs:

● unifi.service - unifi
   Loaded: loaded (/lib/systemd/system/unifi.service; enabled; vendor preset: enabled)
   Active: activating (start-post) since Fri 2023-06-16 17:33:51 CEST; 57s ago
  Process: 23644 ExecStartPre=/usr/sbin/unifi-network-service-helper init (code=exited, status=0/SUCCESS)
  Process: 23682 ExecStartPre=/usr/sbin/unifi-network-service-helper init-uos (code=exited, status=0/SUCCESS)
 Main PID: 23692 (java); Control PID: 23693 (unifi-network-s)
    Tasks: 49 (limit: 4915)
   CGroup: /system.slice/unifi.service
           ├─ 1355 bin/mongod --dbpath /usr/lib/unifi/data/db --port 27117 --unixSocketPrefix /usr/lib/unifi/run --logappend --logpath /usr/lib/unifi/l
           ├─23692 /usr/bin/java -Dfile.encoding=UTF-8 -Djava.awt.headless=true -Dapple.awt.UIElement=true -Dunifi.core.enabled=false -Dunifi.mongodb.s
           ├─23693 /bin/bash /usr/sbin/unifi-network-service-helper healthcheck
           └─24019 sleep 3

Jun 16 17:33:51 PiePi unifi-network-service-helper[23682]: <13>Jun 16 17:33:51 unifi: Skipping init-uos...
Jun 16 17:33:51 PiePi systemd[1]: unifi.service: Found left-over process 1355 (mongod) in control group while starting unit. Ignoring.
Jun 16 17:33:51 PiePi systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 16 17:33:51 PiePi unifi[23703]: health-check max retry count: 100
Jun 16 17:33:51 PiePi unifi-network-service-helper[23693]: <13>Jun 16 17:33:51 unifi: health-check max retry count: 100
Jun 16 17:34:00 PiePi java[23692]: WARNING: An illegal reflective access operation has occurred
Jun 16 17:34:00 PiePi java[23692]: WARNING: Illegal reflective access by org.apache.catalina.loader.WebappClassLoaderBase (file:/usr/lib/unifi/lib/tomc
Jun 16 17:34:00 PiePi java[23692]: WARNING: Please consider reporting this to the maintainers of org.apache.catalina.loader.WebappClassLoaderBase
Jun 16 17:34:00 PiePi java[23692]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Jun 16 17:34:00 PiePi java[23692]: WARNING: All illegal access operations will be denied in a future release

I'm not sure what to make of it. Something like this happened to me a long time ago too and I think it was a drive space issue, so I run a df -h and see:

Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 26G 2.3G 92% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 4.4M 1.9G 1% /dev/shm
tmpfs 1.9G 9.7M 1.9G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mmcblk0p6 253M 49M 204M 20% /boot
/dev/sda 13T 3.0T 9.8T 24% /mnt/USB1
/dev/sdb1 1.9T 1.8T 121G 94% /mnt/USB2
tmpfs 384M 4.0K 384M 1% /run/user/1000

My 32GB SD card is 92% full and I can't see what's using all my space. I have baobab (Hard Disk Analyzer) installed and even running it with "sudo baobab" so it has permissions to all directories, it will only show 9.9GB of files on the disk:

These two things may be unrelated, but I'd still like to see what's actually using up so much space on my OS drive as well.

Any insight or assistance is much appreciated! I'd really like to regain access to my Unifi controller without blowing it away and reinstalling with an old backup (although I think I have it set up to backup every month at least).

 

[Vengance_01]

I have seen this before where you max out the number of inodes or something similar on the file system which will not let you create anymore files even with space on the disk. I would look for a CL script and run as root for the top x files and see if anything stand out

 

[T4rd]

I have seen this before where you max out the number of inodes or something similar on the file system which will not let you create anymore files even with space on the disk. I would look for a CL script and run as root for the top x files and see if anything stand out

Turns out the new Unifi 7.4 requires 64-bit MongoDB/OS installed. I'll have to upgrade my Pi4 with a clean install to Raspbian 64-bit. Yay.. now I'll have to reconfigure my other services/apps on the new build as well. Guess it'll be good training for my upcoming Linux+ class/cert that I'm prepping for.