Operate · Knowledge base
Troubleshooting
Start read-only, preserve the exact error, and fix the narrowest confirmed cause.
First-response workflow
./pauiops -check\n./pauiops -doctor\n./pauiops -service=statusUse -doctor -output=json when structured evidence must accompany a support case. Preview the only automatic repair with sudo ./pauiops -doctor -fix -dry-run; -fix is not a general repair-all command.
Searchable error index
Showing 25 troubleshooting entries
Service stays in activating (auto-restart)+
Meaning
systemd starts Manager, the process exits, and the restart policy tries again. This is not healthy.
Why it happens
The final startup or Failed to bootstrap line usually identifies a fatal configuration, account, database, or managed-root problem.
How to verify
./pauiops -service=status
journalctl -u pauiops.service -n 50 --no-pagerFix
Correct the last fatal error, run ./pauiops -check again, then restart the service. Do not treat repeated activation as readiness.
How to prevent it
Run -check before service changes and keep the managed root, database, and bootstrap configuration valid.
A Manager binary upgrade failed or rolled back+
Meaning
The candidate failed validation, systemd did not become active, or the optional health URL did not return 2xx before timeout.
Why it happens
Common causes are a wrong architecture, missing executable bit, checksum mismatch, bad install path, or a failing /api/ready gate.
How to verify
./pauiops -service=status
journalctl -u pauiops.service -n 100 --no-pager
curl -i http://127.0.0.1:51800/api/ready
curl -s http://127.0.0.1:51800/api/versionFix
Verify the published SHA-256 checksum, ELF architecture, PAUIOPS_MANAGER_INSTALL_PATH, health URL, and timeout. A failed upgrade exits non-zero even when the previous binary was restored successfully.
How to prevent it
Download to a temporary path, verify it first, and use a loopback /api/ready URL for the application health gate.
status=200/CHDIR or working directory permission denied+
Meaning
systemd cannot enter the configured working directory before Manager starts.
Why it happens
An older unit can point at a removed or inaccessible directory.
How to verify
systemctl cat pauiops.service
namei -l <configured-working-directory>Fix
Run sudo ./pauiops -service=enable. The current installer uses /var/lib/pauiops and installs the executable at /usr/local/bin/pauiops by default.
How to prevent it
Let the idempotent service installer own unit and working-directory synchronization.
inspect PAUIOPS_ROOT_PATH /root/repos: permission denied+
Meaning
The non-root service cannot traverse /root, even if the repository has different ownership.
Why it happens
Linux parent-directory permissions block traversal before Manager reaches the repository.
How to verify
namei -l /root/repos
./pauiops -doctorFix
Run sudo ./pauiops -service=enable. For root-account installs, it can safely relocate an eligible root to /srv/pauiops/repos while leaving the source as backup.
How to prevent it
Use a non-root absolute project root such as /home/deploy/repos. Never make /root broadly accessible.
Why /srv/pauiops/repos or /etc/pauiops/pauiops.env appears+
Meaning
/srv/pauiops/repos is the safe relocation target only for projects below /root; /etc/pauiops/pauiops.env is the stable root-owned systemd environment.
Why it happens
Services need a traversable project location and a stable protected configuration file. Neither path forces normal home-based projects into /srv.
How to verify
systemctl cat pauiops.service
namei -l <configured-project-root>Fix
Keep a valid /home/<username>/repos root if that is your intended non-root location. Re-run -service=enable only to synchronize installer-managed values.
How to prevent it
Avoid conflicting copies of the same key in exported variables, repository .env, and the systemd environment.
Startup warns about deprecated DOCKER_MGR_* variables+
Meaning
Manager accepted an older name as a compatibility fallback; the PAUIOPS_* equivalent wins when both exist.
Why it happens
The protected environment has not completed the prefix migration.
How to verify
journalctl -u pauiops.service -n 100 --no-pagerFix
Rename only the keys listed in the warning, run -service=enable for installer-managed changes, and restart. Do not delete or blindly replace the environment file.
How to prevent it
Use current PAUIOPS_* names in deployment automation. Warnings print names, never configured values.
chown: invalid user: pauiops:docker+
Meaning
The pauiops account, Docker group, or both do not exist on that host.
Why it happens
Ownership was changed before the installer resolved a valid service identity.
How to verify
getent passwd pauiops
getent group docker
id pauiopsFix
Run sudo ./pauiops -service=enable before changing ownership manually.
How to prevent it
Do not recursively chown projects to an account that has not been verified.
A dedicated service user cannot traverse /home/<username>/repos+
Meaning
A parent home directory blocks execute traversal even when the project root owner or group is correct.
Why it happens
Private home-directory modes intentionally stop unrelated accounts from traversing the path.
How to verify
namei -l /home/<username>/reposFix
Install ACL support, then rerun service enable: sudo apt-get install acl && sudo ./pauiops -service=enable. PAUIOps grants execute-only traversal, not listing access.
How to prevent it
Keep acl installed when a dedicated service identity must reach a project below a private home.
sudo requires a terminal, has no password, or rejects the password+
Meaning
The background service lacks valid non-interactive input or an exact allowed command for its current identity.
Why it happens
SUDO_PASSWORD can be missing, stale, owned by another account, or paired with an outdated sudo policy.
How to verify
sudo -l
./pauiops -service=status
ls -l /etc/sudoers.d/pauiops-dashboardFix
Place SUDO_PASSWORD only in the protected environment and rerun -service=enable. Dedicated pauiops accounts can be synchronized; PAUIOps never resets a human/custom service account password.
How to prevent it
Never place the password in a command-line argument, and keep the sudo policy limited to the fixed Dashboard commands.
The no new privileges flag is set+
Meaning
An older unit has NoNewPrivileges=true, preventing the deliberately configured sudo workflow.
Why it happens
The unit predates the current scoped Dashboard privilege model.
How to verify
systemctl cat pauiops.serviceFix
Run sudo /usr/local/bin/pauiops -service=enable. Enabling Dashboard sudo weakens part of the service sandbox, so retain the exact allowlist.
How to prevent it
Regenerate the unit after changing privileged-action configuration.
user database is empty+
Meaning
No account loaded, no eligible legacy database migrated, and no first-admin values are present.
Why it happens
This is a first start or a database path changed without a clear migration source.
How to verify
sudo find /root /var/lib/pauiops /home/<username> -maxdepth 3 -type f \( -name 'pauiops.db' -o -name 'docker-manager.db' \) -printf '%p %s bytes\n'Fix
Set all PAUIOPS_BOOTSTRAP_ADMIN_* values, restart, sign in, then remove the bootstrap password and restart again. Do not overwrite an active database; ambiguous SQLite candidates stop migration and MySQL is never auto-migrated.
How to prevent it
Persist /var/lib/pauiops, back up databases, and remove one-time bootstrap secrets after setup.
Git pull blocked: unable to verify repository status: signal terminated+
Meaning
The safety preflight could not finish, so PAUIOps intentionally refused to pull.
Why it happens
A slow filesystem, huge untracked tree, unreachable mount, stuck helper, or overly small command timeout can block status.
How to verify
time sudo -u <service-user> git -c safe.directory=<actual-repository-path> -C <actual-repository-path> status --porcelain=v2 --branch --untracked-files=normalFix
Use the real repository path, diagnose the delay, and only then adjust PAUIOPS_COMMAND_TIMEOUT. There is no pull-anyway bypass.
How to prevent it
Keep generated trees ignored and mounts healthy so the fail-closed status check remains fast.
fatal: cannot change to /.../REPO+
Meaning
REPO was documentation shorthand, not a literal directory name.
Why it happens
An example command was copied without replacing its placeholder.
How to verify
ls -la <configured-project-root>Fix
Replace REPO with the actual project directory shown in PAUIOps, such as SCA-Frontend.
How to prevent it
Review every angle-bracket or all-caps placeholder before running a documentation command.
fatal: could not read Username for https://github.com+
Meaning
Raw Git could not obtain HTTPS credentials and the non-interactive service could not answer a prompt.
Why it happens
Credential values are incomplete, the host differs from the remote, setup ran for another Linux account, or the runtime home/helper is unavailable.
How to verify
getent passwd <service-user>
sudo -u <service-user> -H git config --global --get-all credential.helper
sudo -u <service-user> -H git -C <actual-repository-path> remote get-url origin
sudo -u <service-user> -H git -C <actual-repository-path> ls-remote --exit-code origin HEADFix
Set username, scoped token, and the bare host in the systemd-loaded environment, then restart. App clone remains separate because it injects a one-invocation header; SSH remotes do not use these values.
How to prevent it
Use a scoped provider token, protect the runtime home, and test as the same service identity without displaying .git-credentials.
Startup cannot configure Git credentials+
Meaning
Manager could not resolve the runtime home, write the mode-0600 store, or add credential.helper=store within its bounded setup.
Why it happens
The systemd User, passwd home, home ownership/traversal, disk, Git executable, or global config permissions are inconsistent.
How to verify
systemctl show pauiops.service -p User
getent passwd <service-user>
sudo -u <service-user> -H git config --global --get-all credential.helperFix
Correct the runtime home or permissions and restart. If Manager prints a manual helper command, run it only as the same service user with -H.
How to prevent it
Keep the service user’s home writable by that user; helper setup adds store without replacing existing helpers.
Rotate or remove a stored Git token+
Meaning
Restart replaces the same-host token, but clearing environment values does not erase the already stored credential.
Why it happens
Auto-configuration is intentionally a no-op whenever either credential value is empty, and host changes preserve old host entries.
How to verify
sudo -u <service-user> -H git config --global --get-all credential.helperFix
For rotation, update PAUIOPS_GIT_PASSWORD and restart. For removal, pipe protocol=https and host=<old-git-host> to sudo -u <service-user> -H git credential reject, then clear/change the environment and restart.
How to prevent it
Never print the store, delete the whole file when it contains other hosts, or paste tokens into command arguments or shell history.
insufficient permission for adding an object to .git/objects+
Meaning
Git metadata still belongs to an earlier root or service identity.
Why it happens
A previous deployment or manual command created repository objects under another owner.
How to verify
namei -l <actual-repository-path>/.git/objects
./pauiops -doctorFix
Run sudo ./pauiops -repair-permissions, then restart. Manager already attempts one scoped metadata repair, but a root-owned tree needs the explicit administrator operation.
How to prevent it
Never use chmod -R 777 or safe.directory=*; keep repository work under the service identity.
Docker socket permission denied+
Meaning
The service identity cannot open the Docker Engine socket.
Why it happens
The account is not in the socket’s group or the service has not restarted since membership changed.
How to verify
ls -l /var/run/docker.sock
id <service-user>
getent group dockerFix
Run sudo ./pauiops -service=enable so it adds only missing membership and restarts with the new supplementary group.
How to prevent it
Never make the socket world-writable. Treat Docker group membership as highly privileged Linux access.
Center cannot reach a staging Manager+
Meaning
The registered API or WebSocket path is unreachable or rejected from the browser’s actual network context.
Why it happens
Wrong environment URL, ws/wss mismatch, missing exact origin, proxy upgrade headers, TLS, or firewall rules are common causes.
How to verify
curl -i https://<manager-host>/api/health
# Then use Center > Settings > Server Registry > Test ConnectionFix
Correct the staging API URL, use wss:// with HTTPS, allow the exact Center origin, forward WebSocket upgrades, and validate the Manager certificate and firewall path.
How to prevent it
Test each server entry before enabling it and keep API/WebSocket destinations server-specific.
Users are logged out or login returns HTTP 429+
Meaning
A session expired/restarted/switched servers, or one normalized username and source IP reached 10 failed attempts in 15 minutes.
Why it happens
Manager sessions are per-server and in-memory; HTTPS, proxy, CORS, URLs, stale tokens, and throttling can also affect auth.
How to verify
Use browser Network tools for /api/auth/status, login responses, 401/429, blocked CORS, and WebSocket handshakes.Fix
Confirm the intended Manager URL, cookie/proxy/origin configuration, and wait for the throttle window when applicable. Do not disable throttling.
How to prevent it
Use exact origins and trusted proxy settings. Expect separate sign-in when switching servers and full revocation after password reset.
Password-reset email or code is unavailable+
Meaning
The separate eight-character recovery flow could not deliver or accept its single-use 15-minute code.
Why it happens
SMTP delivery, request throttling, or the five wrong-code cap can apply; forgot always returns a generic response.
How to verify
Check SMTP configuration and delivery logs without exposing credentials or codes.Fix
Restore SMTP. In controlled diagnosis only, PAUIOPS_OTP_LOG_CODES=true also logs reset codes; remove it immediately afterward. The forgot endpoint never changes the password itself.
How to prevent it
Test recovery delivery, protect SMTP settings, and distinguish recovery from the optional six-digit login OTP.
AWS RDS allowlisting cannot determine the public IP+
Meaning
Manager could not validate a public IPv4/IPv6 source from the connection or bounded server-side echo endpoint.
Why it happens
Forwarded headers are untrusted, outbound access is blocked, or the custom echo URL is invalid.
How to verify
Check PAUIOPS_TRUST_PROXY_HEADERS, proxy replacement behavior, outbound access, and PAUIOPS_PUBLIC_IP_ECHO_URL.Fix
Use a controlled proxy or a single non-secret-bearing echo endpoint. The result must be public and, when a trustworthy source exists, must match it.
How to prevent it
Never trust forwarded headers by default or accept an arbitrary IP supplied by the browser.
Login OTP email is unavailable+
Meaning
The optional six-digit second factor is required but mail delivery is unavailable.
Why it happens
SMTP can be misconfigured or the provider can be temporarily down.
How to verify
Check SMTP configuration and non-secret delivery logs.Fix
Temporarily set PAUIOPS_OTP_BYPASS_EMAIL for one admin or PAUIOPS_OTP_LOG_CODES=true only in a controlled recovery, then remove it immediately after mail is restored.
How to prevent it
Monitor SMTP delivery and maintain a documented, time-limited break-glass procedure.