Operate · Knowledge base
Troubleshooting
Start read-only, preserve the exact error, and fix the narrowest confirmed cause.
First-response workflow
pauiops -check\npauiops -doctor\npauiops --service statusUse -doctor -output=json when structured evidence must accompany a support case. Preview the only automatic repair with sudo /usr/local/bin/pauiops -doctor -fix -dry-run; -fix is not a general repair-all command.
Searchable error index
Showing 34 troubleshooting entries
Service stays in activating (auto-restart)+
Meaning
systemd starts Manager, the process exits, and the restart policy tries again. This is not healthy.
Why it happens
The final startup or Failed to bootstrap line usually identifies a fatal configuration, account, database, or managed-root problem.
How to verify
pauiops --service status
journalctl -u pauiops.service -n 50 --no-pagerFix
Correct the last fatal error, run pauiops -check again, then restart the service. Do not treat repeated activation as readiness.
How to prevent it
Run -check before service changes and keep the managed root, database, and bootstrap configuration valid.
A Manager binary upgrade failed or rolled back+
Meaning
The candidate failed validation, systemd did not become active, or the optional health URL did not return 2xx before timeout.
Why it happens
Common causes are a wrong architecture, missing executable bit, checksum mismatch, bad install path, or a failing /api/ready gate.
How to verify
pauiops --service status
journalctl -u pauiops.service -n 100 --no-pager
curl -i http://127.0.0.1:51800/api/ready
curl -s http://127.0.0.1:51800/api/versionFix
Verify the published SHA-256 checksum, ELF architecture, PAUIOPS_MANAGER_INSTALL_PATH, health URL, and timeout. A failed upgrade exits non-zero even when the previous binary was restored successfully.
How to prevent it
Download to a temporary path, verify it first, and use a loopback /api/ready URL for the application health gate.
dependency changes require confirmation+
Meaning
One or more Git, Docker, Compose, Nginx, or ACL requirements are missing or below policy, and the installer has no interactive terminal from which to receive approval.
Why it happens
Package repository changes, conflicting-package removal, and service restarts are infrastructure mutations. PAUIOps refuses to infer approval from a curl pipe or unattended job.
How to verify
curl -fsSL <official-domain>/install.sh | sh -s -- --dry-runFix
Review the complete dependency plan and host backups. Run interactively to answer the prompt, or explicitly approve the same reviewed plan in automation.
curl -fsSL <official-domain>/install.sh | sh -s -- --yes
# Equivalent environment form
curl -fsSL <official-domain>/install.sh |
PAUIOPS_ACCEPT_DEPENDENCY_CHANGES=1 shHow to prevent it
Run --dry-run in deployment review and require an explicit approval step before production package or service changes.
no stable package candidate meets the PAUIOps minimum+
Meaning
The recognized stable package repositories do not publish a sufficiently new build for this distribution, release, or architecture.
Why it happens
The operating system can be unsupported or end-of-life, repository metadata can be stale or unreachable, or the vendor may not publish that component for the host architecture.
How to verify
pauiops -doctor
cat /etc/os-release
uname -m
git --version
docker --version
docker compose version
nginx -vFix
Use a vendor-supported operating-system release and architecture, restore stable repository access, or install a compliant stable package manually. Then rerun --dry-run. PAUIOps does not fall back to a test, beta, release-candidate, mainline Nginx, or unknown third-party package.
How to prevent it
Keep hosts within the published vendor support matrix and test dependency upgrades before an operating-system release reaches end of support.
A project shows Manual fix needed+
Meaning
The project Compose watcher exhausted its recovery-attempt budget and opened the circuit breaker. PAUIOps has stopped issuing automatic repair commands for this project.
Why it happens
The underlying service, image, dependency, network, healthcheck, published port, or application problem survived the targeted and escalated recovery levels, or the service kept flapping inside the configured recovery window.
How to verify
sudo journalctl -u pauiops.service -n 200 --no-pager | grep "compose watcher"
docker compose ps --all
docker compose logs --since 30m <service>
docker inspect <container> --format '{{json .State}}'Fix
Repair and verify the underlying project problem manually. Then toggle Auto-start / restart off and back on for that project. Re-enabling clears the circuit breaker, counters, attempt window, recovery level, and remembered one-time-job classification.
How to prevent it
Add meaningful application healthchecks, keep dependencies and published ports accurate, and tune thresholds only after reviewing real incident timing. Do not use a larger recovery budget to hide a persistent fault.
status=200/CHDIR or working directory permission denied+
Meaning
systemd cannot enter the configured working directory before Manager starts.
Why it happens
An older unit can point at a removed or inaccessible directory.
How to verify
systemctl cat pauiops.service
namei -l <configured-working-directory>Fix
Run sudo /usr/local/bin/pauiops -service=enable. The URL installer uses /var/lib/pauiops and installs the executable at /usr/local/bin/pauiops by default.
How to prevent it
Let the idempotent service installer own unit and working-directory synchronization.
inspect PAUIOPS_ROOT_PATH /root/repos: permission denied+
Meaning
The non-root service cannot traverse /root, even if the repository has different ownership.
Why it happens
Linux parent-directory permissions block traversal before Manager reaches the repository.
How to verify
namei -l /root/repos
pauiops -doctorFix
Run sudo /usr/local/bin/pauiops -service=enable. For root-account installs, it can safely relocate an eligible root to /srv/pauiops/repos while leaving the source as backup.
How to prevent it
Use a non-root absolute project root such as /home/deploy/repos. Never make /root broadly accessible.
Why /srv/pauiops/repos or /etc/pauiops/pauiops.env appears+
Meaning
/srv/pauiops/repos is the safe relocation target only for projects below /root; /etc/pauiops/pauiops.env is the stable root-owned systemd environment.
Why it happens
Services need a traversable project location and a stable protected configuration file. Neither path forces normal home-based projects into /srv.
How to verify
systemctl cat pauiops.service
namei -l <configured-project-root>Fix
Keep a valid /home/<username>/repos root if that is your intended non-root location. Re-run -service=enable only to synchronize installer-managed values.
How to prevent it
Avoid conflicting copies of the same key in exported variables, repository .env, and the systemd environment.
Startup warns about deprecated DOCKER_MGR_* variables+
Meaning
Manager accepted an older name as a compatibility fallback; the PAUIOPS_* equivalent wins when both exist.
Why it happens
The protected environment has not completed the prefix migration.
How to verify
journalctl -u pauiops.service -n 100 --no-pagerFix
Rename only the keys listed in the warning, run -service=enable for installer-managed changes, and restart. Do not delete or blindly replace the environment file.
How to prevent it
Use current PAUIOPS_* names in deployment automation. Warnings print names, never configured values.
chown: invalid user: pauiops:docker+
Meaning
The pauiops account, Docker group, or both do not exist on that host.
Why it happens
Ownership was changed before the installer resolved a valid service identity.
How to verify
getent passwd pauiops
getent group docker
id pauiopsFix
Run sudo /usr/local/bin/pauiops -service=enable before changing ownership manually.
How to prevent it
Do not recursively chown projects to an account that has not been verified.
A dedicated service user cannot traverse /home/<username>/repos+
Meaning
A parent home directory blocks execute traversal even when the project root owner or group is correct.
Why it happens
Private home-directory modes intentionally stop unrelated accounts from traversing the path.
How to verify
namei -l /home/<username>/reposFix
Install ACL support, then rerun service enable: sudo apt-get install acl && sudo /usr/local/bin/pauiops -service=enable. PAUIOps grants execute-only traversal, not listing access.
How to prevent it
Keep acl installed when a dedicated service identity must reach a project below a private home.
sudo requires a terminal, has no password, or rejects the password+
Meaning
The background service lacks valid non-interactive input or an exact allowed command for its current identity.
Why it happens
SUDO_PASSWORD can be missing, stale, owned by another account, or paired with an outdated sudo policy.
How to verify
sudo -l
pauiops --service status
ls -l /etc/sudoers.d/pauiops-dashboardFix
Place SUDO_PASSWORD only in the protected environment and rerun -service=enable. Dedicated pauiops accounts can be synchronized; PAUIOps never resets a human/custom service account password.
How to prevent it
Never place the password in a command-line argument, and keep the sudo policy limited to the fixed Dashboard commands.
The no new privileges flag is set+
Meaning
An older unit has NoNewPrivileges=true, preventing the deliberately configured sudo workflow.
Why it happens
The unit predates the current scoped Dashboard privilege model.
How to verify
systemctl cat pauiops.serviceFix
Run sudo /usr/local/bin/pauiops -service=enable. Enabling Dashboard sudo weakens part of the service sandbox, so retain the exact allowlist.
How to prevent it
Regenerate the unit after changing privileged-action configuration.
user database is empty+
Meaning
No account loaded, no eligible legacy database migrated, and no first-admin values are present.
Why it happens
This is a first start or a database path changed without a clear migration source.
How to verify
sudo find /root /var/lib/pauiops /home/<username> -maxdepth 3 -type f \( -name 'pauiops.db' -o -name 'docker-manager.db' \) -printf '%p %s bytes\n'Fix
Set all PAUIOPS_BOOTSTRAP_ADMIN_* values, restart, sign in, then remove the bootstrap password and restart again. Do not overwrite an active database; ambiguous SQLite candidates stop migration and MySQL is never auto-migrated.
How to prevent it
Persist /var/lib/pauiops, back up databases, and remove one-time bootstrap secrets after setup.
Git pull blocked: unable to verify repository status: signal terminated+
Meaning
The safety preflight could not finish, so PAUIOps intentionally refused to pull.
Why it happens
A slow filesystem, huge untracked tree, unreachable mount, stuck helper, or overly small command timeout can block status.
How to verify
time sudo -u <service-user> git -c safe.directory=<actual-repository-path> -C <actual-repository-path> status --porcelain=v2 --branch --untracked-files=normalFix
Use the real repository path, diagnose the delay, and only then adjust PAUIOPS_COMMAND_TIMEOUT. There is no pull-anyway bypass.
How to prevent it
Keep generated trees ignored and mounts healthy so the fail-closed status check remains fast.
fatal: cannot change to /.../REPO+
Meaning
REPO was documentation shorthand, not a literal directory name.
Why it happens
An example command was copied without replacing its placeholder.
How to verify
ls -la <configured-project-root>Fix
Replace REPO with the actual project directory shown in PAUIOps, such as SCA-Frontend.
How to prevent it
Review every angle-bracket or all-caps placeholder before running a documentation command.
fatal: detected dubious ownership in repository+
Meaning
Git refuses to use a repository whose filesystem owner does not match the Linux account running PAUIOps Manager.
Why it happens
The repository was commonly cloned, copied, or updated as root or another user. A direct root-account installation normally runs Manager as pauiops and manages /srv/pauiops/repos; an installation invoked with sudo by a non-root operator normally runs as that original username and can keep projects below /home/<username>.
How to verify
systemctl show pauiops.service -p User -p Group --no-pager
ls -ld <actual-repository-path> <actual-repository-path>/.git
namei -l <actual-repository-path>Fix
Confirm the User and Group reported by systemd, stop any write operation, and repair only the configured PAUIOps project root. Use the root-account command for the default dedicated pauiops service, or replace the non-root example values with the actual service username and project folder. If the unit reports a different group from docker, use that reported group.
# Direct root-account install: dedicated pauiops service
sudo chown -R pauiops:docker /srv/pauiops/repos
# Install invoked by a non-root operator: edit these two values
pauiops_owner="deploy"
pauiops_project_folder="cktPMSWebApp"
sudo chown -R "$pauiops_owner":docker "/home/$pauiops_owner/$pauiops_project_folder"
# Confirm Git works as the same account that runs Manager
sudo -u <service-user> -H git -C <actual-repository-path> statusHow to prevent it
Run repository Git and deployment writes as the Manager service identity. Do not chown an entire home directory, use chmod -R 777, or configure safe.directory=*; a safe.directory exception suppresses Git’s ownership check but does not repair write ownership.
fatal: could not read Username for 'https://github.com': terminal prompts disabled+
Meaning
The git subcommand needed HTTPS credentials, but PAUIOPS_GIT_USERNAME/PAUIOPS_GIT_PASSWORD are not both configured, or the remote's host does not match PAUIOPS_GIT_HOST.
Why it happens
Every git pull, fetch, push, and ls-remote — admin terminal, non-admin terminal, preset commands, and the structured Git API — authenticates by injecting a host-scoped http.https://<PAUIOPS_GIT_HOST>/.extraHeader config entry, computed fresh from those two values on every single invocation. The header is only added when both values are set and the remote's host equals PAUIOPS_GIT_HOST.
How to verify
getent passwd <service-user>
sudo -u <service-user> -H git -C <actual-repository-path> remote get-url origin
systemctl show pauiops.service -p Environment --no-pager | tr ' ' '\n' | grep PAUIOPS_GITFix
Set PAUIOPS_GIT_USERNAME, PAUIOPS_GIT_PASSWORD, and PAUIOPS_GIT_HOST (if the remote isn't github.com) in the systemd-loaded environment, then restart. No credential file is written anywhere, so the very next command authenticates correctly — there is no separate credential-store step to repeat.
How to prevent it
Use a scoped provider token and keep PAUIOPS_GIT_HOST matched to the remote's actual host; a mismatch now names PAUIOPS_GIT_HOST and the mismatch directly in the error instead of a generic prompt-disabled failure.
Non-admin terminal git pull/push fails to authenticate+
Meaning
A non-admin session's isolation from global Git config previously left no working credential path for HTTPS remotes at all.
Why it happens
Non-admin sessions run with GIT_CONFIG_NOSYSTEM=1 and GIT_CONFIG_GLOBAL=/dev/null so a session can never reach user-controlled global git config, alias injection, or core.sshCommand. The same host-scoped credential header used by the admin terminal is now injected for exactly the four network subcommands (pull, fetch, push, ls-remote) a non-admin can run, so authentication works without weakening that isolation.
How to verify
sudo -u <service-user> -H git -C <actual-repository-path> ls-remote --exit-code origin HEADFix
Configure PAUIOPS_GIT_USERNAME/PAUIOPS_GIT_PASSWORD/PAUIOPS_GIT_HOST as above and restart. Every other git subcommand a non-admin can run (config, status, log, diff, ...) never receives the credential header, so git config --list cannot print it back out.
How to prevent it
Do not attempt to route non-admin Git access through a shared credential file or global config change; the fresh-per-command header is the supported path.
Rotate a Git provider token+
Meaning
Credentials are never written to disk, so rotation is just an environment change, not a stored-file cleanup.
Why it happens
PAUIOPS_GIT_USERNAME/PAUIOPS_GIT_PASSWORD are read fresh on every git invocation and injected as a one-command config header; nothing persists between commands or across a restart.
How to verify
systemctl show pauiops.service -p Environment --no-pager | tr ' ' '\n' | grep PAUIOPS_GIT_PASSWORDFix
Update PAUIOPS_GIT_PASSWORD (and PAUIOPS_GIT_USERNAME if it changed) in the systemd-loaded environment, then restart. There is no stored credential to revoke, clear, or reject afterward.
How to prevent it
Rotate the provider token at the source (e.g. revoke the old personal access token) in addition to updating the environment value; PAUIOps has nothing left on disk to clean up.
insufficient permission for adding an object to .git/objects+
Meaning
Git metadata still belongs to an earlier root or service identity.
Why it happens
A previous deployment or manual command created repository objects under another owner.
How to verify
namei -l <actual-repository-path>/.git/objects
pauiops -doctorFix
Run sudo /usr/local/bin/pauiops -repair-permissions, then restart. Manager already attempts one scoped metadata repair, but a root-owned tree needs the explicit administrator operation.
How to prevent it
Never use chmod -R 777 or safe.directory=*; keep repository work under the service identity.
Docker socket permission denied+
Meaning
The service identity cannot open the Docker Engine socket.
Why it happens
The account is not in the socket’s group or the service has not restarted since membership changed.
How to verify
ls -l /var/run/docker.sock
id <service-user>
getent group dockerFix
Run sudo /usr/local/bin/pauiops -service=enable so it adds only missing membership and restarts with the new supplementary group.
How to prevent it
Never make the socket world-writable. Treat Docker group membership as highly privileged Linux access.
Compose port collision or port scan incomplete+
Meaning
Two visible Compose services declare overlapping host bindings for the same protocol and published port, or Manager could not resolve every authorized Compose file.
Why it happens
Projects can independently publish the same host port, wildcard bindings can overlap a specific address, or one Compose file can be unreadable, invalid, or dependent on unavailable interpolation values.
How to verify
docker compose -f <first-compose-file> config
docker compose -f <second-compose-file> config
docker compose ps --all
ss -lntupFix
Open the collision details, confirm both project/service declarations, and change the intended host port in source configuration. For an incomplete scan, repair every unreadable or invalid authorized Compose file before treating an empty result as clean.
How to prevent it
Allocate published ports centrally, validate resolved Compose configuration in deployment review, and remember that the PAUIOps comparison is declared configuration—not a complete operating-system socket inventory.
A Center domain shows Needs attention+
Meaning
Center saved the domain record, but its assigned Manager did not complete the privileged Nginx apply operation.
Why it happens
DNS may not point to the assigned Manager, ports 80/443 may be blocked or occupied, Certbot may fail, Nginx configuration may be invalid, the host may not match the current Ubuntu/systemd/apt provisioner, or the Manager admin/sudo path may be unavailable.
How to verify
dig +short A <domain>
dig +short CNAME <domain>
sudo nginx -t
sudo systemctl status nginx --no-pager
sudo journalctl -u nginx -n 100 --no-pager
sudo certbot certificatesFix
Preserve the exact Center error. Confirm DNS and public port reachability, repair the narrow package/Nginx/Certbot/admin cause, restore the backed-up HTTPS config if an active route regressed, then use Retry deployment.
How to prevent it
Preview and publish the DNS handoff before Add Domain, use a real certificate email, back up Nginx and certificates, and schedule active-route edits because apply does not currently provide atomic HTTPS rollback.
Another instance of Certbot is already running+
Meaning
A recurring provisioning failure caused by a plain certbot renew --dry-run health check that needed Certbot's global lock even when nothing had to be renewed, made worse by a timed-out provisioning run that could orphan its certbot child and hold that lock indefinitely.
Why it happens
The renewal decision is now a local OpenSSL certificate-expiry check with no network call and no lock involved; a certificate valid for more than 30 days skips Certbot entirely. When Certbot genuinely must run, a retry wrapper waits out lock contention (10s intervals, up to about 3 minutes) instead of failing immediately, and a provisioning run that times out now terminates its whole process group so it can no longer orphan a lock-holding Certbot child.
How to verify
sudo journalctl -u pauiops.service -n 100 --no-pager | grep -i certbot
ps aux | grep certbot
sudo certbot certificatesFix
Update to a Manager build that includes the Nginx provisioning reliability fix, then retry the domain operation. If a stale certbot process from before the upgrade is still holding the lock, stop it manually first. A second domain request arriving mid-provision now returns a clear "Another domain operation is still being applied" error instead of the cryptic Certbot lock failure.
How to prevent it
Let Manager's serialized per-instance domain queue handle concurrent requests instead of running certbot manually against a PAUIOps-managed domain.
Center cannot reach a staging Manager+
Meaning
The registered API or WebSocket path is unreachable or rejected from the browser’s actual network context.
Why it happens
Wrong environment URL, ws/wss mismatch, missing exact origin, proxy upgrade headers, TLS, or firewall rules are common causes.
How to verify
curl -i https://<manager-host>/api/health
# Then use Center > Settings > Server Registry > Test ConnectionFix
Correct the staging API URL, use wss:// with HTTPS, allow the exact Center origin, forward WebSocket upgrades, and validate the Manager certificate and firewall path.
How to prevent it
Test each server entry before enabling it and keep API/WebSocket destinations server-specific.
Users are logged out or login returns HTTP 429+
Meaning
A session expired/restarted/switched servers, or one normalized username and source IP reached 10 failed attempts in 15 minutes.
Why it happens
Manager sessions are per-server and in-memory; HTTPS, proxy, CORS, URLs, stale tokens, and throttling can also affect auth.
How to verify
Use browser Network tools for /api/auth/status, login responses, 401/429, blocked CORS, and WebSocket handshakes.Fix
Confirm the intended Manager URL, cookie/proxy/origin configuration, and wait for the throttle window when applicable. Do not disable throttling.
How to prevent it
Use exact origins and trusted proxy settings. Expect separate sign-in when switching servers and full revocation after password reset.
Password-reset email or code is unavailable+
Meaning
The separate eight-character recovery flow could not deliver or accept its single-use 15-minute code.
Why it happens
SMTP delivery, request throttling, or the five wrong-code cap can apply; forgot always returns a generic response.
How to verify
Check SMTP configuration and delivery logs without exposing credentials or codes.Fix
Restore SMTP. In controlled diagnosis only, PAUIOPS_OTP_LOG_CODES=true also logs reset codes; remove it immediately afterward. The forgot endpoint never changes the password itself.
How to prevent it
Test recovery delivery, protect SMTP settings, and distinguish recovery from the optional six-digit login OTP.
AWS RDS allowlisting cannot determine the public IP+
Meaning
Manager could not validate a public IPv4/IPv6 source from the connection or bounded server-side echo endpoint.
Why it happens
Forwarded headers are untrusted, outbound access is blocked, or the custom echo URL is invalid.
How to verify
Check PAUIOPS_TRUST_PROXY_HEADERS, proxy replacement behavior, outbound access, and PAUIOPS_PUBLIC_IP_ECHO_URL.Fix
Use a controlled proxy or a single non-secret-bearing echo endpoint. The result must be public and, when a trustworthy source exists, must match it.
How to prevent it
Never trust forwarded headers by default or accept an arbitrary IP supplied by the browser.
Login OTP email is unavailable+
Meaning
The optional six-digit second factor is required but mail delivery is unavailable.
Why it happens
SMTP can be misconfigured or the provider can be temporarily down.
How to verify
Check SMTP configuration and non-secret delivery logs.Fix
Temporarily set PAUIOPS_OTP_BYPASS_EMAIL for one admin or PAUIOPS_OTP_LOG_CODES=true only in a controlled recovery, then remove it immediately after mail is restored.
How to prevent it
Monitor SMTP delivery and maintain a documented, time-limited break-glass procedure.