# Production Deployment Runbook

This directory contains reference templates for deploying the Laravel application on Ubuntu with Nginx, PHP-FPM, Supervisor, MySQL, and cron. Replace example hostnames, paths, usernames, and PHP socket versions for the target server. Do not commit production secrets.

## 1. Prepare the server

Install the runtime packages required by the application, including PHP 8.3+, PHP-FPM, required PHP extensions, Composer, Node.js 22.16.0 with npm for the locked asset build, MySQL/MariaDB client tools including `mysql` and `mysqldump`, Nginx, and Supervisor.

The repository pins the frontend Node runtime in `.nvmrc`. Before a production build, verify the installed Node runtime matches it:

```bash
node --version
cat .nvmrc
```

Recommended application path used by these templates:

```text
/var/www/microfinance/current
```

The web server must expose only:

```text
/var/www/microfinance/current/public
```

## 2. Prepare production environment

Copy `.env.production.example` to `.env` on the server and replace every placeholder with target-environment values.

Required minimums before release:

```text
APP_ENV=production
APP_DEBUG=false
APP_URL=https://your-domain.example
DB_CONNECTION=mysql
QUEUE_CONNECTION=database
SESSION_SECURE_COOKIE=true
SESSION_HTTP_ONLY=true
SESSION_ENCRYPT=true
```

Generate the application key once for the production environment and store it securely:

```bash
php artisan key:generate
```

Do not rotate `APP_KEY` casually after encrypted production data exists.

## 3. File ownership

The application code should not be globally writable. Grant the web/process user write access only where Laravel requires it:

```bash
sudo chown -R deploy:www-data /var/www/microfinance/current
sudo find /var/www/microfinance/current -type d -exec chmod 755 {} \;
sudo find /var/www/microfinance/current -type f -exec chmod 644 {} \;
sudo chown -R www-data:www-data /var/www/microfinance/current/storage /var/www/microfinance/current/bootstrap/cache
sudo chmod -R ug+rwX /var/www/microfinance/current/storage /var/www/microfinance/current/bootstrap/cache
```

Adapt the deploy user/group to the server policy.

### PHP-FPM runtime baseline

Install the dedicated pool from `deploy/php-fpm/microfinance.conf.example` and keep the Nginx site on the matching `/run/php/php8.3-fpm-microfinance.sock` socket. The baseline gives this application its own upload, POST, memory, execution, and hard FPM request limits instead of inheriting unknown global defaults. See `deploy/php-fpm/README.md` for installation and capacity-tuning notes.

After installing or changing the pool/site configuration, validate and reload services, then run the fail-closed preflight against the installed files:

```bash
sudo php-fpm8.3 -tt
sudo nginx -t
sudo systemctl restart php8.3-fpm
sudo systemctl reload nginx

PHP_FPM_POOL_FILE=/etc/php/8.3/fpm/pool.d/microfinance.conf \
NGINX_SITE_FILE=/etc/nginx/sites-enabled/microfinance \
bash deploy/scripts/php-runtime-preflight.sh
```

Repository CI validates the reference pool, Nginx template, and preflight logic. The target host must still pass this command and a representative document upload before go-live.

## 4. Install and build

From the release directory:

```bash
composer install --no-dev --prefer-dist --no-interaction --optimize-autoloader
npm ci --no-audit --no-fund
npm run build
php artisan storage:link --force
```

`package-lock.json` is committed and must remain synchronized with `package.json`. Production and CI use `npm ci`, so a missing or stale lockfile fails the build instead of silently resolving a different dependency graph. Frontend dependency changes must include the corresponding lockfile update in the same reviewed release.

Private application documents remain under private storage and must never be exposed through the public symlink.

## 5. Run pre-deployment gates

Before migrations or traffic cutover:

```bash
php artisan production:check
php artisan branches:repair-user-assignments --dry-run
```

`production:check` is a blocking configuration gate. Provider and disabled-backup notices may be warnings; unsafe application, database, queue, session, URL, or backup-path configuration fails the command.

If the branch repair dry-run reports inconsistencies, review them before running:

```bash
php artisan branches:repair-user-assignments
```

## 6. Backup before schema changes

Once the production backup destination is configured and verified:

```bash
php artisan security:backup
```

For automation, the same command can output only the verified bundle path:

```bash
BACKUP_BUNDLE="$(php artisan security:backup --path-only)"
```

Verify a backup bundle explicitly when required:

```bash
php artisan security:backup-verify /absolute/path/to/backup-bundle
```

Do not treat a local on-server backup as the only recovery copy. Maintain protected off-server/offsite copies and perform restore drills.

## 7. Deploy application changes

A conservative in-place deployment sequence is:

```bash
php artisan down --retry=60
php artisan migrate --force
php artisan config:cache
php artisan route:cache
php artisan view:cache
php artisan queue:restart
php artisan up
```

If route caching ever fails because a future route is not cacheable, fix or intentionally document that route before release rather than silently skipping the error.

For higher availability, use versioned release directories and an atomic `current` symlink switch; the same validation and migration rules still apply.

## 8. Queue worker

Install `deploy/supervisor/microfinance-worker.conf.example` as a Supervisor program after adapting paths and user. Then:

```bash
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl status
```

After every deployment:

```bash
php artisan queue:restart
```

`release:verify` dispatches a short queue probe and waits for the worker to process it. A stopped or misconfigured worker therefore blocks release acceptance instead of being discovered later by business traffic.

## 9. Scheduler

Install the scheduler cron entry from `deploy/cron/microfinance-scheduler.example` for the application user. Laravel runs `operations:scheduler-heartbeat` every minute in addition to arrears updates, management work, recurring expenses, reminders, and optional scheduled backups.

Verify registration with:

```bash
php artisan schedule:list
```

After cron is installed or restarted, allow at least one scheduler minute to run before the final `release:verify`. The verifier rejects a missing or stale heartbeat by default.

## 10. Internal release verification and manifest

After migrations, queue worker startup, scheduler startup, and creation of the pre-release backup, run:

```bash
RELEASE_GIT_SHA="$(git rev-parse HEAD)" php artisan release:verify \
  --require-backup \
  --output=storage/app/release-manifest.json
```

The command verifies:

- `production:check` still passes;
- database connectivity;
- database queue tables;
- required scheduled commands;
- scheduler heartbeat freshness;
- actual queue-worker execution through a queued probe;
- checksum validity of the newest backup bundle.

The generated manifest is stored with restrictive permissions and contains no credentials. It records the release Git SHA, environment, runtime summary, backup summary, individual check results, warnings, and errors. Preserve the accepted manifest with deployment records.

The `--skip-queue-probe` and `--skip-scheduler-heartbeat` switches exist for diagnostics/tests. Do not use them for a normal production release acceptance.

## 11. Nginx and external HTTP smoke check

Adapt `deploy/nginx/microfinance.conf.example`, install the TLS certificate, test configuration, and reload Nginx:

```bash
sudo nginx -t
sudo systemctl reload nginx
```

Then run the external smoke script from a host that reaches the deployed HTTPS origin:

```bash
BASE_URL=https://your-domain.example bash deploy/scripts/release-smoke-check.sh
```

The smoke check requires `/up`, `/login`, HTTPS, HSTS, CSP, anti-framing, MIME-sniffing protection, and the configured referrer policy. A failure blocks acceptance.

## 12. Continuous operational monitoring

Release verification proves the deployment at one point in time. After go-live, configure the production monitoring platform to run the runtime health command regularly and alert whenever it exits non-zero:

```bash
php artisan operations:health-check
```

For machine ingestion use:

```bash
php artisan operations:health-check --json
```

The runtime check covers database connectivity, recent failed queue jobs, scheduler heartbeat freshness, free disk space, and backup freshness. Thresholds are controlled by the `OPERATIONS_*` values in `.env.production.example`. When scheduled backups are disabled, backup freshness is reported as a warning; once `BACKUP_SCHEDULE_ENABLED=true`, a missing or stale backup fails the health check.

A practical starting interval is every five minutes. The monitoring platform should alert on a non-zero exit and retain the JSON payload for incident diagnosis. Do not rely on a local logfile as the only alert path.

HTTP reachability must be monitored externally because an application process cannot prove that DNS, TLS, Nginx, load balancers, or the public network path are healthy. Configure an external HTTPS monitor against:

```text
https://your-domain.example/up
```

Alert on non-2xx responses, TLS failures, or sustained latency. Keep the release smoke check as the stronger post-deployment test for security headers and the login surface.

## 13. Isolated backup restore drill

A checksum-valid backup is not enough; restoration must be proven. Prefer a staging or dedicated restore-test MySQL host. Use credentials that are allowed to create and drop only the disposable restore-test database, not application credentials.

The script refuses database names outside the `microfinance_restore_...` namespace and also refuses the active `DB_DATABASE` value:

```bash
BACKUP_BUNDLE=/absolute/path/to/backup-bundle \
RESTORE_DB_HOST=127.0.0.1 \
RESTORE_DB_PORT=3306 \
RESTORE_DB_NAME=microfinance_restore_20260814 \
RESTORE_DB_ADMIN_USER=restore_operator \
RESTORE_DB_ADMIN_PASSWORD='use-a-secret-source' \
bash deploy/scripts/backup-restore-drill.sh
```

The drill first re-verifies checksums, restores `database.sql` into the disposable database, confirms key tables can be queried, and drops the disposable database on exit. Never point this workflow at the live application database as the restore target.

Private-file checksums are already verified by `security:backup-verify`; periodically perform a broader recovery exercise that also mounts/restores the private-file tree in an isolated environment and opens representative customer/loan documents.

## 14. Final post-deployment acceptance

A normal release is accepted only when all of the following succeed:

```bash
php artisan production:check
php artisan schedule:list
RELEASE_GIT_SHA="$(git rev-parse HEAD)" php artisan release:verify --require-backup
php artisan operations:health-check
PHP_FPM_POOL_FILE=/etc/php/8.3/fpm/pool.d/microfinance.conf NGINX_SITE_FILE=/etc/nginx/sites-enabled/microfinance bash deploy/scripts/php-runtime-preflight.sh
BASE_URL=https://your-domain.example bash deploy/scripts/release-smoke-check.sh
```

Also verify the isolated backup restore drill, login, one safe branch-scoped workflow, a representative contract/document upload, dashboard/report access, private document access, and external providers required at launch.

Record together:

- deployed Git SHA;
- release manifest;
- pre-deployment backup bundle identifier;
- restore-drill result;
- operational health-check result;
- PHP runtime preflight result;
- smoke-check result;
- deployment timestamp and operator/change reference.

## 15. Rollback

Application rollback must use a known previous release commit. Database rollback is not automatically safe for migrations that transformed or deleted data. For schema/data failures, restore from the pre-deployment backup or execute a reviewed forward-fix migration according to the incident plan.

Never run `migrate:rollback` blindly on a production financial database.
