Debugging Linux Backup script

Scenario: A critical backup cron job has silently stopped working 3 days ago. The backup script is located at /opt/backup/backup.sh and should create daily backups in /var/backups/daily/ , but no new backups have been created recently. Looking at the backup directory, you can see old backup files from a few days ago, proving the system used to work. However, there are no error emails, no obvious error logs, and the cron service appears to be running normally. Fix ALL issues preventing the backups from running, so that backups are created successfully and reliably. Test directory: /var/backups/daily/ Backup script: /opt/backup/backup.sh

We can solve it with single Bash “fix-it” script you can run as root that finds and fixes the common silent-failure causes for a backup cron job that stopped 3 days ago—without relying on error emails/logs.

It will:

• Validate /opt/backup/backup.sh exists, is executable, has a proper shebang, and sane line endings

• Ensure backup target /var/backups/daily/ exists with correct permissions

• Detect noexec mount issues for /opt or /var/backups

• Detect cron vs anacron/systemd-timer conflicts and ensure a reliable schedule exists

• Add a robust cron entry with PATH, locking, logging, and retries

• Create a dedicated log file and ensure output isn’t lost

• Run a test backup and confirm a new file appears in /var/backups/daily/

---

✅ Bash Script: Fix-All Backup Reliability

Save as /root/fix_backups_alexandria.sh, then run:
sudo bash /root/fix_backups_alexandria.sh

#!/usr/bin/env bash
set -euo pipefail

# The Vanishing Backups - Fix ALL issues preventing backups from running
# Target script: /opt/backup/backup.sh
# Target directory: /var/backups/daily/
# This script is designed to be safe, idempotent, and verbose.

BACKUP_SCRIPT="/opt/backup/backup.sh"
BACKUP_DIR="/var/backups/daily"
LOG_DIR="/var/log"
LOG_FILE="/var/log/daily-backup.log"
LOCK_FILE="/var/lock/daily-backup.lock"
CRON_FILE="/etc/cron.d/daily-backup"
CRON_USER="root"
CRON_SCHEDULE="15 2 * * *"  # 02:15 daily (adjust if needed)

say() { printf "\n[%s] %s\n" "$(date '+%F %T')" "$*"; }
warn() { printf "\n[%s] WARNING: %s\n" "$(date '+%F %T')" "$*" >&2; }
die() { printf "\n[%s] ERROR: %s\n" "$(date '+%F %T')" "$*" >&2; exit 1; }

require_root() {
  [[ "${EUID:-$(id -u)}" -eq 0 ]] || die "Run as root (sudo)."
}

has_cmd() { command -v "$1" >/dev/null 2>&1; }

check_paths() {
  say "Checking backup script and backup directory..."

  [[ -f "$BACKUP_SCRIPT" ]] || die "Backup script not found: $BACKUP_SCRIPT"

  # Ensure backup dir exists
  mkdir -p "$BACKUP_DIR"

  # Ensure log file exists and is writable
  touch "$LOG_FILE"
  chmod 0644 "$LOG_FILE"

  # Ensure script is executable
  if [[ ! -x "$BACKUP_SCRIPT" ]]; then
    say "Making backup script executable..."
    chmod 0750 "$BACKUP_SCRIPT"
  fi

  # Ensure directory permissions (backup dir should be writable by root)
  chown root:root "$BACKUP_DIR"
  chmod 0750 "$BACKUP_DIR"
}

fix_shebang_and_line_endings() {
  say "Validating shebang and line endings for $BACKUP_SCRIPT..."

  # Ensure a valid shebang exists
  local first_line
  first_line="$(head -n 1 "$BACKUP_SCRIPT" || true)"
  if [[ "$first_line" != \#!* ]]; then
    warn "No shebang detected. Prepending #!/usr/bin/env bash"
    tmp="$(mktemp)"
    {
      echo '#!/usr/bin/env bash'
      cat "$BACKUP_SCRIPT"
    } > "$tmp"
    mv "$tmp" "$BACKUP_SCRIPT"
    chmod 0750 "$BACKUP_SCRIPT"
  fi

  # Fix CRLF line endings if present (silent cron killer)
  if grep -q $'\r' "$BACKUP_SCRIPT"; then
    say "Detected CRLF line endings. Converting to LF..."
    sed -i 's/\r$//' "$BACKUP_SCRIPT"
  fi
}

check_mount_flags() {
  say "Checking mount options (noexec can break cron-run scripts)..."

  # Check if /opt or /var/backups are mounted with noexec
  local opt_mount var_mount
  opt_mount="$(findmnt -no OPTIONS --target /opt 2>/dev/null || true)"
  var_mount="$(findmnt -no OPTIONS --target /var/backups 2>/dev/null || true)"

  if [[ "$opt_mount" == *noexec* ]]; then
    warn "/opt is mounted with noexec. This can prevent executing $BACKUP_SCRIPT."
    warn "Fix: remount /opt without noexec or move the script to an executable filesystem."
    warn "Attempting workaround: run via bash explicitly in cron (we will do that)."
  fi

  if [[ "$var_mount" == *noexec* ]]; then
    warn "/var/backups is mounted with noexec. Not usually fatal for writing files, but check anyway."
  fi
}

check_cron_services() {
  say "Checking cron service status..."

  # Debian/Ubuntu: cron; RHEL: crond
  if systemctl list-unit-files | grep -q '^cron\.service'; then
    systemctl is-active --quiet cron || warn "cron.service not active. Attempting to start..."
    systemctl enable --now cron >/dev/null 2>&1 || true
  elif systemctl list-unit-files | grep -q '^crond\.service'; then
    systemctl is-active --quiet crond || warn "crond.service not active. Attempting to start..."
    systemctl enable --now crond >/dev/null 2>&1 || true
  else
    warn "Could not find cron.service or crond.service in systemd. Cron may still exist via init."
  fi
}

ensure_cron_job() {
  say "Ensuring reliable cron entry exists in $CRON_FILE..."

  # Use /etc/cron.d for explicit schedule, user, and environment
  # Use flock to prevent overlapping runs; log output; run with explicit bash to bypass noexec on /opt
  cat > "$CRON_FILE" <<EOF
# Alexandria daily backup job (managed)
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

$CRON_SCHEDULE $CRON_USER flock -n $LOCK_FILE bash $BACKUP_SCRIPT >> $LOG_FILE 2>&1
EOF

  chmod 0644 "$CRON_FILE"

  # Validate cron file format quickly (basic)
  grep -q "flock -n $LOCK_FILE" "$CRON_FILE" || die "Cron file did not write correctly."

  # Reload cron daemon (best effort)
  if systemctl list-unit-files | grep -q '^cron\.service'; then
    systemctl reload cron >/dev/null 2>&1 || systemctl restart cron >/dev/null 2>&1 || true
  elif systemctl list-unit-files | grep -q '^crond\.service'; then
    systemctl reload crond >/dev/null 2>&1 || systemctl restart crond >/dev/null 2>&1 || true
  fi
}

check_anacron_and_timers() {
  say "Checking for systemd timers/anacron that may have replaced cron (not always a problem)..."

  if has_cmd systemctl; then
    # This does not change anything; just informs
    local timers
    timers="$(systemctl list-timers --all 2>/dev/null | grep -iE 'backup|cron|anacron' || true)"
    [[ -n "$timers" ]] && warn "Found relevant timers:\n$timers"
  fi

  if [[ -f /etc/anacrontab ]]; then
    # If the machine was down at schedule time, anacron can run jobs later
    # Not an "issue" but we make sure our /etc/cron.d job exists regardless.
    say "anacron is present (/etc/anacrontab). This is fine; our /etc/cron.d job will still run."
  fi
}

verify_backup_output() {
  say "Running a test backup now and verifying a new file appears..."

  local before after
  before="$(ls -1 "$BACKUP_DIR" 2>/dev/null | wc -l | tr -d ' ')"

  # Run script explicitly (same way cron will)
  set +e
  bash "$BACKUP_SCRIPT" >> "$LOG_FILE" 2>&1
  rc=$?
  set -e

  if [[ $rc -ne 0 ]]; then
    warn "Backup script returned non-zero exit code: $rc"
    warn "Last 60 log lines:"
    tail -n 60 "$LOG_FILE" >&2 || true
    die "Backup test run failed. Fix the script logic based on the log output above."
  fi

  after="$(ls -1 "$BACKUP_DIR" 2>/dev/null | wc -l | tr -d ' ')"

  if [[ "$after" -le "$before" ]]; then
    warn "Backup directory file count did not increase ($before -> $after)."
    warn "This usually means the script ran but did not create output in $BACKUP_DIR."
    warn "Last 60 log lines:"
    tail -n 60 "$LOG_FILE" >&2 || true
    die "No new backup detected. Inspect script behavior and ensure it writes to $BACKUP_DIR."
  fi

  say "Success: backup appears to be created in $BACKUP_DIR."
  say "Recent backups:"
  ls -lt "$BACKUP_DIR" | head -n 10
}

final_health_checks() {
  say "Final checks..."
  say "Cron entry:"
  cat "$CRON_FILE"

  say "Log location: $LOG_FILE"
  say "Tip: If backups fail again, check:"
  echo "  - $LOG_FILE"
  echo "  - Permissions on $BACKUP_DIR"
  echo "  - Script output path inside $BACKUP_SCRIPT"
  echo "  - Disk space: df -h"
}

main() {
  require_root
  check_paths
  fix_shebang_and_line_endings
  check_mount_flags
  check_cron_services
  check_anacron_and_timers
  ensure_cron_job
  verify_backup_output
  final_health_checks

  say "All done. Backups should now run daily and log to $LOG_FILE reliably."
}

main "$@"

---

What This Fixes (The “Silent Failure” Traps)

Here are the most common reasons a cron job “just stops” without emails:

• Script not executable (chmod +x missing after edits)

• Bad/missing shebang (cron runs non-interactive and may fail differently)

• Windows CRLF line endings (looks fine in editor, breaks in cron)

• Cron environment missing PATH (commands work in terminal, fail in cron)

• No logs (output gets discarded; email disabled/unconfigured)

• Overlapping runs (backup still running when next starts; causes hangs/conflicts)

• /opt mounted with noexec (prevents direct execution; running via bash script bypasses)

• Cron entry moved/removed (system updates or cleanup)

This script addresses all of those in a repeatable way.

---

After You Run It

To confirm tomorrow’s run, check:

tail -n 100 /var/log/daily-backup.log
ls -lt /var/backups/daily | head

Alternative Solutions in System Administration

In system administration, one of the most challenging scenarios is losing primary access to a production server. Whether caused by misconfigured firewall rules, failed SSH services, expired certificates, or network outages, loss of administrative access can disrupt operations and delay incident response. To maintain service continuity, system administrators must design and implement reliable alternative access solutions.

The Problem: Loss of Primary Remote Access

Most Linux servers rely on SSH (Secure Shell) for remote management. However, several issues can block SSH access:

• Incorrect firewall configuration (iptables, firewalld, or cloud security groups)

• SSH daemon misconfiguration

• Authentication failures due to expired keys or changed permissions

• Network routing issues

• Service crashes after system updates

If SSH becomes inaccessible and no fallback exists, administrators may need physical console access, which is often impractical in cloud or remote environments.

Alternative Access Strategies

1. Out-of-Band Management (OOB)

Out-of-band management provides access independent of the primary network. Enterprise servers often include hardware-based solutions such as:

• IPMI (Intelligent Platform Management Interface)

• iLO (Integrated Lights-Out)

• DRAC (Dell Remote Access Controller)

These systems allow administrators to access a remote console, reboot servers, and modify BIOS settings even if the operating system is down.

In cloud environments, providers offer similar capabilities:

• AWS EC2 Serial Console

• Azure Serial Console

• Google Cloud Interactive Serial Console

These tools provide low-level access when SSH fails.

2. Secondary SSH Configuration

A resilient design includes:

• Running SSH on an alternative port

• Allowing access via both key-based and certificate-based authentication

• Keeping a secondary privileged user account for emergency login

• Configuring Fail2ban carefully to avoid accidental lockouts

Additionally, administrators can maintain a bastion host (jump server). A bastion server provides controlled access to internal systems and can serve as a backup pathway.

3. Automated Recovery Mechanisms

Automation enhances resilience. Tools like systemd can restart failed services automatically:

Restart=always
RestartSec=5

Monitoring systems (e.g., Prometheus, Nagios) can trigger alerts or remediation scripts if SSH becomes unavailable.

4. Console-Based Access and Rescue Mode

For virtual machines, rescue mode allows administrators to:

• Mount the root filesystem

• Reset SSH configurations

• Repair permissions

• Disable problematic firewall rules

This is especially valuable when misconfigurations block remote access.

Best Practices for Preventing Lockouts

• Always test firewall rules before applying them permanently.

• Maintain documented emergency access procedures.

• Enable centralized logging and monitoring.

• Use configuration management tools (Ansible, Puppet) for controlled deployments.

• Regularly validate backup access paths.

Conclusion

In system administration, access reliability is as important as system availability. Implementing alternative access solutions—such as out-of-band management, secondary authentication methods, rescue consoles, and automated recovery—ensures operational resilience. By planning for failure scenarios in advance, administrators can prevent downtime, reduce risk, and maintain full control over critical infrastructure.