Alerts Management Tool

A comprehensive alerts management system that provides flexible alert creation, notification, tracking, and reporting capabilities for Linux and macOS systems.

Features

Alert Management

• Create alerts with customizable severity levels (critical, high, medium, low)
• Acknowledge and resolve alerts with user tracking
• Alert status tracking (active, acknowledged, resolved)
• Alert history and retention management
• JSON-based alert storage for programmatic access

Notification System

• Multiple notification channels:
  • Email notifications
  • Slack integration with rich formatting
  • PagerDuty integration for on-call management
• Severity-based notification routing
• Customizable notification templates
• Rate limiting and deduplication

Reporting

• HTML reports with interactive charts
• JSON reports for programmatic analysis
• Alert statistics and trends
• Customizable report formats
• Historical data analysis

Cross-Platform Support

• Linux (RHEL, Debian, and variants)
• macOS
• Consistent alert handling across platforms
• Platform-specific optimizations

Installation

Prerequisites

• Bash shell
• Python 3 (for HTML reports)
• jq (for JSON processing)
• yq (for YAML processing, optional)

Installation Steps

1. Download the script from the Lambda Softworks Customer Portal:

   • Log in to your Lambda Softworks Customer Portal account
   • Navigate to the Files section
   • Download the alerts.sh script

2. Make the script executable:

chmod +x alerts.sh

3. Install dependencies:

For Debian/Ubuntu:

sudo apt-get update
sudo apt-get install -y python3 jq

For RHEL/CentOS:

sudo yum install -y python3 jq

For macOS:

brew install python3 jq

4. Create configuration file (optional):

mkdir -p ~/.config/alerts
cat > ~/.config/alerts/config.yml << EOF
email_notifications: true
alert_email: "admin@example.com"
slack_webhook_url: "https://hooks.slack.com/services/your/webhook/url"
pagerduty_api_key: "your-api-key"
pagerduty_routing_key: "your-routing-key"
alert_url: "https://your-alert-dashboard.com"
retention_days: 30
EOF

Usage

Basic Usage

1. Create an alert:

./alerts.sh --create "High CPU Usage" "CPU usage is at 90%" high "system-monitor"

2. List active alerts:

./alerts.sh --list active

3. Acknowledge an alert:

./alerts.sh --acknowledge alert-123

4. Generate HTML report:

./alerts.sh --report html

Advanced Usage

1. Create alert with custom severity:

./alerts.sh --create "Disk Space Low" "Root partition at 95%" critical "disk-monitor"

2. List alerts with filters:

./alerts.sh --list active high "system-monitor"

3. Generate JSON report:

./alerts.sh --report json

4. Clean up old alerts:

./alerts.sh --cleanup 60

Command Line Options

• --create "TITLE" "MESSAGE" [SEVERITY] [SOURCE]: Create a new alert
• --acknowledge ALERT_ID [USER]: Acknowledge an alert
• --resolve ALERT_ID [USER]: Resolve an alert
• --list [STATUS] [SEVERITY] [SOURCE]: List alerts with optional filters
• --report [html|json]: Generate alert report
• --cleanup [DAYS]: Clean up old alerts
• --config FILE: Use custom configuration file
• --help: Show help message

Configuration

Configuration File Location

• Default: ~/.config/alerts/config.yml
• Custom: Specified via --config option

Configuration Options

# Notification settings
email_notifications: true    # Enable email notifications
alert_email: ""             # Email address for notifications
slack_webhook_url: ""       # Slack webhook URL
pagerduty_api_key: ""       # PagerDuty API key
pagerduty_routing_key: ""   # PagerDuty routing key
alert_url: ""              # URL to alert dashboard

# Alert settings
retention_days: 30         # Days to keep alert history
default_severity: "medium" # Default alert severity

# Report settings
report_format: "html"      # Default report format
chart_theme: "light"       # Chart theme (light/dark)

Logs and Data Storage

Log File Location

• Linux: ~/.local/log/alerts.log
• macOS: ~/Library/Logs/alerts.log

Alerts Storage

• Location: ~/.local/share/alerts/alerts/
• Format: JSON files with timestamps
• Retention: Configurable (default: 30 days)

History Storage

• Location: ~/.local/share/alerts/history/
• Format: JSON files with timestamps
• Retention: Configurable (default: 90 days)

Reports Storage

• Location: ~/.local/share/alerts/reports/
• Format: HTML and JSON files
• Retention: Configurable (default: 365 days)

Cross-Platform Compatibility

Linux Support

• RHEL/CentOS 7+
• Ubuntu 18.04+
• Debian 10+
• Other Linux distributions with systemd

macOS Support

• macOS 10.15+
• Intel and Apple Silicon Macs
• Native macOS services support

Examples

Basic Alert Creation

# Create a high severity alert
./alerts.sh --create "Service Down" "Nginx service is not running" high "service-monitor"

# Create a critical alert
./alerts.sh --create "Database Error" "Connection failed to MySQL" critical "db-monitor"

Alert Management

# List all critical alerts
./alerts.sh --list active critical

# Acknowledge an alert
./alerts.sh --acknowledge alert-123 admin

# Resolve an alert
./alerts.sh --resolve alert-123 admin

Report Generation

# Generate HTML report
./alerts.sh --report html

# Generate JSON report
./alerts.sh --report json

# Clean up old alerts
./alerts.sh --cleanup 60

Custom Configuration

# Use custom config file
./alerts.sh --config /path/to/config.yml --create "Test Alert" "This is a test" low "test"

Troubleshooting

Common Issues

1. Permission Errors

   • Ensure the script is run with sudo
   • Check directory permissions
   • Verify user has necessary privileges

2. Notification Issues

   • Verify email/Slack configuration
   • Check network connectivity
   • Review notification logs

3. Alert Rule Issues

   • Check rule syntax
   • Verify metric names
   • Review threshold values

Debug Mode

To enable debug logging:

sudo DEBUG=1 ./alerts.sh [OPTIONS]