## HTTP(S) proxies and private CA chains

HTTP and HTTPS checks can use a corporate web proxy without extra Python packages. **Proxy use is opt-in**; the default `proxy.mode: none` ignores ambient proxy variables to keep checks deterministic.

- `proxy.mode: environment` uses the standard `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, and `NO_PROXY` variables (including lowercase variants), as supported by Requests. `NO_PROXY` bypasses the proxy for matching hosts.
- `proxy.mode: explicit` uses `proxy.http` / `proxy.https` or their safer environment-based forms `proxy.http_env` / `proxy.https_env`. Values must be HTTP(S) proxy URLs, e.g. `http://proxy.internal.example:3128`. Avoid credentials in YAML and command output; use environment variables if proxy credentials are necessary.
- `proxy.mode: none` (default) disables environment proxy inheritance.
- `tls.ca_file` accepts a path to a PEM CA bundle; `tls.ca_file_env` reads the PEM bundle path from an environment variable. Use these for internal/private PKI. Certificate verification stays enabled by default. The bundle must be readable and contain the trusted issuing chain (normally root CA and any needed intermediates).

Example environment:

```sh
export PROBE_HTTPS_PROXY='http://proxy.internal.example:3128'
export PROBE_CA_BUNDLE='/etc/ipsonic-probe/private-roots.pem'
python3 ipsonic-probe.py examples/118-https-explicit-proxy-private-ca.yaml
```

For conventional environment proxy discovery, set `HTTPS_PROXY` and optionally `NO_PROXY`, and use example 117. For direct private-CA verification and certificate expiry inspection, use example 120.

**Important limits:** Proxy routing applies to HTTP(S) requests, not TCP or DNS checks. `connection.resolve` cannot be combined with proxy mode because the proxy determines upstream routing. Detailed TLS certificate/version/cipher inspection (`min_valid_days`, `min_version`, `max_version`, `certificate`, `cipher`) currently uses a direct TLS socket, so it is **rejected when proxy mode is enabled** rather than silently bypassing the proxy. Through a proxy, normal HTTPS certificate chain and hostname verification still occurs via Requests and the configured CA bundle. For an HTTPS-intercepting proxy, the trusted CA must match the certificate chain presented by that proxy. A custom CA bundle is a **trust store**, not a client certificate for mutual TLS authentication.

The existing native Linux packages `python3`, `python3-requests`, and `python3-yaml` (Debian/Ubuntu) or `python3-pyyaml` (AlmaLinux/Rocky) remain sufficient. No `pip`, SOCKS extension, or external package source is required.

# ipsonic-probe — YAML-driven TCP and HTTP(S) validation

## Why ipsonic-probe exists — decades of operating experience, one simple idea

Operating systems and applications **for decades**, we have worked with many monitoring products and integration approaches. Powerful tools have their place, but too often a straightforward health check comes with a complex stack, numerous dependencies, costly integration and ongoing maintenance.

We built **ipsonic-probe** around the Unix philosophy: **KISS — Keep It Simple, Stupid.** A small CLI, plain YAML files, standard Linux distribution packages and meaningful exit codes should be enough to verify a service and pass the result to an existing monitoring system.

**Bridge development and operations:** developers describe expected application behavior in readable YAML; operators reuse and adjust those checks for deployment, acceptance, production monitoring, patching, upgrades and migrations. As an application changes through its lifecycle, its checks can evolve without introducing another monitoring platform to maintain.

**Complement, don't replace:** ipsonic-probe fits alongside Nagios, Icinga 2 and Checkmk. It is not a replacement for metrics, logging, tracing or a full monitoring platform. The goal is a focused, low-resource, low-dependency tool that does one job well.

A lightweight, YAML-driven network validation tool for **Linux**, including Debian, Ubuntu, AlmaLinux, and Rocky Linux. It supports TCP, HTTP(S), DNS, and monitoring integrations in connected or air-gapped environments. Installation uses **distribution-managed packages only**: no `pip`, PyPI, virtual environment, or third-party package repository is required.

## First test — TCP 127.0.0.1:22

New to ipsonic-probe? Follow the [Hello first TCP test guide](first-test.html) from release download and Linux package installation to your first local `tcp://127.0.0.1:22` check. The ready-to-run example is `examples/138-hello-first-tcp-ssh.yaml`.

## Best practices

See the [ipsonic-probe best-practice guide](best-practices.html) for reliable monitoring design, alert severities, secrets, private CAs, CI/CD integration, application lifecycle use and operational runbooks.

## Installation and release downloads

- [Raspberry Pi OS installation](raspberry-pi.html) — ARM-based Debian-family systems.
- [Ansible installation](ansible-installation.html) — optional host provisioning from an Ansible controller.
- [Release downloads](downloads/index.html) — versioned `tar.gz` release archives and SHA-256 checksums, generated by `tools/build-release.sh`.

The release includes a `requirements.txt` **documentation file** listing standard Linux distribution packages. It is not a pip requirements file: install dependencies using APT or DNF, never pip.

## Requirements and installation (Linux)

Runtime dependencies are Python 3, Requests, and PyYAML, all provided as **standard distribution packages**. Requests and PyYAML are Python libraries, but no external package source is necessary. The DNS implementation and website generator use the Python standard library.

### Debian / Ubuntu

Use the official distribution repositories or an approved internal APT mirror:

```sh
sudo apt-get update
sudo apt-get install -y python3 python3-requests python3-yaml
```

### Raspberry Pi OS (Raspbian)

Raspberry Pi OS is Debian-based. Install from the official Raspberry Pi OS repositories or an approved internal APT mirror:

```sh
sudo apt-get update
sudo apt-get install -y python3 python3-requests python3-yaml ca-certificates
```

The same packages work on supported ARM 32-bit and ARM 64-bit releases where available in the configured repository. No pip or external repositories are required. For automated host preparation, see [Raspberry Pi OS installation](raspberry-pi.html). For optional automated provisioning, see [Ansible installation](ansible-installation.html). Ansible runs on a separate controller, not on the probe host.

### AlmaLinux / Rocky Linux

Use the official distribution repositories or an approved internal DNF mirror:

```sh
sudo dnf install -y python3 python3-requests python3-pyyaml
```

### Air-gapped installation

Configure a local APT/DNF mirror containing the required packages **and their transitive dependencies**. On RPM-based systems, for example:

```sh
sudo dnf --disablerepo='*' --enablerepo='YOUR_LOCAL_REPO_ID' install -y \
  python3 python3-requests python3-pyyaml
```

On Debian/Ubuntu, configure an internal APT source and use `apt-get install` as above. If no mirror is available, prepare a complete set of matching DEB/RPM packages on a connected machine and transfer them through the approved process. The project ZIP contains no DEB/RPM files. Do not use `pip` or public repositories on an isolated host.

### Verify installation

```sh
python3 -c 'import requests, yaml, urllib3; print("OS packages OK")'
python3 ipsonic-probe.py --help
```

Probes need network connectivity only to their configured targets. No internet service is required by the framework itself.

## Why use ipsonic-probe for existing applications?

ipsonic-probe is not limited to new software projects. Application operators can monitor **already installed applications** from outside the target, without changing application code or installing an agent. This is particularly useful for legacy systems, commercial/vendor products, WordPress sites, internal portals, reverse-proxy frontends and private-PKI services.

- **Monitor the user-visible contract:** check an expected HTTP response, body marker, TLS trust and certificate expiry, DNS records and response time rather than only process state or port availability.
- **Start with one YAML:** capture a known-good baseline and reuse it after maintenance, patches, application upgrades, DNS changes and data-center migrations.
- **Fit existing operations:** execute from cron or a deployment script, or consume Nagios, Icinga 2 and Checkmk output with existing alerting. No replacement monitoring platform is required.
- **Avoid target changes:** probe from a monitoring host or approved network location. For internal services, configure HTTP(S) proxies and private CA trust where necessary.
- **Make handover easier:** keep small, reviewable YAML checks with application runbooks and version-control changes to expected behavior.
- **Help SecOps spot drift:** monitor certificate validity, HTTPS behavior, DNS answers and selected security headers as observable signals; these are not substitutes for vulnerability scans or a SIEM.
- **Keep the dependency inventory small:** use Linux distribution packages (`python3`, Requests and PyYAML), without pip, hosted services, databases, message brokers or a container orchestrator. The OS and transitive packages still belong in the SBOM.

### Incremental adoption for operators

1. Choose a business-critical installed application and identify its stable health or public URL.
2. Copy a relevant YAML from `examples/` and set the expected status, content, latency and TLS policy.
3. Run it from the actual monitoring network; verify proxy and private-CA settings if needed.
4. Review failure severity and expected maintenance behavior with the application owner.
5. Schedule the check through an existing monitoring platform or scheduler; retain YAML in version control.
6. Reuse the check after upgrades, infrastructure changes and disaster-recovery tests.

**Across the lifecycle:** development → CI/CD → deployment → technical acceptance → operations → security monitoring → upgrade/migration → retirement. Use ipsonic-probe as a focused assertion runner alongside, not instead of, full observability and security tools.

## Microsoft Exchange infrastructure monitoring

Use the [Exchange infrastructure monitoring guide](exchange-monitoring.html) and examples `121`–`137` to monitor OWA, ECP, Autodiscover, EWS, MAPI, ActiveSync, SMTP, IMAP, POP3, mail DNS and Exchange HTTPS backends. These are layered reachability and HTTPS/DNS checks, not authenticated mailbox or end-to-end mail-delivery tests. Enable only services present in your deployment.

## Web project acceptance guide

For a step-by-step customer acceptance workflow, see [WEB-PROJECT-ACCEPTANCE.md](WEB-PROJECT-ACCEPTANCE.md) and [example 91](examples/91-web-project-acceptance.yaml). The guide explains requirements, executable YAML acceptance criteria, evidence collection, sign-off, and operational reuse.

## Quick start

```bash
/usr/bin/python3 ipsonic-probe.py examples/02-http-get-status-allow.yaml
/usr/bin/python3 ipsonic-probe.py /etc/httptest/*.yaml
printf 'exit code: %s\n' "$?"
```

Examples are **templates, not an immediately runnable test suite**: change example hostnames, URLs, paths, expected response contents, and credentials to match your service. `.invalid` hostnames are intentionally nonresolvable placeholders. Do not run the whole `examples/*.yaml` directory without adapting it.

Exit status: **0** when every configuration passes; **1** if any configuration fails or raises an error; argparse may return **2** for invalid CLI usage.

## Configuration reference

```yaml
name: Example name                   # optional, informational
url: https://service.example.invalid/health  # required; tcp://host:port also accepted
request:                              # HTTP(S) only; optional
  method: GET                         # GET or POST, default GET
  user_agent: httptest/1.0            # optional
  headers:                            # optional request headers
    Accept: application/json
  json:                               # optional JSON-serializable POST payload
    check: ping
  # body: 'raw request body'          # alternative to json
  follow_redirects: false             # default false

auth:                                 # optional, HTTP(S) only
  type: bearer                        # none | basic | bearer
  token_env: HTTPTEST_API_TOKEN       # or token: literal
  # For basic: username/password or username_env/password_env

timeouts:
  connect: 3                          # seconds, default 3
  response: 10                        # seconds, default 10

tls:                                  # HTTPS only
  verify: true                        # default true
  min_valid_days: 30                  # optional, only checked when verify=true

tests:                                # all assertions optional
  status:
    allow: [200, 204]                 # if provided, status must be in list
    deny: [401, 403, 500]            # if provided, status must not be in list
  headers:
    exists: [Content-Type]            # header names are case-insensitive
    absent: [X-Debug]
    contains:                          # each listed substring must appear
      Content-Type: [application/json]
    not_contains:                      # none of these substrings may appear
      Server: [nginx/1.14]
  body:
    contains: ['"status"']            # case-sensitive substring checks
    not_contains: [traceback]
```

### Assertion semantics

- All specified assertions are combined using **AND**. Every failed assertion is printed.
- `status.allow` and `status.deny` can be used independently or together. An empty `allow` list rejects every status.
- Header **names** are case-insensitive; header **values** are checked by case-sensitive substring match. `contains` on a missing header fails; `not_contains` on a missing header passes unless `exists` is also set.
- Body assertions operate on decoded response text (`requests.Response.text`), not raw bytes, JSON paths, regular expressions, or exact equality.
- HTTP errors such as 404 are only failures if the configured assertions reject them. Without status assertions, receiving an HTTP response is sufficient.
- A `tcp://` URL only checks whether a TCP connection can be established; HTTP request, auth, TLS, and response assertions do not apply.

### Authentication

`auth.type: basic` requires exactly one of `username` / `username_env` and one of `password` / `password_env`. `auth.type: bearer` requires exactly one of `token` / `token_env`. Missing/empty environment variables fail the configuration. Do **not** define a manual `Authorization` request header together with `auth.type: basic` or `bearer`.

```bash
export HTTPTEST_USER='monitor'
export HTTPTEST_PASSWORD='example-secret'
export HTTPTEST_API_TOKEN='example-token'
/usr/bin/python3 ipsonic-probe.py examples/20-http-bearer-env.yaml
```

Use secret injection (for example Ansible Vault plus `no_log: true` on sensitive tasks). Avoid literal credentials in version control. Bearer tokens and Basic credentials should normally be sent only over HTTPS.

### TLS behavior

- `tls.verify: true` uses the OS/Requests trust configuration for certificate chain and hostname validation.
- `tls.verify: false` disables both certificate chain and hostname validation. **Only for isolated test systems**, never as a production default.
- `tls.min_valid_days` performs an **additional TLS handshake** to inspect the leaf certificate expiry; this requires `verify: true`. The script uses the system trust store for this check. The extra handshake is separate from the measured HTTP response duration.
- For internal CAs, deploy the CA into the OS trust store using your organization's trust-management procedure (typically `update-ca-trust`), and ensure the Python Requests CA bundle is configured appropriately. On RPM installations, confirm the effective CA path with `python3 -c 'import requests; print(requests.certs.where())'`. No per-YAML custom CA file option is implemented.

### Timeout behavior and limitations

- `timeouts.connect` sets the socket connection timeout used by Requests and the TCP check.
- `timeouts.response` is the maximum wait for the **HTTP operation from the caller's perspective**, implemented with a worker thread and a deadline. On timeout the CLI returns an error without waiting for the worker. **The underlying request/thread is not forcibly terminated**, so this is not a hard socket cancellation guarantee.
- Requests additionally uses `response` as its read-inactivity timeout. DNS resolution, the separate certificate-expiry handshake, and some OS socket operations may not obey the same overall deadline.
- A TCP connection is timed using `connect`; its measured elapsed time is also compared with `response`.
- For strict wall-clock enforcement across the entire process, use an external supervisor such as `timeout 15s /usr/bin/python3 ipsonic-probe.py ...` (GNU coreutils).

### Output example

```text
PASS /etc/httptest/health.yaml (0.081s)
FAIL /etc/httptest/api.yaml (0.142s)
  - HTTP 503 not in allowed statuses [200]
  - Body does not contain 'healthy'
ERROR /etc/httptest/tcp.yaml: ConnectionRefusedError: [Errno 111] Connection refused
```

### Ansible example

```yaml
- name: Run httptest checks
  ansible.builtin.command:
    argv:
      - /usr/bin/python3
      - /opt/httptest/httptest.py
      - /etc/httptest/api.yaml
  environment:
    HTTPTEST_API_TOKEN: "{{ vault_api_token }}"
  register: httptest_result
  changed_when: false
  no_log: true  # prevents sensitive environment values leaking to logs
```

To retain non-sensitive output for monitoring, separate the secret-bearing execution task from a safe summary/reporting step. `no_log: true` hides the registered output in normal Ansible logs.

## Example catalog

All files are located in [`examples/`](examples/). These are **representative feature and interaction combinations**, not a Cartesian product of every possible parameter/value. You can freely combine independent assertions in a single file; see examples 11, 25, and 26.

- `01-tcp-connect.yaml` — TCP connect
- `02-http-get-status-allow.yaml` — GET allowed status codes
- `03-http-get-status-deny.yaml` — GET denied status codes
- `04-http-status-allow-and-deny.yaml` — Combined status acceptance and rejection
- `05-http-header-exists.yaml` — Required response headers
- `06-http-header-absent.yaml` — Forbidden response headers
- `07-http-header-contains.yaml` — Required response header substrings
- `08-http-header-not-contains.yaml` — Forbidden response header substrings
- `09-http-body-contains.yaml` — Required body substrings
- `10-http-body-not-contains.yaml` — Forbidden body substrings
- `11-http-headers-body-combined.yaml` — Combined HTTP acceptance checks
- `12-http-post-json.yaml` — POST JSON payload
- `13-http-post-form.yaml` — POST form-encoded payload
- `14-http-user-agent-custom-headers.yaml` — Custom User-Agent and request headers
- `15-http-follow-redirects.yaml` — Follow redirects
- `16-http-do-not-follow-redirects.yaml` — Inspect redirect response
- `17-http-basic-auth-literal.yaml` — HTTP Basic auth literal (demo only)
- `18-http-basic-auth-env.yaml` — HTTP Basic auth from environment
- `19-http-bearer-literal.yaml` — Bearer token literal (demo only)
- `20-http-bearer-env.yaml` — Bearer token from environment
- `21-https-verify-secure.yaml` — Verify TLS certificate and hostname
- `22-https-insecure.yaml` — Disable TLS validation for isolated preprod ONLY
- `23-https-min-valid-days.yaml` — Verify TLS and require 30 days certificate validity
- `24-http-timeouts.yaml` — Short connection and total response deadline
- `25-https-post-bearer-full.yaml` — Full HTTPS POST with bearer token and assertions
- `26-https-post-basic-full.yaml` — HTTPS POST with Basic authentication
- `27-http-no-assertions.yaml` — HTTP connectivity only
- `28-https-insecure-bearer.yaml` — Insecure preprod HTTPS with bearer auth

## Operational notes

- The tool intentionally makes **one HTTP request per YAML file**, plus an extra TLS handshake if certificate lifetime is checked.
- A URL can include a custom port. No `curl --resolve`-style DNS override or SNI override is implemented.
- Redirects are disabled by default. `follow_redirects: true` follows them. Review redirect destinations carefully when using credentials.
- The HTTP client disables environment proxy configuration (`session.trust_env = False`), which also affects some Requests environment settings.
- This tool does not currently provide JSON/JUnit output, regex/JSONPath assertions, client certificates, or retries.
- Keep test YAML files access-controlled if they contain credentials.


## Extended HTTP/HTTPS checks (version 2)

The following checks use the existing Python `requests`, `urllib3` (a dependency of requests), and `PyYAML` RPM packages. No pip installation or outbound internet access is required. Examples `29`–`36` illustrate the additions.

### Per-request DNS override (preproduction)

```yaml
url: https://api.example.com/health
connection:
  resolve:
    api.example.com: 10.20.30.40
```

Connects directly to the supplied IP while retaining the original URL hostname for the HTTP Host header, TLS SNI, and certificate hostname verification. Only a single mapping matching the URL hostname is supported. Overrides apply to the initial target; **redirect destinations are not automatically overridden**. Proxies inherited from the environment are disabled. IPv4 and IPv6 literals are accepted. TLS certificate validity is still checked against the URL hostname.

### Redirect assertions

```yaml
request:
  follow_redirects: true
tests:
  redirects:
    max_count: 2
    require_https: true
    allowed_hosts: [example.com, www.example.com]
    final_url: https://www.example.com/new
```

`max_count` checks followed redirects; `require_https` checks all URLs in the followed chain; `allowed_hosts` checks the hostname of every URL in the chain. `final_url` compares the resulting URL exactly. With `follow_redirects: false`, checks only observe the initial response, and `max_count` will be zero. Redirect checks do not prevent a redirect before it occurs: do not enable following redirects to untrusted hosts with credentials.

### Response metadata

```yaml
tests:
  response:
    min_bytes: 20
    max_bytes: 1048576
    content_type: application/json
```

Size is the length of the downloaded/decoded response bytes. Content type matches the media type case-insensitively, ignoring parameters such as charset. This is a post-download assertion, **not a streaming download-size cap**.

### Structured JSON assertions

```yaml
tests:
  json:
    paths:
      - {path: status, equals: healthy}
      - {path: database.connected, equals: true}
      - {path: services, min_items: 2}
      - {path: version, exists: true}
      - {path: error, exists: false}
```

Paths are dot-separated dictionary keys; numeric segments address list indexes. This is a simple path selector, **not full JSONPath**. `equals` compares both JSON type and value; `min_items` / `max_items` apply to lists or objects. Keys containing dots are not supported.

### Cookies and HSTS

```yaml
tests:
  cookies:
    require:
      sessionid:
        secure: true
        httponly: true
        samesite: Lax
  security:
    hsts_min_age: 31536000
```

Cookie rules check the named cookie's attributes on the final response. HSTS parses numeric `max-age` from the final response's Strict-Transport-Security header. Cookie attributes are parsed using Python's `http.cookies.SimpleCookie`; unusual or non-standard cookie syntax may not be recognized.

### Linux air-gapped deployment

```sh
dnf --disablerepo='*' --enablerepo=YOUR_INTERNAL_REPO install -y python3 python3-requests python3-pyyaml
python3 httptest.py examples/36-combined-preprod.yaml
python3 httptest.py examples/*.yaml
```

Use the appropriate standard packages for your distribution (`python3-yaml` on Debian/Ubuntu; `python3-pyyaml` on AlmaLinux/Rocky Linux). The archive does not bundle DEB/RPM packages. Validate the installed Python and urllib3 versions. Example hostnames and addresses are placeholders.

### Limitations

The total request deadline uses a background worker; it bounds the CLI's waiting time but does not guarantee immediate termination of the underlying socket operation. TLS min-valid-days uses a separate certificate handshake. Redirect chains, JSON paths, and cookies are evaluated after the request completes. No direct TLS socket inspection, cipher auditing, OCSP, or HTTP/2 support has been added.


## Extended checks (v2)

All features below are implemented in `ipsonic-probe.py`. See `examples/29-*` through `examples/40-*` for individual and combined YAML files. One YAML file describes one request and multiple assertions (logical AND).

### Direct target IP / curl --resolve equivalent

```yaml
url: https://api.example.invalid/health
connection:
  resolve:
    api.example.invalid: 10.20.30.40
```

This maps the URL hostname to a literal IPv4/IPv6 destination without modifying DNS or `/etc/hosts`. The HTTP Host header, TLS SNI and certificate hostname remain the URL hostname. Proxy environment variables are ignored (`Session.trust_env = False`). The mapping must contain exactly the URL hostname. **Automatic redirects cannot be combined with `connection.resolve`** to prevent a redirected connection from silently bypassing the override. Use separate YAML files to test redirects in this situation. This override is for HTTP(S), not TCP.

### TLS version, certificate and cipher assertions

```yaml
tls:
  verify: true
  ca_file: /etc/pki/httptest/internal-ca.pem # optional custom CA bundle
  min_valid_days: 30
  min_version: TLSv1.2
  max_version: TLSv1.3
  certificate:
    issuer_contains: Example CA
    subject_contains: api.example.invalid
    san_contains: [api.example.invalid]
    # fingerprint_sha256: '64 lowercase hex characters'
  cipher:
    allow: [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256]
    deny: [AES128-SHA]
```

TLS checks establish a **separate handshake** before the HTTP request; they validate that connection's negotiated version, cipher and peer certificate. `min_version`/`max_version` restrict the probe's TLS context; the actual HTTP connection may negotiate independently. `cipher.allow` and `cipher.deny` compare the OpenSSL negotiated cipher name (not IANA names). Fingerprint is SHA-256 of DER-encoded leaf certificate, compared case-insensitively with optional colon separators. SAN checks are exact matches for DNS SAN entries. Issuer/subject are substring checks on flattened distinguished names. CA trust and hostname validation use the OS/OpenSSL trust configuration or `ca_file`. `min_valid_days`, issuer/subject/SAN assertions require `verify: true`; fingerprint also works when insecure. **Certificate revocation / OCSP, all supported TLS versions, and exhaustive server cipher enumeration are not tested.** A minimum-version assertion demonstrates that a connection can negotiate at or above the configured minimum; it does not prove the server rejects older protocols.

### Redirect policy

```yaml
request:
  follow_redirects: true
tests:
  redirects:
    max_count: 2
    require_https: true
    allowed_hosts: [api.example.invalid]
    final_url: https://api.example.invalid/health
```

Redirect assertions inspect `response.history` and the final URL. `require_https` checks every URL in the chain, including the initial URL. `allowed_hosts` checks all visited URL hostnames. Do not enable redirects to untrusted hosts when sending credentials. Redirect behavior is not a substitute for an open-redirect security scan.

### Response size and MIME type

```yaml
tests:
  response:
    min_bytes: 10
    max_bytes: 1048576
    content_type: application/json
```

Byte length is measured after Requests content decoding (including gzip decompression). MIME comparison ignores parameters such as `charset=UTF-8` and is case-insensitive. Response download is buffered; `max_bytes` is an assertion **after** download, not a streaming memory cap.

### Structured JSON paths

```yaml
tests:
  json:
    paths:
      - {path: status, equals: healthy}
      - {path: database.connected, equals: true}
      - {path: services.0.name, exists: true}
      - {path: services, min_items: 2, max_items: 10}
      - {path: error, exists: false}
```

Path syntax is simple dot-separated object keys and zero-based array indices, **not full JSONPath**: no filters, wildcards or escaping dots in keys. `equals` uses exact JSON value and type comparison (boolean is distinct from integer). Missing paths fail unless `exists: false` is requested.

### Cookie flags and HSTS

```yaml
tests:
  cookies:
    require:
      session:
        secure: true
        httponly: true
        samesite: Strict
  security:
    hsts_min_age: 31536000
```

Cookie assertions inspect `Set-Cookie` headers on the **final** response. `Secure`, `HttpOnly` and `SameSite` are checked per named cookie. HSTS verifies the numeric `max-age` directive, not preload eligibility or subdomain inclusion. Cookie parsing uses Python `http.cookies.SimpleCookie`; malformed or unusual cookie headers may not parse as expected.

### Air-gapped Linux constraints

Runtime dependencies remain **only** OS-managed packages: `python3`, `python3-requests`, and `python3-yaml` (Debian/Ubuntu) or `python3-pyyaml` (AlmaLinux/Rocky Linux), plus their distribution-managed dependencies. Python standard library provides TLS, socket, JSON, hashing and cookie parsing. No pip, `cryptography`, `jsonpath-ng`, `httpx`, external `openssl` command or online service is required. Check `/usr/bin/python3` and `/usr/bin/python3 -c 'import requests, yaml, urllib3'` on the target. The archive contains source code, examples and tests **but no DEB/RPM packages**. Use an internal repository or transfer dependency-complete packages matching the OS release and CPU architecture.

Run offline unit tests:

```bash
/usr/bin/python3 -m unittest discover -s tests -v
```

### Operational limitations

`timeouts.response` is an outer deadline for the CLI result implemented using a worker thread. When the deadline expires, the thread may continue until the socket's own timeout; it is **not** a guaranteed immediate cancellation of the underlying socket. The TLS preflight uses a separate connection and is not included in the measured HTTP elapsed time. Avoid pointing these examples at production systems until URLs, secrets and policies are reviewed. TLS probe supports TLS 1.2 and 1.3 configuration with the OpenSSL library supplied by the Linux distribution.


## Monitoring integration (Nagios, Icinga 2, Checkmk)

Each YAML file can specify a `monitoring` mapping. The command-line `--output` option overrides `monitoring.output` for every supplied file. Supported values: `text` (default), `nagios`, `icinga2`, `checkmk`, `json`. Output formatting does not change the assertions or connection behavior.

```yaml
monitoring:
  service_name: "API Health"
  output: nagios
  thresholds:
    response_time:
      warning: 2      # seconds, strictly greater => WARNING
      critical: 5     # seconds, strictly greater => CRITICAL
```

Thresholds are optional, numeric, positive and finite. When both are set, warning must be less than critical. **Failed assertions take precedence over latency warnings and produce CRITICAL**. Invalid YAML, invalid configuration, connection errors, and timeouts produce UNKNOWN. A configured response-time threshold is evaluated against the HTTP operation measured by the engine; the separate TLS inspection handshake is not included. `timeouts.response` remains an independent operation deadline; a deadline error is UNKNOWN rather than a latency threshold violation.

| Code | Status | Meaning |
|---|---|---|
| 0 | OK | All assertions pass, latency below thresholds |
| 1 | WARNING | Assertions pass, warning response-time threshold exceeded |
| 2 | CRITICAL | Assertion failed or critical response-time threshold exceeded |
| 3 | UNKNOWN | Configuration, network, timeout or internal error |

Nagios and Icinga 2 use the conventional plugin output with performance data:

```text
OK - API Health: all assertions passed | response_time=0.184s;2;5;0;
```

Checkmk output follows the **local-check** format (one line per service):

```text
0 "API Health" response_time=0.184s;2;5;0; all assertions passed
```

For Checkmk, the leading state field is authoritative. Deploy the command as a local check under the appropriate agent local-check directory and run it with `--output checkmk`. Avoid using `--output json` for a Checkmk local check. Checkmk performance-data parsing and unit handling should be verified with your installed Checkmk version. The JSON mode emits one JSON object per file (JSON Lines), not one combined JSON array.

```bash
/usr/bin/python3 ipsonic-probe.py --output nagios examples/41-monitoring-nagios.yaml
/usr/bin/python3 ipsonic-probe.py --output icinga2 examples/41-monitoring-icinga2.yaml
/usr/bin/python3 ipsonic-probe.py --output checkmk examples/41-monitoring-checkmk.yaml
/usr/bin/python3 ipsonic-probe.py --output json examples/41-monitoring-json.yaml
```

**Nagios / Icinga 2:** invoke a single YAML file per monitored service so that each service has its own status and performance data. When passing multiple YAML files, the CLI prints one result line per file and returns the maximum status code; this is suitable for batch use but is not a single-service Nagios plugin response. **Checkmk:** multiple files may be used to generate multiple local-check lines, but service names must be unique. Do not mix output modes in one Checkmk local-check invocation.

`service_name` is optional and defaults to `name` or `ipsonic-probe`. Keep names unique and free of quotes for Checkmk integration. See `examples/41-monitoring-*.yaml`, `42-monitoring-tcp.yaml`, `43-monitoring-thresholds.yaml`, and `44-monitoring-no-thresholds.yaml`.

## Packaging

The distribution provides `ipsonic-probe.py` and a compatibility copy named `httptest.py`. Both run on system Python with distribution-managed Requests and PyYAML; no pip or external package repositories are required. The archive contains no DEB/RPM binaries.

## Assertion failure severity (`monitoring.on_failure`)

Each YAML configuration represents one test service. Set `monitoring.on_failure` to control the monitoring status when **one or more assertions fail**. The default is `critical`.

```yaml
monitoring:
  service_name: "API Health"
  output: icinga2       # text, nagios, icinga2, checkmk, json
  on_failure: warning   # info, warning, critical (default: critical)
  thresholds:
    response_time:
      warning: 2
      critical: 5
```

| `on_failure` | Plugin status | Exit code | Notes |
|---|---|---|---|
| `info` | OK | 0 | Failure details remain visible with `INFO -` prefix; Checkmk Local Check uses state 0 |
| `warning` | WARNING | 1 | Assertion failure raises warning |
| `critical` (default) | CRITICAL | 2 | Assertion failure raises critical |

This option applies to **all assertions in the YAML file**, not individual assertion expressions. For different severities, use separate YAML files/services. Network, TLS handshake, timeout, and configuration exceptions remain `UNKNOWN` (3) rather than being downgraded. Response-time monitoring thresholds remain independent and may raise WARNING/CRITICAL if no assertion failed. `info` is a descriptive OK state, not a fifth Nagios/Icinga status.

## Per-assertion monitoring severity

`monitoring.assertions` provides **ordered, per-failure severity overrides**.
Each rule matches a literal, case-sensitive substring of the generated failure
message (`contains`). The first matching rule wins. If no rule matches,
`monitoring.on_failure` applies; its default is `critical`.

```yaml
monitoring:
  output: icinga2
  service_name: API Health
  on_failure: critical
  assertions:
    - contains: "Missing header: X-Request-ID"
      severity: info
    - contains: "Body does not contain 'healthy'"
      severity: warning
    - contains: "HTTP 503"
      severity: critical
```

A single failed assertion can therefore be INFO (Nagios/Icinga OK exit 0),
WARNING (exit 1), or CRITICAL (exit 2). The highest severity of all failures
and independently evaluated response-time thresholds determines the final state.
Errors executing the check remain UNKNOWN (exit 3). Individual failure
severities are included in JSON as `assertion_results` and are prefixed in
human-readable monitoring output. Matching is based on diagnostic text, so
review rules after upgrading the engine; rules are not a general expression
language. Avoid matching broad fragments that may affect unrelated checks.

## Static documentation site (offline)

The static documentation website is generated from the **current `README.md`** and **every YAML file under `examples/`**. It is documentation, not a live monitoring dashboard. No pip packages, Markdown tools, JavaScript libraries, CDN, network access, or external assets are needed.

```sh
./tools/generate-site.sh
# or choose an explicit separate directory:
./tools/generate-site.sh --output /srv/www/ipsonic-probe-docs
```

The default output directory is `site-output/` at the project root. It contains `index.html` (rendered README), `README.md` (original source), `examples/index.html` (example catalog), one HTML page per YAML example, `downloads/examples/` (unmodified YAML files), and `assets/style.css`.

Serve the generated output directory as static files with an existing Apache/nginx installation, or open `index.html` directly. The generator refuses to overwrite an existing directory unless it contains its own `.ipsonic-generated` marker; subsequent runs regenerate it from source. **Do not edit generated files**, as they will be replaced. Markdown rendering supports common headings, lists, tables, code fences, links and emphasis; it is intentionally a lightweight renderer, not a full CommonMark implementation.

## DNS record validation (A, AAAA, CNAME, TXT)

`ipsonic-probe` supports DNS-only YAML files (no `url` field). It sends DNS wire-protocol queries using Python's standard library; **no dnspython or pip installation is required**. DNS checks use the same monitoring outputs, response-time thresholds, `monitoring.on_failure`, and per-assertion severity matching as HTTP checks.

```yaml
name: Internal DNS

dns:
  name: api.example.com
  nameserver: 10.10.10.53  # optional IP literal; IPv4 or IPv6
  port: 53                 # optional, default 53
  transport: auto          # auto (UDP with TCP truncation fallback), udp, tcp
  timeout: 3               # seconds
  records:
    - type: A
      exists: true
      equals: [10.20.30.40]
      not_equals: [127.0.0.1]
      min_count: 1
      max_count: 4
      min_ttl: 60
    - type: AAAA
      exists: false
    - type: TXT
      contains: ["v=spf1"]
      not_contains: ["+all"]
monitoring:
  service_name: Internal DNS
  output: icinga2
  on_failure: critical
```

### DNS configuration reference

| Key | Meaning |
| --- | --- |
| `dns.name` | Required domain name to query |
| `dns.nameserver` | Optional explicit resolver IPv4/IPv6 **address**; if omitted, the first `nameserver` in `/etc/resolv.conf` is used |
| `dns.port` | DNS server port, default `53` |
| `dns.transport` | `auto` (default), `udp`, or `tcp`; auto retries truncated UDP responses over TCP |
| `dns.timeout` | Socket timeout in seconds, default `timeouts.connect` or `3` |
| `dns.records` | Required list of independent record checks |
| `type` | `A`, `AAAA`, `CNAME`, or `TXT` |
| `rcode` | Expected DNS response code, default `NOERROR` (e.g. `NXDOMAIN`) |
| `exists` | `true` requires at least one matching record; `false` requires none |
| `equals` | Every listed value must appear as a complete matching record (CNAME trailing dots ignored) |
| `not_equals` | None of the listed values may appear as complete matching records |
| `contains` | Every listed substring must occur in at least one matching record |
| `not_contains` | No matching record may contain a listed substring |
| `min_count`, `max_count` | Number of answer records of the requested type and owner |
| `min_ttl` | Minimum TTL in seconds for returned records of the requested type |

The DNS client requests one record type per `dns.records` entry. A CNAME check tests the CNAME response itself; it does not recursively chase aliases. TXT character-string fragments belonging to one TXT record are concatenated before assertion checks. Comparison of record content is case-sensitive except for DNS owner names. DNS response codes such as `NXDOMAIN` are assertions; socket errors, timeouts and malformed responses result in UNKNOWN.

**Air-gapped operation:** DNS queries only target the selected nameserver, with no HTTP requests or external dependencies. An explicit resolver must be an IP literal to avoid accidental DNS bootstrap queries to another resolver. Examples use documentation-only IPs and domains and must be adapted. DNSSEC validation, DoH, DoT, SRV, MX and iterative resolution are not implemented.

Examples: `50-dns-a.yaml` through `57-dns-multiple-records.yaml`. Run them using `python3 ipsonic-probe.py examples/54-dns-custom-nameserver.yaml`. Regenerate the static documentation website with `./tools/generate-site.sh`.


## Extended DNS records and execution controls

Supported DNS resource record types: `A`, `AAAA`, `CNAME`, `TXT`, `MX`, `NS`, `SOA`, `PTR`, `SRV`, `CAA`, `DS`, `DNSKEY`, `RRSIG`. The DNS wire implementation uses the Python standard library; no `dnspython` or `pip` is needed.

Records are represented as normalized text for `equals`, `contains`, and other existing assertions:

- `MX`: `priority hostname.`
- `SOA`: `mname. rname. serial refresh retry expire minimum`
- `SRV`: `priority weight port target.`
- `CAA`: `flags tag value`
- `DS`: `keytag algorithm digesttype HEXDIGEST`
- `DNSKEY`: `flags protocol algorithm BASE64KEY`
- `RRSIG`: `typecovered algorithm labels originalttl expiration inception keytag signer. HEXSIGNATURE`

**DNSSEC limitation:** `dnssec: true` sets EDNS DO and enables inspection of DNSKEY, DS and RRSIG data. `authenticated_data: true` checks the AD flag reported by the selected recursive resolver. This is **not independent cryptographic DNSSEC chain validation**. Use a trusted validating resolver when depending on AD; DNSSEC failures may surface as SERVFAIL. No external resolver is contacted unless explicitly configured.

### Compare nameservers

```yaml
dns:
  name: api.example.com
  compare_nameservers: [192.0.2.53, 192.0.2.54]
  require_identical: true
  records:
    - type: A
      exists: true
      max_response_time: 2.0
```

The comparison checks RCODE and sorted, unique answer values; TTLs are deliberately ignored. Different answers from load-balanced DNS may legitimately differ.

### Retries and multiple backends

```yaml
execution:
  retries: 2
  retry_delay: 1
  success_required: 2
  backends:
    - name: node-a
      resolve: {api.example.com: 192.0.2.10}
    - name: node-b
      resolve: {api.example.com: 192.0.2.11}
```

`retries` means additional attempts; default 0. `success_required` counts successful attempts per backend (default 1). All backends must satisfy their success quota. Backends may override `url` or `resolve` for HTTP, or `nameserver` for DNS. Attempts execute sequentially. With retries configured, connection errors are treated as unsuccessful attempts; without `execution`, legacy UNKNOWN handling is retained. Avoid configuring excessive retry counts in latency-sensitive monitoring checks.

### Stable assertion category IDs

```yaml
monitoring:
  on_failure: critical
  assertions:
    - id: dns.a.comparison
      severity: warning
    - id: http.body
      severity: info
```

Rules accept exactly one of `id` or legacy `contains`, plus `severity`. IDs currently identify **assertion categories**, not each individual YAML assertion. DNS identifiers include `dns.<recordtype>.record`, `.rcode`, `.ttl`, `.comparison`, `.dnssec_ad`, `.response_time`. HTTP categories include `http.status`, `http.body`, `http.header`, `response.time` and generic `probe.assertion`. The first matching severity rule wins; unmatched failures use `monitoring.on_failure` (default `critical`). JSON output exposes assertion IDs.

### Air-gapped Linux

Install `python3`, `python3-requests`, and the distribution YAML package (`python3-yaml` on Debian/Ubuntu; `python3-pyyaml` on AlmaLinux/Rocky Linux) from approved local repositories, including dependencies. No `pip`, runtime internet access, or third-party DNS module is required. DNS nameserver addresses must be IP literals. Examples use documentation-only IP ranges and must be adapted.

### Generate the static documentation site

```sh
./tools/generate-site.sh --output ./site-output
```

The site generator reads this README and the YAML examples, and writes its output to a separate directory.


## Extended configuration examples (71–90)

These additional YAML templates illustrate the extended DNS record types, DNSSEC metadata, resolver comparison, retry success quotas, independent backends, stable assertion-category severity IDs and monitoring outputs. All IP addresses in `192.0.2.0/24` are documentation placeholders. Replace names, records and addresses before running checks. `authenticated_data` trusts the resolver AD flag and is not independent DNSSEC cryptographic validation.

| Example | Coverage |
| --- | --- |
| `71-dns-mx-exact.yaml` | MX exact record and forbidden host |
| `72-dns-ns-exact.yaml` | NS exact authoritative server |
| `73-dns-soa-serial.yaml` | SOA serial substring check |
| `74-dns-ptr-negative.yaml` | PTR reverse lookup negative assertion |
| `75-dns-srv-exact.yaml` | SRV priority weight and port |
| `76-dns-caa-forbidden.yaml` | CAA permitted and forbidden issuers |
| `77-dns-ds.yaml` | DNSSEC DS record inspection |
| `78-dns-dnskey.yaml` | DNSSEC DNSKEY record inspection |
| `79-dns-rrsig.yaml` | DNSSEC RRSIG inspection |
| `80-dns-ad-flag.yaml` | Trusted resolver AD flag check |
| `81-dns-compare-relaxed.yaml` | Compare resolvers without requiring identical answers |
| `82-dns-compare-strict.yaml` | Strict resolver comparison with severity |
| `83-dns-udp.yaml` | UDP DNS transport |
| `84-dns-tcp.yaml` | TCP DNS transport |
| `85-dns-retry-quorum.yaml` | DNS retries requiring two successes |
| `86-dns-multi-backend-retry.yaml` | DNS two resolvers with retries |
| `87-http-multi-backend-retry.yaml` | HTTP backend pool with retries |
| `88-monitoring-dns-ids.yaml` | DNS stable assertion ID severity rules |
| `89-monitoring-default-output.yaml` | Default text output (no monitoring.output) |
| `90-monitoring-all-outputs.yaml` | Nagios output selection (CLI may override) |

`monitoring.output` supports `text` (default), `json`, `nagios`, `icinga2`, and `checkmk`. CLI `--output` overrides YAML. To test all formats against the same YAML, run `python3 ipsonic-probe.py --output FORMAT examples/90-monitoring-all-outputs.yaml` with each supported FORMAT.

## WordPress website checks

Start with [`92-wordpress-homepage.yaml`](examples/92-wordpress-homepage.yaml) for a public WordPress homepage smoke test. It checks HTTPS, certificate validity, successful HTML response, redirect policy, response time, and two common WordPress failure messages. Replace `www.example.com` / `example.com` and the allowed hosts with your actual domain.

Optional checks: [`93-wordpress-rest-api.yaml`](examples/93-wordpress-rest-api.yaml) tests `/wp-json/` only when the REST API is intended to be publicly available; [`94-wordpress-login-page.yaml`](examples/94-wordpress-login-page.yaml) checks `/wp-login.php` only when the standard login endpoint is intentionally exposed. Hardened sites may disable or rename either endpoint. These are **separate YAML files** so a customer can explicitly choose which endpoints are contractual acceptance criteria.

Run one check with `python3 ipsonic-probe.py examples/92-wordpress-homepage.yaml`, or multiple files by passing each path. No login credentials, administrative actions, or WordPress plugins are needed. These are availability and response checks, not proof that WordPress is patched, secure, accessible, or functionally correct.


## Combined real-world scenarios (examples 95–109)

These YAML files demonstrate practical combinations of existing assertions rather than isolated features. All hostnames and addresses are **documentation placeholders**; adapt them to your environment, security policy, and acceptance contract before execution.

| Example | Combined checks | Intended use |
| --- | --- | --- |
| 95 Public web security baseline | HTTPS, HSTS, headers, body, latency | Web operations / SecOps |
| 96 API readiness contract | JSON, status, payload size, TLS, latency | Application engineering |
| 97 Authenticated API health | Bearer token from environment, JSON, status | Private API monitoring |
| 98 Load balancer backend parity | Per-backend resolution, retries, TLS, body | Platform engineering |
| 99 WordPress production hardening | WordPress errors, HTTPS, HSTS, headers | WordPress operations |
| 100 Customer staging acceptance | Status, TLS, content, latency | Customer acceptance |
| 101 Mail DNS MX and SPF | MX and TXT/SPF presence | Mail infrastructure |
| 102 DNS resolver consistency | Two resolvers, A records, latency | DNS operations |
| 103 DNSSEC publication metadata | DNSKEY and RRSIG presence | SecOps / DNS |
| 104 TLS expiry and HSTS | Certificate age, HTTPS, HSTS | SecOps |
| 105 CDN edge asset | CSS content type, size, TLS, latency | Frontend operations |
| 106 Maintenance page detection | HTTP 200 plus body denylist | Incident detection |
| 107 Canonical HTTPS redirects | Redirect count, approved hosts, HTTPS | Web infrastructure |
| 108 Internal service health | Private IP, health body, latency | Internal operations |
| 109 Security header baseline | HSTS, CSP, nosniff, debug header | SecOps drift detection |

**Important:** The probe runs one configured target per YAML. For a complete customer journey or infrastructure acceptance, combine multiple YAML files and archive their outputs. A presence check is not equivalent to a security audit, a full mail-delivery test, or cryptographic DNSSEC validation. Mandatory acceptance checks should use CRITICAL severity; the project owner must still sign off on the evidence.

## Example descriptions

The [example catalog](examples/index.html) includes a plain-English explanation for every YAML file, including the checks performed and usage notes. Descriptions are maintained in `examples-descriptions.json` and rendered into each static HTML example page; no runtime dependency is added.


## Maintainer and disclaimer

Maintained by **mati peterson** ([ipsonic@mailbox.org](mailto:ipsonic@mailbox.org)). Official project URL: [https://ipsonic.com/probe/](https://ipsonic.com/probe/).

**Disclaimer:** This software and its examples are provided **AS IS**, without warranty of any kind. Use at your own risk. Review and validate configurations before production deployment.


Project main website: https://ipsonic.com/