Best practices — ipsonic-probe

This guide is for developers, application operators, SecOps teams and service owners running ipsonic-probe against existing or newly deployed applications. It focuses on reliable, actionable checks with a small Linux-native dependency footprint.

1. Start with the service contract, not the YAML

Agree what healthy means for each service: expected URL, DNS answer, HTTP status, content marker, TLS policy, response-time budget and the network location from which it must be reachable. Assign a service owner, escalation contact and business criticality. Start with one important user-facing endpoint before adding dozens of probes.

A successful TCP connection is not proof that an application works. Prefer a lightweight HTTP health endpoint that validates the dependencies needed to serve requests. Avoid checks that create records, send real messages or modify production data unless the side effects are deliberate and controlled.

2. Layer checks to isolate failures

Use separate checks for separate failure domains:

Use the examples catalog to find a starting YAML; do not combine every assertion into one enormous check. Smaller, purpose-specific checks produce clearer alarms.

3. Keep YAML deterministic and readable

Store configuration in Git or another version-controlled repository. Use descriptive filenames such as portal-production-homepage.yaml, portal-production-tls.yaml and portal-staging-api.yaml. Give each probe a stable name and monitoring.service_name. Keep environment-specific targets in separate files, and review changes like application code.

Prefer explicit HTTP status allowlists, a short stable response marker, HTTPS verification and bounded timeouts. Avoid assertions against timestamps, session IDs, dynamic HTML, localized text or release-specific build numbers unless they are part of the agreed contract.

4. Set realistic timeouts, retries and severities

Use a short connect timeout and a bounded total response timeout suitable for the service and the monitoring interval. Define response-time warning and critical thresholds using observed baseline latency and an agreed service objective. Retries can reduce transient noise, but excessive retries can hide real failures and increase alert delay.

Map genuinely service-breaking failures to CRITICAL. Use WARNING for degradation requiring attention and INFO only for non-blocking observations. Note that INFO can yield an OK monitoring exit code: never classify a mandatory acceptance criterion as INFO.

For a fleet, stagger schedules to avoid synchronized bursts. Choose check frequency based on criticality and cost; not every endpoint needs a check every minute.

5. Secure credentials, proxies and private PKI

Never commit passwords, bearer tokens, proxy credentials or private keys to YAML. Prefer the tool's environment-variable-backed authentication fields and restrict the permissions of environment files, monitoring command definitions and CI secrets. Do not print credentials in logs or published acceptance evidence.

Use tls.verify: true. For internal CAs, configure tls.ca_file or tls.ca_file_env with an approved PEM trust bundle. Use proxy.mode: environment or proxy.mode: explicit when the monitoring vantage point must traverse a corporate proxy; direct access remains the default. Do not use tls.verify: false to make an alert disappear.

Detailed certificate, TLS-version and cipher inspection requires direct TLS connectivity in the current implementation; it is rejected when proxy mode is active. HTTP(S) proxy settings do not affect DNS or TCP probes. See examples 117–120.

6. Monitor from the right vantage point

A probe from inside a data center answers a different question than a probe from the public internet. Use monitoring hosts in the networks relevant to your users, and document their DNS resolver, egress route, proxy and trusted CAs. A service may work from the monitoring host while being unavailable to real users.

For load-balanced applications, check the public endpoint and individual backends where routing permits. Preserve the intended Host header and TLS SNI when using backend overrides. Avoid exposing internal backend details in public documentation.

7. Make alerts actionable for operators

Each check should answer: what failed, which service is affected, who owns it, and what should happen next? Pair the YAML with a short runbook describing likely causes and verification steps. Configure alert dependencies in Nagios, Icinga 2 or Checkmk so that an upstream DNS, firewall or load-balancer outage does not generate an unnecessary flood of downstream notifications.

Treat checks as signals, not a complete observability platform. Correlate failures with application logs, metrics, deployment history and security events when available.

8. Reuse the same checks across the lifecycle

• Development: define the intended endpoint contract and maintain example YAML in the application repository.
• CI: run fast smoke checks against an ephemeral or integration environment after the service becomes reachable; preserve the process exit code.
• Deployment: run staging and post-deployment checks, and fail or roll back the release according to the team's policy.
• Acceptance: archive approved YAML, checksums, timestamp, result and sign-off as technical evidence.
• Operations: schedule the same checks through existing Nagios, Icinga 2, Checkmk or cron workflows.
• SecOps: watch certificate expiry, unexpected redirects, DNS changes and security-header drift; investigate deviations.
• Migration / retirement: probe old and new endpoints during cutover, then remove checks when the service is decommissioned.

The same YAML can be reused, but adjust environment-specific URLs, credentials, thresholds and alert routing. A monitoring OK state is not a substitute for user acceptance or a full security assessment.

9. Keep installation and releases boring

Use Linux distribution packages only. On Debian, Ubuntu and Raspberry Pi OS:

sudo apt-get install -y python3 python3-requests python3-yaml ca-certificates

On AlmaLinux and Rocky Linux:

sudo dnf install -y python3 python3-requests python3-pyyaml ca-certificates

Use an approved internal APT/DNF mirror for air-gapped systems. No pip, virtual environment, Node.js, container runtime, database or hosted service is required for the probe. Requests and PyYAML remain runtime libraries, supplied by the distribution; the full transitive package inventory depends on the OS release.

For manual releases, use ./tools/build-release.sh VERSION and verify the published tar.gz with its SHA-256 checksum before deployment. Review changes to probe code and YAML separately.

10. Validate before rolling out

1. Run the probe's unit tests and validate all YAML files.
2. Test against known-good and deliberately broken endpoints, including TLS failures and timeouts.
3. Verify that monitoring output and exit codes match the severity policy.
4. Confirm proxy and CA settings on the actual monitoring host.
5. Test alert routing, ownership and runbooks with a controlled failure.
6. Review check schedules, credentials, permissions and evidence retention.
7. Promote checks gradually from staging to production.

Minimal practical example

name: "Production website availability"
url: "https://www.example.com/health"
timeouts:
  connect: 3
  response: 10
tls:
  verify: true
  min_valid_days: 30
tests:
  status:
    allow: [200]
  body:
    contains: ["healthy"]
monitoring:
  service_name: "Website health"
  output: nagios
  on_failure: critical
  thresholds:
    response_time:
      warning: 2
      critical: 5

Replace the documentation hostname and response marker with values approved for your service. For a proxy-based check, omit tls.min_valid_days and monitor certificate expiry separately from a direct-connect vantage point.

11. Know the limits

ipsonic-probe is designed for simple, reproducible active checks. It does not replace a full browser, synthetic transaction platform, load test, SIEM, vulnerability scanner, application tracing, mailbox login test or cryptographic DNSSEC validator. An HTTP 200 response does not prove business functionality, and a TCP port check does not prove a successful SMTP/IMAP/POP3 protocol exchange. Use dedicated tooling when these deeper assurances are required.

Further reading

• Technical web project acceptance
• Exchange monitoring
• Ansible installation
• YAML examples
• Release downloads

Maintainer and disclaimer

Maintained by mati peterson (ipsonic@mailbox.org). Official project URL: https://ipsonic.com/probe/.

Disclaimer: This software and its examples are provided AS IS, without warranty of any kind. Use at your own risk. Review and validate configurations before production deployment.