If an external domain is added and a Require TLS policy is assigned to it, when somebody tries sending a message from this domain with no TLS, the connection is accepted with no issues.

If the same sending relay's IP is now added under Hosts with a Require TLS policy, any attempt to establish an SMTP session is rejected after the "MAIL FROM:" command with "Must issue a STARTTLS command first".

IMPORTANT:   Require TLS policy dictates that the connecting host or domain use TLS. If a connecting host does not use TLS, the connection will not be established for this host. If a connecting domain does not use TLS, the message will not be accepted.

Connection tracking shows "Bypassing spoof check. spoof_protect=1, outbound=0, inWhilelist=1".

The keyword is "Whitelisted". This behavior is only observed when IP Reputation (IPR) is turned on (feature available in license) under Admin UI -> Relay Policies and the action for sources with known good reputation is set to "Accept: Bypass Relay Policies".

If the IPR is turned off or the policy changed to "Allow after a full policy scan", any non-TLS connection from the same domain will be rejected.

When it comes to a single IP, even though the IPR is enabled with the same accept policy and the same whitelisted category is assigned to the IP in question, connection gets rejected.

The technical explanation is that the receiving connection policy for hosts takes place before IPR checking, and the receiving connection policy for domain comes after IPR (i.e. after the MAIL FROM: command).

By design, the "Accept: Bypass Relay Policies" action is not meant to override connection policies applied on Internal or External hosts. Still, domains are being overridden.

WORKAROUND:   Change the default IPR policy to Throttle or Allow.