Skip to content

Best Practices

LLM Selection

  • Experiment with LLM Choices: Both the primary LLM, used for the main inference tasks, and utility LLM, used (in a guardrail-specific way) for supplementary tasks, can be selected for all AI-driven guardrail actions. Larger models are typically more capable, but come with the tradeoff of cost and speed.

  • Fine-tuning: Fine-tuning is an option, after everything else has been tuned. Your administrator can include fine-tuned models (and locally-hosted models) as available options.

Guardrail Type Selection

  • Automation vs Rigor: Agentic guardrails (LLM-as-a-Judge, Consensus, Consistency Checking, and Critique & Review) require a low effort level to configure, and can cover broad topical domains. If the specific application does not require the extra burden of proof that the Policy Rules Guardrail offers, then these may offer a quick path to more trustworthy verification.

Policy Rules Design

  • Test with Realistic Examples: Always validate your rules against a set of development examples. Run the rules with the playground or via the client and manually review each assertion’s outcome —both the final evaluation and individual assertions- confirm correct behavior. Specifically watch out for rules that turn out UNSAT when the relevant topic is not even present in the text.

  • Clear, Question-Oriented Variable Names: Ensure that DSAIL variables have clear (verbose) names that lend themselves to automated question generation. landlordMentionedRecentRenovation is a better name than x, since automation can then produce Did the landlord mention a recent renovation?.

  • Favor One-sided Assertions: Design assertions so that missing (unbound) variables default to the more common, non-flagged scenario. If a conversation topic is absent, the assertion should typically remain SAT. This ensures that exceptional or irregular conditions explicitly require evidence in the conversation to go UNSAT.

  • Combine Assertions Thoughtfully: “Is this compliant?” is answered by checking if all your assertions can be SATisfied by the solver. If at least one assertion cannot be satisfied, then overall compliance fails. Make sure each assertion is meaningful in representing a distinct aspect of compliance or violation.

  • Understand Value Checking as Binding: When an assertion checks a variable’s value (e.g., x == True)`, the solver may treat that as an assignment for unbound variables. Avoid surprises by remembering that such an assertion can force the variable’s value in the logical solution space.

  • Distinguish Context from the Main Question: Information in supporting context (like reference documents) might introduce details you don’t want to trigger additional constraints. Ensure only truly relevant context is included, or scope it carefully, so it doesn’t inadvertently UNSATisfy your main rules.

  • Assign Logical Defaults Where Needed: If you need to avoid ambiguous outcomes for a rule, declare a default value using let. This default remains in effect unless the user or data explicitly overrides it, keeping your policy rules from incorrectly concluding SAT or UNSAT due to incomplete information.

  • Keep Rules Atomic: Each rule should check one narrow concern. This avoids confusion and makes it easy to see which parts of a policy are satisfied or not.