Tutorials

Fair Housing: Policy Rules Guardrail

What you will learn

How to create a Policy Rules Guardrail.
- Extracting rules from a policy document.
- Codifying rules as DSAIL.
- Crafting data extraction questions.
How to test a policy rules guardrail using the playground.

Scenario

A landlord/property manager has created a chatbot to manage new tenant applications and schedule apartment showings. While the chatbot performs sufficiently well at its primary function, it exposes the landlord to the risk of fair housing law violations. In order to mitigate this risk, we will set up a Policy Rules Guardrail to detect and flag such violations. This guardrail will focus on Section 8 compliance; in a real application several complementary guardrails may function together, verifying different aspects of the same overall policy. For example, Section 8 compliance addresses financial inequality, but does not directly address other types of discrimination also outlined in different states' fair housing laws; these might be implemented as complementary guardrails.

flowchart LR U[("User")] --> C[["Chatbot"]] G <-.-> LLM C --> Jaxon Jaxon --> V{"Violation?"} V -- Yes --> Flag["Fallback to\nHuman Support"] %%V -- No --> U subgraph Jaxon["Jaxon"] direction TB G["Policy Rules Guardrail"] LLM["LLM"] end

Guardrail Inputs

Context: The chat transcript between user and chatbot
Question: "Is this chat Section 8 compliant?"
Answer: N/A We will report the answer in the guardrail outputs

Guardrail Outputs

Eval: A YES/NO decision, along with a confidence score, on whether the context is compliant with the policy.
Proof: A breakdown of individual rules/clauses from the policy, so we know which are/aren't satisfied.

Creating the guardrail

Create an application. We'll name it section8.
- [Applications] -> [Add Application] -> Fill form with id: section8 / name: section8 -> [Create].
Create the guardrail.
- [Guardrails] -> [Add Guardrail] will display the Basic Information form for a new guardrail.
  - Application should already have section8 selected; if not do so.
  - Guardrail Name is a human readable label. Section 8 Tutorial is a good choice.
  - Guardrail Type should be set to Policy Rules.
- Basic Configuration
  - Pick the best LLM, Utility LLM, and Embedding Model that you have access to.
  - Good choices from OpenAI are o4-mini, o3, and text-embedding-3-large respectively.
  - Check Human Review Enabled
- Policy Document - this is a prose capture/extraction of the policy you want to enforce. For this tutorial, copy/paste the following:
```
Fair housing law in Massachusetts, as governed by both federal and state regulations, prohibits landlords 
from discriminating against tenants based on race, color, religion, national origin, sex, disability, 
familial status, and source of income—including Section 8 housing vouchers.

Under the Massachusetts Fair Housing Law (M.G.L. c. 151B) and the federal Fair Housing Act, landlords 
cannot refuse to rent to an otherwise qualified tenant solely because they use a Section 8 voucher.

Additionally, landlords must follow reasonable accommodation requirements for tenants with disabilities, 
such as allowing service animals despite a no-pet policy or making necessary structural modifications.

To comply with Section 8 requirements, landlords must ensure that their rental units meet the Housing 
Quality Standards (HQS) set by the Department of Housing and Urban Development (HUD) and pass an 
inspection before a lease is approved.

They are also required to charge rent within the limits set by the local housing authority and cannot 
demand additional payments outside the terms of the Housing Assistance Payment (HAP) contract.

Furthermore, landlords must maintain habitable conditions, perform necessary repairs, and cannot 
retaliate against tenants for exercising their rights under fair housing laws. Violations of these laws 
can result in penalties, including fines, lawsuits, and potential loss of the ability to participate in 
housing programs.

We want this policy to specifically adhere to analyzing chat transcripts between a landlord and a prospective 
tenant to identify the rules that must be met in order to ensure that fair housing law was not violated. 
Rules should be specific violations that might be found in such a chat.
```
  - Click [Extract Rules] leaving
  - The Additional Prompt Instructions allows you to provide additional instructions to the Utility LLM to help guide it towards topical rules.
  - A RAG Method is available as an alternative to the prompt-based default. This utilizes a sophisticated pipeline that:
    - Chunks the policy document using semantic segmentation (recommended) or fixed/dynamic size strategies
    - Extracts rules from each chunk in parallel for better performance
    - Performs intelligent deduplication using semantic similarity, fuzzy matching, and LLM judgment
    - Filtering Strategy: Choose between two approaches for topical relevance:
    - Embedding Similarity: Fast cosine similarity between rule and query embeddings
    - Provence Reranker: Advanced neural reranker model for more accurate context understanding
    - Relevance Threshold: Controls filtering strictness (0.0-1.0) - higher values are more selective
- Rules - The extraction will populate the Rules area with a bullet-point extraction of specific rules that have to be true in order to comply with the policy.
- Note you can start the process directly at this step if you have "bullet point" rules; the policy document and extraction is not required.
- You may continue with your generated rules, or you may copy/paste the below version into your Rules area in order to keep your system synchronized with this tutorial (as these steps use LLM-based transformations, the exact outputs will vary with each run.)
```
* Anti-Discrimination: Do not discriminate based on race, color, religion, national origin, sex, disability, familial status, or source of income (including Section 8 vouchers).
* Voucher Acceptance: Do not refuse a qualified tenant solely because they use a Section 8 voucher.
* Disability Accommodation: Provide reasonable accommodations for tenants with disabilities, including allowing service animals and necessary modifications.
* Housing Quality: Ensure rental units meet HUD’s Housing Quality Standards and pass required inspections before leasing.
* Rent Compliance: Charge rent within limits set by the local housing authority and do not require extra payments outside the HAP contract.
* Habitability Maintenance: Keep units in habitable condition and carry out necessary repairs.
* No Retaliation: Do not retaliate against tenants for exercising their rights under fair housing laws.
```
- Prove Compliance vs Prove Violations describes the "orientation" of the DSAIL code that we're about to generate from these rules. Consider whether a rule (in text) or an assertion (in DSAIL) proves adherence to the policy, or proves a violation of the policy. This is pertinent when uncertainty is introduced due to some questions (see later in this tutorial) being unanswered - a common case, in practice. A satisified assertion means that some choice of unknown variables can make this assertion True; an unsatisfied assertion means that no choice of unknown variables can do so. This means that a satisfied assertion may indicate the truth of the rule it describes, but cannot guarantee it.
  - Prove Compliance - Orient each assert so that proven compliance produces an unsatisfied result. assert noDiscrimination {{ landlordRefusedSection8 }} # When landlordRefusedSection8 is false (compliance), assert becomes UNSAT
  - Prove Violation - Orient each assert so that a violation produces an unsatisfied result. assert noDiscrimination {{ Not(landlordRefusedSection8) }} # When landlordRefusedSection8 is true (a violation), assert becomes UNSAT
  For this tutorial, we will select Prove Violations
- Click [Generate DSAIL] or copy the below program into Code Generated
```
declare landlordDiscriminatedOnRace as boolean;
declare landlordDiscriminatedOnColor as boolean;
declare landlordDiscriminatedOnReligion as boolean;
declare landlordDiscriminatedOnNationalOrigin as boolean;
declare landlordDiscriminatedOnSex as boolean;
declare landlordDiscriminatedOnDisability as boolean;
declare landlordDiscriminatedOnFamilialStatus as boolean;
declare landlordDiscriminatedOnSourceOfIncome as boolean;
declare tenantRefusedDueToSection8Voucher as boolean;
declare disabilityAccommodationsNotProvided as boolean;
declare unitPassedInspection as boolean;
declare unitMeetsHousingQuality as boolean;
declare rentWithinLimits as boolean;
declare extraPaymentsOutsideHAP as boolean;
declare unitHabitable as boolean;
declare repairsCompleted as boolean;
declare tenantRetaliationOccurred as boolean;

assert antiDiscrimination { Not(Or(landlordDiscriminatedOnRace, landlordDiscriminatedOnColor, landlordDiscriminatedOnReligion, landlordDiscriminatedOnNationalOrigin, landlordDiscriminatedOnSex, landlordDiscriminatedOnDisability, landlordDiscriminatedOnFamilialStatus)) };
assert voucherAcceptance { Not(Or(tenantRefusedDueToSection8Voucher, landlordDiscriminatedOnSourceOfIncome)) };
assert disabilityAccommodation { Not(disabilityAccommodationsNotProvided) };
assert housingQuality { And(unitPassedInspection, unitMeetsHousingQuality) };
assert rentCompliance { And(rentWithinLimits, Not(extraPaymentsOutsideHAP)) };
assert habitabilityMaintenance { And(unitHabitable, repairsCompleted) };
assert noRetaliation { Not(tenantRetaliationOccurred) };
```
  - Note the declarations at the top of this program; these form the basis for the questions in the final configuration step.
  - One should always review this code and ensure that (being LLM-generated) it accurately reflects the semantics and logic from the rules. See best practices for additional advice.
    - You can review the DSAIL Language Guide for DSAIL background and documentation.
  - As with the previous steps, the process can start with the code section, skipping the natural language steps above, but this is considered poor practice in most situations.
- Click [Generate Questions] - This converts each declared variable into a data extraction question for use at inference time.
  - You can click into any of these questions if the text should be altered, or if additional context (to an LLM performing the extraction) would be helpful.
** This must be done for each rule. Note how rule status changes to green when the rule is populated. **
- Click [Create] and find your new guardrail at the top of the list, as with the Hello World Tutorial.

Testing the guardrail in the playground

This test will function similarly to the Hello World Tutorial, however there is an additional complexity: the inclusion of Human Review (a checkbox we selected early in the configuration process). For general playground usage, see the Playground documentation.

Set Up for Human Review Testing:

This guardrail uses Human Review, so we need to set up both the Playground and Human Review dashboard:

Navigate to the Playground: Click the Playground tab in the top navigation bar to open the testing environment
Open Human Review Dashboard: Open a second browser tab/window and navigate to the Human Review tab (or ctrl/cmd-click the Human Review tab from the navbar). Position this side-by-side with your Playground tab
Prepare for Testing: You'll use the Playground to submit requests and the Human Review dashboard to approve data extractions

Human Review Nav Link

Test the Guardrail:

In the Playground tab, select the new guardrail (named Section 8 Tutorial) from the Select Target dropdown
Submit the following sample in the playground:

Context

APPLICANT: I would like to schedule a tour. 1 bedroom I have a section 8 voucher. Thanks

Landlord: We're excited to hear that you are interested in renting with us at Mosaic! I'm sorry but we cannot accept section 8 vouchers. To view all of our available listings, schedule a tour, or fill out an application, please visit {{URL}}

If you have any questions, you can reply to this email. We look forward to hearing back from you!
Hugs & Kisses,
Landlord

Click [Submit]. Note: don't dawdle before completing the following review step, as the playground has a timeout period.

This time, before you receive a playground response, you'll see the request appear in the dashboard. This is a user's opportunity to review the LLM extraction of answers (to the generated questions), and correct any mistakes.

Human Review Dashboard

For now, we'll simply assume correctness and click [Accept] (but feel free to correct a few answers if you like). This will return a final response to the playground. This response includes those answers in JSON format, but also includes the Eval and Proof sections. Each entry for proof corresponds to one of our rule assertions.

Results

The confidence score provided is on a scale from 0 to 1, where 1 is the most confident. This guardrail can now be run in inference, just as described in the Hello World Tutorial.

LLM-as-a-Judge

What you will learn

How to create an agentic guardrail using LLM-as-a-Judge.
How to structure a RAG-style LLM interaction as an Entailment Frame.
How to test a guardrail using the playground.
How to run inference on a guardrail using the Python client.
- For this you will need a terminal window with a Python interpreter installed.

Data Model: Intro to the Entailment Frame

We are going to verify a completed RAG-LLM interaction using an agentic guardrail to perform a "second opinion" on whether the original LLM got the answer right.

flowchart LR %% Main flow C[(RAG Documents)] Q[User Question] LLM[Application LLM] A[Unverified Output] Q -.-> LLM C -.-> LLM LLM -.-> A subgraph Jaxon[Jaxon] direction TB G[LLM-as-a-Judge Guardrail] JLLM[LLM] G <-.-> JLLM end subgraph Result direction TB Proof{{Proof}} FinalEval{{Eval}} Proof -.-> FinalEval end Q -->|Question| Jaxon C -->|Context| Jaxon A -->|Answer| Jaxon Jaxon --> Result

Guardrail Inputs

Context: Facts injected by the RAG retrieval.
Question: The specific query put to the LLM (with the context prepended).
Answer: The original answer produced by the LLM.

Guardrail Outputs

Eval: A YES/NO decision, along with a confidence score, indicating whether the answer is correct.
Proof: Not used for this guardrail.

This complete set of Inputs and Outputs - Context, Question, Answer, Eval, and Proof - comprises an entailment frame. This is the common data structure for interaction with all guardrails. Each guardrail type has its own interactions with the entailment frame. LLM-as-a-Judge evaluates the consistency of Context+Question with Answer, capturing its assessment in Eval. As the LLM's internal reasoning is a "black box", Proof is unused.

Creating the guardrail

Create an application. We'll name it hello.
- [Applications] -> [Add Application] -> Fill form with id: hello / name: hello -> [Create].
- Leave Context Document empty for this tutorial. The Context Document is application-wide background information that gets injected into all requests.
Create the guardrail.
- [Guardrails] -> [Add Guardrail] will display the Basic Information form for a new guardrail.
  - Application should already have hello selected; if not, do so.
  - Guardrail Name is a human readable label. Hello World is a good choice.
  - Guardrail Type should be set to LLM-as-a-Judge.
  - Click [Create].
- This will enable the Configuration section of the form. You do not need to change these default settings. Click [Update] to close.
- Your new guardrail will appear in the list of guardrails. It is now available for use.

Testing the guardrail in the playground

Now we'll test the guardrail using the interactive Playground. For detailed information about using the Playground, see the Playground documentation.

Navigate to the Playground: 1. Click the Playground tab in the top navigation bar of the admin interface 2. This opens the interactive testing environment in a new view

Test Your Guardrail: 1. In the Select Target dropdown, choose your newly created guardrail 2. Fill in the input fields as shown below:

Playground Input

Click Submit to execute the guardrail
Review the response - you can switch between Table View (default) and JSON View for different perspectives:

Playground Response

The confidence score provided is on a scale from 0 to 1, where 1 is the most confident.

💡 Tip: The Playground is your primary tool for testing and refining guardrails. Use it to experiment with different inputs and validate behavior before integrating with your applications.

Testing the guardrail with the client

We can access this guardrail programmatically; this is the primary usage model for deployment.

Setup

In a terminal with Python installed, create a workspace:

mkdir jaxon-hello
cd jaxon-hello
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install jaxon-<version>-py3-none-any.whl  # Replace <version> with current platform version

Client Code

Create a file hello.py with the following program:

from jaxon import Client
from jaxon.vendor.data_model import EntailmentFrame

# Connect to the platform via HTTP (recommended)
client = Client(
    api_base_url="https://your-domain.com:8443/client-api",
    verbose=False
)

try:
    # Create an EntailmentFrame with the test data
    frame = EntailmentFrame(
        C=["All dogs go to heaven.", "Fido is a cat."],
        Q="Does Fido go to heaven?",
        A="Yes"
    )

    # Send to your guardrail
    guardrail_id = "d79c0e0d"  # Copy from Guardrail list in Web UI
    response_topic = "hello-world-response"
    _, trace_id = client.send_message(guardrail_id, response_topic, frame)

    # Get the response (timeout in milliseconds)
    response = client.get_response(trace_id, response_topic, timeout=120000)

    # Parse the evaluation
    import json
    frame_data = json.loads(response['entailment_frame'])
    print(f"Conclusion: {frame_data['E']['conclusion']}")
    print(f"Confidence: {frame_data['E']['confidence']}")

finally:
    client.close()

Run the program:

python hello.py

Expected output:

Conclusion: NO
Confidence: 1.0

The guardrail correctly identifies that the answer "Yes" is inconsistent with the context (Fido is a cat, not a dog).

For detailed information about client setup, authentication, and advanced usage, see the Client API documentation.