Building Intent-Based Policy Graphs with Neo4j

Unstructured and Unlinked Policy Documents

ACME_CORP has policies scattered everywhere: PDF files in shared drives, API responses from regulatory systems, manually created requirement documents, compliance frameworks from vendors. Each source is different. Each format is inconsistent. And yet, your auditors want to know:

• Which rules actually prevent unauthorized access?
• Where do we have conflicting compliance requirements?
• If we breach this rule, what intent were we violating?
• Can we prove every rule maps back to a business goal?
• Can we do a look ahead on the impact of a new regulation?

The answer isn’t to read everything manually. The answer is to build a governance graph that understands intent and not just rules.

Why Intent Graphs Beat Traditional Approaches

Most companies store policies in one of two ways:

1. Spreadsheets: Rules as rows, frameworks in columns. No relationships. Hard to see conflicts.
2. Document databases: PDFs and Word docs organized by folder. Full-text searchable. But structure lost.

ACME_CORP previously tried to solve this by dumping all its policies into a searchable database. But databases don’t understand why a rule exists. They can tell you “users must use MFA” exists in 47 documents, but not that all 47 are trying to achieve the same intent: “prevent unauthorized access.”

Intent graphs are different. They capture meaning through:

• Deduplication at scale: Found the same rule in 3 different policies? The graph knows it’s the same and has one source of truth.
• Conflict detection: Two rules serve the same intent but contradict? The graph flags it.
• Framework mapping: Which NIST controls does your policy satisfy? One query.
• Impact analysis: If I change this rule, which intents are affected? Which frameworks? The graph shows you.
• Audit trails: Full lineage from intent → rule → evidence → sign-off.

Medallion Architecture for Governance

A medallion architecture is a proven data engineering pattern that enforces quality gates at each stage. Think of it as three layers of increasingly trustworthy data:

• Bronze: Raw ingestion (PDFs, APIs, files as-is)
• Silver: Validated, deduplicated, quality-gated data
• Gold: Enriched, approved, audit-ready data with full chain of custody

When you apply this pattern to compliance policies, something magical happens: you don’t just store rules—you capture why they exist through an intent graph.

Instead of a flat list of policies, you build a graph where:

• Policies define Rules
• Rules satisfy Intents
• Intents require other Intents (hierarchical business goals)
• Rules map to Compliance Frameworks (NIST, ISO 27001, SOC2, GDPR, Board Objectives)

Your auditors can now trace backwards: This rule exists to satisfy this intent, which stems from this business goal, which addresses this regulatory requirement, and here’s the timestamped evidence.

Candidate Nodes within Medallion Graph

Bronze Layer: Capturing Everything Unfiltered

The first mistake people make is trying to be perfect at ingestion. Bronze doesn’t need to be perfec. It needs to be complete and traceable.

When you ingest a policy document into Bronze, you preserve everything:

:RawPolicy {
  id: "raw_pdf_SECURITY_POLICY_2024_Q2_pg3_1720329600",
  raw_text: "... full unprocessed PDF text ...",
  source_url: "file:///policies/SECURITY_POLICY_2024_Q2.pdf",
  extraction_confidence: 0.92,
  extracted_at: datetime("2024-07-12T14:30:00Z"),
  processing_status: "success"
}

Notice the ID includes the source, page number, and timestamp. This is deliberate. Bronze is where you answer: “Where did this data come from? How was it extracted? How confident are we?”

Hybrid Extraction Is Better

Regex extraction and LLM extraction work well on their own but they cannot compete with a hybrid approach. Most policies follow predictable patterns. Rules are stated explicitly with standard language. LLM shines when a rule is buried in narrative or contradicted elsewhere. That’s rarer with policy documents.

Regex: Use regex patterns and heuristics to find rules. Look for keywords like “must,” “shall,” “required.” Extract severity based on keyword density. Simple. Fast. ~80% accuracy on well-structured policies.

LLM: Use Claude or GPT-4 with structured prompts. “Extract all rules from this policy. For each rule, identify severity, who it applies to, and any exceptions.” ~90% accuracy. Costly per document.

Hybrid: Use rule-based extraction first (free, fast). Only send low-confidence results to LLM for refinement. Catches 80% of rules for free, uses LLM budget strategically for the hard 20%. ~92% accuracy at 20% of the cost.

Extracting Intent in Bronze

Extracting intent is harder than extracting rules. A rule is a sentence: “Users must authenticate.” However, intent is the why: “Prevent unauthorized access and reduce fraud.”

In Bronze, you extract both, with confidence scores:

:RawRule {
  id: "raw_rule_a1b2c3_pdf_1720329600",
  raw_text: "Users must use multi-factor authentication",
  llm_classification: "rule",
  llm_confidence: 0.88
}
:RawIntent {
  id: "raw_intent_7f3e2d_gpt4_1720329600",
  intent_name: "Prevent unauthorized access",
  intent_level: "operational",
  llm_confidence: 0.75
}
:RawRule -[:TENTATIVELY_SATISFIES {confidence: 0.78}]-> :RawIntent

The mapping between rule and intent is tentative. LLM guessed with 78% confidence that the MFA rule satisfies the “prevent unauthorized access” intent. In Bronze, you capture this guess. In Silver, you validate it.

Silver: Where Quality Gates Matter

If Bronze is the raw ore, Silver is the smelted metal. It’s where you enforce constraints, eliminate duplicates, and make quality decisions.

Quality Gate 1: Deduplication

You ingested policies from 5 different sources. Three of them contain the same authentication policy (copy-pasted, slightly reworded). Bronze has 3 copies. Silver must have 1.

Use fingerprinting: hash the normalized rule text. If the hash matches an existing rule with >95% text similarity, it’s a duplicate. Keep the highest-confidence version. Delete the others.

MATCH (r1:Rule), (r2:Rule)
WHERE r1.id < r2.id
  AND similarity(r1.text, r2.text) > 0.95
RETURN r1.id, r2.id, "DUPLICATE" as issue

Quality Gate 2: Schema Validation

Every rule must have:

• Text (NOT NULL)
• Severity (enum: CRITICAL, HIGH, MEDIUM, LOW)
• Mandatory flag (boolean)
• A policy it’s defined by (cardinality: 1)

Enforce these as Neo4j constraints. If a record violates a constraint, reject the entire batch. Auditors require precision.

Quality Gate 3: Confidence Thresholds

Rules extracted with <75% confidence go to a manual review queue. Engineers or compliance officers spend 5 minutes reviewing and correcting. This feedback loop is critical as it teaches you where your extraction breaks down and why.

Intent Hierarchy in Silver

Intent graphs allow the building of an acyclic intent hierarchy:

:Intent {id: "INT_2024_015", 
         intent_name: "Prevent unauthorized access",
         intent_level: "operational"}
  -[:REQUIRES]->
:Intent {id: "INT_2024_016", 
         intent_name: "Enforce multi-factor authentication",
         intent_level: "tactical"}

This hierarchy is acyclic (Neo4j enforces it). No circular intent dependencies. And critically, you detect conflicts: if Intent A requires MFA but Intent B minimizes login friction, they’re in tension. Document the tension. Have the business owner approve the exception.

Conflict Detection

Before moving to Gold, catch conflicts:

1. Circular dependencies: Rule A requires Rule B which requires Rule C which requires Rule A. Impossible. Flag it.
2. Contradictory severity: Two nearly identical rules, one marked CRITICAL and the other LOW. Why? Investigate.
3. Intent conflicts: Does “maximize security” conflict with “minimize friction”? They’re not mutually exclusive, but they do create tradeoffs that deserve explicit discussion.

Gold: The Audit-Ready Truth

Gold is where everything is enriched, approved, and immutable.

By the time data reaches Gold, it has been:

• Validated: Schema constraints passed
• Deduplicated: No duplicates remain
• Linked: Every rule is mapped to at least one intent
• Approved: Business owner signed off
• Enriched: Business context, risk assessment, test procedure, remediation steps all recorded

In Gold, violations are append-only. You never update a violation record after creation. If a rule is broken, you log it, investigate, remediate, and get auditor sign-off. Then it’s sealed.

:Violation {
  id: "VIO_2024_027",
  description: "Unauthorized API access detected",
  severity: "CRITICAL",
  detectedAt: datetime("2024-07-10T14:23:00Z"),
  detectedBy: "security_scan_v3.1",
  status: "resolved",
  resolvedAt: datetime("2024-07-11T10:00:00Z"),
  auditSignoff: {
    signedBy: "compliance_officer@company.com",
    signedAt: datetime("2024-07-12T09:00:00Z"),
    confirmHash: "sha256_hash_proof_of_evidence"
  }
}

Every field in this violation is immutable after the audit sign-off. The confirmHash proves the evidence wasn’t tampered with. If your auditor says “prove this violation was real,” you can show the complete chain: rule violated → evidence → investigation → remediation → auditor signature.

External API in Gold

Gold also exposes a compliant external API:

GET /api/compliance/audit/{policy_id}
→ Returns full lineage:
  ├─ Policy
  ├─ Rules (with intent mapping)
  ├─ Evidence (with hash verification)
  ├─ Violations (with signatures)
  └─ Framework mappings (NIST, ISO)

Role-based access: auditors see everything, engineers see only their policies, executives see compliance scorecards.

Real-World Example: Authentication Policy

Let’s trace one policy through all three tiers to show how this actually works.

Bronze: Messy Reality

An existing PDF arrives called “SECURITY_POLICY_2024_Q2_FINAL.pdf” containing the following information:

raw_pdf_SECURITY_POLICY_2024_Q2_pg3_1720329600:
  raw_text: "Users must use multi-factor authentication within 30 days of account creation, 
             except for service accounts which are exempt."
  extraction_confidence: 0.92
  extracted_at: 2024-07-12T14:30:00Z

The hybrid extraction tool (hybrid: rules + LLM) finds:

raw_rule_a1b2c3_pdf_1720329600:
raw_text: "Users must use multi-factor authentication"
llm_confidence: 0.88
raw_intent_7f3e2d_gpt4_1720329600:
intent_name: "Prevent unauthorized access"
llm_confidence: 0.82

In this example the rule extraction missed the timeline (30 days) and the exception (service accounts). The confidence result is high enough to move forward.

Silver: Cleaned & Validated

Your Silver process then cleans and validates:

POL_2024_001:
  title: "Information Security Policy"
  version: "2.1.0"
  owner: "security-team@company.com"
  effectiveDate: 2024-07-01
  status: "active"
  quality_gates_passed: true
RUL_2024_042:
  text: "Users must use multi-factor authentication (MFA) for all system access.
         Timeline: 30 days from account creation. 
         Exception: Service accounts are exempt."
  severity: "CRITICAL"
  mandatory: true
  policyId: "POL_2024_001"
  quality_gates_passed: true
INT_2024_015:
  intent_name: "Prevent unauthorized access"
  intent_level: "operational"
  business_value: "Reduce fraud and data breaches"
  status: "active"
  quality_gates_passed: true
RUL_2024_042 -[SATISFIES {confidence: 0.92}]-> INT_2024_015

Notice the rule text was enriched with the timeline and exception. The LLM caught these on second pass. Now everything is normalized, deduplicated, and validated.

Gold: Audit-Ready

At this stage the business owner approves, and data moves to Gold with enrichment:

POL_2024_001 (enriched):
  businessContext: "Comply with ISO 27001-A.9.4 and PCI-DSS Requirement 8.3"
  riskAssessment: "Weak authentication enables credential compromise → data breach"
  owner: "ciso@company.com"
  approvers: [{
    approver: "security-lead@company.com",
    approval_date: 2024-07-15,
    signature: "sig_hash_123..."
  }]
  nextReviewDate: 2025-07-01
RUL_2024_042 (approved):
  businessImpact: "Prevents attackers from compromising accounts via stolen credentials"
  testProcedure: "Attempt login without MFA token; verify access denied"
  testFrequency: "quarterly"
  remediationOwner: "iam-team@company.com"
  approvals: [{approver: "security-lead", approval_date: 2024-07-14}]
  -[MAPPED_TO_CONTROL]-> CTRL_NIST_AC-2 (status: compliant)
  -[MAPPED_TO_CONTROL]-> CTRL_ISO27001_A.9.4 (status: compliant)
INT_2024_015 (enriched):
  business_owner: "ciso@company.com"
  kpi_metric: "% of users with MFA enabled"
  kpi_target: 100.0
  risk_if_violated: "Attackers gain system access via stolen credentials"
  approval: {approver: "ciso@company.com", approval_date: 2024-07-15}

Auditable API

Lastly a GraphQL API can answer the question of “Why do you require MFA?” by presenting auditor with the the full chain:

Intent: Prevent unauthorized access
Business Goal: Comply with ISO 27001-A.9.4 + PCI-DSS
Rule: Enforce MFA for all users within 30 days
Test Procedure: Quarterly automated login tests
Evidence: Last test 2024-07-10, 99.8% compliance
Audit Sign-Off: CISO approved 2024-07-15

This is a complete governance trail. Not a rule in a spreadsheet. A graph showing intent.

Structured Content as an Intent Graph

Most governance initiatives fail because they treat policies as documents to file and forget. But policies are living intent. They encode what your company believes should happen. Over time, they conflict. They change. New ones contradict old ones. Someone adds an exception without updating the broader policy.

A governance graph doesn’t prevent these problem, it makes them visible. Your auditors see not just “did you follow rule X?” but “does your rule enforcement serve your stated intent?” That’s the question they actually care about.

A graph showing the full lineage from intent to evidence makes governance tractable.