Skip to content

symfonic.evals.book_visible_truth

book_visible_truth

Visible polarity and currentness checks for closed-book claims.

visible_currentness_contradiction 

visible_currentness_contradiction(claim: BookClaim, answer: str) -> bool

Detect opposite polarity/currentness within one visible clause.

Cues are compared by whole terms and proximity rather than a fixed sentence template, so subject/value word order does not change the verdict.

Source code in src/symfonic/evals/book_visible_truth.py 
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
def visible_currentness_contradiction(claim: BookClaim, answer: str) -> bool:
    """Detect opposite polarity/currentness within one visible clause.

    Cues are compared by whole terms and proximity rather than a fixed sentence
    template, so subject/value word order does not change the verdict.
    """

    expected = tuple(str(value) for value in claim.expected_current_assertions.values())
    forbidden = tuple(rule.value for rule in claim.forbidden_current_assertions)
    for clause in _CLAUSE.split(answer):
        expected_denied = any(
            _near_in_either_order(clause, cue, value)
            for value in expected
            for cue in _EXPECTED_DENIAL_PREFIXES
        ) or any(
            _near_in_either_order(clause, value, cue)
            for value in expected
            for cue in _EXPECTED_HISTORICAL_SUFFIXES
            if value.casefold() != cue.casefold()
        )
        forbidden_current = any(
            _near_in_either_order(clause, value, cue)
            for value in forbidden
            for cue in _CURRENTNESS_CUES
        )
        if expected_denied or forbidden_current:
            return True
    return False