Clinical Benchmarks
Menu

MedPIC-Bench

Tests whether models apply and withdraw medication-safety rules correctly as patient information changes.

Hou et al.28 rows28 models on the boardmeasured to Aug 2026

Official pagePaper
More about this boardLess

MedPIC-Bench evaluates whether language models use patient information to apply medication-safety rules correctly. Its 467 expert-validated multiple-choice questions combine fixed guideline-following cases with counterfactual cases in which changes to patient information alter whether a rule applies. Models must select the exact correct set of options; linked-pair accuracy requires both cases to be correct. The benchmark also distinguishes activating a safety warning from withdrawing a warning whose trigger is absent.

Published by Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), released 4 Aug 2026. Overall exact-match accuracy (%). 0–100.

  • Guideline-following accuracy (GF). Exact option-set match accuracy on 284 fixed patient cases requiring application of a medication-safety recommendation. Unit: %.
  • Counterfactual accuracy (CF). Exact option-set match accuracy on 183 questions with controlled changes in patient information that alter rule applicability. Unit: %.
  • Linked counterfactual pair accuracy. Share of the 89 linked counterfactual pairs where both questions, presented separately, are answered correctly. The pairing comes from the paper; the public dataset does not include pair identifiers. Unit: %.
  • Risk activation accuracy. Exact option-set match accuracy on the 71 counterfactual questions where patient information activates a medication-safety rule. Unit: %.
  • Risk deactivation accuracy. Exact option-set match accuracy on the 68 counterfactual questions where changed patient information removes a medication-safety rule trigger. Unit: %.
  • Published GF-minus-CF accuracy gap. Guideline-following accuracy minus counterfactual accuracy, in percentage points, as printed in Table 2. It describes how much accuracy drops when patient information changes. The paper does not treat a smaller gap as better, so this board does not rank models on it. Unit: percentage points.
Authors
Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang
License
CC BY 4.0
Limitations
  • Released questions.json has no explicit pair ID, rule ID, source-citation field, raw model responses or evaluator code. Pair scores are paper-reported only.
  • No live leaderboard: these results are the August 2026 study snapshot.
  • Printed gaps can differ by 0.1 from subtracting the rounded guideline-following and counterfactual scores; the printed values are kept.
  • Drug-category labels are multi-valued and can overlap; their counts do not sum to question count.
  • Dataset questions are constructed from selected rules and disproportionately cover older adults; this is not an observed clinical cohort or patient-outcome evaluation.
  • Risk activation/deactivation exclude the other 44 counterfactual items; do not average those two scores to reconstruct total CF.
  • Model-group averages confound architecture, size and training. No claim that medical adaptation generally helps or harms follows from them.
Institutions
The Hong Kong Polytechnic University, InfiX.ai, Sun Yat-sen University
Paper version
arXiv:2608.03028v1
Dataset revision
9ef6db4f13865b14fc2e6be3f94dcfaf3a0cf983

Ranking

Headline score

  1. 1Gemini-3.1-ProGoogle · zero-shot80.7
    Printed as 80.7Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Gemini-3.1-Pro, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Gemini-3.1-Pro | 80.7 | 87.7 | 69.9 | 17.7 | 74.6 | 77.9 | 48.3
    Every result from this document

    Link to this row

  2. 2GPT-5OpenAI · zero-shot75.6
    Printed as 75.6Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row GPT-5, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    GPT-5 | 75.6 | 83.5 | 63.4 | 20.1 | 70.4 | 61.8 | 38.2
    Every result from this document

    Link to this row

  3. 3Qwen3.5-PlusAlibaba · zero-shot72.4
    Printed as 72.4Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Qwen3.5-Plus, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Qwen3.5-Plus | 72.4 | 81.7 | 57.9 | 23.8 | 66.2 | 55.9 | 32.6
    Every result from this document

    Link to this row

  4. 4DeepSeek-V4-ProDeepSeek · zero-shot71.7
    Printed as 71.7Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row DeepSeek-V4-Pro, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    DeepSeek-V4-Pro | 71.7 | 81.3 | 56.8 | 24.5 | 67.6 | 50.0 | 32.6
    Every result from this document

    Link to this row

  5. 5GPT-OSS-120BOpenAI · zero-shot70.7
    Printed as 70.7Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row GPT-OSS-120B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    GPT-OSS-120B | 70.7 | 79.6 | 56.8 | 22.7 | 62.0 | 55.9 | 30.3
    Every result from this document

    Link to this row

  6. 6Qwen3.5-27BAlibaba · zero-shot69.4
    Printed as 69.4Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Qwen3.5-27B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Qwen3.5-27B | 69.4 | 78.2 | 55.7 | 22.4 | 67.6 | 51.5 | 31.5
    Every result from this document

    Link to this row

  7. 7Qwen3.5-35B-A3BAlibaba · zero-shot68.5
    Printed as 68.5Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Qwen3.5-35B-A3B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Qwen3.5-35B-A3B | 68.5 | 77.5 | 54.6 | 22.8 | 70.4 | 45.6 | 30.3
    Every result from this document

    Link to this row

  8. 8GPT-5.2OpenAI · zero-shot68.1
    Printed as 68.1Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row GPT-5.2, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    GPT-5.2 | 68.1 | 78.2 | 52.5 | 25.7 | 63.4 | 42.6 | 29.2
    Every result from this document

    Link to this row

  9. 9Claude-Sonnet-4.6Anthropic · zero-shot64.2
    Printed as 64.2Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Claude-Sonnet-4.6, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Claude-Sonnet-4.6 | 64.2 | 74.6 | 48.1 | 26.6 | 62.0 | 35.3 | 21.3
    Every result from this document

    Link to this row

  10. 10Qwen3.5-9BAlibaba · zero-shot63.8
    Printed as 63.8Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Qwen3.5-9B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Qwen3.5-9B | 63.8 | 73.9 | 48.1 | 25.9 | 54.9 | 47.1 | 24.7
    Every result from this document

    Link to this row

All 28 rowsTop 10 only
  1. 11MedGemma-27B-TextGoogle · zero-shot61.2
    Printed as 61.2Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row MedGemma-27B-Text, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    MedGemma-27B-Text | 61.2 | 67.6 | 51.4 | 16.2 | 76.1 | 32.4 | 23.6
    Every result from this document

    Link to this row

  2. 12Fleming-R1-32BUbiquant · zero-shot60.6
    Printed as 60.6Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Fleming-R1-32B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Fleming-R1-32B | 60.6 | 69.0 | 47.5 | 21.5 | 59.2 | 44.1 | 20.2
    Every result from this document

    Link to this row

  3. 13Lingshu-32BAlibaba · zero-shot59.5
    Printed as 59.5Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Lingshu-32B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Lingshu-32B | 59.5 | 63.7 | 53.0 | 10.7 | 60.6 | 54.4 | 25.8
    Every result from this document

    Link to this row

  4. 14GPT-OSS-20BOpenAI · zero-shot59.1
    Printed as 59.1Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row GPT-OSS-20B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    GPT-OSS-20B | 59.1 | 66.5 | 47.5 | 19.0 | 52.1 | 48.5 | 24.7
    Every result from this document

    Link to this row

  5. 15Llama-3.1-70BMeta · zero-shot55.7
    Printed as 55.7Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Llama-3.1-70B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Llama-3.1-70B | 55.7 | 61.3 | 47.0 | 14.3 | 63.4 | 32.4 | 16.9
    Every result from this document

    Link to this row

  6. 16HuatuoGPT-o1-70BFreedomIntelligence · zero-shot55.2
    Printed as 55.2Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row HuatuoGPT-o1-70B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    HuatuoGPT-o1-70B | 55.2 | 63.7 | 42.1 | 21.7 | 63.4 | 23.5 | 14.6
    Every result from this document

    Link to this row

  7. 17Gemini-2.5-ProGoogle · zero-shot54.4
    Printed as 54.4Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Gemini-2.5-Pro, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Gemini-2.5-Pro | 54.4 | 60.6 | 44.8 | 15.8 | 63.4 | 27.9 | 20.2
    Every result from this document

    Link to this row

  8. 18Fleming-R1-7BUbiquant · zero-shot50.7
    Printed as 50.7Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Fleming-R1-7B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Fleming-R1-7B | 50.7 | 59.9 | 36.6 | 23.2 | 53.5 | 29.4 | 12.4
    Every result from this document

    Link to this row

  9. 19Baichuan-M2-32BBaichuan · zero-shot47.8
    Printed as 47.8Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Baichuan-M2-32B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Baichuan-M2-32B | 47.8 | 54.6 | 37.2 | 17.4 | 49.3 | 30.9 | 11.2
    Every result from this document

    Link to this row

  10. 20HuatuoGPT-o1-8BFreedomIntelligence · zero-shot44.3
    Printed as 44.3Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row HuatuoGPT-o1-8B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    HuatuoGPT-o1-8B | 44.3 | 50.0 | 35.5 | 14.5 | 49.3 | 20.6 | 13.5
    Every result from this document

    Link to this row

  11. 21Lingshu-7BAlibaba · zero-shot43.3
    Printed as 43.3Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Lingshu-7B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Lingshu-7B | 43.3 | 50.7 | 31.7 | 19.0 | 39.4 | 32.4 | 6.7
    Every result from this document

    Link to this row

  12. 22HealthGPT-Pro-8BZJU4HealthCare · zero-shot42.6
    Printed as 42.6Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row HealthGPT-Pro-8B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    HealthGPT-Pro-8B | 42.6 | 51.8 | 28.4 | 23.3 | 42.3 | 20.6 | 5.6
    Every result from this document

    Link to this row

  13. 23Hulu-Med-7BZJU-AI4H · zero-shot42.0
    Printed as 42.0Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Hulu-Med-7B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Hulu-Med-7B | 42.0 | 48.2 | 32.2 | 16.0 | 36.6 | 38.2 | 14.6
    Every result from this document

    Link to this row

  14. 24Gemma-3-27BGoogle · zero-shot40.3
    Printed as 40.3Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Gemma-3-27B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Gemma-3-27B | 40.3 | 44.7 | 33.3 | 11.4 | 53.5 | 10.3 | 5.6
    Every result from this document

    Link to this row

  15. 25Llama-3.1-8BMeta · zero-shot40.0
    Printed as 40.0Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Llama-3.1-8B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Llama-3.1-8B | 40.0 | 44.4 | 33.3 | 11.0 | 54.9 | 10.3 | 5.6
    Every result from this document

    Link to this row

  16. 26Gemma-3-12BGoogle · zero-shot39.6
    Printed as 39.6Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row Gemma-3-12B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    Gemma-3-12B | 39.6 | 44.0 | 32.8 | 11.2 | 57.7 | 7.4 | 3.4
    Every result from this document

    Link to this row

  17. 26MedGemma-4BGoogle · zero-shot39.6
    Printed as 39.6Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row MedGemma-4B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    MedGemma-4B | 39.6 | 45.4 | 30.6 | 14.8 | 46.5 | 10.3 | 5.6
    Every result from this document

    Link to this row

  18. 28HealthGPT-Pro-4BZJU4HealthCare · zero-shot36.0
    Printed as 36.0Independent run, measured Aug 2026Configuration: zero-shot; independent questions; exact option-set match
    Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning paper, Zhitian Hou, Yuhang Liu, Pengkai Wang, Zeyu Liu, Guanghao Zhu, Zheng Liu, Shuo Cai, Congkai Xie, Zhijie Sang, Kun Zeng, Hongxia Yang (The Hong Kong Polytechnic University; InfiX.ai; Sun Yat-sen University), 4 Aug 2026. arXiv:2608.03028v1, Section 4, Table 2 (#S4.T2), row HealthGPT-Pro-4B, column Overall; row order: Model | Overall | GF | CF | Δ_GF−CF | Activation | Deactivation | Pair
    HealthGPT-Pro-4B | 36.0 | 37.7 | 33.3 | 4.3 | 36.6 | 41.2 | 11.2
    Every result from this document

    Link to this row

0–100. Bar colour: the lab; a pale bar with an outline is a number the model’s own lab reported. Select a row for its quote, page and document.

What the items cover

Item counts; an item can carry more than one label.

Benchmark task family

Item counts by benchmark task family
LabelAll itemsCounterfactualGuideline following
Guideline following284–284
Counterfactual183183–

Clinical system

Item counts by clinical system
LabelAll itemsCounterfactualGuideline following
Nervous system1627092
Urinary system1384494
Cardiovascular system623527
Reproductive system492029
Digestive system19613
Lymphatic and immune system1129
Skeletal system10–10
Integumentary system844
Respiratory system826

Drug categories

Item counts by drug categories
LabelAll itemsCounterfactualGuideline following
Nsaids/antipyretic analgesics501634
Opioids412219
Antiarrhythmics331518
Antidiabetics33–33
Antihistamines33–33
Antipsychotics332211
Antipyretic analgesics22220
Factor xa inhibitors/anticoagulants22814
Pde3 inhibitors/peripheral vascular drugs21912
H2-receptor antagonists19127
Alpha-1 receptor blockers18810
Calcium-channel blockers18414
Cholinesterase inhibitors17134
Antibacterials1569
Gabapentinoids1569
Ras inhibitors15–15
Gastrointestinal drugs1367
Antiepileptics1266
Multiple drug categories (composite rules)1239
Statins12210
Anticoagulants1055
Antibiotics927
Thyroid hormones9–9
Antigout agents844
Opioid-like analgesics844
Psychiatric medications844
H2-receptor blockers725
Local anesthetics/topical drugs642
Tricyclic antidepressants624
Benzodiazepines5–5
Lithium523
Theophyllines523
Antidepressants/analgesics422
Antivirals422
Direct thrombin inhibitors/anticoagulants422
Fluoroquinolone antibacterials422
Immunomodulators44–
Low-molecular-weight heparin anticoagulants422
Muscle relaxants422
Nitroimidazole antibacterials4–4
Ophthalmic drugs422
Potassium-sparing diuretics4–4
Thiazolidinedione antidiabetics422
Tricyclic antidepressants/anticholinergics422
Urinary antibacterials422
Z-drugs4–4
Retinoids33–
Sulfonamide antibacterials3–3
Analgesics2–2
Antimetabolites/immunosuppressants22–
Cardiovascular drugs2–2
Central nervous system depressants22–
Hormonal drugs22–
Nonselective alpha-1 receptor blockers2–2
Prostaglandins22–
Ras inhibitors/potassium-sparing diuretics22–
Renin-angiotensin system inhibitors2–2
Aldosterone receptor antagonists1–1
Anesthetics1–1
Antihistamines/antiemetics1–1
Dermatologic drugs1–1
First-generation antihistamines/anticholinergics1–1
Opioid antagonists1–1
Sulfonamides/urinary antibacterials1–1
Topical drugs1–1
Z-class hypnotics1–1

Patient info type

Item counts by patient info type
LabelAll itemsCounterfactualGuideline following
Disease presence1365977
Dose threshold1184474
Composite clinical context662046
Drug interaction451827
Age threshold421824
Gestational-week threshold29–29
Pregnancy status2020–
Pediatric-adult contraindication1147

Population source

Item counts by population source
LabelAll itemsCounterfactualGuideline following
Older adults352137215
Pediatrics662640
Pregnancy492029

Clinical department

Item counts by clinical department
LabelAll itemsCounterfactualGuideline following
Nephrology1184474
Neurology1104664
Pediatrics662640
Cardiology603525
Obstetrics and gynecology492029
Urology20–20
Psychiatry1569
Geriatric medicine10–10
Pain medicine743
Gastroenterology6–6
Pulmonology624

Reasoning operation

Item counts by reasoning operation
LabelAll itemsCounterfactualGuideline following
Static risk identification284–284
Risk activation7171–
Risk deactivation6868–
Risk redistribution2626–
Interaction identification1818–

Results by department

As printed, Table 3.

Mean counterfactual accuracy (%) by model group; departments with at least 20 CF questions

Mean counterfactual accuracy (%) by model group; departments with at least 20 CF questions
DepartmentGeneralProprietaryMedical specific
Neurology31.640.726.4
Cardiology3342.927.9
Pediatrics47.454.942.9
Nephrology5367.241.1
Obstetrics & Gynecology81.196.480

Documents

1 document supports this board