K-12 Accountability Analytics

The project is in three phases

The work is sequenced so that each phase produces an artifact the next phase builds on. We're currently in Phase 0.

How we got here

Before we committed to two specific papers, we worked through a four-step process that's worth being explicit about. The work isn't a single leap from "idea" to "draft" — it's a sequence of decisions that each narrowed the space.

Step 1 — Landscape synthesis

We read three sources: the EDM 2026 accepted-papers list, the IJIET topic page on Educational Data Mining and Learning Analytics, and the IEEE literature review (11204503) on methods, ethics, and AI trends in EDM/LA. From EDM 2026's 126 papers, we identified seven dominant research threads (described in the next section). The single most-cited gap across the 32 full papers: we have no shared, open, real-world K-12 systems that practitioners can build on.

Step 2 — Strategic pressure test

Once we had a candidate paper in mind (K12-Bench as a Text-to-SQL benchmark), we stress-tested the framing against four likely reviewer objections. The reframings we chose in response — "accountability-metric reasoning" instead of "Text-to-SQL benchmark," LM-evaluation instead of system-paper, cost-aware model lineup, schema-portability story — are documented in our strategic pressures doc.

Step 3 — Decisions

We chose to publish a case-study paper first, not K12-Bench first. Rationale: the case study is faster (4–6 weeks), lower risk, and gives K12-Bench a real user story. Two papers, even if the second is the stronger one, beat one paper.

Step 4 — Execution

This is the part we're in now. Co-author search, then case-study draft, then K12-Bench build, then K12-Bench paper draft. The methodology for each step is laid out below and in the case-study plan.

What we've worked with

Three sources form the foundation. We're not claiming a comprehensive review here — these are the sources that shaped the framing. If you know of K-12 NL→SQL or accountability-data work we should be citing, we want to hear about it.

EDM 2026 accepted papers

The 2026 EDM conference accepted 126 papers, which we clustered into seven dominant threads. (Reviewed: educationaldatamining.org/edm2026/accepted-papers/.)

• LLM tutors / LLM-in-the-classroom (~14 papers) — efficacy, behavior, evaluation. Anchor: Vanacore et al. (causal effects of on-demand tutoring), Ahtisham et al. (LLM reasoning predicts correctness).
• Privacy-preserving / synthetic educational data (~3) — Tiukhova et al., Diaz Ramos et al.
• Knowledge tracing beyond DKT/SAKT (~6) — Cook et al. (UNVaMP variational), Bhattacharyya et al. (specialized KT beats LLMs).
• Multimodal learning analytics (~8) — eye gaze, EEG, keystroke, video, audio.
• Auto-grading & item-design analytics (~4) — confusion-aware rubric optimization, item-writing flaws.
• Causal inference in classrooms (~3) — on-demand tutoring, help-seeking, ASSISTments reanalysis.
• Scaffolding / GenAI consensus / agentic tutors (~4) — LLM agent consensus, coordination regimes, PedRAG.

The single most-cited gap across the 32 full papers: no shared, open, real-world K-12 education datasets or systems that practitioners can build on. Most papers use higher-ed (OULAD, ASSISTments, Khan Academy) or controlled lab data.

IJIET — Educational Data Mining and Learning Analytics

IJIET's EDM/LA topic page (covers 2011→2026, 4–12 papers/year) frames the field as student-modeling plus decision-support. The 2026 list skews toward predictive admission / course-recommendation systems, ML classification for student behavior, and algorithmic thinking / CS-education data. The publication tradition is applied, lower-rigor but high-replicability work — system descriptions with a real dataset, a real model, and a real result. (ijiet.org/list-197-1.html.)

Pattern: IJIET is a natural fallback venue if EDM doesn't accept. It's also a useful target for the case-study paper specifically, since case studies with real data fit the IJIET tradition better than the EDM system-paper bar.

IEEE 11204503 — Literature review

The IEEE review structures EDM/LA as a four-pillar field: Prediction, Clustering, Relationship Mining, Discovery/improvement of domain models. It flags three ethical tensions that recur in 2026 work: utility vs. privacy, algorithmic fairness across SES / language / region, and LLM-behavior-vs-learning (does the tool help, or just appear to help?). (ieeexplore.ieee.org/document/11204503.)

Where our stack fits

Cross-walking the stack against EDM 2026's threads:

• The Aeries+CDE warehouse → Privacy (B), System (G). Real, open-schema, K-12-scale warehouse. EDM doesn't have one.
• The fine-tuned Qwen2.5-Coder LoRA → LLM-in-classroom-adjacent (A), but administrator-facing rather than student-facing.
• The eval harness and frozen prompt set → Empirical NL2SQL benchmark gap.
• The Rill dashboards → State-accountability metric transparency. Most EDM papers treat dashboards as a given.

The stack sits at the intersection of threads A, B, E, and G. We're not trying to compete on knowledge-tracing algorithms (thread C) or multimodal sensing (thread D).

What the first paper will study

The case study is the work that's most likely to change with collaborator input. Below is the current spine. If you have a strong view on methodology, RQ design, or which narrative arcs to develop, this is where we'd most want to hear it.

Working title

"Open-Source K-12 Accountability Analytics in Practice: A Case Study of the LFED Framework"

Alternative: "From Warehouse to NL Query in 14B Parameters: How a District Uses Open-Source Tooling to Operationalize State Accountability Data."

One-sentence thesis

Open-source K-12 education analytics — combining a privacy-preserving DuckDB warehouse, a fine-tuned NL-to-SQL model, and state-accountability-aligned Rill dashboards — can be deployed in a district and meaningfully shift who has access to accountability data, and the failure modes of the LLM components are predictable and addressable.

Three research questions

• RQ1 — Adoption. How does a district administrator actually use the LFED stack to interrogate accountability data? What workflows does the NL→SQL layer enable that weren't possible before?
• RQ2 — Failure modes. Where does the system break? What categories of accountability-metric questions does the LoRA fail on, and how does a non-technical user work around it?
• RQ3 — Equity. Does open-source tooling change who asks the data questions? Does it shift from "IT team runs SQL" to "principal runs a natural-language query"?

Methodology

Mixed methods: a 30-day log analysis of NL→SQL usage by 3–5 district administrators, plus 1–2 hour semi-structured interviews with each. The case-study design follows the tradition of practitioner case-study research in design science — we treat the deployment itself as the unit of analysis, not a synthetic lab experiment.

Four narrative arcs

We have four candidate arcs developed. The case study will develop Arc A and Arc C in parallel; Arc B is held in reserve; Arc D is long-tail. Each arc is a self-contained story with a real accountability-data question, a user persona, and a documented failure mode.

Arc A — The principal who wanted chronic absenteeism data

Elementary principal at a 400-student school; her district is on the California School Dashboard red list for chronic absenteeism. She wants the school's chronic absenteeism rate by grade for the last 3 years, broken out by English Learner status. She doesn't write SQL; IT is busy. She opens the LFED Gradio app, types the question, gets a chart, uses it in a staff meeting the next morning. The chart ends up in the LCAP draft.

Documented failure mode: The LoRA returns "2023-24" rows when asked for "2023-2024" attendance. CDE uses "2023-24" year format; Aeries uses "2023-2024." The principal notices the chart looks "off" relative to her mental model and manually edits the SQL.

Arc B — The district analyst's audit

District data analyst preparing the annual equity audit. She needs to verify CDE's pre-computed Status/Change/Color against raw Aeries data. She uses the Rill dashboard to identify suspect schools, then the NL→SQL app to pull raw Aeries counts, then a Python notebook to reconcile. Documented failure mode: the LoRA doesn't know about the is_suppressed column and includes suppressed rows in averages, producing a different number than CDE.

Arc C — The board presentation

Superintendent preparing four charts for tomorrow night's school board meeting: chronic absenteeism, suspension, ELA, ELPI — all broken out by reporting category. She opens Rill, asks the LoRA for the OBT mart, gets a multi-indicator query in 30 seconds. She exports a PDF and sends it to the board chair. Documented failure mode: reporting category code confusion — the LoRA is asked for "Black students" but the CDE reporting category code is RB (with "African American" sometimes a separate code). The LoRA invents an alias; the chart looks plausible but is wrong.

Arc D — Hybrid tutor routing (long-tail, on hold)

A simulator that decides LLM-route vs. human-route for student help-seeking based on signals like keystroke confidence, time-on-task, prior mastery. Held in reserve. Requires a real intervention partner; doesn't fit the 4–6 week case-study window. If a CS-ed faculty member with an active intervention study joins, Arc D becomes the natural research-design conversation.

The benchmark, in brief

K12-Bench is the second paper. The full plan is in the deferred K12-Bench weeks-1-and-2 plan — but here's the shape so collaborators can react to it now.

220 frozen prompts

200 main-set prompts + 20 special-test prompts. Distributed across four indicators (Chronic Absenteeism 30%, Suspension 25%, ELA 25%, ELPI 20%) and five difficulty tiers (Lookup → Multi-indicator). The benchmark is the schema, not the rows — any district that adopts the open-source dbt transforms gets the same tables.

Six models evaluated

The comparison is deliberately cost-tier-aware: districts can't afford GPT-4o, so the benchmark measures whether specialization wins at the cost tier most districts operate at.

• LFED LoRA v2 r=64 (ours) — the primary contribution
• Qwen2.5-Coder-14B-Instruct base (ablation: shows the LoRA effect)
• Qwen2.5-Coder-32B-Instruct (open-weight, larger base)
• GPT-4o-mini (cheap proprietary)
• Claude 3.5 Haiku (cheap proprietary)
• DeepSeek-Coder-V2-Lite-Instruct (open-weight, code-specialized)
• Llama-3.1-70B-Instruct (open-weight, large generalist)

Total eval cost: ≤ $20 per benchmark run.

The framing: accountability-metric reasoning

This is the key reframe. The benchmark is positioned as evaluating LMs on accountability-metric reasoning, not as a generic Text-to-SQL benchmark. The novelty is the failure modes on CDE accountability semantics — not the dataset size.

Examples of the domain semantics that LMs fail on:

• Suppression rules: n ≤ 10 students → field is null. LMs don't know to filter on is_suppressed.
• Reporting category codes: TA/RA/RB/RH/GM/GF/SE/EL/SWD/HOM/FOS. LMs invent aliases like "Black" instead of RB.
• Status/Change/Color grid: the 5×5 placed report CDE uses. LMs have no concept of this structure.
• Year format mismatch: CDE uses "2023-24"; Aeries uses "2023-2024." LMs join across mismatched keys.

Metrics

Three metrics, each capturing something different. Execution accuracy (EX): does the predicted SQL execute and return a result? (Too coarse — a wrong query can return the same answer.) Exact match (EM): does the predicted SQL match the gold SQL character-for-character? (Too strict — different valid SQL often differs.) Component Match (CM): a weighted score — 0.4·correct tables, 0.3·correct joins, 0.2·correct filters, 0.1·correct aggregates. This is a known technique (cf. Spider's test-suite accuracy, BIRD's soft F1); we cite it rather than claim novelty on the metric.

Beyond the metrics, a semi-automated failure-mode tagger flags each miss with a likely cause: wrong_table, wrong_aggregate, wrong_filter, syntax_error, timeout, result_subset, hallucinated_column. The failure-mode breakdown is the qualitative meat of the paper.

Where collaborator feedback would change the work

We're sharing this publicly because the work is at a stage where outside input would be valuable. Below are the specific questions where collaborator feedback could change the plan — not just polish it.

On the case study

• Are the three RQs the right ones? Is "equity" (RQ3) a strong-enough framing for an EDM paper, or would a different RQ3 land better?
• Is 3–5 participants enough for an 8-page EDM case-study paper? What does EDM 2026's strongest case-study work look like in terms of N?
• Are the four narrative arcs the right ones, or are we missing a category of district user we should be studying?
• Is 30 days of log analysis the right window, or is that too short to surface failure modes?

On K12-Bench

• Is "accountability-metric reasoning" the right reframe? Does it survive a skeptical reviewer, or is "yet another Text-to-SQL benchmark" still the headline?
• Is 220 prompts the right size, or are we under-shooting? EDM reviewers have seen 1k+ prompt benchmarks; is that a bar we should clear?
• Is the 6-model lineup the right one? Are we missing a baseline that reviewers will expect (Code Llama, StarCoder, etc.)?
• Is the schema-portability story (the schema is the benchmark) convincing, or do reviewers want a different generalization argument?

On the literature

• Are we missing important K-12 NL→SQL or accountability-data work? Especially in non-EDM venues — IJAIED, JEDM, DGO, IUI, LAK?
• Is the four-pillar framing (Prediction, Clustering, Relationship Mining, Discovery) from IEEE 11204503 the right organizing principle, or is there a more current framing we should use?

On the framing overall

• Is "practitioner case study + benchmark" the right two-paper strategy, or should we be doing something different — e.g., a single comprehensive paper, or three shorter workshop papers?
• Is EDM 2027 the right primary venue, or are we missing a closer-fit venue (LAK, IUI, AIED)?

Three ways to be involved

The minimum viable partner is someone who'll give 1–2 hours of interview time and a quote. But there are three roles — pick the one that fits.

Co-Author

For CS-ed or LA researchers who'd join the byline and shape the failure-mode analysis. Realistic ask: 5–10 hours over 4–6 weeks, mostly async. Includes a 1-hour kickoff call, async review of the failure-mode analysis, a quote for the discussion, and byline on the published paper.

Reviewer

For senior researchers who'd give one round of feedback before submission. Lower time commitment, still a material contribution to acceptance odds. Includes a 30-min call, one round of written feedback, and acknowledgment in the paper.

District Partner

For CA district IT directors, data analysts, or principals who'd test the system in their district for 30 days and share their experience. No SQL required. Includes a 1-hour onboarding call, 30 days of usage, a 1-hour exit interview, and continued access to the system after the study ends.

How to reach us

Email is the easiest. We read everything within 48 hours and reply to substantive notes within a week.

You can also star or comment on the HF model or the live demo — we check those weekly.