The risks we see, and what we decided
A candid look at the four pressures on this research plan, the reframings we chose in response, and the trade-offs we accepted.
Public-facing research notes · Last updated 2026-07-18
Every research plan has pressures — places where the framing, the methodology, or the contribution could collapse under reviewer scrutiny. This document is the public-facing version of the strategic pressure test we ran on our own plan. The four risks below are the ones we think are most likely to land; the reframings are what we did about them.
We're publishing this because the reframings are defensible but not obvious, and outside input could either strengthen them or reveal a better path. If you've reviewed for EDM, JEDM, IUI, LAK, or AIED — or if you know K-12 ed data deeply — we want your reaction. The questions at the bottom of the page are where collaborator feedback would most change the work.
The case for naming the risks
It's tempting to only publish the polished version of a research plan — the one where the framing lands cleanly, the methodology is settled, and the contribution is unambiguous. We chose not to do that, for three reasons.
- It pre-empts reviewer feedback. If a reviewer raises Risk A, we want to be able to point to a public document that says "we saw this, here's how we addressed it." That's a stronger position than a paper that acts like the risk doesn't exist.
- It invites collaborator input at the right moment. The reframings are still soft. If a CS-ed faculty member reads this and says "the accountability-metric framing is too clever — reviewers will see through it," we want to know that before we submit, not after.
- It signals intellectual honesty. The research is in progress. A document that admits the plan could be wrong reads as more credible than a document that pretends the plan is settled.
The four risks below are ranked by how loud a reviewer objection we expect them to be. Risk A is the loudest; Risk D is the quietest. The reframings get sharper as we go.
Risk A — "Yet another Text-to-SQL benchmark"
This is the loudest objection a reviewer will raise. The Text-to-SQL benchmark space is saturated — BIRD, Spider, ScienceBenchmark, SParC, KaggleDBQA, EHRSQL, Spider 2.0. EDM reviewers especially will say: why is this not just a smaller BIRD?
Our reframe
The benchmark isn't really about Text-to-SQL as an NLP problem. It's about whether LMs can reason about state-accountability metric semantics — the Status/Change/Color grid, suppression flags, reporting category codes (TA/RA/RB/...), aggregate-level filters, year-over-year deltas with district rollups. That is genuinely K-12-specific and not captured by BIRD or ScienceBenchmark.
Why we think the reframe works
The title would change. We're not calling it "K12-Bench: a Text-to-SQL benchmark." We're calling it something like "K12-Bench: An Accountability-Metric Reasoning Benchmark for LMs in Education". The contribution becomes the failure modes, not the dataset size. The risk: it could read as a label change rather than a substance change. We have to make the substance back the framing.
Risk B — Solo-author + system-paper combo lands weak at EDM
EDM historically rewards work with educational impact: an intervention, a deployment, a study with students. A pure system/benchmark paper with one author and no empirical study can be seen as "engineering work, not research." EDM 2026 had ~30% system/demo papers, so it's not zero, but acceptance is harder.
Our reframe
Frame the LoRA itself as the "intervention" being studied: "We trained a specialized model; how well does it generalize? What does the gap to GPT-4o tell us about K-12 ed data complexity?" This puts the paper in the LM-evaluation EDM sub-thread (cf. Ahtisham, Park, Scheinberg in EDM 2026) rather than in the system-paper bucket.
Why we think the reframe works (and where it might not)
The reframing moves us into a more crowded LM-evaluation space, but it's a space EDM is actively engaging with. The risk: if our LM evaluation is shallow — just an accuracy table — it doesn't land. The 6-model comparison with failure-mode analysis is the part that has to be deep.
Risk C — Cheap proprietary models will look weak next to full SOTA
GPT-4o-mini and Claude 3.5 Haiku are weaker than their full counterparts. If GPT-4o (full) crushes everything, our paper becomes "we compared cheap models to a 14B LoRA" — not strong. If 4o-mini happens to be similar to 4o on this task, fine. If 4o-mini < 4o by 10+ points, reviewers will say "you should have used the real models."
Our reframe
Lean into the cost framing. Add a third proprietary-tier-free model — Qwen2.5-Coder-32B-Instruct (via OpenRouter, ~$5 for 200 prompts) — so the comparison is 14B LoRA, 14B base, 32B base, two *-mini, 70B, and DeepSeek. The cost table becomes a first-class contribution.
Why we think the reframe works
Cost-aware deployment is a real EDM concern. Districts can't pay for GPT-4o. A 14B local model that beats 4o-mini on K-12 ed data is a publishable finding: "specialization wins at the cost tier most districts operate at."
Risk D — The "real warehouse" claim generalizes from one district
Our warehouse is built on Aeries (one major SIS) and CDE (one state). A reviewer will ask: "Are you benchmarking K-12 ed data, or one district's K-12 ed data?"
Our reframe
The benchmark is the schema, not the rows. Any district that adopts the open-source dbt transforms gets the same main_core / main_analytics / main_staging tables. The benchmark is reproducible across districts. This is the open-source / data-model story.
Why we think the reframe works
The schema-portability argument is the lever, and it pre-empts the generalization objection if we write the schema-portability section properly. The data model documentation we already have (docs/PHASE4A_DATA_MODEL_DESIGN.md) becomes cited material. The risk: a reviewer could push back and ask for actual cross-district validation, which we don't have. We've decided that's a Phase 3 problem.
The pivot record
Beyond the four risk reframings, we made several sequencing and framing decisions that shape the work. They're listed here in priority order.
1. Sequence: case-study paper first, K12-Bench second
The case study is faster (4–6 weeks), lower risk, and gives K12-Bench a real user story. Two papers, even if the second is the stronger one, beat one paper. The case study also gives the failure-mode analysis a human context that pure benchmark numbers don't.
2. Framing for K12-Bench: "accountability-metric reasoning"
Not "Text-to-SQL benchmark." The contribution is failure-mode analysis on CDE accountability semantics. Defends against the "yet another BIRD" objection.
3. Solo author + LM-evaluation framing (for now)
Fits EDM's LM-evaluation thread (Ahtisham, Park, Scheinberg in EDM 2026) better than the system-paper bucket. Co-author being hunted, but not blocked on it. Minimum-viable co-author: someone on the byline + 1–2 hours of interview + a quote. Doesn't need to be senior academic.
4. 6-model lineup for K12-Bench
LFED LoRA v2 r=64 (ours, primary), Qwen2.5-Coder-14B base (ablation), Qwen2.5-Coder-32B base, GPT-4o-mini, Claude 3.5 Haiku, DeepSeek-Coder-V2-Lite, Llama-3.1-70B. Total cost ≤ $20 per benchmark run. The cost tier matters as much as the absolute accuracy.
5. Venues: EDM 2027 first, IJIET fallback
EDM 2027 (Full Paper or Industry / Demo track) is the primary target for both papers. IJIET is a rolling-submission fallback that fits the case-study paper's applied tradition. EDM Posters & Demos was considered as a stepping stone; rejected because the case study is long enough to need a full paper slot.
6. Co-author not required to be a co-author
A CS-ed advisor or district IT director as reviewer materially improves acceptance odds. Acknowledged reviewer contributions are recognized in the paper's acknowledgments section. The minimum-viable co-author threshold is intentionally low.
What we gave up
The reframings and decisions above are not free. Each one closes off a path we could have taken. Listing them here so the trade-offs are explicit.
What the case-study-first sequencing costs us
K12-Bench is the stronger paper — it's the one that fills the EDM 2026 gap. By sequencing case-study first, we're betting that the case study lands well enough to give K12-Bench a citation advantage when it goes out. If the case study doesn't land, K12-Bench loses its grounding.
What the accountability-metric reframe costs us
If the reframe works, we get a defensible paper with a strong contribution. If it doesn't — if reviewers see the same paper under a different name — we've burned credibility. The reframe only works if the failure-mode analysis is genuinely about CDE accountability semantics, not generic SQL errors. We've committed to making the failure-mode analysis the meat of the paper.
What the cost-tier-only model lineup costs us
By excluding GPT-4o (full) and Claude Sonnet (full), we lose the chance to compare against the strongest proprietary models. If a reviewer says "you should have used GPT-4o," we don't have a clean rebuttal beyond "we prioritized cost-tier realism." That argument has to land.
What the schema-portability argument costs us
We don't have cross-district validation of the schema. We've decided that's a Phase 3 problem. The risk: a reviewer asks for it, and we don't have it. The mitigation is to be explicit in the paper that the schema-portability story is a claim, not a result, and that cross-district validation is future work.
What the solo author stance costs us
A senior academic on the byline would increase acceptance odds materially. We've decided to hunt a co-author (CS-ed faculty or district IT director) but not block on it. The risk: solo + system-paper + new framing = three reasons for a reviewer to say no. The mitigation is the LM-evaluation framing + the cost-tier story + the case-study grounding.
Where pushback would change the work
The four risks above are the ones we identified. There may be others we missed. Below are the specific questions where outside input would most change the plan.
On Risk A (the loudest one)
- Is the "accountability-metric reasoning" reframe strong enough to overcome "yet another Text-to-SQL benchmark"? Or does the field see right through it?
- Are the CDE accountability semantics genuinely novel contributions, or are they just SQL quirks with a domain wrapper?
- Has anyone seen a K-12 NL→SQL or accountability-data paper that we should be citing? Especially in IJAIED, JEDM, or LAK?
On Risk B (the framing risk)
- Does the "LoRA-as-intervention" framing work in EDM's LM-evaluation sub-thread, or is it still seen as system-paper work in disguise?
- What does the strongest EDM 2026 case-study paper look like — what's its N, methodology, and contribution? Are we aiming for the right bar?
On Risk C (the cost-tier risk)
- Is "specialization wins at the cost tier districts operate at" a strong enough story to anchor a paper? Or do EDM reviewers want the strongest models regardless of cost?
- Should we add a budget "real" SOTA model (GPT-4o or Claude Sonnet) to the comparison just to pre-empt the "you should have included it" objection?
On Risk D (the generalization risk)
- Is the schema-portability argument convincing on its face, or do we need actual cross-district validation to defend it?
- Has anyone published a portable K-12 ed data schema we should be aligning with?
On the trade-offs
- Are there trade-offs we're missing?
- Is there a path that gives us more of what we want without paying the costs we've identified?
Tell us where we're wrong
The reframings are defensible but not obvious. If you see a stronger path — or a fatal flaw — say so. We'd rather learn it now than from a reviewer.