Methodology
AI SCREENPLAY EVALUATION
A quantitative, explainable framework for scoring, benchmarking, and discovering emerging screenwriters.
White Paper — Draft Version 1.0 (July 2026)
Prepared for methodology design, product development, and industry review.
Core Proposition
Use LLMs as scalable, evidence-based evaluators—while preserving transparent weights, calibrated score anchors, version control, and human oversight.
Executive Summary
This white paper proposes a quantitative methodology for evaluating screenplays and television pilots using large language models (LLMs). The system is designed as a discovery engine rather than a conventional coverage service: its purpose is to identify strong, manager-ready work among high submission volumes while giving writers transparent, actionable feedback.
The methodology uses ten weighted categories—Premise, Story, Structure, Characters, Dialogue, Pacing, Theme, Tone, Marketability, and Craft. Each category contains five adjustable subcategories whose weights sum to 100%. The LLM assigns evidence-backed subcategory scores; deterministic code applies the published weights and calculates category and overall scores. This separation allows the rubric to evolve without requiring every script to be re-analyzed.
The proposed launch calibration assumes an emerging-writer population with a mean overall score near 6.3, a median near 6.2, and a standard deviation near 1.0. Under that illustrative distribution, approximately 10% of submissions score 7.6 or higher, 4% score 8.0 or higher, and 1% score 8.7 or higher. These values are hypotheses to be validated against observed platform data—not empirical industry facts.
The framework rests on five principles:
Quality scores must be explainable and tied to page- or scene-level evidence.
Awards, box office, and production acclaim are not substitutes for screenplay craft.
Objective facts, LLM inferences, and evaluative opinions must be stored separately.
The top of the scale must remain intentionally sparse to preserve trust.
Discovery should combine quality thresholds with creative fit, not rank writers solely by one number.
1. Problem Definition and Product Thesis
Emerging screenwriters face a discovery problem. High-volume platforms, contests, and coverage marketplaces can generate feedback and rankings, but strong writers may still be lost in large pools. Industry representatives face the inverse problem: they need a credible mechanism that reduces hundreds of submissions to a small, relevant set worth reading.
The proposed product addresses both sides. For writers, it provides a transparent diagnostic profile and peer-relative context. For managers and producers, it provides structured search, filters, creative-neighbor matching, and a monthly spotlight drawn from the strongest scripts in each genre.
The key success metric is not whether a model can produce a plausible score. It is whether the system repeatedly surfaces scripts that experienced industry readers consider worth requesting, discussing, or representing.
2. Rubric Architecture
Each category is scored on a 1–10 scale. The category score is the weighted sum of its five subcategory scores. The overall score is the weighted sum of the ten category scores; the initial implementation may weight all ten categories equally, while preserving the option to introduce format- or genre-specific weights later.
Category Score = Σ (Subcategory Score × Subcategory Weight)
Overall Score = Σ (Category Score × Category Weight)
3. Score Anchors and Calibration
The scoring scale should be anchored to craft and execution—not awards. Awards reflect the finished film, campaign dynamics, cultural moment, direction, performance, and other factors beyond the screenplay. A screenplay may be award-winning without functioning as a near-perfect benchmark across every category.
10.0 One of the finest screenplays ever written in its genre; a benchmark of the craft; exceptionally rare.
9.5–9.9 Elite professional work that can credibly be discussed alongside the best modern screenplays.
9.0–9.4 Outstanding professional screenplay comparable to top-tier produced work.
8.0–8.9 Strong professional screenplay with realistic commercial or artistic viability.
7.0–7.9 Good screenplay with clear strengths and material weaknesses.
6.0–6.9 Competent work that demonstrates promise but is not yet consistently industry-ready.
<6.0 Developing work requiring substantial revision.
Calibration rule: scores above 9.5 should occur only when the evidence demonstrates sustained excellence across nearly every relevant category. A compelling premise alone cannot justify an elite overall score.
4. Quantitative Model and Expected Distribution
For launch planning, the following distribution is an illustrative prior for 1,000 emerging-writer submissions. It is not a claim about the true population. The platform should replace these assumptions with observed distributions once a sufficient sample exists and should report both raw scores and cohort percentiles.
5. LLM Role, Prompt Design, and Failure Modes
5.1 What the LLM should do
Extract descriptive metadata such as genre, tone, setting, story engine, and audience.
Assign subcategory scores using explicit anchors and cited evidence.
Identify strengths, weaknesses, and revision priorities.
Produce structured JSON that downstream code validates and scores.
Generate similarity features for benchmark matching.
5.2 What deterministic code should do
Validate schema, ranges, required fields, and weight totals.
Calculate category and overall scores.
Recalculate historical scores when weights change.
Enforce threshold logic and ranking rules.
Store model, prompt, rubric, and benchmark versions.
5.3 Principal LLM failure risks
5.4 Prompt requirements
A production prompt should: define every subcategory; provide 1–10 anchors; prohibit knowledge of the writer; require independent scoring before weighting; distinguish objective observations from judgments; demand page- or scene-based evidence; explain uncertainty; and return machine-readable output. The prompt should explicitly state that a 7 is good, an 8 is professional, and a 9 is exceptional.
6. Metadata and Data Architecture
The system should retain structured metadata even when the uploaded document itself is processed ephemerally and discarded. Four layers are recommended:
Privacy Note: derived metadata, scene summaries, and embeddings can still encode information about the screenplay. Product language should distinguish “document not retained” from “no information retained.” Writers should be told exactly what is stored and why.
7. Benchmarking and “Creative Neighbors”
Benchmarking should compare a script to curated peers in the same format and genre. Similarity is not quality. A script can be highly similar in tone or dialogue style while scoring lower in structure or craft. For that reason, the interface should avoid presenting “83% similar” without explanation; users may interpret it as “83% as good.”
A recommended pipeline is:
Eligibility filter: format, genre, audience, runtime, period, and production scale.
Feature comparison: premise, tone, themes, character configuration, dialogue style, pacing, and score profile.
Neighbor selection: rank comparables by weighted similarity.
Explainability: show the strongest and weakest dimensions of the match.
A composite similarity score may combine category profile, premise semantics, character archetypes, tonal profile, pacing, and dialogue fingerprint. The platform should retain the component similarities so the formula can be adjusted later and the match can be explained.
8. Discovery, Monthly Spotlights, and Manager Search
The platform should not rank purely by overall score. A defensible process uses a quality threshold, then considers creative fit, representation across subgenres, and standout strengths. For example, a script with an 8.1 overall score and 9.4 dialogue may be more relevant to one manager than an 8.6 script with weaker character work.
A monthly spotlight can combine several selection types: highest overall score, most original premise, strongest dialogue, strongest characters, strongest commercial potential, and editor or human-review pick. This reduces the false claim that the system has identified an objectively perfect top ten while still highlighting manager-ready work.
For industry users, metadata should power queries such as: contained thriller; female lead; budget under $8 million; fast pacing; high dialogue score; first-time writer; or comparable to a selected set of produced works.
9. Validation, Governance, and Continuous Improvement
Trust depends on demonstrated consistency and downstream outcomes. The system should therefore operate as a versioned measurement program rather than a static prompt.
Blind validation: score scripts without writer identity or career information.
Inter-rater testing: compare LLM scores with multiple professional readers.
Reliability testing: rescore a fixed test set after prompt or model changes.
Calibration testing: examine whether score bands correspond to reader judgments and request rates.
Outcome tracking: monitor reads, downloads, requests, meetings, representation, options, and staffing outcomes.
Bias audits: test for genre, demographic, cultural, and stylistic disparities.
Appeals and overrides: allow human review of anomalies without silently rewriting model history.
The most important business metric is precision at the top of the funnel: among the scripts spotlighted, what proportion earn serious reads or positive professional judgments? The platform does not need perfect ordering across all 1,000 submissions to create value; it must be dependable where discovery decisions are made.
10. Implementation Roadmap
Phase 1 — Controlled pilot: Build the uploader, deterministic document checks, rubric prompt, structured output, and internal reviewer console. Score a small curated set and test repeatability.
Phase 2 — Calibration cohort: Process several hundred submissions, compare with professional readers, refine score anchors, and establish empirical percentile tables.
Phase 3 — Discovery launch: Introduce genre-specific monthly spotlights, creative-neighbor comparisons, and searchable manager profiles.
Phase 4 — Outcome validation: Use request and representation outcomes to validate rankings, refine thresholds, and publish methodology updates.
Conclusion
A credible AI screenplay platform should not claim to make taste objective. It should make evaluation more structured, consistent, inspectable, and useful. The proposed methodology combines a transparent weighted rubric, evidence-based LLM analysis, deterministic scoring, calibrated distributions, rich metadata, peer benchmarking, and human governance.
The platform’s defensible advantage is discovery: narrowing large pools to a small number of high-quality, well-matched scripts while giving emerging writers a clearer path to improvement and industry visibility. Over time, trust should be earned through validation data and real outcomes—not by asserting that an algorithm can identify greatness with certainty.
Appendix A. Example Structured Output
{
"rubric_version": "feature-v1.0",
"prompt_version": "analysis-v1.3",
"model": "<model-id>",
"scores": {
"dialogue": {
"naturalness": 7.8,
"subtext": 7.4,
"voice": 9.0,
"memorability": 8.4,
"efficiency": 7.5,
"weighted_category_score": 8.36
}
},
"evidence": [
{"page": 12, "category": "dialogue", "finding": "Distinct lead voice"}
],
"confidence": 0.84
}
Appendix B. Statistical Notes
All distributions, category means, thresholds, and funnel counts in this draft are modeling assumptions developed for product planning. They should be labeled “illustrative” until the platform has enough observations to estimate empirical distributions. Confidence intervals and cohort definitions should accompany public percentile claims once live data is available.
Made by a writer. For writers.