Codebook

Version 1.0, September 2026. This is the rulebook coders use. It states what each score means at each step, which documents can support it, how silence is recorded, and how two coders resolve a disagreement. A worked audit of one live deployment follows, from the first document to the published number.

General rules

1. One line, one document. Every score above 0 cites at least one source of Tier A or B. Tier C (company statements) caps a score at 2. Tier D (independent sources) can lower a score or confirm a 0; it supports a positive score only when it quotes a Tier A document.
2. Silence is ND. A coder who cannot find a document writes ND, not 0. ND is excluded from averages and counted separately. A 0 requires either a document that shows absence or a structural fact: for example, an alert that precedes any contact with the driver cannot offer the driver a chance to be heard before it fires.
3. Score the role, not the system. Each row names one role. The same step is scored separately for the operator and for the affected person, and the two rows may cite different documents.
4. Score the documented period. Where a regulator or court describes a specific period, the profile scores that period and says so. A company's statement that practice has changed since then is Tier C and is recorded in the profile without changing the historic score until a Tier A or B document confirms it.
5. Several coders, human verification always. Data are collected and verified jointly by AI and several human coders; every score and every addition to the dataset is checked by a human. Each case receives at least two independent passes by coders who do not see each other's scores, and at least one of them is human. Scores that differ by one point are averaged and rounded toward the lower score; the simple average is published alongside the rounded score so that the effect of the rule is visible. Scores that differ by two or more points go to a third coder, and the disagreement and its resolution are logged. The disagreement log of each edition is published with the edition. A profile published before its passes are reconciled is marked "first reading" until the log is posted. From the second edition, case files are assembled jointly by a human coder and an AI assistant, the human scores first, the AI scores independently with recorded reasoning, and the human reviews the AI scores and reasoning line by line before reconciliation.
6. Weakest step. Every profile names the affected person's lowest-scoring step. A 0 at notification, explanation or reaching a person with authority marks the profile "standing blocked" whatever the average.
7. Models named. Every AI pass records the system, its version and the date of the pass in the case file and in the edition's methodology note, so that a pass can be reproduced.
8. Saved before scored. No score is entered until the document that supports it is saved in the case folder with the date of saving, and the file name is recorded in the case sheet. A Wayback Machine save is requested for every cited URL on the day of scoring. Pages change and disappear; the score refers to the saved copy.
9. Live and closed. A deployment is live if the scored system is in use in the scored form at the time of publication. Otherwise it is a closed case, scored as a benchmark and reported in a separate table.

Scale anchors by step

Each table gives the meaning of 4, 2 and 0 at that step and names a first-edition case at that level. Scores of 3 and 1 fall between the anchors: 3 where the power is documented with a material limit, 1 where only a trace of it exists.

1. Data collection

Score	Meaning	Example
4	The person can see the data used about them, by right, before or at the decision	Seattle: the ordinance requires access to the records that substantiate a deactivation
2	Categories of data are disclosed in general terms; the specific inputs are not	Upstart: variable count disclosed to investors, categories in the privacy policy
0	Data is collected without notice and cannot be seen	Rotterdam: 315 inputs, including caseworker assessments, unknown to recipients

2. Assessment

Score	Meaning	Example
4	The person is told, at the decision, that an automated system took part and what it did	Natural Cycles: the algorithm is the product and is described as such
2	Automation is disclosed at the corporate level, not per decision	Lemonade: automation share disclosed in the annual report
0	No disclosure; the person has no way to know a machine was involved	Clearview, Vermont: the person is never told a match was run

3. Decision

Score	Meaning	Example
4 (operator)	A named role holds documented authority to override the output	Aidoc: the clearance letters keep the radiologist responsible
2 (operator)	Authority exists on paper but the record shows it is not exercised	SafeRent: the landlord could not adjust the score and wrote that it could not override
0 (operator)	No person is in the decision	Uber 2018 to 2022: software acted on threshold breaches without a person
4 (person)	The person can be heard before the decision is final	Seattle: fourteen days' notice before most deactivations
0 (person)	The decision precedes any contact	PayPal: the hold is placed before any contact; Flock: the alert fires before any contact

4. Notification

Score	Meaning	Example
4	Notice is required by law or binding rule and given at the decision	Upstart: adverse action notice within thirty days; UnitedHealth: denial notice under Medicare rules
2	Notice is given in practice but not required, or given late	Deliveroo: ranking visible in the app, without notice of a change
0	The person is not told	Flock: no individual notice of a search or alert; Aidoc: the patient receives the report, not the flag

5. Explanation

Score	Meaning	Example
4	Reasons are specific to the person and sufficient to act on, including what would change the outcome	No first-edition case reaches 4
3	Reasons are specific to the person but not counterfactual	Upstart: principal reasons derived per applicant
2	Reasons are generic categories	Cigna: "not medically necessary"
0	No reasons, or reasons withheld by design	Workday: no reasons to applicants; bias-testing data held privileged

6. Correction

Score	Meaning	Example
4	The person can correct the inputs and the decision is re-run	Natural Cycles: entries can be edited and the algorithm recalculates
3	Correction exists through a third party or a formal dispute	Upstart: bureau data disputed under FCRA
0	No route to correct; or correction arrived only collectively, years later	Deliveroo: the algorithm ignored lawful absences and could not be told

7. Review and appeal

Scored on the recourse depth scale: 0 no meaningful recourse; 1 automated reconsideration; 2 human review without authority to change; 3 human review with authority; 4 independent or external contestability that is practical to use.

Score	Example
4	UnitedHealth: multi-level Medicare appeal to an independent review entity; Seattle: challenge procedure plus city enforcement
3	Lemonade: state insurance department complaint as an external route; internal appeal not documented
2	Uber: the regulator's decision came six years after the complaint
1	Flock: the only route is a court; Workday: EEOC and a collective action, years later
0	SafeRent: "we do not accept appeals"; Evolv: cancellation rights went to the school, not the student; Clearview, Vermont: no forum

8. Exit

Score	Meaning	Example
4	The person can leave without losing the service, the history or their standing	No first-edition case reaches 4
3	Leaving is possible with a modest, documented loss	Natural Cycles: deleting the account deletes the cycle history
1	Leaving costs the income, the coverage or the service	Seattle and Uber: losing the platform means losing the work
0	There is no exit	Flock: not driving on public roads; CAF: the sole benefits agency

A worked audit: Seattle deactivations

System: platform deactivation decisions for app-based workers in Seattle, governed by the App-Based Worker Deactivation Rights Ordinance, Seattle Municipal Code chapter 8.40, in force from 1 January 2025. Roles: the company reviewer (operator) and the worker (affected person). Status: live. Sources: the ordinance text on the city's site; the Office of Labor Standards rules of 24 June 2025; the OLS enforcement bulletin reporting the first fourteen months; reporting on the Ninth Circuit decision of 4 March 2026.

Step	Worker	Why	Source tier
1 Data collection	4	The ordinance requires the company to give the worker access to the records that substantiate the deactivation	A
2 Assessment	3	Reasons and records are required; disclosure that automation was used is not, but human review of every deactivation is	A
3 Decision	4	Fourteen days' notice before most deactivations; immediate deactivation is allowed only for egregious conduct, which is the reason the score is not marked down further	A
4 Notification	4	Notice must state the reasons, including the specific incidents, and provide the records relied on	A
5 Explanation	4	Same requirement: reasons specific to the incidents	A
6 Correction	3	Correction runs through the internal challenge procedure; there is no separate right to correct a record before the notice issues	A
7 Review and appeal	4	Internal challenge, then a complaint to OLS or a private action; OLS reported more than 30 reactivations and more than 20 reruns of the process in the first fourteen months	A, D
8 Exit	1	Losing the platform still means losing the income; the ordinance does not change the cost of exit	E

Worker average: 3.4 of 4, or 84 on the 100 scale. Operator average: 4.0, or 100. Human Option Gap: 17. Weakest step: exit, at 1. ND count: 0. Standing blocked: no. Reading: the ordinance created standing at every step the law can reach; the one step it cannot reach is the cost of leaving.

A worked ND: Upstart

Three of the borrower's steps are ND: reaching a person with authority before the decision, an appeal with authority to change the outcome, and a non-automated path at comparable cost. The coder searched the lender-facing compliance pages, the consumer terms, the help center and the public complaint database and found no document describing any of the three. None of the three is structurally impossible, so none is a 0. The borrower's average is computed over the documented steps only, and the profile shows "ND ×3" next to the number. The right of reply asks Upstart for the documents; a single help-center page describing a human review would turn the second ND into a score.

Change log

Version 1.0, 22 September 2026: first public version, with the rule "saved before scored" added on 22 September. Changes to any anchor or rule will be logged here with the date and the cases re-scored.

Back to the methodology