How the data is made

Question. Some official sources publish raw records instead of finished tables: a spreadsheet with one row per batch of permits (Greece), or a pivot-table workbook whose visible sheet shows only totals while the breakdowns appear only after setting filters (Australia, Home Affairs BP0068). The figure you want, for example permits by nationality in a year, is printed nowhere; it exists only once someone adds up the records. Do such figures belong in the dataset?

Decision. Yes, but only as a separate, clearly marked kind of figure:

• Status derived. Never actual or any other status reserved for printed figures.
• Only simple sums or counts of one official source's own records, as the source publishes them (rows of its data sheet, records in its pivot table). No estimates, rates, shares, averages, currency conversions or combining of different sources.
• Every derived number carries its recipe: the file, the sheet or table, and the exact filters or rows that were summed, so anyone can repeat it. The recipe is written in notes, starting with Recipe:.
• Before a derived number enters the data, a second, independent calculation from the raw file must give the same result.
• Pages and summaries show derived figures as "calculated by us from official records", never as quotes, and leave them out wherever only printed figures are claimed.

Why. These are often the only source for figures that matter (who buys, by nationality), and the arithmetic is plain and repeatable from official records. But they are not printed, so there is no page to check them against. Marking them, keeping the recipe and recalculating independently keeps the line clear between "printed in the source" and "calculated by us from the source".

Applied (2026-10-09). Greek permit spreadsheets 2016-2019: 497 figures derived, 6 printed summary cells kept as printed; all 503 matched an independent recalculation. Australian BP0068 workbook: 3,547 figures derived (3,511 new, 36 older rows converted); all matched an independent recalculation, which also reproduced every cell of the workbook's visible table. Ten Australian sums come out negative because the source data is masked (privacy perturbation); they are held back (open question 1a below).

Question. Sometimes the number isn't printed as digits: it is written as a word ("three applications"), given as a limit ("more than 700") or rounded ("about 20"), or only implied by a sentence ("there were no investment inflows"). Sometimes it can only be worked out by reasoning ("as in 2011, nearly 20 last year too"). Which of these belong in the data, and how do we tell them apart?

Decision. Every number keeps its verbatim sentence in quote, and a column reading says how the number was read:

Pages show implied figures as stated in words (e.g. "stated as none"), not as a bare number. An inferred figure is kept in the data with its sentence but never shown on pages (amended 2026-10-09 under the general principle; first decided as "not recorded").

Why. The sentence is the evidence, so it must travel with the number. Words, limits and rounded figures are still what the source says. An implied zero is real information, but readers should see that no digit was printed. An inferred number is a calculation in disguise, which decision 1 only allows in the narrow, recalculated form of a derived figure.

Applied (2026-10-09). The column is in both topics; scripts/reading.py gives a first reading from the quote and value, merge keeps reviewed readings, and scripts/check.py rejects inferred and unknown values. Every row the classifier couldn't settle, and every row it tagged implied or words, was reviewed against the document (809 rows). 45 inferred rows were removed: 39 Egyptian naturalisation counts tallied from lists of names (the decrees print no totals), and 6 whistleblowing rows (Hungary 2011 enquiries carried over from 2012; two Slovak counts made from descriptions; an SEC footnote taken as a yearly total; a CFTC sum of 1 + 23; an SEC ordinal count of awards stored as people).

Clarified (2026-10-10). approximate is for figures the text itself hedges ("about", "around", "nearly", "some", "approximately", "close to", "roughly"). A rounded figure printed without a hedge word ("EC$402.2 million") is digits.

Question. Some statistics offices add random noise to record-level data so small counts can't identify people ("masking"). Large sums are barely affected, but a sum over a few records can be noise: it can even come out negative (-5 permits). How do we treat derived figures from such data?

Decision.

• Never publish a negative count. Negative derived sums are held back and noted on the source.
• When the source is masked, derived values below 5 are stored as a range from 0 to 4 (the way publishers print small numbers, e.g. Australia's "<5"), with a note that the source is masked.
• Derived values of 5 and more are kept as calculated, with the masking noted on the source.

Why. Masking makes small sums unreliable whether or not they look wrong; a range keeps the information without claiming a precision the source doesn't have.

Question. Sometimes a document prints a number without its unit: a chart labelled "432,746" with no "in thousands of pesos", or a table headed "EC$ millions" whose cells are clearly full amounts. Keep such numbers?

Decision. Keep them, with a warning:

• A column unit_basis says printed (the default), inferred (the same document settles the unit) or guessed (nothing in the document settles it; our best judgement from outside evidence, such as the programme's own law, with the reasoning in the notes; amended 2026-10-09).
• For inferred, the notes say how the unit was chosen and what supports it (e.g. "the same quantity is printed as ₱584,270,596 in the text, so the chart is in thousands").
• Inferred-unit figures carry a warning wherever they are shown, guessed ones a stronger one; whether pages show them at all is decided separately.
• When we can, we look for confirmation elsewhere (another edition, the source's methodology note, the agency) and switch the row to printed with that source, or correct it.

Why. A wrong unit is a factor-of-a-thousand error that looks plausible, so these figures must never pass as ordinary. But the number itself is printed and the unit is usually clear from context, so dropping them would lose real data.

Applied (2026-10-09). Column unit_basis in both topics (merge fills printed by default; check.py requires notes for inferred). 77 CBI rows whose notes say the unit, scale or currency was chosen from context are tagged inferred (Philippine charts read as thousands, tables printing "$" or no currency where the currency was taken from the rest of the document, and similar). Rows where a bare "$" was recorded as the local currency are left for decision 1.4. Rows that inferred a unit without saying so in their notes can't be found this way; the extraction brief now requires the field.

Question. Many countries call their currency the dollar (US, Canada, Australia, New Zealand, Hong Kong, Singapore, Bermuda, the Eastern Caribbean states). Documents often print only "$". Which currency do we record?

Decision. Decision 4 applied to currency:

• An official document's "$" is the dollar of the issuing government (a Grenadian budget's "$" is EC$, a Canadian federal document's "$" is CAD). It is recorded as that currency with unit_basis = inferred and a note saying why.
• US federal documents: "$" is US dollars, recorded as printed.
• Where a document mixes dollars (EC$ and US$ in one report), "$" is assigned only when the table or section settles it; otherwise the currency stays unspecified, with a note.
• unspecified is kept only for genuine ambiguity (e.g. a press article where "$" could be either).

Applied (2026-10-09). 356 CBI and 21 whistleblowing rows already recorded in the issuing government's dollar are flagged inferred. Of 185 CBI rows recorded as unspecified, 122 got the currency their document settles (CAD 59, NZD 49, AUD 7, USD 4, XCD 3), flagged inferred; 63 stay unspecified (press articles; Antigua tables that print a bare "$" in a report that mixes EC$ and US$).

Why. Without a currency the number can't be compared or converted; guessing silently hides the risk of a 2.7-times error (EC$ vs US$). Recording the likely currency with a warning keeps the number usable and honest.

Question. Statistical tables often list every country or month, most with 0. Does each printed 0 become a row?

Decision. Yes. Every printed 0 or "-" (nil) is stored as a row like any printed figure (reading digits). Blank cells are not recorded; the source's notes say what a blank means in that table ("blank = zero in this edition", "blank = not available"). What pages show is decided separately (most charts leave zeros out).

Why. A zero is information ("no investors from X in 2019"). If it isn't stored, "zero", "not published" and "not extracted" look the same, which is the silent gap the coverage ledger exists to prevent.

Applied (2026-10-09). 29,517 printed zeros added to CBI: Canadian IRCC workbooks (18,382), Latvian permit tables (9,233), New Zealand residence decision reports (1,358), Malta 2021 report (544). The New Zealand 2024 edition prints "." with no legend; those 926 cells are stored as 0 with reading = implied, because the rows only add up if "." means nil. Sources that list only non-zero combinations (UK extensions dataset) have no zeros to add. Malta regulator reports 2016-2024 followed (1,344 more zeros); a table that uses blanks for none (Malta 2024 Tables 21-22) gets no rows, and its ledger entry says what the blank means.

Question. IMF (and World Bank, ECCB...) tables show a column per year. Past years are often the government's own outturns reprinted; the latest years are marked "Est." or "Prel."; future years "Proj.". What status does each figure get?

Decision. The column's own marking decides:

• unmarked past year: actual, with a note that the body relays the government's figure;
• "Est." (or a staff estimate): estimate; "Prel.": provisional; "Proj.": projection; "Budget": budget.

Revisions between report years are kept side by side: each report's figure is its own row.

Why. Status tells a reader whose figure it is. Labelling a government's outturn "estimate" because an international body reprinted it hides that it is official, and it contradicts the body's own headings.

Applied (2026-10-09). 475 IMF rows checked against their column headings: 255 changed (228 estimate -> actual, 25 estimate -> projection, 2 others); 8 unsettled (e.g. model outputs in unmarked columns, a forecast bracket that looks copied from the previous report) stay estimate with a note.

Question. A price, fee or deadline is restated by later laws, consolidated texts, official pages and new editions of a factsheet. One row per rule, or one per source?

Decision. One row per source that states it, quoted from that source. A column restates links a row to the earlier row stating the same rule (same programme, route, amount, currency, applies_to). Pages show one rule with "stated by N sources" and the latest confirmation; a restatement that differs is flagged as a possible change.

Why. Restatements are evidence: they confirm a rule is still current, and a restatement that quietly differs is how unannounced changes are caught. Linking keeps charts and "current price" from double-counting.

Applying (2026-10-09). Column restates in both rules tables, filled by merge when programme, rule, amount, unit and applies_to match exactly (257 CBI and 187 whistleblowing links). Groups that share an amount but are worded differently (645) were reviewed by hand and linked through corrections. The 429 documents whose passes skipped restated rules were re-read: about 1,100 rules added (CBI 3,149 -> 4,239 rows; whistleblowing 860 -> 1,171). The gap from one batch (rs-cbi-5), which skipped rules stated by a single source, was filled: 12 rules added.

Question. Studies, reviews, press and international comparisons often copy figures and rules from elsewhere (an EPRS study's table of minimum investments, a review quoting SEC totals). The copies are sometimes stale or wrong (the 2018 EPRS study gives Gozo as EUR 250,000 where Malta's law says EUR 220,000). Keep them?

Decision. Keep everything, and rank sources when choosing what to show:

1. Primary: the law itself, the government or agency that runs the programme, a court. Source types legislation, government (the programme's own government), court.
2. International bodies reporting their own work or relaying official figures: imf, eu, international_org (status follows decision 7).
3. Secondary: studies, NGOs, press: academic, ngo, media. Figures carry status reported (or claim), with a note naming where the source says the figure comes from.
4. Weakest: agents' and promoters' material (industry), used only when nothing else exists, labelled so.

A copied figure or rule never sets a headline value (such as the current minimum price) when a higher-ranked source states the same thing. Where it contradicts a higher-ranked source, both are kept and shown side by side as a disagreement. Observations carry the rank through their status; rules, which have no status column, carry it through their source's type.

Why. Copies are evidence of what was publicly said and sometimes the only source for a period; a contradiction between a study and the law is worth seeing. But a stale copy must not outrank the law.

Question. Agency releases list settlements (e.g. the US Justice Department's yearly False Claims Act releases) without saying whether a whistleblower brought them. Record them as cases?

Decision. Yes. Cases carry whistleblower_link: stated (the document ties the case to a whistleblower, relator or qui tam suit) or not_stated. Counts and "whistleblower cases" pages use only stated.

Applied (2026-10-10). 497 settlements added as not_stated cases from the DOJ False Claims Act year-end releases (FY2000-FY2025) and the Texas reports: many more than the 60 first counted, because earlier passes had marked most releases complete while leaving out settlements not tied to a whistleblower. Three California cases stored as stated by default corrected to not_stated. To do: check each not_stated case against court dockets or agency case pages and switch it to stated with that source where a relator is named.

Why. Most False Claims Act money comes from whistleblower suits, so these settlements are part of the record, but counting them as whistleblower cases without evidence would inflate the figures.

Question. Extraction passes propose new indicators when nothing fits. CBI ended up with 160 proposed indicators (12,379 rows), many overlapping (processing_time_months, _days, _weeks...). How do we keep them consistent?

Decision.

• One concept, one indicator. The unit and the breakdown carry the variation, not the name: one processing_time with the unit in days, weeks or months (no conversion by us, decision 2).
• Every indicator has a plain definition: what it counts, what is included and excluded, and its kind (count, money, share, duration, rate).
• New indicators are proposed by extraction passes and reviewed before publishing. Rows using an indicator that is still proposed stay in the data but are not shown.
• Consolidations are applied through data/corrections.csv, so they are reversible.

Why. Indicators decide what can be compared. Near-duplicates split one measure across names, so charts miss data; vague definitions lump unlike things together.

Applied to CBI (2026-10-09). All 267 indicators reviewed: 236 kept with reviewed definitions and kinds, 29 merged, 2 renamed, leaving 238, none still proposed. 9,888 rows moved to their merged indicator; distinctions kept as breakdown keys (applicant=, permit_type=, measure=goal...), no values converted. Main merges: one processing_time (days, weeks, months as printed; goals marked measure=goal), one residence_permits_valid (investor and family permits as applicant=), one applicant_share. The review map is in topics/cbi/reports/indicator-review-2026-10-09.json. Rows the review found misfiled (e.g. bond counts in a money indicator) are queued as corrections to existing data.

Applied to whistleblowing (2026-10-10). All 73 indicators reviewed: 63 kept with plain definitions and kinds, 9 merged, 1 renamed, 3 added, leaving 67 (map: topics/whistleblowing/reports/indicator-review-2026-10-10.json). 3,515 rows moved by merges and renames, with breakdown keys keeping the distinctions (no new disagreements appeared). 485 percentages filed in count or money indicators moved to share indicators (each checked: unit percent). 167 other misfiled rows (IRS claims, longest days in status, counts of fines, single highest awards) go through the decision 12 check before moving. The review needs a decision on the three large False Claims Act merges (relator shares into awards paid, qui tam recoveries into collections attributable, qui tam suits into tips received): applied, reversible.

Question. Agents reading documents for other purposes find errors in rows already in the data (a wrong date, an amount the source doesn't print, a figure filed under the wrong indicator). How do such errors get fixed?

Decision. Every correction to existing data:

• cites the page that shows the right value (document, location, verbatim quote);
• is drafted by one reader and confirmed independently by a second reader against the page;
• is applied only through data/corrections.csv, with the reason, the evidence and both checks named;
• if the second reader disagrees, is not applied, and the disagreement is logged for a person to decide.

Value, date and classification fixes come before note improvements.

Why. A wrong row is worse than a missing one, but a "fix" can itself be wrong. Two independent readings and a cited page keep corrections as trustworthy as the data they change, and corrections.csv keeps every change reversible.

Applied to CBI (2026-10-09). 108 queued problems. 79 fixed (5 of them in part), 18 found correct, 1 rejected by the second reader, 8 left for a person, 2 fixed earlier. 749 lines in corrections.csv, including 9 duplicate sources merged. Item-by-item outcome: topics/cbi/reports/corrections-2026-10-09.md. A second CBI round (14 items) followed on 2026-10-10: 13 fixed. Whistleblowing (2026-10-10): 482 changes suggested by extraction passes, each checked blind: 435 confirmed and applied; the other 47 redrafted with every unsupported phrase removed (mostly labels the orders don't print: "insider", "outsider", "former employee", "reduced", "criminal", or filled-in redactions), checked blind again and applied (5 with the second reader's exact correction); 2 kept as they were (a "more than $1 million" sanctions floor is not an amount collected).

Question. A document records something several countries did together (the five Eastern Caribbean programmes jointly notifying the European Commission, a regional agreement on a minimum price). One event, or one per country?

Decision. One event, filed under the region (jurisdiction INTL), with a countries column listing every country the document names. Each of those countries' pages shows it. No per-country copies.

Why. It is the most accurate record: one thing happened once. Copies per country double-count it on a combined timeline, and keeping only some copies (as the data did) makes it look as if only those countries took part.

Applied (2026-10-10). Column countries in both topics. The three country copies (Grenada, Dominica, Saint Kitts and Nevis) deleted, and their region events list the countries. Of the other 44 region events, 32 now list their countries (drafted and checked blind against the documents); 12 stay empty because the document names no country, the text is addressed to all member states, or the source isn't stored.

Question. Programmes often rest on an older general law (North Macedonia's 1992 citizenship law allows naturalisation for "special economic interest") and only later get rules that make investing the route (the 2005 decree). And sources sometimes disagree about the start (Grenada 1997: "enabled" per the US State Department, "made more attractive" per an agent's blog). Which date is the start?

Decision. Keep both.

• launched is the first official text that makes investing a route to citizenship or residence; the event gets category launch.
• legal_basis (new column in programs) is the date of an earlier law the programme rests on, if there is one; its event gets category legal_basis. Pages show both ("launched 2005, legal basis from 1992").
• When sources disagree about the start, each account is its own event, side by side; launched follows the higher-ranked source (decision 9), and the programme's notes mention the other account.

Why. Both dates are true and answer different questions (since when has it been possible in law; since when has there been a programme). Picking one silently would hide the other.

Added 2026-10-10 (Oscar left these to Claude's judgement):

• launched is the date the founding text comes into force; the publication date is used only when no in-force date is stated (the notes say which).
• A general clause allowing naturalisation for "special interest" (Austria 1965, North Macedonia 1992, Malta 2020) is the legal_basis, not the launch.
• The day applications opened (St Lucia, Hong Kong's New CIES) is its own event (category applications_open), beside the launch.
• An earlier scheme is its own programme when it had its own legal basis or a gap separates it from the current one (Grenada 1997-2001); otherwise it is the same programme with a rule change.

Applied (2026-10-10). North Macedonia: launched 2005-02-10, legal basis 1992-11-03. Grenada (1997-2001 scheme): launch event from the State Department report added; the blog's account kept as a separate event. All 129 programmes then checked against the rule: 37 proposed fixes, of which 33 confirmed by a blind second reader and applied (launch dates for Egypt, Malta IIP, Nauru, Panama, Papua New Guinea, Vanuatu, Bahrain, Argentina, Australia; legal basis for Dominica, Ireland, Malta, Panama, Qatar, Turkey, the US Gold Card, Vanuatu; 17 launch or legal-basis events added); the rest left for a person, mostly because the founding law isn't stored. Open: whether launched is the publication or the in-force date, and the law's date or the day applications opened (asked 2026-10-10).

Question. Stored documents describe routes the dataset had no programme for (Panama's forestry, agrarian and business investor categories, UAE green and retiree residence, Sierra Leone's GO-FOR-GOLD, Canada's Prince Edward Island investor stream, France's exceptional-contribution residence, UAE residence for property owners on an income). Which belong in the dataset?

Decision. A route is in scope when putting money in — an investment, donation, property purchase or deposit — is what gets the applicant residence or citizenship. Routes that rest on income or a pension (Panama's Rentista Retirado, UAE residence for property owners on a monthly income with no property minimum) are out. Rows that state general naturalisation law (the ordinary waiting period) are kept as context rows without a programme.

Why. It is the line the dataset already drew (retiree routes bought with a deposit are in, as in the Philippines). Income routes would turn the dataset into one about residence visas in general.

Applied (2026-10-10). Added: Sierra Leone GO-FOR-GOLD; Panama forestry, agrarian, small-business, City of Knowledge, call-centre, processing-zone and Panama-Pacifico investor categories; UAE green residence for investors and retiree residence; Prince Edward Island investor stream; France exceptional economic contribution residence; Panama film-industry investor. Refined (2026-10-10, after reviewing 42 programmes found by a search for uncovered countries):

• Routes that require running or managing a business (business-manager, self-employment, entrepreneur permits) are in only if a minimum investment by itself qualifies the applicant; otherwise out.
• Thresholds on a company's money (capital, assets) rather than the applicant's are out, as is mere proof of funds held anywhere.
• Buying property is in even without a minimum price (it is a purchase). Narrowed by decision 18: a minimum value or fixed payment is needed.
• Investor routes whose amount the law leaves to a later decision are in (amount recorded as unset).

Already in the data but out of scope by these tests: Croatia's business residence (investment plus employees and salary). Its rows are kept, the programme is marked out of scope, and pages don't show it.

Already in the data but out of scope by this test: Mauritius's residence permit for retired non-citizens (an income transfer, not an investment). Its rows are kept, the programme is marked out of scope, and pages don't show it. Left out until a stored document shows it requires an investment: Panama's "Inversionista o Empresario" (341 approvals 2010-2021).

Question. St Kitts and Nevis audit reports show government capital spending on the Hurricane Relief Programme, which one CBI investment option financed (its proceeds were collected as unit revenue). Is that spending of CBI money?

Decision. Record it as printed under its own indicator (spending on a programme financed through a CBI option), with a note that the report does not trace which money came from CBI. It is never added to "CBI spending" totals.

Why. Extract everything, but claim no more than the source says: money is fungible, and the report links the programme to the option, not each dollar spent.

Question. Some documents hold things in scope that no amount of work will get out: a chart with no printed values, a page missing from the only copy, text behind a paywall or bot check we don't get around, digits too blurred to read. The coverage ledger marked these documents partial, the same as documents with work still to do. How do we tell the two apart?

Decision. The document stays partial, and its to_do starts with the words "Not extractable:" followed by what and why. If something extractable is also left, it is listed first and the "Not extractable:" part follows. Values are never estimated from a chart's bar heights or a picture (decision 2). Coverage summaries count the two kinds separately.

Why. "Partial" must keep meaning that something in scope is missing, so nothing is presented as complete when it isn't. But a reader, and the next extraction pass, need to know which gaps are work left and which are limits of the sources.

Applied (2026-10-11). Of 1,681 stored documents, 30 are partial: 29 only for things that can't be extracted (mostly charts without printed values, missing pages and paywalls) and 1 with an optional extra calculation left (more cross-tabulations from the Australian pivot workbook, which prints none of them).

Question. The dataset grew from citizenship by investment into investment migration in general: 44 citizenship programmes, 146 residence programmes, and about 26,000 numbers about immigration in general (Canada's and New Zealand's full residence tables) or context (St Kitts sugar). Some routes in scope are ordinary rules for foreigners who own a home, with no price. Should the dataset be split, and where is the edge?

Decision.

• One dataset, published as two products: citizenship by investment and residence by investment, each with its own page, counts and download, built from the same tables (programs.kind). Sources, the coverage ledger, corrections and checks stay shared, because many documents cover both and residence often leads to citizenship.
• A route is in scope when a set investment or payment is what gets the applicant residence or citizenship: a minimum amount, or a fixed payment, printed in law or official guidance (or left by law to a later decision that sets one). Owning a home with no minimum value (Albania, Belarus, Serbia) is out: it is a general rule for property owners, not something sold. Tax residence without a residence permit (Uruguay) is out. A residence permit granted for paying a set tax (Swiss lump-sum taxation) stays in, since a fixed payment is the reason for the permit. This narrows decision 15's "buying property is in even without a minimum price".
• Numbers about immigration in general, and context figures, are kept (decision "extract everything") but carry no programme, and pages and counts leave them out.

Why. Readers of a page called citizenship by investment should not count golden visas as passports, and the residence side is a real subject of its own. One set of tables keeps every fix in one place. A price, or a set payment, is what makes a route something a country sells.

Applied (2026-10-11). Out of scope (rows kept, not shown): alb-property-residence, blr-property-residence, srb-property-residence, ury-tax-residence-investment. Programmes whose minimum is set by a decision not yet stored stay in with the amount unset (Turkey property residence since November 2024).