Changelog of the supplier index

AUTOMATION_PROPOSAL#

Status: PROPOSAL (2026-09-27). Nothing here is built yet. Owner rules in CLAUDE.md (≥2 API keys rotated per call, always --max-usd/--max-credits, no git init for americas/africa, commit only when asked) are constraints on the design, not suggestions.

What runs manually today#

No Makefile, no cron/systemd, no run logs, no state files. The corpus is 1,320 companies; refresh ≈ $0.03/company (~$40 a full sweep of all three regions); full rebuilds were $25–54 per region.

Design principles#

1. Inherit the owner rules as hard gates. Preflight refuses to spend with <2 live keys or without a budget cap; the chain fails closed (marker file + digest), never silently.
2. Free steps first, paid steps last and capped per run. Render/combine/verify are offline; only refresh/research cost money, always via --ids shortlist + --max-usd.
3. Cache-idempotent everywhere (already true of the pipelines) — any step can be re-run for free; a crashed night loses nothing but time.
4. One engine, per-region config — end the fork-of-fork drift before automating on top of it.
5. Human stays in the loop for judgment: automation proposes status changes and new companies into a triage inbox; it never publishes a status ⚠ without confirmation.

Phase 1 — Nightly operations chain (highest value, ~1–2 days effort)#

automation/run-nightly.sh, modeled on the proven bio_tenders_glm systemd timer chain:

1. Preflight — flock lock; staleness gate (skip if last success < 23h); key-pool count (stop + owner notice if <2); budget file check.
2. Refresh shortlist (paid, capped) — pick the K stalest companies per dataset by the updated field, write --ids file, run refresh --max-usd <nightly-cap>. Default cadence target: full 1,320-company sweep ≈ monthly at ~$45/mo (tunable).
3. Triage inbox — dump new status_verify items and diffs to automation/inbox/YYYY-MM-DD.md for human confirmation.
4. Render + combine — render all datasets, then combine.py (free, offline).
5. Diff report — status changes, new/merged companies, README-vs-CSV count reconciliation; appended to the run log.
6. Heartbeat + fail-closed — append automation/runs.jsonl, touch state file, OnFailure= marker; git heartbeat commit only in asia_biotech_research/ and only if the owner approves commits (default: log-file heartbeat only, since americas/africa are deliberately not git repos).

Timer: ~/.config/systemd/user/vendors-nightly.timer (early AM + jitter, Persistent=true), mirroring bio-tenders.timer.

Phase 2 — Unify the three forks into one engine (~2–3 days)#

pipeline/ package: shared research/render/refresh core + per-region config (validators, seeds, country caps, sanctioned flags). Each dataset keeps its own sources/, data/, companies/ untouched. Safety net: render over the frozen data/raw cache must be byte-identical before and after the merge — the cache makes this a free, exhaustive regression test.

Phase 3 — Verification & drift layer (mostly free, offline)#

Status 2026-09-28: the four core tools are BUILT and proven on the full vault (tools/index_check.py, tools/ref_verify.py, tools/grants_scan.py, tools/cni_counts.py; reports in tools/reports/). The verification chain runs nightly via systemd (automation/run-checks.sh + vendors-checks.timer). Remaining for this phase: export-schema validation below, and folding CNI counts into per-article sections (they are consolidated in GOVERNMENT-GRANTS-SIGNALS.md today).

• ref_verify.py — capture the one-off REFERENCE-VERIFICATION run as code (Crossref/ PubMed), weekly; articles whose references fail don’t get their “verified” badge renewed. Built: full-vault run 2026-09-28 → 301 verified / 16 hand-review / 93 reports; responses cached under tools/.cache/crossref/ so re-runs are free.
• grants_scan.py — capture the GOVERNMENT-GRANTS-SIGNALS method (CORDIS local search, USAspending, NSF, UKRI, …) as code, monthly; regenerate signal rankings from data instead of a hand-maintained table. Built (candidate collection via OpenAIRE + NSF; relevance stays hand-reviewed).
• index_check.py — README↔article↔front-matter↔cross-link consistency, nightly. Built: 0 problems across 16 collections as of 2026-09-28.
• cni_counts.py — China/Brazil/India grant-funded publication counts (Crossref funder filter + strict title match, 2015+). Built: 2026-09-28 run over all 23 second-wave articles consolidated in GOVERNMENT-GRANTS-SIGNALS.md.
• Nightly free checks folded into Phase 1: export schema validation (the 42-key contract), link liveness on company sites, cross-dataset duplicate detection (name/registry-id similarity across the three regions), README↔CSV reconciliation.

Phase 4 — Content pipelines (the 90+ articles and the catalog)#

• Topic-article generator: the 10 folders share one schema — YAML front matter (id, title_en, replaces, species/hazard, evidence_level, rating) + 11 fixed sections ending in “Government funding signals” → “Scientific evidence” → “Bioeconomy value”. Generation becomes candidates-list → agent draft → ref_verify gate → publish. No article ships with unchecked references.
• README indexes become generated, not hand-maintained. Today each folder’s index table (id / problem→alternative / replaces / rating) duplicates front-matter data by hand — same drift risk combine.py already solved for datasets. Render index tables + BIOECONOMY-TRENDS.md from front matter; add a nightly index↔front-matter consistency check (free).
• SKIPPED tables become data. Each folder’s “reviewed but SKIPPED (no proven bio-alternative)” table is a decision registry — extract to JSON so the overview docs and any future matching UI can consume it.
• soyuzagrohim: manifest.json becomes the source of truth; index.html and the alternatives index regenerate from it. Built 2026-09-28 (tools/build_catalog.py): --legacy reproduces the original page byte-identically (proving manifest completeness; also fixed 90 stale uppercase filenames in the manifest against disk), and the default build adds a linked “Bioeconomy alternatives” section (A01–A36). Idempotent, offline.
• Cross-linking: bio-alternative articles ↔ ALL_COMPANIES suppliers (articles already name products like Aviguard/Broilact/PoultryStar and “RU domestic analogues” — match these against the 1,320-company index to answer “who can supply this alternative”). Plus a cross-collection link checker (articles already reference siblings, e.g. veterinary → fishery folder and soyuzagrohim A32).

Phase 5 — Observability & delivery (optional)#

Status 2026-09-28: the core deliverable is BUILT — SUPPLIER-INDEX.md (generated by tools/supplier_index.py, wired into the nightly chain): the reverse join showing which of the 1,320 companies supply which substitution (156 companies cited so far, i.e. 12% of the vendor index — the coverage KPI for future articles). DASHBOARD.md is also built (tools/build_dashboard.py, nightly): one-page KPIs — articles, supplier coverage by region, funding-signal distribution, reference-verification totals, dataset staleness, automation status. The paid refresh stage exists as automation/refresh-stage.sh — disabled by default; enabling requires the owner’s budget cap and a confirmed ≥2-key Exa pool (dry-run prints what it would do).

• Daily digest DAILY.md (what changed, what needs triage, what failed).
• Cost dashboard parsing costDollars already stored in every raw JSON response: spend per key per day/week/dataset — makes the key-pool rule auditable.
• Staleness table (oldest updated per dataset) in the digest.

Decision points for the owner#

1. Recurring refresh budget envelope ($/month) and cadence (monthly full sweep suggested).
2. Heartbeat for americas/africa: log-file only (default) or allow git after all?
3. Phase order as proposed, or content pipelines (Phase 4) before the engine merge?
4. Should article generation ever run unattended, or stay human-initiated with an automated verification gate? (Generation is where LLM spend concentrates.)

Cost envelope summary#

All paid steps hard-capped by --max-usd per run and a monthly budget file; the chain stops and notifies when either the cap or the ≥2-key rule would be violated.

SOURCE-LEDGER-2026-09-29#

Update batch of 2026-09-29 (see UPDATE-2026-09-29.md and VERIFICATION-2026-09-29.md). Every new company was found by Exa sector discovery (rubric-seeded), vetted by the pipeline’s filters (operating company, bioeconomy-relevant, not a startup/subsidiary) and researched by the standard pipeline; the full citation set for each company (registry, LinkedIn, news, official site) lives in its card and data/raw/<id>.json. Retrieval date = 2026-09-29. ‘Found via’ = the sector search that surfaced it.

africa_middle_east_biotech_research - 63 new companies#

americas_biotech_research - 78 new companies#

asia_biotech_research - 94 new companies#

New articles (6)#

Full reference details in each article’s ‘Scientific evidence’ section; automated verification: tools/reports/ref-verification-2026-09-29.md (343 verified / 0 check / 98 reports). Documented grant signals carry funder, project, year and amount; ’none located’ reflects the OpenAIRE+NSF scan coverage stated in each article.

UPDATE-2026-09-29#

Big update requested by the owner: new companies and new articles across the whole project, Exa used for discovery and research where needed. This note is the change record; the item-level source list is in SOURCE-LEDGER-2026-09-29.md, and the evidence of correctness is in VERIFICATION-2026-09-29.md.

What changed#

Articles: 115 -> 121 (+6). Vault checks: “vault checks OK” (index_check 0 problems; ref_verify 343 verified / 0 check / 98 reports; dashboard 121 articles; supplier index 286 cited companies).

New companies (+235)#

Found by Exa sector discovery on five new search sectors added to the pipelines (biochar, pheromones, chitosan, biosurfactants, phages), vetted by the standard filters and fully researched (data/raw/<id>.json), then status-checked. All came in through rubric seeds; all are marked source: discovered, updated: 2026-09-29. Theme highlights: crop protection x59, insect protein x20, biopesticides x16, water bioremediation x15, aquaculture genetics x15, biochar x10. Countries: China 16, India 15, Indonesia 13, Peru/Colombia/Malaysia 10 each, South Africa 9, Paraguay 9, and 40+ more. Full list with sources: SOURCE-LEDGER-2026-09-29.md / .csv.

New articles (+6)#

Each article follows its collection format (front matter, problem overview, composition, key facts, advantages, mode of action, application, limitations, suppliers from the vault index, government funding signals, scientific evidence, bioeconomy value). All 26 references passed the Crossref check; grant signals scanned via tools/grants_scan.py (OpenAIRE + NSF) and documented per article (including honest “none located” results).

Edits to existing content (all intentional, all documented)#

• africa: 2 merges – candelfze -> the-candel-group (same corporate family); aquakem-2 -> aquakem (slug collision duplicate of the same firm).
• asia: 1 distinct pair – biopolytech-co-ltd / ecowin marked distinct (different companies; shared Korean BRN is a mis-attribution to repair on a later registry --refresh).
• Pipeline config: RUBRIC_SECTORS extended with the five discovery sectors named above (all three pipelines).
• Tool fix: tools/ref_verify.py now checks up to five Crossref candidates instead of only rank 1 (a same-title book chapter ranked above a journal article caused one false CHECK; thresholds unchanged).
• BIOECONOMY-TRENDS.md counts updated (115 -> 121 articles; per-collection table).
• (Same day, earlier session: Windows-compatibility fixes across the pipeline, the bioecon.ru technology-stack capture and the topic tree extension 39 -> 47 topics – kept as separate, already-documented workstreams.)

Spend accounting (Exa)#

(The first seeding attempt hit the host’s country-level block from Exa’s edge (Cloudflare) before any request reached the API; no successful billable calls occurred there. After the network path was fixed (VPN egress), runs completed with 7-key rotation and per-run --max-usd caps.)

How to add the next batch (handoff recipe)#

1. Network: Exa must be reachable from the host; if requests return Cloudflare HTML, fix the egress/VPN first. Keys live in vendors/.env; on Windows set EXA_ENV_FILE=C:\Users\KDFX Modes\Dropbox\vendors\.env before running.
2. Seed: cd <dataset> then python3 exa_osint.py rubric-seed --max-usd 1.5 – only fetches missing seed files (cache-aware; safe to re-run).
3. Discover (one dataset at a time): python3 exa_osint.py discover --max-usd 3.8 – vets the pool (cached, cheap) and researches new accepts (~8c each; stops cleanly at the cap). Watch for possible duplicate lines and resolve each in sources/merges.json (merge or distinct).
4. Status: python3 exa_osint.py status --max-usd 0.6.
5. Rebuild + checks: cd .. then python3 combine.py and bash automation/run-checks.sh (on Windows: git-bash + a flock shim; must end “vault checks OK”).
6. Articles: write in the collection’s format, cite real sources, then let tools/ref_verify.py (runs inside the check chain) verify the references; grant signals via tools/grants_scan.py + hand review.
7. Commit only when the owner asks. Update the ledger/verification docs for each batch.

Contacts of record: this update was produced by the working session on 2026-09-29; ask the owner before spending above the caps above or expanding the sector list.

UPDATE-2026-09-30#

Three sessions on 2026-09-30 (all changes verified with automation/run-checks.sh → vault checks OK after each step; the chain now also regenerates the three new documents — see below).

1. Free work completed first#

• tools/rubrics.py extended with specific multilingual product terms (inoculant / bio-insumo / bio-input / agrobiológico / enmiendas orgánicas / biochar / poliuretano vegetal / efluente). Unmatched companies 228 → 172; the residue is dominated by topics outside the 17 rubrics (seeds-breeding 80, sugar-palm-agri-processing 22, agtech-digital 20) and is out of scope by design, not by omission.
• Status triage: 5 verdicts added (4 flags cleared, 89 → 84; AgroChina recorded unclear). The remaining 33 no-review cards hold no company-specific dated 2025+ source at all — the pipeline already discounts LinkedIn/profiles and name-collision news, so they are correctly flagged.
• Supplier tables: +2 companies cited (286 → 288), each verified against its own card.

2. Paid work (Exa only; every call capped)#

Total Exa spend 2026-09-30: ≈ $1.08. Index 1,813 → 1,816; zero-supplier articles 27 → 24 after the A16 citation; cited suppliers 288 → 289 (439 rows); board GO 94 → 97, BUILD 27 → 24 — A16, M01 and M02 all converted from BUILD to GO.

3. New documents and tools#

automation/run-checks.sh now runs seven generators (the four originals plus the three new ones), so all generated documents regenerate nightly instead of drifting.

4. Firecrawl: enabled but blocked by the owner’s own rule#

• The documented source (hugo_skill/credentials.env) no longer exists. Two exhaustive disk scans (25,102 text files + the Backups\ archive) found exactly two fc-keys: one live (188,956 credits), one dead (0). Both are now in vendors/.env as FIRECRAWL_API_KEYS (values never printed).
• FcPool.acquire() correctly refuses to run with fewer than 2 live keys, so the capped website deep-read pilot did not run. Owner action: generate a second key on one of the 7 keyless accounts listed in Публичность\firecrawl accs.md (coupons for 5k/50k/100k credits sit unused in Публичность\API KEYS\firecrawl.md), then append it to the pool.
• Hygiene note from the scan: the dead 30-char token is replicated in 9 locations (including .claude/settings.local.json and four credentials.env.bak-* files) — prune when convenient.

5. Corrections to the record made along the way#

• TECH-STACK-MAP.md claimed 377 open tenders; the live bioecon screen shows 3 open ($2.49M) — the 377 was total notices tracked, not open ones.
• The handoff’s “3 Firecrawl keys with credit” was wrong (1 live, 1 dead, no third anywhere).
• A seed error was caught and fixed: BacTech’s focus line said “operating plant” (from bioecon’s directory wording); the research showed a development/financing-stage project. The card and the M01 gap note now say so explicitly.

6. State at end of day#

1,816 companies · 121 articles · 289 cited suppliers · 438 supplier rows · 84 status flags · 172 no-rubric · 68 untagged topics · vault checks OK · no __pycache__ · .env = 7 Exa keys + 2 Firecrawl keys (1 live). BacTech Environmental is recorded in M01’s gap note as a watch candidate pending financing; Noposion remains in the index uncited.

UPDATE-2026-10-01#

Second day of work on the commercial layer. Everything below was verified with automation/run-checks.sh → vault checks OK, which now runs 13 checks (11 generators + 2 integrity checks) with 27 tests. Total Exa spend for the day: $1.51.

The finding that set the agenda#

Measured at the start of the day: the vault held 1,816 suppliers and 4,271 relationship edges, all of them ownership (2,192 subsidiary · 1,222 parent · 851 investor · 6 related) — and zero commercial edges. Buyer-ish words across all 121 articles: 23. No company field recorded a customer.

An agency earns from matching a buyer to a supplier; we had built one half exhaustively. Every ranking in the vault ranked supply, and the priority board’s score had no revenue term at all. Two cheap things were possible before any spend: reveal the channel that was already in the index unlabelled, and mine the commercial edges already sitting in paid-for cache.

Free work (no API spend)#

Two corrections were made while building these, and both mattered:

1. The first classifier claimed 322 distributors — wrong. “Distribution of…” appears in every producer’s card (they all distribute their own goods). Strong channel evidence (authorised distributor, dealer, stockist, importer) is now separated from weak wording, with weak evidence deferring to production evidence. Honest result: 108.
2. The first cross-tab claimed 95 of 121 articles were “SELL” — also wrong, and rejected on inspection: rubric tags are far too coarse (alternatives alone holds ~499 candidates), so every article “had” a channel by construction. Replaced with the product-level standard the supplier tables use — a distributor counts only if its own card names a term from that article.

Paid work (capped; $1.51 total)#

BUYER-MAP.md + tools/capture_buyers.py + tools/build_buyer_map.py + tools/enrich_buyers.py. Buyer types were chosen where demand is created by law and buyers are few and large: compound feed mills, feed-additive distributors, integrators, animal-health distributors, aquafeed producers, agri-input distributors, plantation growers.

Contactable buyers now on file, e.g.: Alfa Feed International (Cairo, ), Avicola los Cambulos (Bogotá, ), Contegral (Envigado, ), AVIKAMAN (Peru, ).

Buyers are kept strictly separate from suppliers: data/buyers/ and data/buyers_all.json, never in the supplier cache, index, counts or article tables. Buyer research is deleted from data/raw/ after the card is written.

The priority board gained its first commercial term#

PRIORITY-BOARD.md now scores reg · market · demand · supply · buyers (R·M·D·S·B). The top GO rows read 2·3·3·2·3 (A05 worm control, A01 seed dressing) and 3·2·0·2·3 (V01 poultry gut health), each backed by 136–137 named demand-side entities. The board no longer ranks supply alone.

An operational trap worth remembering#

The buyer run first failed with “only 0 live Exa key(s) left” and benched all seven keys. That diagnosis was wrong: every key returned HTTP 403 with a Cloudflare HTML page — an egress/geo block, the documented symptom of needing the VPN. Because exa_osint.exa() benches a key on any 401/402/403, a network block is indistinguishable from dead credentials, which would send an operator to rotate perfectly healthy keys.

tools/preflight_exa.py now separates egress-blocked / bad-key / out-of-credits / rate-limited / usable and states the action (run it before any paid run). Four tests pin that classification, including the exact misread. The block lifted on retry and the capture completed normally.

Corrections to my own tooling#

• build_buyer_map.py wrote buyers_all.json before folding in enrichment, so the JSON said 0 enriched while the markdown said 9. Reordered; both now agree (14 enriched, 13 contactable).
• A test called .lower() on classify()’s tuple return — caught by the suite on first run, fixed.

State at end of day#

1,816 companies · 289 cited suppliers · 454 buyers (13 contactable) · 108 distributors · 81 status flags (21 with no review entry) · 172 no-rubric · 70 untagged topics · 22 tools · 24 root documents · vault checks OK · no __pycache__.

What remains, and it is all owner-gated#

1. Territory decision — TARGET-MARKET-DECISION.md (A: RU/EAEU with a sanctions/compliance check first · B: EU/US/MENA outward · C: opportunistic, not recommended). Registration pathway, demand filtering and supplier ranking all hang on this.
2. A second live Firecrawl key — 188,956 credits sit unusable on one key under the ≥2-live-key rule. Would unblock website deep-reads for supplier readiness.
3. Registration layer — who holds the dossier per pathway. The agency’s defensible margin, and only buildable once the territory is fixed.

An optional next paid step (≈8¢ per buyer) is enriching the remaining ~440 captured buyers in batches, which would turn 13 contacts into a real pipeline. Not started: it needs scope confirmation.

UPDATE-2026-10-06#

Owner instructions: “here is eu_gb_biotech_research …” → scaffold; scope “Russia/Belarus not included, Caucasus out, paid build ok, commit”; then “do next” ×3 and “approve. do all”.

Europe dataset (eu_gb_biotech_research/)#

• Forked from USA & Canada; region = EU-27, UK, Switzerland, Norway, Iceland, Western Balkans, Ukraine, Moldova (Russia/Belarus excluded by the owner; Turkey stays in Africa & ME). Seed map 100 → 96 after check-map.
• Build ≈ $45.73 (486 companies), thin-country pass ≈ $11.37 (+87 companies in 18 CEE/Balkan/Baltic countries; new --countries option for rubric-seed/discover; --max-new added earlier). Owner-approved exclusions of 10 out-of-scope firms → 558 companies.
• Gap articles: 35 + 7 articles gained European rows (fish vaccines, modified wood, bio-concrete, BIOX/bioleaching, struvite, enteric methane, biosurfactants, fermentation equipment, …); SELL 128 → 144, SOURCE 22 → 7. FI08 lipases stays a gap (Fermentia cited as closest capability only).
• Status triage: 23 confirmed verdicts. One mistake caught and fixed: a BSA Ambiente verdict was based on news about a different “BSA” — check that news is about the same company before using it.

Freshness refresh (four older datasets) ≈ $69.9#

• One run was killed at the 2-hour background limit; resumed with --ids lists of companies not yet refreshed. 10 companies whose calls errored at a budget stop were redone ($0.31). Note: refresh sets meta.updated before its calls, so “updated today” does not prove the calls succeeded — check for "error" in news/risk/status.
• Cited suppliers no longer active after the refresh were removed from 6 articles with a dated note: Insectta (liquidation), Vibha Seeds (insolvency), Braskem (extrajudicial reorganisation), Entologics (no current activity). Nutrition Technologies (renamed Sentara Group) stays.

Website deep reads (Firecrawl) ≈ 17,900 credits#

• All 509 cited suppliers attempted; 471 yielded website facts, 38 sites unreachable or without facts.
• Fix in all five pipelines: website used to skip companies whose seed URL was “N/A” even when research had found the site (the Exa company record); it now falls back to that URL (junk domains and exa.ai pages excluded).

Keys#

• Exa key #7 ran out of credits during the refresh (9 usable). Firecrawl balances moved without our runs — another machine may share the pool.

Territory decided: Russia / EAEU (owner answers 2026-10-06)#

• Buyer capture for Russia, Belarus, Kazakhstan, Kyrgyzstan, Armenia: 3 new buyer types (agroholding, food processor, water utility), Russian-language queries (--lang ru), 248 buyers; enrichment of 160 (--countries filter added). Total ≈ $12.45. Stopped on the owner’s request with ~38 left.
• Bugs fixed: Cyrillic buyer names all keyed to company (cards overwrote each other) → domain-based key (build_buyer_map.buyer_key); groups found in several countries’ searches now get one deal sheet.

Disruption evidence (owner: “I need clear evidence how bioeconomy practices disrupt old tech processes”)#

• Back to the vault’s founding claim (BIOECONOMY-TRENDS.md: alternatives “already proven in practice”): only 28 of 152 articles held any numeric displacement claim, none graded.
• tools/disruption_evidence.py: Exa /answer with a claim schema (displacement / performance / economics / driver, exact quote + URL) → every cited page fetched and each number checked inside the quoted sentence, sentence checked on the page; source tier from the URL domain (the model’s own labels were wrong in the pilot); grade D3–D0 from verified claims; second, displacement-only pass for D1/D0; hint pass for 8 likely under-found cases.
• Hand review of every automatic D3/D2 (77 reviews): 19 of 38 automatic D3s did not hold (wrong topic, old process replaced by non-bio chemistry, or figures showing no displacement). Final: D3 17 · D2 29 · D1 99 · D0 7. Cost ≈ $2.12.
• Second hint pass (owner “1”): 24 likely under-found D1/D0 articles, $0.35. Hand review: F04 SPF shrimp → D3; M01, M02, V01, V09, FD03, C02, EN05 → D2; 16 stay D1 (market-research or company figures only, or the old process was replaced by non-bio chemistry). Source-tier list extended with institutions the domain rules had missed (ICSG, IEA Bioenergy, Canada Energy Regulator, AMCRA, EEA, FEFAC, CPI, SAIMM, ASBP). Final: D3 18 · D2 36 · D1 92 · D0 6.

Evidence-first article expansion (owner: “a need all. much more articles”)#

• tools/candidate_screen.py screened 30 candidate substitutions ($0.44). Hand review: 12 qualified, Marek’s disease vaccine then dropped (no old chemical process replaced) → 11 new articles: FI09 citric acid, FI10 enzymatic HFCS, FI11 MSG, FI12 riboflavin, V13 screwworm SIT, E12 cassava mealybug, E13 prickly pear, E14 medfly SIT, W05 EBPR, W06 biological desulfurisation, WC04 biowaste composting/digestion. 4 candidates folded into existing articles (A35 → D3, EN02, BR01, EN05); 15 dropped (only market-research, company or pilot figures).
• Articles use only verified figures; suppliers by the card rule (honest gaps for riboflavin, screwworm and medfly SIT, prickly pear); funding: grants_scan returned only unrelated awards (stated as such), CN/BR/IN Crossref counts added; references all Crossref-verified (two unverifiable ones replaced or removed).
• Totals: 163 articles · D3 25 · D2 40 · D1 92 · D0 6 · 549 cited suppliers · SELL 153.
• Round 2 (10 candidates, $0.15): 5 new articles — A38 biological nematicides (D3, Brazil ~75% of the market), A39 sugarcane biocontrol (D2), CE03 greenhouse biocontrol (D3, Almería/NL), CE04 bumblebee pollination (D2), EN06 biomass district heating (D3, Sweden >70%); folded E15 → E01 (D2) and BR04 → BR02 (D3); dropped Salmonella vaccination, synthetic-ethanol and egg-packaging candidates. A38–A39 sit in alternatives/ as additions outside the SoyuzAgroHim catalogue. Totals: 168 articles · D3 29 · D2 43 · D1 90 · D0 6 · focus list 44.

Business models — Novozymes/Novonesis as role model (owner request)#

• tools/company_research.py: 27 companies (Novonesis + Chr. Hansen, IFF H&B, AB Enzymes, Amano, Advanced Enzyme, dsm-firmenich, Ajinomoto, CJ BIO, Corbion, Lesaffre, Lallemand, Kerry, Kemin, Koppert, BioFirst, Certis, Pivot Bio, Marrone, Neste, NatureWorks, Novamont, Raízen, Paques, Amyris, Zymergen, Ginkgo), 189 verified facts, ~$0.55. Zymergen segment figures that belonged to Ginkgo were excluded.
• 69 verified product-substitution facts mapped to 45 articles (Industry pioneers sections); section order made stable (evidence → pioneers → suppliers) so nightly runs don’t churn files.
• New candidates from the research: detergent enzymes → I07 article (D2); L-methionine (no displacement — a counter-example), DVS cultures, enzymatic fat splitting, natural curing → dropped (no strong evidence).

Merge and a data-integrity fix#

• Owner: merged the duplicate novozymes card into novonesis (Europe sources/merges.json); I07 now cites the merged card (its profile names household care / “cleaner laundry”).
• Found while checking the merge: buyer enrichment had deleted two Europe supplier caches (agrico, nor-feed) because buyers with the same names (Agrico, Russia; Nor-Feed, Kazakhstan) were researched under the same ids — the buyer cards got the European companies’ data and the supplier files were removed. Supplier caches restored from those cards, the two corrupted buyer cards deleted (left unresearched — owner wants no more buyers), and tools/enrich_buyers.py now researches buyers under buyer--<id> and never touches an existing file. Other datasets checked: no losses.

Deep evidence search and round 3 (owner: “do next”)#

• disruption_evidence.py now merges refreshed claims with earlier ones (never drops). Targeted search on 11 D1 articles ($0.17) + E07/E02 ($0.03): WC03 → D3 (UK 66% of sludge digested, 2008), A24 → D3 (Trichogramma on 80–85% of Moldova’s farmland 1984–92), E07 → D2 (Lake Victoria water hyacinth −95%), A37/EN03/E02 notes strengthened. Phytase, algal DHA, secondary treatment, Cedomon, bromate: still only company/trade figures → D1.
• Round 3 ($0.04): new article A41 Anticarsia baculovirus in Brazil (D2) — peak ~2 M ha, then a documented reversal after 2007; jinggangmycin and chestnut-blight hypovirulence dropped. CSIRO added to strong sources.

Digital and robotic enablers (owner: “bioinformatics, robots in bioeconomy” → “go”)#

• New collection digital-robotics/ registered in index_check, ref_verify, supplier_index, build_dashboard, build_priority_board, sellable_now and rubrics.py. Screened 7 candidates ($0.12): DR01 genomic selection (D3), DR02 spraying drones (D3 as an application method; chemical savings only company-claimed), DR03 drone release of beneficials (D2); spot spraying, robotic weeding, variable-rate N → D1; AI traps folded into DX02 (→ D2).
• 11 enabler companies researched ($0.16): XAG/DJI profitable at scale; Western weeding robots mostly unprofitable (Naïo in court-supervised recovery 2025); sensing subscriptions (Semios profitable 2021); AI protein design valuable only tied to manufacturers (Arzeda–Unilever, Basecamp–P&G). XAG prospectus figures misattributed to DJI were excluded.

Close-out (owner: “finish the project we did all”)#

• Project closed 2026-10-06. Final state: 2,807 companies in 5 datasets; 173 articles in 29 collections graded on verified, hand-reviewed displacement evidence (D3 33 · D2 46 · D1 88 · D0 6); territory Russia/EAEU with routes, origin tiers and a focus short-list; 702 buyers (148 contactable in-territory deal sheets); business models of 38 companies. Exa spend to date ≈ $200; Firecrawl ≈ 17,900 credits. Nightly free checks keep generated pages current; no paid automation runs.
• Open owner items at close: lawyer’s sanctions check (gates all outreach), registration-consultant review of routes, Asia-repo uncommitted changes, optional robot/drone supplier discovery. Future work starts from NEXT-STEPS.md.

Owner items after close-out (“1 not needed, 2 do, 3 do”)#

• Sanctions check: not needed (owner decision) — gate wording replaced in all tools and docs.
• Registration routes: tools/route_verify.py ($0.16) checked all 22 routes against official/legal sources (garant, consultant, eaeunion, fsvps, …; quotes verified on the pages). Main corrections: feed registration ≤45 working days (Minselkhoz Order 316/2025); food SGR ≤30 days (declaration for standard additives); veterinary medicines 155–235 working days under EEC Council Decision 1/2022 (61-FZ route until 31.12.2027); TR EAEU 041/2017 in force since 2021 with Minpromtorg as registrar; agrochemicals Order 327/2025; vet diagnostics Decision 140/2022; vet DDD agents Decision 150/2023; EAEU-registered applicant required (agency can act). Bands updated (FOOD, WATER, FUEL, COSM, PACK → 3; VET, BREED → 2). CLASSIC not verifiable.
• Asia repo: ~1,580 pending changes reviewed (no secrets; data + one code fix) and committed in two commits.

2026-10-07 — review work (owner “do”)#

• Africa & ME: giloil-company-ltd-2 merged into giloil-company-ltd; Kenana Engineering excluded (out of scope); four oil mills kept (scope_keep with notes). Index 2,805.
• Status ⚠ triage from the 2026-10-05/06 news refresh: 64 verdicts recorded only where the news was dated and about the same company (Africa 16 · LatAm 17 · Asia 17 · USA/Canada 8 · Europe 3); look-alikes (e.g. Spiber Inc vs Spiber Technologies AB, Lecico, BSA, Integro24, DVO) left flagged. New distressed: Calysta, NSL Sugars.
• DEAL-SHEETS.md drafts reviewed: fixed broken audience plurals (“waters”, “animal healths”), mid-word truncated product names and lower-cased acronyms; removed the unsupported “regulators increasingly restrict” claim and the “we represent” overclaim; every in-territory sheet now has an English and a Russian draft.

VERIFICATION-2026-09-29#

Evidence for the acceptance criteria of the 2026-09-29 update. Build/toolchain: this vault is a local Dropbox folder (no web server); “build and page checks” are the pipeline’s own render/combine/check chain, which is exactly the artifact users read (READMEs, cards, DASHBOARD, indexes, exports).

1. Build result (verbatim from automation/run-checks.sh)#

[index_check] rc=0 index_check: 0 problem(s) across 17 collections
[ref_verify] rc=0 343 verified / 0 check / 98 report -> tools/reports/ref-verification-2026-09-29.md
[build_catalog] rc=0 index.html unchanged: 121 products / 8 categories / + alternatives section
[supplier_index] rc=0 SUPPLIER-INDEX.md: 286 companies cited across 16 collections, 433 table rows
[rubric_coverage] rc=0 RUBRIC-COVERAGE.md: 17 rubrics, 3695 candidate links, gaps: mining
[build_dashboard] rc=0 DASHBOARD.md: 121 articles, 286/1813 suppliers (16%)
vault checks OK

2. Schema / required fields / duplicates#

• Every new company carries name, country, sector, website, card file and full research JSON; a required-field scan over all 235 new entries returned 0 missing.
• Duplicate scan across all three datasets (domain + validated registry ID): 0 remaining after resolver runs (2 merges in africa, 1 distinct pair in asia – see UPDATE note). The pipeline’s own possible duplicate warnings are resolved.
• Intra-batch duplicate check: no repeats among the 235 (slugs unique, domains unique).

3. New content reachable from the listings#

• 235/235 new companies: card file exists, name present in the dataset README, id present in ALL_COMPANIES.csv and ALL_COMPANIES.json – 0 problems (automated check, script output retained in the session log).
• 6/6 new articles: file exists and is linked from its collection README; alternatives README count and the product cross-reference rows (086/087/089/091) updated; index_check 0 problems across all collections.
• Counters: DASHBOARD 121 articles / 286 of 1,813 suppliers cited; SUPPLIER-INDEX 433 rows; RUBRIC-COVERAGE 3,695 candidate links (grew +387 with the new companies); soyuzagrohim-2026-english/index.html shows the new alternatives section row.

4. Sources and references#

• Source ledger: 235 company rows (name, country, website, seed sector, retrieval date) + 6 article rows – SOURCE-LEDGER-2026-09-29.md / .csv.
• References: ref_verify full pass 343 verified / 0 hand-check / 98 reports; all 26 references in the six new articles verified against Crossref (one initially flagged false CHECK due to a rank-1 book chapter; fixed in the tool by widening candidate depth to 5 with unchanged thresholds).
• Grant signals: scanned 2026-09-29 via tools/grants_scan.py (OpenAIRE + NSF); A37/P06/FI06 honestly report “none located”; FD06/F07/I06 list funder, project, year, amount.

5. Link checks (new content)#

• Website reachability for the 235 new companies, checked from this host: 175 answered HTTP 200; 14 answered with non-200 (bot blocks / moved URLs: e.g. 403/404/503); 46 did not respond through this host’s network path (TLS/connect timeouts – disproportionately .cn/.ir-style hosts under the current VPN egress). The company’s citations in its card (registry, coverage, LinkedIn) are the substantive sourcing; the website field is a convenience link. Recommended: re-check the non-200/timeout list from a different network when convenient.
• Internal links (cards, cross-collection paths, article footnotes): index_check = 0 problems.

6. Spot-reads (accuracy against sources)#

• A37 pheromones: mechanism and dosing ranges match the cited reviews (Witzgall 2010; Carde & Minks 1995); supplier rows verified against card text (e.g. Pherobio NEEQ-listed; Agri Phero traps).
• F07 phages: trial/regulatory framing matches Nakai 2002 and Richards 2014; suppliers’ card wording kept conservative (“livestock and poultry” for PhageLab).
• P06 coatings: sensory-limitation and efficacy caveats match the cited reviews; suppliers listed as “closest capabilities” under an explicit gap note.
• FI06 omega-3: DHA content ranges and approval framing match the cited literature; only non-fish producers cited as suppliers (fish-oil makers deliberately excluded).

7. Ledger matches the diff#

• Companies: 560+63=623, 470+78=548, 548+94=642, total 1578+235=1813 – verified against exports.
• Articles: 115+6=121 – verified against DASHBOARD and folder counts.
• No previously published company was dropped: asia git shows 0 deleted company files; africa/americas changes are the documented merges only; counts reconcile exactly.

8. Residual items#

• mining remains the only rubric below the candidate threshold (real gap; owner guidance: do not re-run discovery there).
• Website timeouts listed in section 5 – re-check on another network if it matters.
• The Korean registry-ID mis-attribution on biopolytech-co-ltd/ecowin cards is marked distinct and flagged for a future registry pass.

Addendum - no-regression close-out (fresh evidence, 2026-09-29 evening)#

1. Build completes - full chain re-run:

[index_check] rc=0 0 problem(s) across 17 collections
[ref_verify] rc=0 343 verified / 0 check / 98 report
[build_catalog] rc=0 index.html unchanged: 121 products / 8 categories / + alternatives section
[supplier_index] rc=0 SUPPLIER-INDEX.md: 286 companies cited across 16 collections, 433 table rows
[rubric_coverage] rc=0 RUBRIC-COVERAGE.md: 17 rubrics, 3695 candidate links, gaps: mining
[build_dashboard] rc=0 DASHBOARD.md: 121 articles, 286/1813 suppliers (16%)
vault checks OK        (EXIT=0)

2. No entries dropped or silently rewritten - git diff --name-status HEAD in asia_biotech_research: 0 deleted, 0 renamed, 111 modified. The modified files are exactly the expected outputs of the render + documented edits: companies/ 101, exports/ 3, sources/ 3, data/ 2, exa_osint.py 1, README.md 1. A sampled card diff shows additive render changes only (new rubric seeds in similar_to; relative card links enriched).
3. Old-entry integrity (all datasets) - every company, old or new, has its card on disk (cards missing: 0 / 0 / 0); untouched-since-yesterday companies spot-checked in READMEs; totals reconcile with the pre-update baselines (560+63, 470+78, 548+94).
4. Link sweep - 119,730 relative internal links checked across every .md in the vault. Findings:
   • 6 occurrences inside generated reports (tools/reports/ref-verification-*.md): quoted article notes such as (method: REFERENCE-VERIFICATION.md) resolve correctly in their source articles (energy/... -> vault root) but not when quoted from tools/reports/. Pre-existing, cosmetic, inherent to quoting; left untouched to avoid rewriting historic content.
   • 1 prose placeholder card in NEXT-STEPS example text - not a link.
   • Net: no real broken internal links introduced; index_check (the vault’s link validator) = 0 problems.