Waystone survival AI benchmark: open data Version of 25 October 2026 Page: https://waystoneoffline.com/guides/survival-ai-benchmark/ WHAT THIS IS A small, repeatable test of how well small AI models that run offline on a laptop answer emergency-preparedness questions from a fixed library, in seven languages: English, Dutch, French, German, Spanish, Polish and Swedish. Every answer was read and judged by hand. Waystone is a planned offline preparedness assistant that is still in development. Nothing here is a product, and nothing here is a medical or safety reference: in an emergency, call 112 (Europe) or 911 (US) first. FILES library.json 32 short passages written from official advice: Ready.gov, CDC, EPA and FoodSafety.gov (US), and government sources from the Netherlands, Belgium, France, Germany, Spain, Poland and Sweden. Each passage has its source link. A passage whose "country" is not "all" is only offered to questions from that country. These are our own summaries, not official texts: check the source before relying on any number. questions.json Test 1 (22 October 2026): 33 questions in English, Dutch and French. questions_lang2.json Test 2 (25 October 2026): 44 questions in German, Spanish, Polish and Swedish. Each question has: id, lang, country, q (the question), expect (the passage ids that answer it, or null when the library does not cover it), must (regular expressions a good answer should match) and neg (a regular expression for a needed "no" or "never"). The last block of each file is one conversation that switches language at every turn. spike.py The test runner. Python 3 standard library only; it talks to a local Ollama server. summarize.py Automatic scores per model and language. results/retrieval.json, results/retrieval_lang2.json The 3 passages the search model found for each question, with their similarity scores. results/answers_.json, results/answers__lang2.json Every answer, with timings and automatic scores. Models: llama3.2:latest (Llama 3.2 3B, Q4_K_M) and gemma3:4b (Gemma 3 4B, Q4_K_M). Search model: bge-m3. results/hand_review.csv One row per answer (154 rows): our verdict, whether the answer was fully correct, whether it was a serious safety error, whether an out-of-library question was declined, a short note, and the answer itself. results/summary_auto.txt, results/summary_auto_lang2.txt The automatic scores. They miss wrong meanings; the hand verdicts are the real score. HAND REVIEW RULES "Fully correct" means: correct, grounded in the passages the model was given, in the language of the question, and citing the right passage. Serious: a wrong number or unit on a safety topic, an invented or lost warning, or ungrounded advice on a dangerous topic. Not fully correct, but not serious: a wrong citation, a language slip, a garbled sentence. A decline written in English to a question in another language counts as declined, but not as fully correct. A new claim that is not in the cited passage makes an answer not fully correct, even when the claim is true. HOW TO RERUN IT 1. Install Ollama (https://ollama.com) and download the models: ollama pull bge-m3 ollama pull llama3.2 ollama pull gemma3:4b 2. Put these files in one folder and keep results/ as a subfolder. The runner overwrites the files in results/, so keep a copy of ours to compare with. 3. Test 1: python3 spike.py embed python3 spike.py run llama3.2:latest python3 spike.py run gemma3:4b python3 summarize.py Test 2: the same commands with SPIKE_SET=lang2 in front, for example SPIKE_SET=lang2 python3 spike.py embed and then: python3 summarize.py lang2 Settings: temperature 0, seed 1, a 4096-token context, the top 3 passages. Answers can still differ a little between Ollama versions and machines. Our runs: an Apple M2 Mac with 8 GB of memory, Ollama 0.40.7 (test 1) and 0.40.10 (test 2), one model at a time. LIMITS Small: 32 passages, 77 questions, one laptop, one reviewer. The questions in languages other than English were written by us and not checked by native speakers. Each language has its own questions on similar topics, so compare languages with care. LICENCE Questions, passages, results and hand verdicts: Creative Commons Attribution 4.0 (CC BY 4.0), https://creativecommons.org/licenses/by/4.0/ Credit "Waystone survival AI benchmark" with a link to the page above. The two Python scripts: MIT licence, with the same credit. The AI models have their own licences (Llama 3.2 Community License, Gemma Terms of Use).