Operations Scripts

Overview

Technical guide for docs/operational/scripts/ — operational helper scripts referenced by the runbook and maintenance docs.

Files

File	Purpose
rotate-logs.sh	Rotates and compresses Hermes agent logs older than 30 days; cleans project pipeline logs older than 90 days

Key Conventions

• Scripts are standalone shell scripts (set -euo pipefail).
• Run from repo root unless noted otherwise.
• rotate-logs.sh is also documented in ../maintenance.md.

Quick Commands

# Manual log rotation
bash docs/operational/scripts/rotate-logs.sh

See Also

• README.md — Quick navigation
• ../runbook.md — Operations runbook
• ../maintenance.md — Maintenance procedures

🚀 Research Project Template

📄 Published: A template/ approach to Reproducible Generative Research: Architecture and Ergonomics from Configuration through Publication — DOI: 10.5281/zenodo.19139090

Template Repository - Click "Use this template" to create a research project with this structure

Quickstart

Just cloned the repo? Do this:

1. git clone <this-repo> && cd template
2. uv sync (installs the root environment, including deterministic dependencies for all public template exemplars)
3. ./run.sh (interactive menu) or ./run.sh --pipeline --project templates/template_code_project --core-only (non-interactive, no LLM)
4. PDFs land in output/templates/<project>/pdf/. Logs in output/templates/<project>/logs/.
5. Run ./run.sh --help for all flags. The always-present roster is generated from PUBLIC_PROJECT_NAMES in docs/_generated/active_projects.md.

Repurposing for your research? See docs/repurposing-architectures.md — maps every reusable architecture (DAG pipeline, two-layer separation, evidence registry, multi-format rendering, MCP server, publishing stack) to its module and adoption path.

For deeper guidance see docs/guides/getting-started.md and docs/RUN_GUIDE.md.

Thin-orchestrator gates: uv run python scripts/audit/check_template_drift.py --strict, uv run python scripts/gates/module_line_count_check.py, uv run python -m infrastructure.core.health — details in docs/architecture/thin-orchestrator-summary.md.

Assistants and editors: .cursorrules summarizes architecture and tooling for Cursor; CLAUDE.md is the command cheat sheet; AGENTS.md is the full system manual (pipeline, validation, configuration). For routable agent workflows, start at docs/prompts/SKILL.md and the generated skill index docs/_generated/skills_index.md.

Contributors and CI: GitHub Actions, Dependabot, and PR/issue templates live under .github/README.md (agent entry point, doc map, CI inventory) and .github/AGENTS.md (job names, thresholds, troubleshooting).

Local hooks: After uv sync, run pre-commit install and pre-commit install --hook-type pre-push to mirror Ruff, mypy, Bandit, and smoke tests locally (see .pre-commit-config.yaml).

Strict Mermaid/PDF checks: Run npm ci at the repository root. The Python documentation and rendering gates automatically resolve the pinned node_modules/.bin/mmdc; add that directory to PATH only when invoking mmdc directly. The existing Chrome resolver supplies the browser executable.

A system for research and development projects. This template provides a test-driven structure with automated PDF generation, professional documentation, and validated build pipelines.

🧭 Positioning (honest framing)

This is primarily Daniel Ari Friedman's research operating system, made public and Apache 2.0-licensed so other researchers can fork it if helpful. It is not a one-size-fits-all template — it is opinionated, Python+pytest+LaTeX-flavored, and tuned to the kind of work Daniel does (Active Inference, computational biology, cognitive security). Honest framing ages better than wishful adoption metrics.

If your workflow looks similar (TDD-on-research-code, Markdown→PDF, multi-project monorepo, optional local-LLM draft assistance, deterministic + watermarked outputs, Zenodo DOI publishing), the template will probably save you time. If it doesn't look similar, a lighter alternative (Quarto, MyST, Cookiecutter-data-science) may serve you better. See MAINTAINERS.md for ownership and STATUS.md for per-subsystem freshness so you can judge what's actively maintained vs dormant.

Long-horizon viability guides — toolchain migration, regression testing, archival redundancy, local CI, and the design for a future executable-bundle stage — live in docs/maintenance/.

🎯 What This Template Provides

This is a GitHub Template Repository that gives you:

• ✅ Multi-project support - Run multiple projects in one repository
• ✅ Project structure with clear separation of concerns
• ✅ Test-driven development setup with coverage requirements
• ✅ Automated PDF generation from markdown sources
• ✅ Thin orchestrator pattern for maintainable code
• ✅ Methods orchestration linking pipeline contracts, methods prose, artifacts, and evidence
• ✅ Executable methods contracts with DAG, script, artifact, and verification validation
• ✅ Bounded parallel quick testing with isolated project processes and serial oracles
• ✅ Public exemplar capability inventory covering structure and declared skip reasons
• ✅ Ready-to-use utilities for any research project
• ✅ Professional documentation structure (full inventory: docs/documentation-index.md)
• ✅ Advanced quality analysis and document metrics
• ✅ Reproducibility tools for scientific workflows
• ✅ Integrity verification and validation
• ✅ Publishing tools for academic dissemination
• ✅ Scientific development best practices
• ✅ Reporting with error aggregation and performance metrics
• ✅ Local Ollama workflow documented in infrastructure/llm/README.md and docs/operational/troubleshooting/llm-review.md

🗺️ Choose Your Path

Pick the entry point that matches your goal:

• Cold start, human or agent (install prerequisites, run the canonical exemplar once): START_HERE.md.
• New users (write docs, generate PDFs): start with Quickstart, then docs/guides/getting-started.md and docs/reference/quick-start-cheatsheet.md.
• Developers (figures, data analysis, automation): docs/core/architecture.md, docs/architecture/thin-orchestrator-summary.md, docs/guides/methods-orchestration.md, docs/core/workflow.md, docs/guides/figures-and-analysis.md.
• Contributors (improve the template): docs/development/contributing.md, docs/development/code-of-conduct.md, docs/development/roadmap.md, docs/development/security.md.
• Advanced (system internals, modules, deep technical dive): AGENTS.md, docs/RUN_GUIDE.md, docs/modules/modules-guide.md, docs/documentation-index.md.

🧭 Documentation Hub

📚 Documentation Index | 📖 Documentation Guide | 🔍 Quick Reference

The template ships with a large documentation corpus under docs/. The full hierarchical map (with mermaid diagram) lives in docs/AGENTS.md; the authoritative per-file index lives in docs/documentation-index.md (rely on that index, not a hard-coded file count, which drifts). Top-level layout:

• docs/core/ — essential reading: how-to-use, architecture, workflow
• docs/guides/ — progressive walkthroughs by skill level (1–12)
• docs/operational/ — build, configuration, troubleshooting, performance
• docs/reference/ — FAQ, cheatsheet, common workflows, API reference
• docs/architecture/ — two-layer architecture, thin orchestrator, decision tree
• docs/usage/ — examples, showcase, markdown writing guide
• docs/modules/ — module-by-module guides
• docs/development/ — contributing, testing, roadmap

🤖 Agentic operation and SKILLS

Agents should load the smallest applicable workflow before editing. The routing surface is first-class and available via two skill discovery channels:

Hermes / agentskills.io project-local skills

Every canonical exemplar under projects/templates/ now ships its own .agents/skills/<name>/SKILL.md with YAML frontmatter discoverable by Hermes and agentskills.io runtimes. Each skill captures when to use, quick reference (pytest / analysis / render commands), pitfalls, and cross-refs for that template. Load the skill by its name (e.g. template-code-project, template-active-inference) when working inside that exemplar.

Skill	Template	When to load
template-active-inference	template_active_inference	Use this template when several independent research tracks must compose into one manuscript whose claims stay consistent where the tracks overlap — here: a closed-form analytical oracle, a pymdp simulation harness, a Lean formalization boundary, and shared GNN/ontology notation.
template-advanced-literature-review	template_advanced_literature_review	Use this template when one review question needs distinct retrieval phases, phase-specific filters, and explicit cross-phase provenance.
template-autoresearch-project	template_autoresearch_project	Use this template when you need a bounded, offline AutoResearch loop: deterministic ML candidate evaluation over a fixed local dataset, with evidence-linked claims, machine-readable ledgers, artifact-integrity manifests, and deferred human-review gates.
template-autopoiesis	template_autopoiesis	Use this template when you need to generate runnable project trees deterministically — not a manuscript, but a whole child project (its own src/, tests/, scripts/, and manuscript/) selected by a seed from a combinatoric grammar, with recompute-based provenance verification and a falsifiable honesty manifest against green-by-construction test theater.
template-code-project	template_code_project	Use this template for code-driven computational research: algorithms in src/, numerical experiments with deterministic seeds, automated publication-quality figures, and a manuscript that reports the computed results.
template-data-descriptor	template_data_descriptor	Use this template when the contribution is a published dataset or data paper: a schema contract, file inventory, data dictionary, provenance chain, license boundary, quality checks, and machine-readable descriptor must all stay consistent before publication.
template-eda-notebook	template_eda_notebook	Use this template for exploratory data analysis on tabular data: load a dataset, surface missingness, compute descriptive statistics and per-group means, rank features by correlation, and produce a few diagnostic figures.
template-formal	template_formal	Use this template when the research subject is the type architecture itself: illegal-state-unrepresentable design, session-typed protocols, affine/linear resource-handle discipline, or a decentralized (no-shared-global-state) multiagent simulation that needs its own local storage and local networking per agent.
template-literature-meta-analysis	template_literature_meta_analysis	Use it whenever the research object is a body of literature about a topic and you want every reported number to trace to committed, regenerable artifacts.
template-madlib	template_madlib	Use this template when you need configuration-driven manuscript generation with auditable token provenance, conditional section structure, explicit/default field visibility, failure-boundary reporting, and a reviewer-visible authoring contract: lexicon categories, section titles, narrative moves, method steps, design principles, pipeline phases, evaluation criteria, QA probes, failure modes, authoring obligations, visualization controls, audit rules, and slots are declared in YAML; src/ deterministically expands those declarations; and the manuscript receives large-grain {{TOKEN}} bodies only after the source code has generated the supporting artifacts.
template-newspaper	template_newspaper	Use this template when you need data-driven, large-format print layout: multi-page broadsheets/tabloids with precise column geometry, typography-first constraints, and strict content/engine separation (YAML editions in content/, pure-Python ReportLab engine in src/).
template-pools-rules-tools	template_pools_rules_tools	Use this project when you need to: - Demonstrate how a research project integrates multiple resource directories (fonds, tools, rules) in a single pipeline - Validate that your fonds, tools, and rule infrastructure modules are correctly wired and discoverable - Onboard new teams to the three-resource architecture with a concrete, runnable example - Test cross-cutting concern integration where fonds supply data, rules govern validation, and tools execute transforms - Extend the architecture by adding new resource types; copy this project as a starting point for integration testing This template is ideal for platform teams, CI engineers, and infrastructure maintainers who need a self-validating meta-project that exercises all three resource layers without coupling to a specific domain science.
template-prose-project	template_prose_project	Use this template for manuscript-focused editorial pipelines: readability gates (Flesch-Kincaid bands), structural checking, citation/BibTeX validation, and prose-quality review workflows where the document itself is the artifact under test.
template-autoscientists	template_autoscientists	Use this template when you need to isolate and measure agent-coordination mechanisms: coordinated teams vs single-thread baselines under matched budgets, deterministic ablation studies, and honest per-mechanism effect reporting (including nulls).
template-gold-refinement	template_gold_refinement	Use this template for analogical manuscript composition research: projects that map a scientific domain onto a refinement pipeline and generate the manuscript through deterministic mega-madlib token injection.
template-methods-paper	template_methods_paper	Use this template when the paper you are writing describes a methodology rather than reports results: a procedure, protocol, or specification language, where the contribution is the controlled vocabulary and its guarantees (dimensional safety, staged validation, deterministic compilation) rather than a numeric outcome.
template-redacted-report	template_redacted_report	Use this template when a report needs formal redaction before release: classification ceilings, source-control markings, redaction decisions, authority review, reviewer approvals, source-safe hash ledgers, residual-risk checks, mosaic-risk checks, and a public audit packet must be validated before any sanitized narrative is published.
template-registered-report	template_registered_report	Use this template when the core method is a registered report or preregistered replication: hypotheses, outcomes, exclusion rules, analysis plan, power or sensitivity rationale, deviation ledger, and confirmatory-versus-exploratory claim boundaries must be locked before results are interpreted.
template-search-project	template_search_project	Use this template for literature-review and evidence-synthesis pipelines: multi-backend search (arXiv, Crossref, local corpora, optional Paperclip), deduplication, BibTeX generation, and LLM-assisted per-paper and corpus-level synthesis.
template-sia	template_sia	Use this template when you need a self-improvement-agent evaluation harness: Meta → Target → Feedback generation loops, public/private task splits to detect overfitting, deterministic fixture replay for testability, and fail-closed loop validation.
template-storybook	template_storybook	Use this template when you need full-page illustrated PDF storytelling: picture-book pages, symbolic scenes, character-generation methods, page-level orchestrators, and deterministic raster art assembled into a print-ready PDF.
template-template	template_template	Use this template when your research subject is the repository itself — programmatic introspection of architecture, pipeline DAGs, module inventories, and security layers, rendered as a manuscript whose every metric is computed live (autopoietic: the paper regenerates itself from the code it describes).
template-textbook	template_textbook	Use this template for book-length manuscripts: parts → chapters → labs → question banks declared in a single config.yaml, with auto-numbering, deterministic figure/diagram generation, and structural-contract tests that keep hundreds of pages from drifting.
template-pitch-deck	template_pitch_deck	Use this template when you need a pitch, grant report, or recurring stakeholder update treated as a build artifact — bound to live repository facts, validated for unresolved tokens and pitch-deck cliché, and regenerable byte-for-byte from source rather than hand-maintained in a proprietary slide tool.

Generated infrastructure skills

In addition to the project-local skills, the repo generates infrastructure-level SKILL.md files from live discovery:

• Shared context-engineering skills: .agents/skills/ carries a pinned, provenance-checked Agent Skills collection discoverable by Codex/OpenAI and Hermes-compatible runtimes. Use uv run python -m infrastructure.skills runtime-status to audit Codex, Claude Code, and Hermes parity, or ... runtime-install to create reversible user-level links from the pinned shared store.
• Workflow router: docs/prompts/SKILL.md (template-workflows) routes broad requests such as full audits, pipeline debugging, code changes, tests, validation, manuscript work, and release checks to exactly one child workflow.
• Agentic-use hardening: docs/prompts/agentic-use/SKILL.md covers skill inventory, routing checks, .cursor/skill_manifest.json, and generated skill-index maintenance.
• Infrastructure module skills: infrastructure/SKILL.md is the Layer-1 hub; pair the relevant infrastructure/<module>/SKILL.md with that module's AGENTS.md before editing code.
• Script and resource-pool skills: scripts/*/SKILL.md files and public fonds/templates/, rules/templates/, and tools/templates/ skills are included when present, without scanning private lifecycle roots.
• Human skill index: docs/_generated/skills_index.md lists all discovered skills. Regenerate after skill changes with uv run python -m infrastructure.skills write-index; refresh the editor manifest with uv run python -m infrastructure.skills write; verify both with uv run python -m infrastructure.skills check and uv run python -m infrastructure.skills check-contracts. The same inventory is returned by the opt-in stdio MCP server's list_skills tool via uv run python -m infrastructure.mcp_server.

🔀 Multi-Project Support

The repo can host multiple research projects in parallel. Each project owns its own src/, tests/, manuscript/, scripts/, and output/ directory under projects/<name>/. Layer-1 infrastructure is shared.

Permanent canonical exemplars — always present and tracked in git:

Exemplar	When to use
template_active_inference	Use this template when several independent research tracks must compose into one manuscript whose claims stay consistent where the tracks overlap — here: a closed-form analytical oracle, a pymdp simulation harness, a Lean formalization boundary, and shared GNN/ontology notation.
template_advanced_literature_review	Use this template when one review question needs distinct retrieval phases, phase-specific filters, and explicit cross-phase provenance.
template_autoresearch_project	Use this template when you need a bounded, offline AutoResearch loop: deterministic ML candidate evaluation over a fixed local dataset, with evidence-linked claims, machine-readable ledgers, artifact-integrity manifests, and deferred human-review gates.
template_autopoiesis	Use this template when you need to generate runnable project trees deterministically — not a manuscript, but a whole child project (its own src/, tests/, scripts/, and manuscript/) selected by a seed from a combinatoric grammar, with recompute-based provenance verification and a falsifiable honesty manifest against green-by-construction test theater.
template_code_project	Use this template for code-driven computational research: algorithms in src/, numerical experiments with deterministic seeds, automated publication-quality figures, and a manuscript that reports the computed results.
template_data_descriptor	Use this template when the contribution is a published dataset or data paper: a schema contract, file inventory, data dictionary, provenance chain, license boundary, quality checks, and machine-readable descriptor must all stay consistent before publication.
template_eda_notebook	Use this template for exploratory data analysis on tabular data: load a dataset, surface missingness, compute descriptive statistics and per-group means, rank features by correlation, and produce a few diagnostic figures.
template_formal	Use this template when the research subject is the type architecture itself: illegal-state-unrepresentable design, session-typed protocols, affine/linear resource-handle discipline, or a decentralized (no-shared-global-state) multiagent simulation that needs its own local storage and local networking per agent.
template_literature_meta_analysis	Use it whenever the research object is a body of literature about a topic and you want every reported number to trace to committed, regenerable artifacts.
template_madlib	Use this template when you need configuration-driven manuscript generation with auditable token provenance, conditional section structure, explicit/default field visibility, failure-boundary reporting, and a reviewer-visible authoring contract: lexicon categories, section titles, narrative moves, method steps, design principles, pipeline phases, evaluation criteria, QA probes, failure modes, authoring obligations, visualization controls, audit rules, and slots are declared in YAML; src/ deterministically expands those declarations; and the manuscript receives large-grain {{TOKEN}} bodies only after the source code has generated the supporting artifacts.
template_newspaper	Use this template when you need data-driven, large-format print layout: multi-page broadsheets/tabloids with precise column geometry, typography-first constraints, and strict content/engine separation (YAML editions in content/, pure-Python ReportLab engine in src/).
template_pools_rules_tools	Use this project when you need to: - Demonstrate how a research project integrates multiple resource directories (fonds, tools, rules) in a single pipeline - Validate that your fonds, tools, and rule infrastructure modules are correctly wired and discoverable - Onboard new teams to the three-resource architecture with a concrete, runnable example - Test cross-cutting concern integration where fonds supply data, rules govern validation, and tools execute transforms - Extend the architecture by adding new resource types; copy this project as a starting point for integration testing This template is ideal for platform teams, CI engineers, and infrastructure maintainers who need a self-validating meta-project that exercises all three resource layers without coupling to a specific domain science.
template_prose_project	Use this template for manuscript-focused editorial pipelines: readability gates (Flesch-Kincaid bands), structural checking, citation/BibTeX validation, and prose-quality review workflows where the document itself is the artifact under test.
template_autoscientists	Use this template when you need to isolate and measure agent-coordination mechanisms: coordinated teams vs single-thread baselines under matched budgets, deterministic ablation studies, and honest per-mechanism effect reporting (including nulls).
template_gold_refinement	Use this template for analogical manuscript composition research: projects that map a scientific domain onto a refinement pipeline and generate the manuscript through deterministic mega-madlib token injection.
template_methods_paper	Use this template when the paper you are writing describes a methodology rather than reports results: a procedure, protocol, or specification language, where the contribution is the controlled vocabulary and its guarantees (dimensional safety, staged validation, deterministic compilation) rather than a numeric outcome.
template_redacted_report	Use this template when a report needs formal redaction before release: classification ceilings, source-control markings, redaction decisions, authority review, reviewer approvals, source-safe hash ledgers, residual-risk checks, mosaic-risk checks, and a public audit packet must be validated before any sanitized narrative is published.
template_registered_report	Use this template when the core method is a registered report or preregistered replication: hypotheses, outcomes, exclusion rules, analysis plan, power or sensitivity rationale, deviation ledger, and confirmatory-versus-exploratory claim boundaries must be locked before results are interpreted.
template_search_project	Use this template for literature-review and evidence-synthesis pipelines: multi-backend search (arXiv, Crossref, local corpora, optional Paperclip), deduplication, BibTeX generation, and LLM-assisted per-paper and corpus-level synthesis.
template_sia	Use this template when you need a self-improvement-agent evaluation harness: Meta → Target → Feedback generation loops, public/private task splits to detect overfitting, deterministic fixture replay for testability, and fail-closed loop validation.
template_storybook	Use this template when you need full-page illustrated PDF storytelling: picture-book pages, symbolic scenes, character-generation methods, page-level orchestrators, and deterministic raster art assembled into a print-ready PDF.
template_template	Use this template when your research subject is the repository itself — programmatic introspection of architecture, pipeline DAGs, module inventories, and security layers, rendered as a manuscript whose every metric is computed live (autopoietic: the paper regenerates itself from the code it describes).
template_textbook	Use this template for book-length manuscripts: parts → chapters → labs → question banks declared in a single config.yaml, with auto-numbering, deterministic figure/diagram generation, and structural-contract tests that keep hundreds of pages from drifting.
template_pitch_deck	Use this template when you need a pitch, grant report, or recurring stakeholder update treated as a build artifact — bound to live repository facts, validated for unresolved tokens and pitch-deck cliché, and regenerable byte-for-byte from source rather than hand-maintained in a proprietary slide tool.

Test and coverage figures are representative; confirm against docs/_generated/COUNTS.md after substantive changes.

Choosing an exemplar: every exemplar README opens with a ## When to use this template section, and the generated differentiation map in docs/_generated/exemplar_roster.md collects them into one "copy THIS when…" table (regenerate with uv run python scripts/docgen/exemplar_roster.py; sync is test-enforced).

The permanent exemplars share the same core layout and verification checklist. The code/prose exemplars also carry the 12-file project docs/ hub (agent_instructions.md, style_guide.md, syntax_guide.md, testing_philosophy.md, rendering_pipeline.md, faq.md, quickstart.md, output_conventions.md, troubleshooting.md, architecture.md, AGENTS.md, README.md). New projects copy whichever exemplar is closest in shape and adjust from there. See projects/AGENTS.md for the full comparison.

Publication metadata for every public exemplar is generated from project config and sidecars into docs/_generated/publication_records.md, and the GitHub-facing table in .github/README.md is auto-injected from that same source. To publish a project modularly, start with the Publication runbook: it covers the standalone public GitHub mirror, first real Zenodo DOI, new-version releases, optional mirrors, status blocks, and archival handoff.

Public Exemplar Outputs And Mirrors

Every canonical exemplar under projects/templates/ is tracked in this monorepo, including its project-local output/ tree. The copied release artifacts under output/templates/<name>/ are tracked as well, so a clone of docxology/template contains both the source and the latest rendered public artifacts. Public output files above 50 MB remain excluded by the generated-artifact guard; private or rotating project outputs remain blocked.

Each exemplar also has a standalone docxology/template_* GitHub repository linked to its Zenodo concept and latest version DOI. The current matrix is docs/_generated/publication_records.md. The standalone repository must exist before scripts/publish/publish_project_release.py can create a release there; the release script publishes the GitHub release asset and Zenodo deposit, but it does not create the repository itself. To regenerate any exemplar from the monorepo:

git clone https://github.com/docxology/template
cd template
uv sync
./run.sh --project templates/template_code_project --pipeline --core-only
uv run python scripts/pipeline/stage_04_validate.py --project templates/template_code_project
uv run python scripts/pipeline/stage_05_copy.py --project templates/template_code_project

Replace template_code_project with any public exemplar name from the table above. The standalone repositories are publication mirrors; use this monorepo when you need the shared infrastructure, full render pipeline, or cross-template validation.

The canonical exemplars also ship project-local composability overlays: domain_profile.yaml declares review gates, source policy, artifact expectations, and benchmark rubric preferences; experiment_plan.yaml declares design-validation conditions, primary metric direction, expected figures/tables, baselines, and ablations. These files are declarative inputs for validation and benchmark tooling; they do not generate experiments or run autonomous agents.

🔒 Confidentiality. This is a public template repo. Only the canonical exemplars above (under projects/templates/) are git-tracked/pushed — .gitignore ignores projects/* and negates only projects/templates/. Any other project you add under projects/ (research, client, or confidential work) stays local-only and is never committed; scripts/audit/check_tracked_all.py blocks any accidental commit in the pre-push hook and CI.

Private lifecycle projects. In Daniel's working checkout, confidential projects live outside this public repo at $TEMPLATE_PRIVATE_PROJECTS_ROOT. The simplified sidecar uses working/ and archive/; optional ongoing/ (long-lived projects with no publication target) plus legacy active/, published/, and other/ folders are still supported when present. run.sh and python -m infrastructure.orchestration auto-sync existing folders into matching typed subfolders under projects/: working/* into projects/working/*, ongoing/* into projects/ongoing/*, archive/* into projects/archive/*, and optional active/* into projects/active/*. templates/ and optional active/ links behave like native rendered entries; working/, ongoing/, and archive/ links are visible for explicit targeted work but are not default-rendered. Inspect without changing the tree: uv run python -m infrastructure.orchestration link-projects --dry-run. Override the sibling path with TEMPLATE_PRIVATE_PROJECTS_ROOT or .private_projects_root; disable auto-sync with TEMPLATE_SKIP_LINK_SYNC=1. The symlinked project keeps working outputs at projects/<subfolder>/<name>/output/ (the private target), while final deliverables still copy to output/<subfolder>/<name>/ in this template checkout.

Other entries rotate between projects/working/ and projects/archive/ as work progresses. Never hard-code their paths in long-lived docs — consult docs/_generated/active_projects.md (authoritative public scope, regenerated from infrastructure.project.public_scope) and docs/_generated/COUNTS.md instead.

Common commands:

./run.sh                                     # Interactive project selection
./run.sh --project templates/template_code_project --pipeline
./run.sh --all-projects --pipeline           # All discovered projects sequentially
./secure_run.sh --steganography-only --project templates/template_code_project  # Re-watermark PDFs
mkdir -p projects/my_research/{src,tests,manuscript,scripts}  # Scaffold new project

Lifecycle: rendered = projects/templates/ plus optional projects/active/ (discovered, executed). The simplified private sidecar normally uses working/ and archive/ (plus optional ongoing/ for long-lived work with no publication target); render sidecar projects explicitly with a qualified name such as working/{name} or ongoing/{name}. See projects/PROJECTS_PARADIGM.md for lifecycle, slug rules, and discovery semantics.

🚀 Quick Start {#quick-start}

See the Quickstart at the top of this file for the canonical clone-to-PDF flow. For headless cloud deployment use docs/CLOUD_DEPLOY.md (uv is installed automatically when you run ./run.sh --pipeline). Beginner walkthrough: docs/guides/getting-started.md. One-page command reference: docs/reference/quick-start-cheatsheet.md. Twelve-level usage guide: docs/core/how-to-use.md.

System Status

Current state is captured in docs/_generated/COUNTS.md (updated from discovery, test runs, and CI configuration).

Key elements:

• Active projects listed via discover_projects()
• Coverage enforced at 60% (infrastructure) and 90% (projects)
• Tests run with real data and computations
• Commands standardized to uv run
• Outputs organized per project under output/{name}/

See docs/_generated/COUNTS.md and docs/development/testing/testing-guide.md for details.

🎓 Skill-Based Learning Paths

Twelve progressive levels — Document Creation (1–3), Figures & Automation (4–6), Test-Driven Development (7–9), System Architecture (10–12) — are documented end-to-end in docs/core/how-to-use.md. Per-band walkthroughs: docs/guides/getting-started.md, docs/guides/figures-and-analysis.md, docs/guides/testing-and-reproducibility.md, docs/guides/extending-and-automation.md.

🏗️ Project Structure

Two-layer architecture:

• Layer 1 — infrastructure/ (generic, reusable): build, validation, rendering, LLM, publishing, etc. Plus scripts/ (entry-point orchestrators) and tests/ (infrastructure tests, ≥60 % coverage).
• Layer 2 — projects/<name>/ (project-specific, customizable): src/ (algorithms, ≥90 % coverage), tests/, scripts/ (thin orchestrators), manuscript/ (markdown sections + config.yaml).
• Output is generated under each exemplar's project-local projects/templates/<name>/output/ tree. Final deterministic evidence (publication PDFs, figures, analysis data, hydrated manuscripts, and release/validation registries) may be tracked for public exemplars; checkpoints, logs, telemetry, pipeline snapshots, and LaTeX/slide build intermediates are disposable, ignored, and regeneratable.
• Docs live under docs/ (full hierarchy in docs/AGENTS.md); per-directory AGENTS.md files document every leaf.

System Architecture Overview

A short summary lives here; full architecture diagrams (system overview, module-dependency graph, per-stage data flow, configuration-system flow) are maintained in AGENTS.md and docs/core/architecture.md. In short:

• Entry points: ./run.sh (interactive or --pipeline) and uv run python scripts/runner/execute_pipeline.py --project <name> [--core-only]; numbered orchestrators under scripts/ include 00_*.py through 07_*.py (setup → copy, LLM, executive report — see scripts/AGENTS.md).
• Orchestration: the pipeline runs Setup → Tests → Analysis → Render → Validate → Copy, with optional LLM Review and LLM Translations stages.
• Core systems: importable infrastructure/ packages (Layer 1; live list in docs/_generated/COUNTS.md) plus per-project projects/{name}/src/ algorithms (Layer 2); see docs/_generated/COUNTS.md for the live module list.
• Data flow: project source + manuscript markdown + config.yaml flow through the pipeline into output/<name>/{pdf,figures,data,reports}/.
• Quality assurance: infra ≥60 % and project ≥90 % coverage gates, no-mocks policy, deterministic seeds, real PDF/markdown validation.
• Configuration: projects/{name}/manuscript/config.yaml plus environment overrides feed PDF metadata, LaTeX preamble, figure labels, and validation rules.

Directory Overview with Documentation Links:

Directory	Purpose	Documentation
infrastructure/	Generic build/validation tools (Layer 1)	infrastructure/AGENTS.md
scripts/	Entry point orchestrators	scripts/AGENTS.md
tests/	Infrastructure test suite	tests/AGENTS.md
projects/{name}/src/	Project-specific scientific code (Layer 2)	Per-project AGENTS.md
projects/{name}/tests/	Project test suite	Per-project AGENTS.md
docs/	Documentation hub	docs/documentation-index.md
projects/{name}/manuscript/	Research manuscript sections	Per-project AGENTS.md
output/	Generated public evidence plus local build output	Final evidence is tracked selectively; runtime residue is ignored

📚 Explore Documentation: See docs/documentation-index.md for documentation structure

🔑 Key Architectural Principles

The repository follows a thin orchestrator pattern: business logic lives only in infrastructure/ and projects/{name}/src/; scripts coordinate, never implement. Tests prohibit mock frameworks and prefer real execution; remaining pytest.monkeypatch dependency replacements are explicitly inventoried rather than treated as proof of a mock-free suite. Coverage gates remain strict. Full narrative + benefits: docs/architecture/thin-orchestrator-summary.md, docs/core/architecture.md.

✨ Key Features

• Test-driven development with ≥60 % infra and ≥90 % project coverage gates (docs/core/workflow.md).
• Automated script execution via thin-orchestrator scripts under projects/{name}/scripts/ (scripts/AGENTS.md).
• Markdown-to-PDF pipeline with cross-referenced manuscripts and figure integration (docs/usage/markdown-template-guide.md, docs/modules/pdf-validation.md).
• Validated build system with 16 declared stages, a default 10-stage core+LLM path, an 8-stage --core-only path, and CI gates (docs/RUN_GUIDE.md).
• Generic + reusable — drop the same infrastructure/ into any project that follows the layout (docs/usage/template-description.md).

🔒 Security & Monitoring

LLM input sanitization (infrastructure.llm.core.sanitization), security validators (infrastructure.core.security), runtime health checks (infrastructure.core.runtime.health_check), rate limiting, and HTTP security headers. Full surface and worked usage examples: docs/development/security.md.

🛠️ Installation & Setup

Prerequisites: pandoc and a TeX distribution (texlive-xetex on Debian/Ubuntu, MacTeX on macOS). Python deps install with uv sync (project interpreter is .venv/bin/python; the template targets Python 3.10+ (requires-python in pyproject.toml) and CI tests infrastructure on 3.10–3.13, with .python-version pinning 3.12 as the local default). Add per-project deps with uv run python scripts/maintenance/manage_workspace.py add <package> --project <name>. To generate a manuscript, follow the Quickstart at the top.

Layer 1 is also a standard Python distribution. Build it with uv build, or install the wheel attached to a GitHub release. Installation provides the research-template command (research-template --help). Optional pip extras mirror the major capability groups, for example research-project-template[rendering,publishing]; repository contributors should continue to use uv sync --group ... for development groups.

🐳 Docker Support

docker-compose up (or docker build -t research-template . && docker run -it research-template) builds a reproducible image with pandoc, TeX, Ollama LLM server support, persistent model/output volumes, and hot-reload. See Dockerfile and docker-compose.yml.

🔧 Customization

Project Metadata Configuration

Two configuration paths exist: edit projects/{name}/manuscript/config.yaml (recommended) or export AUTHOR_NAME / AUTHOR_ORCID / AUTHOR_EMAIL / PROJECT_TITLE / DOI environment variables (env vars override the YAML file). The YAML schema (paper title, authors with ORCID, publication DOI, keywords, optional LLM translations block) and a worked example are documented once in CLAUDE.md and AGENTS.md; both files also list every available field. See projects/{name}/manuscript/config.yaml.example for the full template. Applied configuration drives PDF metadata, LaTeX document properties (see docs/reference/copypasta.md for preamble examples), generated file headers, and cross-reference systems.

Adding Project-Specific Scripts

Place Python scripts under projects/{name}/scripts/. They must follow the thin orchestrator pattern: import computation from projects/{name}/src/ or infrastructure/, handle only I/O / visualization / orchestration, print output paths to stdout for manifest collection, and never implement algorithms inline. Worked examples and the full pattern walkthrough live in scripts/AGENTS.md and docs/architecture/thin-orchestrator-summary.md.

Manuscript Structure

Per-project manuscript files live in projects/{name}/manuscript/: config.yaml, preamble.md, zero-padded numbered chapter files (00_abstract.md onward), optional S01_*.md supplements, and 99_references.md. Exact chapter slugs vary per project — the canonical exemplar is projects/templates/template_code_project/manuscript/; the numbering system and slug rules are authoritative in docs/usage/manuscript-numbering-system.md.

📊 Testing

TDD with strict coverage gates: infrastructure ≥ 60 %, projects ≥ 90 %. No unit-level mock frameworks — tests use real data, real files; HTTP boundaries use pytest-httpserver (in-process test server). The project pipeline runs a focused pipeline-smoke infrastructure contract plus the selected project's full coverage suite, so ordinary renders do not rerun the entire repository test matrix. Run the full infrastructure gate explicitly with uv run python scripts/pipeline/stage_01_test.py --infra-only --infra-scope full; run a project suite with uv run python scripts/pipeline/stage_01_test.py --project-only --project <name>. Per-suite commands and coverage report flags are documented in tests/AGENTS.md and docs/development/testing/testing-guide.md; live coverage / test counts live in docs/_generated/COUNTS.md.

Output

Working outputs: projects/{name}/output/. Public exemplars retain only deterministic deliverables and evidence under that tree; runtime state (.checkpoints/, .pipeline/, logs/, telemetry, snapshots) and renderer intermediates are local-only and regenerated by the pipeline. Multi-project mode adds output/executive_summary/ as a disposable copied-output surface.

🔍 How It Works

Two entry points — ./run.sh (interactive or --pipeline) and uv run python scripts/runner/execute_pipeline.py --project <name> [--core-only].

Pipeline (canonical phrasing — keep in sync with CLAUDE.md and AGENTS.md): The default pipeline.yaml declares 16 named stages: 8 core stages, 2 optional LLM stages, 2 opt-in ebook/metadata stages, 2 opt-in bundle/archival stages, and 2 opt-in science/provenance stages (Connector Search, Provenance Record). Default full runs include the 10 core+LLM stages (Clean Output Directories plus nine numbered stages). --core-only runs 8 stages by excluding LLM-tagged and opt-in stages. Ebook, metadata, bundle, archival, science, and provenance stages are declared for contracts but invoked separately when needed (directly via their scripts/pipeline/stage_*.py entry points).

Stage	Script	Tags	Failure mode
0 Clean Output Directories	built-in _run_clean_outputs	core, clean	soft fail
1 Environment Setup	scripts/pipeline/stage_00_setup.py	core	hard fail
2 Infrastructure Tests	scripts/pipeline/stage_01_test.py --infra-only --verbose --infra-scope pipeline-smoke	core, tests	configurable tolerance
3 Project Tests	scripts/pipeline/stage_01_test.py --project-only --verbose	core, tests	configurable tolerance
4 Project Analysis	scripts/pipeline/stage_02_analysis.py	core	hard fail
5 Connector Search	scripts/pipeline/stage_08_connector_search.py	science	skipped if not configured
6 Provenance Record	scripts/pipeline/stage_09_provenance_record.py --stage Connector Search	provenance	skipped if not configured
7 PDF Rendering	scripts/pipeline/stage_03_render.py	core	hard fail
8 Output Validation	scripts/pipeline/stage_04_validate.py	core	PDF/bookends and artifact/provenance failures block; optional-format structure remains a warning + report
9 LLM Scientific Review	scripts/pipeline/stage_06_llm_review.py --reviews-only	llm	skipped if Ollama absent
10 LLM Translations	scripts/pipeline/stage_06_llm_review.py --translations-only	llm	skipped if Ollama absent
11 Copy Outputs	scripts/pipeline/stage_05_copy.py	core	soft fail
12 Ebook Generation	scripts/pipeline/stage_11_ebook.py	core, ebook	soft fail
13 Metadata Package	scripts/pipeline/stage_12_metadata.py	core, metadata	soft fail
14 Executable Bundle	scripts/runner/bundle_executable.py	bundle	soft fail
15 Archival Publication	scripts/runner/archive_publication.py	archival	soft fail

Full per-stage flowchart, failure/skip transitions, and the script-to-stage mapping for --core-only live in AGENTS.md and docs/RUN_GUIDE.md. Workflow narrative: docs/core/workflow.md. Architecture narrative: docs/core/architecture.md.

📚 Documentation Index

The full per-file documentation index lives in docs/documentation-index.md (authoritative; counts drift, so it is not duplicated here). Top-level entry points:

• System reference: AGENTS.md, CLAUDE.md, docs/AGENTS.md
• Walkthroughs: docs/guides/getting-started.md, docs/core/how-to-use.md (12 skill levels), docs/reference/quick-start-cheatsheet.md
• Architecture: docs/core/architecture.md, docs/architecture/thin-orchestrator-summary.md, docs/architecture/two-layer-architecture.md
• Pipeline & build: docs/RUN_GUIDE.md, docs/CLOUD_DEPLOY.md
• Modules: docs/modules/modules-guide.md, infrastructure/AGENTS.md
• Quality / testing: tests/AGENTS.md, docs/development/testing/testing-guide.md
• Agent code navigation: docs/guides/codegraph-local.md, docs/guides/leann-local.md (optional local indexes; never committed artifacts)
• Best practices: docs/best-practices/best-practices.md
• Live facts (auto-derived): docs/_generated/COUNTS.md, docs/_generated/active_projects.md
• Per-directory AGENTS.md files in infrastructure/, scripts/, tests/, docs/, and every projects/{name}/{src,scripts,tests,manuscript}/.

🤝 Contributing

contribution guide | Code of conduct | Roadmap

We welcome contributions! To contribute:

1. Ensure all tests pass with coverage requirements met - Testing Guide
2. Follow the established project structure - Architecture
3. Add tests for new functionality - Workflow
4. Update documentation as needed - Documentation Guide
5. Maintain thin orchestrator pattern - scripts use src/ methods - Pattern Guide

Recent Improvements:

• Build system optimizations - Performance Optimization
• Test suite enhancements
• Simplified directory structure with markdown/ elimination

📄 License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

📚 Citation

The machine-readable CITATION.cff is the single source of truth (GitHub's "Cite this repository" widget reads it). If you use this template in your research, please cite:

Cite the current release. Earlier versions retain their own Zenodo DOIs; the version-independent concept DOI always resolves to the latest.

BibTeX:

@software{friedman_template_2026,
  author    = {Daniel Ari Friedman},
  title     = {A template/ approach to Reproducible Generative Research:
               Architecture and Ergonomics from Configuration through Publication},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19139090},
  url       = {https://doi.org/10.5281/zenodo.19139090}
}

Plain text: Daniel Ari Friedman. (2026). A template/ approach to Reproducible Generative Research: Architecture and Ergonomics from Configuration through Publication. Zenodo. https://doi.org/10.5281/zenodo.19139090

🆘 Troubleshooting

Common issue catalog — failing tests, missing pandoc/xelatex, PDF quality, LLM unavailability — lives in docs/operational/troubleshooting/README.md and docs/reference/faq.md. Pipeline entry points and flags: docs/RUN_GUIDE.md. PDF validator: docs/modules/pdf-validation.md.

🔄 Migration from Other Projects

To adapt this template: copy infrastructure/ and scripts/, mirror the projects/{name}/{src,tests,scripts,manuscript}/ layout, adopt config.yaml (see AGENTS.md), and validate by running the pipeline. Worked examples: docs/usage/examples.md, docs/best-practices/migration-guide.md.

🏗️ Architecture Benefits

Thin orchestrator pattern delivers single-source-of-truth business logic, high testability (≥90 % project coverage), reusability across projects, and CI-gated quality. Full benefits + rationale: docs/core/architecture.md.

---

Quick Navigation by Task

Task	Start here
Assistants / Cursor	.cursorrules, CLAUDE.md, AGENTS.md
Write documents	docs/guides/getting-started.md, docs/usage/markdown-template-guide.md
Add figures	docs/guides/figures-and-analysis.md, docs/usage/visualization-guide.md
Fix issues	docs/operational/troubleshooting/README.md, docs/reference/faq.md
Understand architecture	docs/core/architecture.md, docs/architecture/two-layer-architecture.md
Configure system	docs/operational/config/configuration.md, AGENTS.md
Run pipeline	docs/RUN_GUIDE.md
Contribute code	docs/development/contributing.md, docs/rules/AGENTS.md
Find all docs	docs/documentation-index.md
Check the backlog	TO-DO.md
See what changed	CHANGELOG.md

---

🎉 Get Started Now

Ready to begin? Choose your path:

1. New User? → Start with Quick Start or docs/guides/getting-started.md
2. Developer? → Read docs/core/architecture.md and docs/core/workflow.md
3. Need Help? → Check docs/reference/faq.md or docs/operational/troubleshooting/README.md
4. Explore All Docs? → Browse docs/documentation-index.md

📚 Documentation Hub: All documentation is organized in the docs/ directory with guides for every aspect of the template.