An LLM writes complete architecture programs.
A population-based evolutionary search decides which ones survive.

Ruoyu Zhao^* · Jiaqi Wu^* · Chenyu Zhu · Zhichao Lu^†
City University of Hong Kong
_{^* Equal contribution · ^† Corresponding author}

Project page · Paper · Architecture zoo · Citation

---

Most neural architecture search works inside a fixed space of cells and operators. OpenArchEvo searches over code: an LLM acts as the mutation operator and rewrites whole architecture programs, while a population-based evolutionary loop with islands, a surrogate and a three-view novelty measure decides what survives and what gets trained. The pipeline is problem-agnostic; in the paper we use it to discover native spiking sequence models.

Highlights

• Open program space. Candidates are executable PyTorch blocks, so the search can invent mechanisms no predefined space contains.
• Evolution, not a single agent. Multi-island populations, NSGA-II selection over predicted fitness and novelty, and a TabPFN surrogate screen thousands of LLM proposals per iteration; only ≤16 are trained.
• Three-view representation. Code, design rationale and a behavioural fingerprint (spike-pattern and firing-rate probes plus module and structural statistics) describe every program before training.
• Results. 10 iterations, 151 new trained architectures, about 132 V100 GPU-days. NeuroGate reaches 26.4 WikiText-103 PPL, ahead of the ANN DeltaNet (27.5), with 31.7× lower estimated arithmetic energy than a dense Transformer.

Replay ten iterations of the search and click any of the 171 trained architectures.	Every architecture has a block diagram, its source code, the LLM's design rationale and metrics.

How it works

_{Pipeline. The inner loop (blue) evolves LLM-written programs on islands and screens them with a surrogate; the outer loop (purple) trains a small, selected batch and updates the archive.}

_{Three-view representation. Every program is described by its code (CodeBLEU), its design rationale (text embedding) and a behavioural fingerprint (21 statistics, incl. spike-pattern and firing-rate probes, measured before training). They drive novelty, duplicate screening and the surrogate.}

1. Inner loop. On islands of related programs, the LLM rewrites parent programs (code + design rationale); thousands of candidates per iteration, none trained.
2. Outer loop. Picks a small batch (≤16) by predicted fitness and novelty, hands it off for real training, and adds the measured results to the archive.
3. Surrogate. TabPFN, refitted on the archive each iteration, predicts task metrics for every candidate in the inner loop.
4. Three-view representation. Each program is compared by its code, its rationale and a behavioural fingerprint (21 statistics measured once at initialisation); this drives novelty and duplicate rejection.
5. Proxy training. Short, fixed-protocol training runs (here WikiText-2 and ListOps) give the measured fitness.

Details: docs/pipeline.md.

Quickstart (CPU, no GPU or LLM)

git clone https://github.com/Optima-CityU/OpenArchEvo && cd OpenArchEvo
pip install -e ".[dev]"
python scripts/run_round.py --run-dir runs/demo --dry-run --auto

--dry-run replaces the LLM, surrogate and training with offline fakes and runs three small outer iterations in seconds; inspect runs/demo/ to see what each stage writes. For a real search (LLM endpoint, GPU surrogate service, training machines) follow docs/real-run.md.

Use it for your own problem

• Task: write a training script that takes --program and --result_json and writes {"metrics": {...}}; add a fitness mode in openarchevo/fitness.py.
• Search space: edit the program contract in openarchevo/contract.py and the prompt templates in openarchevo/llm/templates/; seed the archive with a few hand-written programs.
• Fingerprint: replace the spiking-specific probes in openarchevo/fingerprint/ with ones that suit your models.
• LLM / surrogate: any OpenAI-compatible endpoint works from configs/search.yaml; other backends implement one small interface.
• Keep the dry run passing after every change. Guide: docs/extending.md.

Architecture zoo

Source code of every architecture trained in the paper's search, with metrics in zoo/index.csv. See zoo/README.md.

Folder	Files	Contents
zoo/experts/	20	Initial archive: spiking versions of published linear-attention / SSM families
zoo/discovered/	151	Programs discovered and trained during the 10 search iterations, with LLM rationales
zoo/elites/	3	The three architectures analysed in the paper: NeuroGate, HomeoResSSM, LoopMem
zoo/asi_arch/	152	ASI-Arch-derived programs from an earlier study (separate protocol)

Citation

@article{zhao2026openarchevo,
  title   = {Large Language Model-Guided Evolutionary Discovery of Native Neural Architectures for Spiking Sequence Modeling},
  author  = {Zhao, Ruoyu and Wu, Jiaqi and Zhu, Chenyu and Lu, Zhichao},
  journal = {arXiv preprint arXiv:2609.40258},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.40258}
}

Apache License 2.0 (LICENSE); third-party code, weights and datasets: THIRD_PARTY_NOTICES.md. No model weights or datasets are included.