LLM Code Authorship Effect — Replication Package

Paper: "Who Tests Better When They Built It? An Empirical Study of LLM Test Generation for LLM-Coded vs. Human-Coded REST APIs" Author: Md. Mainul Islam Venue: ICSE 2027 NIER — IEEE/ACM International Conference on Software Engineering, New Ideas and Emerging Results track, Dublin, Ireland, April 25 – May 1, 2027 Status: Under review

---

What this repo contains

Complete replication package for the first empirical study isolating the effect of LLM code authorship on LLM test generation quality. We measure whether LLM-generated tests align better with LLM-authored code than with human-authored code that implements the same specification.

llm-code-authorship-effect/
├── apis/
│   ├── human_coded/             # Three REST APIs written by a human developer (ports 5001–5003)
│   │   └── api_servers.py
│   └── llm_coded/               # Same three APIs re-implemented by Claude from the OpenAPI spec (ports 6001–6003)
│       └── llm_coded_servers.py
├── tests/
│   ├── developer/               # Human-written Postman collections
│   └── llm_generated/           # LLM-generated Postman collections
├── analysis/
│   ├── 06_validate_and_analyse.py   # Validation + B vs C authorship effect
│   └── 07_rq3_failure_analysis.py   # Per-test failure categorisation (F1–F3)
├── results/
│   ├── paper2_results.json      # Aggregate metrics across all 4 conditions
│   └── rq3_analysis.json        # Per-test outcome flips
└── paper/
    ├── icse2027_authorship_effect.tex
    ├── icse2027_authorship_effect.pdf
    └── fig_authorship_matrix.png

---

Relationship to prior work

This study extends our prior empirical comparison of LLM-generated and developer-written REST API tests at RASE 2026 (under review):

Islam, M.M. Do LLMs Generate Adequate REST API Tests? An Empirical Comparison with Developer-Written Test Suites. RASE @ ASE 2026. Replication package: github.com/Mainul003/llm-api-test-study

The prior work established Conditions A (human tests / human code) and B (LLM tests / human code). This paper contributes Conditions C (LLM tests / LLM code) and D (human tests / LLM code), enabling four pairwise comparisons that isolate authorship as the experimental variable.

---

Experimental design

A 2×2 factorial design. Test generator (Claude claude-sonnet-4-20250514, temperature 0) and OpenAPI specification are held constant. Code authorship is the sole independent variable.

	Human-coded API	LLM-coded API
Human tests	Condition A (baseline)	Condition D
LLM tests	Condition B (prior work)	Condition C (new)

---

Key results

RQ1 — Endpoint Coverage (%)

Identical between B and C — endpoint coverage is a property of the test suite, not the code.

RQ2 — Test pass rate (%)

Project	B (human code)	C (LLM code)	Δ
TaskManager	95.7	100.0	+4.3
ProductStore	100.0	100.0	+0.0
UserVault	100.0	100.0	+0.0
Mean	98.6	100.0	+1.4

LLM tests are the only ones that ever fail, and only on human-coded APIs — evidence of an authorship-consistency effect: LLM-generated tests align with the LLM's own implementation choices.

RQ3 — Failure mode analysis

Two tests flipped from FAIL to PASS between conditions B and C, both concerning empty-request-body handling on POST /tasks and POST /users/login. The LLM expected HTTP 400 for empty bodies; its own implementation returned exactly that, while the human's equally-valid implementation returned a different status. This is a concrete instance of the self-oracle effect: LLM tests verify the LLM's interpretation of the specification, not the specification itself.

Validation

Developer tests pass at 100% against all three LLM-coded APIs, confirming functional equivalence to the human implementations.

---

How to reproduce

Requirements

pip install flask flask-cors
npm install -g newman

Run the full experiment

# 1. Start both server sets in parallel
python3 apis/human_coded/api_servers.py &       # ports 5001–5003
python3 apis/llm_coded/llm_coded_servers.py &   # ports 6001–6003

# 2. Run Newman for each condition (see analysis scripts for exact commands)
#    Condition B: tests/llm_generated/ against ports 5001–5003
#    Condition C: tests/llm_generated/ against ports 6001–6003
#    Condition D: tests/developer/ against ports 6001–6003

# 3. Compute the authorship effect
python3 analysis/06_validate_and_analyse.py

# 4. Compute the failure-mode analysis
python3 analysis/07_rq3_failure_analysis.py

---

Citation

@inproceedings{islam2027authorshipeffect,
  title     = {Who Tests Better When They Built It? An Empirical Study of
               {LLM} Test Generation for {LLM}-Coded vs.\ Human-Coded {REST} {API}s},
  author    = {Islam, Md. Mainul},
  booktitle = {Proc. 49th IEEE/ACM Int. Conf. Software Engineering (ICSE),
               New Ideas and Emerging Results Track},
  year      = {2027},
  address   = {Dublin, Ireland}
}

---

Contact: Md. Mainul Islam — mainulfuad95@gmail.com ORCID: 0009-0006-6368-2987 Department of CSE, United International University, Dhaka, Bangladesh