Skip to content

fix(vllm): default attention backend to TRITON_ATTN on ROCm - #507

Merged
HuangJoJo merged 1 commit into
alibaba:mainfrom
amd-fuweiy:rocm-vllm-triton-attn-default
Oct 19, 2026
Merged

HuangJoJo merged 1 commit into
alibaba:mainfrom
amd-fuweiy:rocm-vllm-triton-attn-default

Conversation

@amd-fuweiy

Copy link
Copy Markdown
Contributor

What

create_async_llm (roll/third_party/vllm/__init__.py) defaults attention_config.backend to FLASH_ATTN when the caller does not pass one. This PR keeps that default on every non-ROCm platform and uses TRITON_ATTN on ROCm.

Why

On ROCm the FLASH_ATTN backend depends on the CUDA-only vllm_flash_attn package. vLLM's ROCm path returns None from get_flash_attn_version() on purpose (vllm/v1/attention/backends/fa_utils.py, "ROCm doesn't use vllm_flash_attn"). FlashAttentionImpl.forward then asserts that value is not None, so every rollout engine fails during profile_run, before the first training step:

AssertionError: FlashAttention version not detected.

TRITON_ATTN is the Triton FlashAttention implementation vLLM ships as its ROCm default.

Behavior

  • CUDA and other platforms: unchanged, still FLASH_ATTN.
  • ROCm: TRITON_ATTN.
  • An explicit attention_config from the caller still takes precedence, because the existing "attention_config" not in kwargs guard is untouched.

Without this change, ROCm users have to set actor_infer.strategy_args.strategy_config.attention_config.backend: TRITON_ATTN in every pipeline config.

Testing

  • 8x AMD Instinct MI308X (gfx942), base image rocm/primus:v26.7 (torch 2.12.0+rocm10.0.0), vLLM v0.31.0.
  • ROLL RLVR pipeline with vLLM rollout and an FSDP2 trainer in colocate mode, Qwen2.5-0.5B. Smoke run of 3 steps at learning rate 0: engines initialize, the trainer-to-vLLM weight sync is bit-exact, and the checkpoint save/resume works.
  • The same pipeline without this change fails at engine init with the assertion above.
  • Not tested on CUDA in this PR; the CUDA branch of the expression is the existing default.

create_async_llm defaults attention_config.backend to FLASH_ATTN when the
caller passes none. On ROCm that backend relies on the CUDA-only
vllm_flash_attn package: get_flash_attn_version() returns None there, so
every engine fails during profile_run with
"AssertionError: FlashAttention version not detected."

Keep FLASH_ATTN on other platforms and use TRITON_ATTN on ROCm. An
explicit attention_config from the caller still takes precedence.

Co-authored-by: Cursor <cursoragent@cursor.com>
@CLAassistant

CLAassistant commented Oct 14, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@amd-fuweiy
amd-fuweiy marked this pull request as ready for review October 14, 2026 03:47
@HuangJoJo
HuangJoJo merged commit d3506ea into alibaba:main Oct 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants