Help needed: No clear documentation/examples for implementing speculative decoding with backend serve

[e1ijah1]

Hi there,

I'm trying to implement speculative decoding using TensorRT-LLM backend, specifically with Qwen2.5-72B-Instruct-AWQ as the target model and Qwen2.5-3B-Instruct-AWQ as the draft model. However, I've encountered some difficulties:

1. There seems to be no clear documentation or examples demonstrating how to configure tensorrtllm_backend to serve speculative decoding.
2. The trtllm-serve doesn't appear to support speculative decoding functionality.

Could someone provide guidance on:

• Proper configuration steps for speculative decoding with tensorrtllm_backend
• Whether there are any workarounds for trtllm-serve to support this feature
• Any existing examples or reference implementations

Any help would be greatly appreciated. Thanks!