README.md:
- Added new quants (Q2_K, Q4_K_M)
TTS_ENGINE:
- Updated the External Inference Server section:
- Made the model parameter configurable via ORPHEUS_MODEL_NAME environment variable
Environment:
- Updated .env.example to include this new parameter
Updated tts_engine/speechpipe.py:
- Enhanced the turn_token_into_id function with better documentation
- Added proper type hints and parameter descriptions
- Made the token prefix constant explicitly defined (CUSTOM_TOKEN_PREFIX)
- Increased the cache size from 1000 to 10000 entries for better performance
Updated tts_engine/inference.py:
- Imported the token handling from speechpipe.py
- Removed the duplicate implementation of turn_token_into_id
- Added a comment explaining the function is now imported
| Metric | Original Model | Q8 Model | Improvement |
|--------|---------------|----------|-------------|
| Token generation | ~280 tokens/sec | ~390 tokens/sec | ~40% faster |
| Realtime factor | 1.4-1.7x | 2.1-2.3x | ~50% faster |
| Audio generation | ~19-20 chunks/sec | ~26 chunks/sec | ~35% faster |
| Overall latency | Lower | Much lower | Substantial
Allows the inference engine to correctly yield a single token, even when the streaming response from the backend contains multiple tokens concatenated together.
