v1.1.0

lgtm

.env.example

@@ -0,0 +1,18 @@
+# Orpheus-FastAPI Configuration
+# Copy this file to .env and customize as needed
+
+# Server connection settings
+ORPHEUS_API_URL=http://127.0.0.1:1234/v1/completions
+ORPHEUS_API_TIMEOUT=120 # You should scale this value based on max tokens and your inference speed
+
+# Generation parameters
+ORPHEUS_MAX_TOKENS=8192 # If you want longer completions, increase this value
+ORPHEUS_TEMPERATURE=0.6
+ORPHEUS_TOP_P=0.9
+# Repetition penalty is now hardcoded to 1.1 for stability (this is a model constraint) - this setting is no longer used
+# ORPHEUS_REPETITION_PENALTY=1.1
+ORPHEUS_SAMPLE_RATE=24000
+
+# Web UI settings (keep in mind that the web UI is not secure and should not be exposed to the internet)
+ORPHEUS_PORT=5005
+ORPHEUS_HOST=0.0.0.0

README.md

@@ -6,6 +6,15 @@
 
 High-performance Text-to-Speech server with OpenAI-compatible API, 8 voices, emotion tags, and modern web UI. Optimized for RTX GPUs.
 
+## Changelog
+
+**v1.1.0** (2025-03-23)
+- ✨ Added long-form audio support with sentence-based batching and crossfade stitching
+- 🔊 Improved short audio quality with optimized token buffer handling
+- 🔄 Enhanced environment variable support with .env file loading (configurable via UI)
+- 🖥️ Added automatic hardware detection and optimization for different GPUs
+- 📊 Implemented detailed performance reporting for audio generation
+
 [GitHub Repository](https://github.com/Lex-au/Orpheus-FastAPI)
 
 ## Voice Demos
@@ -27,7 +36,11 @@ Listen to sample outputs with different voices and emotions:
 - **High Performance**: Optimized for RTX GPUs with parallel processing
 - **Multiple Voices**: 8 different voice options with different characteristics
 - **Emotion Tags**: Support for laughter, sighs, and other emotional expressions
-- **Long-form Audio**: Efficient generation of extended audio content in a single request
+- **Unlimited Audio Length**: Generate audio of any length through intelligent batching
+- **Smooth Transitions**: Crossfaded audio segments for seamless listening experience
+- **Web UI Configuration**: Configure all server settings directly from the interface
+- **Dynamic Environment Variables**: Update API endpoint, timeouts, and model parameters without editing files
+- **Server Restart**: Apply configuration changes with one-click server restart
 
 ## Project Structure
 
@@ -187,6 +200,55 @@ This server works as a frontend that connects to an external LLM inference serve
 
 For best performance, adjust the API_URL in `tts_engine/inference.py` to point to your LLM inference server endpoint.
 
+### Hardware Detection and Optimization
+
+The system features intelligent hardware detection that automatically optimizes performance based on your hardware capabilities:
+
+- **High-End GPU Mode** (dynamically detected based on capabilities):
+  - Triggered by either: 16GB+ VRAM, compute capability 8.0+, or 12GB+ VRAM with 7.0+ compute capability
+  - Advanced parallel processing with 4 workers
+  - Optimized batch sizes (32 tokens)
+  - High-throughput parallel file I/O
+  - Full hardware details displayed (name, VRAM, compute capability)
+  - GPU-specific optimizations automatically applied
+
+- **Standard GPU Mode** (other CUDA-capable GPUs):
+  - Efficient parallel processing
+  - GPU-optimized parameters
+  - CUDA acceleration where beneficial
+  - Detailed GPU specifications
+
+- **CPU Mode** (when no GPU is available):
+  - Conservative processing with 2 workers
+  - Optimized memory usage
+  - Smaller batch sizes (16 tokens)
+  - Sequential file I/O
+  - Detailed CPU cores, threads, and RAM information
+
+No manual configuration is needed - the system automatically detects hardware capabilities and adapts for optimal performance across different generations of GPUs and CPUs.
+
+### Token Processing Optimization
+
+The token processing system has been optimized with mathematically aligned parameters:
+- Uses a context window of 49 tokens (7²)
+- Processes in batches of 7 tokens (Orpheus model standard)
+- This square relationship ensures complete token processing with no missed tokens
+- Results in cleaner audio generation with proper token alignment
+- Repetition penalty fixed at 1.1 for optimal quality generation (cannot be changed)
+
+### Long Text Processing
+
+The system features efficient batch processing for texts of any length:
+- Automatically detects longer inputs (>1000 characters) 
+- Splits text at logical points to create manageable chunks
+- Processes each chunk independently for reliability
+- Combines audio segments with smooth 50ms crossfades
+- Intelligently stitches segments in-memory for consistent output
+- Handles texts of unlimited length with no truncation
+- Provides detailed progress reporting for each batch
+
+**Note about long-form audio**: While the system now supports texts of unlimited length, there may be slight audio discontinuities between segments due to architectural constraints of the underlying model. The Orpheus model was designed for short to medium text segments, and our batching system works around this limitation by intelligently splitting and stitching content with minimal audible impact.
+
 ### Integration with OpenWebUI
 
 You can easily integrate this TTS solution with [OpenWebUI](https://github.com/open-webui/open-webui) to add high-quality voice capabilities to your chatbot:
@@ -194,7 +256,7 @@ You can easily integrate this TTS solution with [OpenWebUI](https://github.com/o
 1. Start your Orpheus-FASTAPI server
 2. In OpenWebUI, go to Admin Panel > Settings > Audio
 3. Change TTS from Web API to OpenAI
-4. Set APIBASE URL to your server address (e.g., `http://localhost:5005`)
+4. Set APIBASE URL to your server address (e.g., `http://localhost:5005/v1`)
 5. API Key can be set to "not-needed"
 6. Set TTS Voice to one of the available voices: `tara`, `leah`, `jess`, `leo`, `dan`, `mia`, `zac`, or `zoe`
 7. Set TTS Model to `tts-1`
@@ -214,10 +276,22 @@ The inference server should be configured to expose an API endpoint that this Fa
 
 ### Environment Variables
 
-You can configure the system by setting environment variables:
+You can configure the system using environment variables or a `.env` file:
 
 - `ORPHEUS_API_URL`: URL of the LLM inference API (tts_engine/inference.py)
 - `ORPHEUS_API_TIMEOUT`: Timeout in seconds for API requests (default: 120)
+- `ORPHEUS_MAX_TOKENS`: Maximum tokens to generate (default: 8192)
+- `ORPHEUS_TEMPERATURE`: Temperature for generation (default: 0.6)
+- `ORPHEUS_TOP_P`: Top-p sampling parameter (default: 0.9)
+- `ORPHEUS_SAMPLE_RATE`: Audio sample rate in Hz (default: 24000)
+- `ORPHEUS_PORT`: Web server port (default: 5005)
+- `ORPHEUS_HOST`: Web server host (default: 0.0.0.0)
+
+The system now supports loading environment variables from a `.env` file in the project root, making it easier to configure without modifying system-wide environment settings. See `.env.example` for a template.
+
+![Server Configuration UI](https://lex-au.github.io/Orpheus-FastAPI/ServerConfig.png)
+
+Note: Repetition penalty is hardcoded to 1.1 and cannot be changed through environment variables as this is the only value that produces stable, high-quality output.
 
 Make sure the `ORPHEUS_API_URL` points to your running inference server.
 
@@ -233,6 +307,27 @@ Make sure the `ORPHEUS_API_URL` points to your running inference server.
 
 To add new voices, update the `AVAILABLE_VOICES` list in `tts_engine/inference.py` and add corresponding descriptions in the HTML template.
 
+## Using with llama.cpp
+
+When running the Orpheus model with llama.cpp, use these parameters to ensure optimal performance:
+
+```bash
+./llama-server -m models/Orpheus-3b-FT-Q8_0.gguf \
+  --ctx-size={{your ORPHEUS_MAX_TOKENS from .env}} \
+  --n-predict={{your ORPHEUS_MAX_TOKENS from .env}} \
+  --rope-scaling=linear
+```
+
+Important parameters:
+- `--ctx-size`: Sets the context window size, should match your ORPHEUS_MAX_TOKENS setting
+- `--n-predict`: Maximum tokens to generate, should match your ORPHEUS_MAX_TOKENS setting
+- `--rope-scaling=linear`: Required for optimal positional encoding with the Orpheus model
+
+For extended audio generation (books, long narrations), you may want to increase your token limits:
+1. Set ORPHEUS_MAX_TOKENS to 32768 or higher in your .env file (or via the Web UI)
+2. Increase ORPHEUS_API_TIMEOUT to 1800 for longer processing times
+3. Use the same values in your llama.cpp parameters (if you're using llama.cpp)
+
 ## License
 
 This project is licensed under the Apache License 2.0 - see the LICENSE.txt file for details.

app.py

@@ -4,14 +4,36 @@
 
 import os
 import time
+import asyncio
 from datetime import datetime
 from typing import List, Optional
+from dotenv import load_dotenv
+
+# Function to ensure .env file exists
+def ensure_env_file_exists():
+    """Create a default .env file if one doesn't exist"""
+    if not os.path.exists(".env") and os.path.exists(".env.example"):
+        try:
+            # Copy .env.example to .env
+            with open(".env.example", "r") as example_file:
+                with open(".env", "w") as env_file:
+                    env_file.write(example_file.read())
+            print("✅ Created default configuration file at .env")
+        except Exception as e:
+            print(f"⚠️ Error creating default .env file: {e}")
+
+# Ensure .env file exists before loading environment variables
+ensure_env_file_exists()
+
+# Load environment variables from .env file
+load_dotenv(override=True)
 
 from fastapi import FastAPI, Request, Form, HTTPException, Depends
 from fastapi.responses import HTMLResponse, FileResponse, JSONResponse
 from fastapi.staticfiles import StaticFiles
 from fastapi.templating import Jinja2Templates
 from pydantic import BaseModel
+import json
 
 from tts_engine import generate_speech_from_api, AVAILABLE_VOICES, DEFAULT_VOICE
 
@@ -22,6 +44,10 @@ app = FastAPI(
     version="1.0.0"
 )
 
+# We'll use FastAPI's built-in startup complete mechanism
+# The log message "INFO:     Application startup complete." indicates
+# that the application is ready
+
 # Ensure directories exist
 os.makedirs("outputs", exist_ok=True)
 os.makedirs("static", exist_ok=True)
@@ -53,6 +79,9 @@ async def create_speech_api(request: SpeechRequest):
     """
     Generate speech from text using the Orpheus TTS model.
     Compatible with OpenAI's /v1/audio/speech endpoint.
+    
+    For longer texts (>1000 characters), batched generation is used
+    to improve reliability and avoid truncation issues.
     """
     if not request.input:
         raise HTTPException(status_code=400, detail="Missing input text")
@@ -61,12 +90,19 @@ async def create_speech_api(request: SpeechRequest):
     timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
     output_path = f"outputs/{request.voice}_{timestamp}.wav"
     
-    # Generate speech
+    # Check if we should use batched generation
+    use_batching = len(request.input) > 1000
+    if use_batching:
+        print(f"Using batched generation for long text ({len(request.input)} characters)")
+    
+    # Generate speech with automatic batching for long texts
     start = time.time()
     generate_speech_from_api(
         prompt=request.input,
         voice=request.voice,
-        output_file=output_path
+        output_file=output_path,
+        use_batching=use_batching,
+        max_batch_chars=1000  # Process in ~1000 character chunks (roughly 1 paragraph)
     )
     end = time.time()
     generation_time = round(end - start, 2)
@@ -95,9 +131,20 @@ async def speak(request: Request):
     timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
     output_path = f"outputs/{voice}_{timestamp}.wav"
     
-    # Generate speech
+    # Check if we should use batched generation for longer texts
+    use_batching = len(text) > 1000
+    if use_batching:
+        print(f"Using batched generation for long text ({len(text)} characters)")
+    
+    # Generate speech with batching for longer texts
     start = time.time()
-    generate_speech_from_api(prompt=text, voice=voice, output_file=output_path)
+    generate_speech_from_api(
+        prompt=text, 
+        voice=voice, 
+        output_file=output_path,
+        use_batching=use_batching,
+        max_batch_chars=1000
+    )
     end = time.time()
     generation_time = round(end - start, 2)
 
@@ -120,11 +167,99 @@ async def root(request: Request):
 @app.get("/web/", response_class=HTMLResponse)
 async def web_ui(request: Request):
     """Main web UI for TTS generation"""
+    # Get current config for the Web UI
+    config = get_current_config()
     return templates.TemplateResponse(
         "tts.html",
-        {"request": request, "voices": AVAILABLE_VOICES}
+        {"request": request, "voices": AVAILABLE_VOICES, "config": config}
     )
 
+@app.get("/get_config")
+async def get_config():
+    """Get current configuration from .env file or defaults"""
+    config = get_current_config()
+    return JSONResponse(content=config)
+
+@app.post("/save_config")
+async def save_config(request: Request):
+    """Save configuration to .env file"""
+    data = await request.json()
+    
+    # Convert values to proper types
+    for key, value in data.items():
+        if key in ["ORPHEUS_MAX_TOKENS", "ORPHEUS_API_TIMEOUT", "ORPHEUS_PORT", "ORPHEUS_SAMPLE_RATE"]:
+            try:
+                data[key] = str(int(value))
+            except (ValueError, TypeError):
+                pass
+        elif key in ["ORPHEUS_TEMPERATURE", "ORPHEUS_TOP_P"]:  # Removed ORPHEUS_REPETITION_PENALTY since it's hardcoded now
+            try:
+                data[key] = str(float(value))
+            except (ValueError, TypeError):
+                pass
+    
+    # Write configuration to .env file
+    with open(".env", "w") as f:
+        for key, value in data.items():
+            f.write(f"{key}={value}\n")
+    
+    return JSONResponse(content={"status": "ok", "message": "Configuration saved successfully. Restart server to apply changes."})
+
+@app.post("/restart_server")
+async def restart_server():
+    """Restart the server by touching a file that triggers Uvicorn's reload"""
+    import threading
+    
+    def touch_restart_file():
+        # Wait a moment to let the response get back to the client
+        time.sleep(0.5)
+        
+        # Create or update restart.flag file to trigger reload
+        restart_file = "restart.flag"
+        with open(restart_file, "w") as f:
+            f.write(str(time.time()))
+            
+        print("🔄 Restart flag created, server will reload momentarily...")
+    
+    # Start the touch operation in a separate thread
+    threading.Thread(target=touch_restart_file, daemon=True).start()
+    
+    # Return success response
+    return JSONResponse(content={"status": "ok", "message": "Server is restarting. Please wait a moment..."})
+
+def get_current_config():
+    """Read current configuration from .env.example and .env files"""
+    # Default config from .env.example
+    default_config = {}
+    if os.path.exists(".env.example"):
+        with open(".env.example", "r") as f:
+            for line in f:
+                line = line.strip()
+                if line and not line.startswith("#") and "=" in line:
+                    key, value = line.split("=", 1)
+                    default_config[key] = value
+    
+    # Current config from .env
+    current_config = {}
+    if os.path.exists(".env"):
+        with open(".env", "r") as f:
+            for line in f:
+                line = line.strip()
+                if line and not line.startswith("#") and "=" in line:
+                    key, value = line.split("=", 1)
+                    current_config[key] = value
+    
+    # Merge configs, with current taking precedence
+    config = {**default_config, **current_config}
+    
+    # Add current environment variables
+    for key in config:
+        env_value = os.environ.get(key)
+        if env_value is not None:
+            config[key] = env_value
+    
+    return config
+
 @app.post("/web/", response_class=HTMLResponse)
 async def generate_from_web(
     request: Request,
@@ -145,9 +280,20 @@ async def generate_from_web(
     timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
     output_path = f"outputs/{voice}_{timestamp}.wav"
     
-    # Generate speech
+    # Check if we should use batched generation for longer texts
+    use_batching = len(text) > 1000
+    if use_batching:
+        print(f"Using batched generation for long text from web form ({len(text)} characters)")
+    
+    # Generate speech with batching for longer texts
     start = time.time()
-    generate_speech_from_api(prompt=text, voice=voice, output_file=output_path)
+    generate_speech_from_api(
+        prompt=text, 
+        voice=voice, 
+        output_file=output_path,
+        use_batching=use_batching,
+        max_batch_chars=1000
+    )
     end = time.time()
     generation_time = round(end - start, 2)
     
@@ -166,5 +312,43 @@ async def generate_from_web(
 
 if __name__ == "__main__":
     import uvicorn
-    print("🔥 Starting Orpheus-FASTAPI Server (CUDA)")
-    uvicorn.run("app:app", host="0.0.0.0", port=5005, reload=True)
+    
+    # Check for required settings
+    required_settings = ["ORPHEUS_HOST", "ORPHEUS_PORT"]
+    missing_settings = [s for s in required_settings if s not in os.environ]
+    if missing_settings:
+        print(f"⚠️ Missing environment variable(s): {', '.join(missing_settings)}")
+        print("   Using fallback values for server startup.")
+    
+    # Get host and port from environment variables with better error handling
+    try:
+        host = os.environ.get("ORPHEUS_HOST")
+        if not host:
+            print("⚠️ ORPHEUS_HOST not set, using 0.0.0.0 as fallback")
+            host = "0.0.0.0"
+    except Exception:
+        print("⚠️ Error reading ORPHEUS_HOST, using 0.0.0.0 as fallback")
+        host = "0.0.0.0"
+        
+    try:
+        port = int(os.environ.get("ORPHEUS_PORT", "5005"))
+    except (ValueError, TypeError):
+        print("⚠️ Invalid ORPHEUS_PORT value, using 5005 as fallback")
+        port = 5005
+    
+    print(f"🔥 Starting Orpheus-FASTAPI Server on {host}:{port}")
+    print(f"💬 Web UI available at http://{host if host != '0.0.0.0' else 'localhost'}:{port}")
+    print(f"📖 API docs available at http://{host if host != '0.0.0.0' else 'localhost'}:{port}/docs")
+    
+    # Read current API_URL for user information
+    api_url = os.environ.get("ORPHEUS_API_URL")
+    if not api_url:
+        print("⚠️ ORPHEUS_API_URL not set. Please configure in .env file before generating speech.")
+    else:
+        print(f"🔗 Using LLM inference server at: {api_url}")
+        
+    # Include restart.flag in the reload_dirs to monitor it for changes
+    extra_files = ["restart.flag"] if os.path.exists("restart.flag") else []
+    
+    # Start with reload enabled to allow automatic restart when restart.flag changes
+    uvicorn.run("app:app", host=host, port=port, reload=True, reload_dirs=["."], reload_includes=["*.py", "*.html", "restart.flag"])

requirements.txt

@@ -1,15 +1,30 @@
-# Core dependencies
-fastapi>=0.103.1
-uvicorn>=0.23.2
-jinja2>=3.1.2
-pydantic>=2.3.0
-numpy>=1.24.0
-requests>=2.31.0
-sounddevice>=0.4.6
-python-multipart>=0.0.6
+# Web Server Dependencies
+fastapi==0.103.1
+uvicorn==0.23.2
+jinja2==3.1.2
+pydantic==2.3.0
+python-multipart==0.0.6
 
-# SNAC is required for audio generation from tokens
-snac>=0.3.0
+# API and Communication
+requests==2.31.0
+python-dotenv==1.0.0
 
-# PyTorch - Note: Install PyTorch with CUDA 12.4 support separately:
-# pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
+# Audio Processing
+numpy==1.24.0
+sounddevice==0.4.6
+snac==1.2.1       # Required for audio generation from tokens
+
+# System Utilities
+psutil==5.9.0
+
+# PyTorch - Install separately with CUDA support:
+# On Windows/Linux: 
+#   pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
+# On macOS:
+#   pip3 install torch torchvision torchaudio
+
+# Optional Dependencies
+# For MP3 conversion (not currently implemented)
+# pydub==0.25.1
+# For better sentence splitting (potential future improvement)
+# nltk==3.8.1

templates/tts.html

@@ -144,6 +144,7 @@
                     name="text" 
                     id="text" 
                     rows="4" 
+                    maxlength="8192"
                     class="block w-full rounded-md border-dark-600 bg-dark-700 text-white shadow-sm focus:border-primary-500 focus:ring-primary-500 focus:ring-offset-dark-800 sm:text-sm px-3 py-2"
                     placeholder="Enter text to convert to speech..."
                     required
@@ -191,6 +192,8 @@
                       </svg>
                     </span>
                   </summary>
+                  
+                  <!-- Audio generation options -->
                   <div class="mt-4 grid grid-cols-1 md:grid-cols-2 gap-4">
                     <div>
                       <label for="model" class="block text-sm font-medium text-white mb-1">Model</label>
@@ -216,6 +219,86 @@
                       </div>
                     </div>
                   </div>
+                  
+                  <!-- Server configuration section -->
+                  <div class="mt-6 border-t border-dark-700 pt-4">
+                    <h3 class="text-sm font-medium text-white mb-3">Server Configuration</h3>
+                    <p class="text-xs text-purple-300 mb-3">These settings will be saved to a <code class="bg-dark-700 px-1 rounded">.env</code> file. Restart the server to apply changes.</p>
+                    
+                    <!-- Form fields for all .env parameters -->
+                    <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
+                      <!-- Connection settings -->
+                      <div>
+                        <label for="api_url" class="block text-xs font-medium text-white mb-1">API URL</label>
+                        <input type="text" id="api_url" name="ORPHEUS_API_URL" 
+                               placeholder="http://127.0.0.1:1234/v1/completions" 
+                               class="block w-full rounded-md bg-dark-700 border-dark-600 text-white text-sm focus:border-primary-500 focus:ring-primary-500 focus:ring-offset-dark-800 px-3 py-2">
+                      </div>
+                      
+                      <div>
+                        <label for="api_timeout" class="block text-xs font-medium text-white mb-1">API Timeout (seconds)</label>
+                        <input type="number" id="api_timeout" name="ORPHEUS_API_TIMEOUT" min="10" max="1800" step="1"
+                               class="block w-full rounded-md bg-dark-700 border-dark-600 text-white text-sm focus:border-primary-500 focus:ring-primary-500 focus:ring-offset-dark-800 px-3 py-2">
+                      </div>
+                      
+                      <!-- Generation parameters -->
+                      <div>
+                        <label for="max_tokens" class="block text-xs font-medium text-white mb-1">Max Tokens</label>
+                        <input type="number" id="max_tokens" name="ORPHEUS_MAX_TOKENS" min="100" max="200000" step="1"
+                               class="block w-full rounded-md bg-dark-700 border-dark-600 text-white text-sm focus:border-primary-500 focus:ring-primary-500 focus:ring-offset-dark-800 px-3 py-2">
+                      </div>
+                      
+                      <div>
+                        <label for="temperature" class="block text-xs font-medium text-white mb-1">Temperature</label>
+                        <input type="number" id="temperature" name="ORPHEUS_TEMPERATURE" min="0.1" max="1.5" step="0.1"
+                               class="block w-full rounded-md bg-dark-700 border-dark-600 text-white text-sm focus:border-primary-500 focus:ring-primary-500 focus:ring-offset-dark-800 px-3 py-2">
+                      </div>
+                      
+                      <div>
+                        <label for="top_p" class="block text-xs font-medium text-white mb-1">Top P</label>
+                        <input type="number" id="top_p" name="ORPHEUS_TOP_P" min="0.1" max="1" step="0.05"
+                               class="block w-full rounded-md bg-dark-700 border-dark-600 text-white text-sm focus:border-primary-500 focus:ring-primary-500 focus:ring-offset-dark-800 px-3 py-2">
+                      </div>
+                      
+                      <div>
+                        <div class="flex justify-between">
+                          <label class="block text-xs font-medium text-white mb-1">Repetition Penalty</label>
+                          <span class="text-xs text-primary-400">Fixed at 1.1</span>
+                        </div>
+                        <div class="bg-dark-700 text-gray-500 border-dark-600 rounded-md px-3 py-2 text-sm">
+                          Value hardcoded to 1.1 for optimal generation quality
+                        </div>
+                      </div>
+                      
+                      <!-- Server settings -->
+                      <div>
+                        <label for="server_host" class="block text-xs font-medium text-white mb-1">Server Host</label>
+                        <input type="text" id="server_host" name="ORPHEUS_HOST" placeholder="0.0.0.0"
+                               class="block w-full rounded-md bg-dark-700 border-dark-600 text-white text-sm focus:border-primary-500 focus:ring-primary-500 focus:ring-offset-dark-800 px-3 py-2">
+                      </div>
+                      
+                      <div>
+                        <label for="server_port" class="block text-xs font-medium text-white mb-1">Server Port</label>
+                        <input type="number" id="server_port" name="ORPHEUS_PORT" min="1024" max="65535" step="1"
+                               class="block w-full rounded-md bg-dark-700 border-dark-600 text-white text-sm focus:border-primary-500 focus:ring-primary-500 focus:ring-offset-dark-800 px-3 py-2">
+                      </div>
+                      
+                      <!-- Save button and restart button -->
+                      <div class="col-span-1 md:col-span-2 mt-4 flex flex-col md:flex-row gap-4">
+                        <button id="save-config-btn" type="button" 
+                                class="btn-primary bg-purple-600 hover:bg-purple-700 active:bg-purple-800 w-full md:w-auto">
+                          Save Configuration
+                        </button>
+                        <button id="restart-server-btn" type="button" 
+                                class="btn-primary bg-red-600 hover:bg-red-700 active:bg-red-800 w-full md:w-auto hidden">
+                          Restart Server
+                        </button>
+                        <p class="text-xs text-purple-300 mt-2 flex-grow">
+                          <span id="config-status" class="hidden"></span>
+                        </p>
+                      </div>
+                    </div>
+                  </div>
                 </details>
               </div>
             </div>
@@ -482,6 +565,166 @@
         generation_time: "{{ generation_time }}"
       });
       {% endif %}
+      
+      // Configuration management
+      async function loadConfigValues() {
+        try {
+          const response = await fetch('/get_config');
+          if (!response.ok) {
+            throw new Error('Failed to load configuration');
+          }
+          
+          const config = await response.json();
+          
+          // Populate form fields with config values
+          document.querySelectorAll('input[name^="ORPHEUS_"]').forEach(input => {
+            const name = input.name;
+            if (config[name] !== undefined) {
+              input.value = config[name];
+            }
+          });
+          
+          console.log('Configuration loaded successfully');
+        } catch (error) {
+          console.error('Error loading configuration:', error);
+        }
+      }
+      
+      // Save configuration
+      async function saveConfiguration() {
+        const configData = {};
+        
+        // Collect values from all configuration inputs
+        document.querySelectorAll('input[name^="ORPHEUS_"]').forEach(input => {
+          configData[input.name] = input.value;
+        });
+        
+        try {
+          const response = await fetch('/save_config', {
+            method: 'POST',
+            headers: {
+              'Content-Type': 'application/json'
+            },
+            body: JSON.stringify(configData)
+          });
+          
+          if (!response.ok) {
+            throw new Error('Failed to save configuration');
+          }
+          
+          const result = await response.json();
+          
+          // Show success message
+          const statusElem = document.getElementById('config-status');
+          statusElem.textContent = result.message;
+          statusElem.classList.remove('hidden');
+          statusElem.classList.add('text-green-400');
+          
+          // Hide message after 5 seconds
+          setTimeout(() => {
+            statusElem.classList.add('hidden');
+          }, 5000);
+          
+        } catch (error) {
+          console.error('Error saving configuration:', error);
+          
+          // Show error message
+          const statusElem = document.getElementById('config-status');
+          statusElem.textContent = 'Error saving configuration. Please try again.';
+          statusElem.classList.remove('hidden');
+          statusElem.classList.add('text-red-400');
+          
+          // Hide message after 5 seconds
+          setTimeout(() => {
+            statusElem.classList.add('hidden');
+          }, 5000);
+        }
+      }
+      
+      // Function to restart the server
+      async function restartServer() {
+        const restartBtn = document.getElementById('restart-server-btn');
+        restartBtn.disabled = true;
+        restartBtn.textContent = 'Restarting...';
+        
+        try {
+          const response = await fetch('/restart_server', {
+            method: 'POST',
+            headers: {
+              'Content-Type': 'application/json'
+            }
+          });
+          
+          if (!response.ok) {
+            throw new Error('Failed to restart server');
+          }
+          
+          const result = await response.json();
+          
+          // Show server restarting message and overlay
+          const statusElem = document.getElementById('config-status');
+          statusElem.textContent = result.message;
+          statusElem.classList.remove('hidden');
+          statusElem.classList.add('text-yellow-400');
+          
+          // Show loading overlay with different message
+          const loadingOverlay = document.getElementById('loading-overlay');
+          loadingOverlay.querySelector('p').textContent = 'Server is restarting...';
+          loadingOverlay.classList.remove('hidden');
+          
+          // Poll for server readiness, then do a complete page reload with cache busting
+          function checkServerReady() {
+            console.log("Checking if server is ready...");
+            fetch('/get_config?cache=' + Date.now(), {
+              cache: 'no-store',
+              headers: { 'pragma': 'no-cache' }
+            })
+              .then(response => {
+                if (response.ok) {
+                  console.log("Server is up and running, forcing complete page reload");
+                  // Force a complete reload with cache busting
+                  window.location = window.location.pathname + '?t=' + Date.now();
+                } else {
+                  console.log("Server not ready yet, status: " + response.status);
+                  setTimeout(checkServerReady, 1000);
+                }
+              })
+              .catch(error => {
+                console.log("Server not responding yet, waiting...");
+                setTimeout(checkServerReady, 1000);
+              });
+          }
+
+          // Start checking after initial delay
+          setTimeout(checkServerReady, 2000);
+          
+        } catch (error) {
+          console.error('Error restarting server:', error);
+          
+          // Show error message
+          const statusElem = document.getElementById('config-status');
+          statusElem.textContent = 'Error restarting server. Please try again.';
+          statusElem.classList.remove('hidden');
+          statusElem.classList.add('text-red-400');
+          
+          // Reset button
+          restartBtn.disabled = false;
+          restartBtn.textContent = 'Restart Server';
+        }
+      }
+      
+      // Add event listeners
+      document.getElementById('save-config-btn').addEventListener('click', function() {
+        saveConfiguration();
+        // Show restart button after saving
+        const restartBtn = document.getElementById('restart-server-btn');
+        restartBtn.classList.remove('hidden');
+      });
+      
+      document.getElementById('restart-server-btn').addEventListener('click', restartServer);
+      
+      // Load configuration values when page loads
+      loadConfigValues();
     });
   </script>
 </body>

tts_engine/inference.py

@@ -12,32 +12,126 @@ import queue
 import asyncio
 from concurrent.futures import ThreadPoolExecutor
 from typing import List, Dict, Any, Optional, Generator, Union, Tuple
+from dotenv import load_dotenv
 
-# Detect if we're on a high-end system like RTX 4090
+# Helper to detect if running in Uvicorn's reloader
+def is_reloader_process():
+    """Check if the current process is a uvicorn reloader"""
+    return (sys.argv[0].endswith('_continuation.py') or 
+            os.environ.get('UVICORN_STARTED') == 'true')
+
+# Set a flag to avoid repeat messages
+IS_RELOADER = is_reloader_process()
+if not IS_RELOADER:
+    os.environ['UVICORN_STARTED'] = 'true'
+
+# Load environment variables from .env file
+load_dotenv()
+
+# Detect hardware capabilities and display information
 import torch
+import psutil
+
+# Detect if we're on a high-end system based on hardware capabilities
 HIGH_END_GPU = False
 if torch.cuda.is_available():
-    gpu_name = torch.cuda.get_device_name(0).lower()
-    if any(x in gpu_name for x in ['4090', '3090', 'a100', 'h100']):
-        HIGH_END_GPU = True
-        print(f"High-end GPU detected: {torch.cuda.get_device_name(0)}")
-        print("Enabling high-performance optimizations")
+    # Get GPU properties
+    props = torch.cuda.get_device_properties(0)
+    gpu_name = props.name
+    gpu_mem_gb = props.total_memory / (1024**3)
+    compute_capability = f"{props.major}.{props.minor}"
+    
+    # Consider high-end if: large VRAM (≥16GB) OR high compute capability (≥8.0) OR large VRAM (≥12GB) with good CC (≥7.0)
+    HIGH_END_GPU = (gpu_mem_gb >= 16.0 or 
+                    props.major >= 8 or 
+                    (gpu_mem_gb >= 12.0 and props.major >= 7))
+        
+    if HIGH_END_GPU:
+        if not IS_RELOADER:
+            print(f"🖥️ Hardware: High-end CUDA GPU detected")
+            print(f"📊 Device: {gpu_name}")
+            print(f"📊 VRAM: {gpu_mem_gb:.2f} GB")
+            print(f"📊 Compute Capability: {compute_capability}")
+            print("🚀 Using high-performance optimizations")
+    else:
+        if not IS_RELOADER:
+            print(f"🖥️ Hardware: CUDA GPU detected")
+            print(f"📊 Device: {gpu_name}")
+            print(f"📊 VRAM: {gpu_mem_gb:.2f} GB")
+            print(f"📊 Compute Capability: {compute_capability}")
+            print("🚀 Using GPU-optimized settings")
+else:
+    # Get CPU info
+    cpu_cores = psutil.cpu_count(logical=False)
+    cpu_threads = psutil.cpu_count(logical=True)
+    ram_gb = psutil.virtual_memory().total / (1024**3)
+    
+    if not IS_RELOADER:
+        print(f"🖥️ Hardware: CPU only (No CUDA GPU detected)")
+        print(f"📊 CPU: {cpu_cores} cores, {cpu_threads} threads")
+        print(f"📊 RAM: {ram_gb:.2f} GB")
+        print("⚙️ Using CPU-optimized settings")
+
+# Load configuration from environment variables without hardcoded defaults
+# Critical settings - will log errors if missing
+required_settings = ["ORPHEUS_API_URL"]
+missing_settings = [s for s in required_settings if s not in os.environ]
+if missing_settings:
+    print(f"ERROR: Missing required environment variable(s): {', '.join(missing_settings)}")
+    print("Please set them in .env file or environment. See .env.example for defaults.")
+
+# API connection settings
+API_URL = os.environ.get("ORPHEUS_API_URL")
+if not API_URL:
+    print("WARNING: ORPHEUS_API_URL not set. API calls will fail until configured.")
 
-# Orpheus-FASTAPI settings - make configurable for different endpoints
-API_URL = os.environ.get("ORPHEUS_API_URL", "http://your-server-ip:port/v1/completions or v1/chat/completions")
 HEADERS = {
     "Content-Type": "application/json"
 }
 
-# Better timeout handling for API requests
-REQUEST_TIMEOUT = int(os.environ.get("ORPHEUS_API_TIMEOUT", "120"))  # 120 seconds default for long generations
+# Request timeout settings
+try:
+    REQUEST_TIMEOUT = int(os.environ.get("ORPHEUS_API_TIMEOUT", "120"))
+except (ValueError, TypeError):
+    print("WARNING: Invalid ORPHEUS_API_TIMEOUT value, using 120 seconds as fallback")
+    REQUEST_TIMEOUT = 120
 
-# Model parameters - optimized defaults for high-end GPUs
-MAX_TOKENS = 8192 if HIGH_END_GPU else 1200  # Significantly increased for RTX 4090 to allow ~1.5-2 minutes of audio
-TEMPERATURE = 0.6 
-TOP_P = 0.9
+# Model generation parameters from environment variables
+try:
+    MAX_TOKENS = int(os.environ.get("ORPHEUS_MAX_TOKENS", "8192"))
+except (ValueError, TypeError):
+    print("WARNING: Invalid ORPHEUS_MAX_TOKENS value, using 8192 as fallback")
+    MAX_TOKENS = 8192
+
+try:
+    TEMPERATURE = float(os.environ.get("ORPHEUS_TEMPERATURE", "0.6"))
+except (ValueError, TypeError):
+    print("WARNING: Invalid ORPHEUS_TEMPERATURE value, using 0.6 as fallback")
+    TEMPERATURE = 0.6
+
+try:
+    TOP_P = float(os.environ.get("ORPHEUS_TOP_P", "0.9"))
+except (ValueError, TypeError):
+    print("WARNING: Invalid ORPHEUS_TOP_P value, using 0.9 as fallback")
+    TOP_P = 0.9
+
+# Repetition penalty is hardcoded to 1.1 which is the only stable value for quality output
 REPETITION_PENALTY = 1.1
-SAMPLE_RATE = 24000  # SNAC model uses 24kHz
+
+try:
+    SAMPLE_RATE = int(os.environ.get("ORPHEUS_SAMPLE_RATE", "24000"))
+except (ValueError, TypeError):
+    print("WARNING: Invalid ORPHEUS_SAMPLE_RATE value, using 24000 as fallback")
+    SAMPLE_RATE = 24000
+
+# Print loaded configuration only in the main process, not in the reloader
+if not IS_RELOADER:
+    print(f"Configuration loaded:")
+    print(f"  API_URL: {API_URL}")
+    print(f"  MAX_TOKENS: {MAX_TOKENS}")
+    print(f"  TEMPERATURE: {TEMPERATURE}")
+    print(f"  TOP_P: {TOP_P}")
+    print(f"  REPETITION_PENALTY: {REPETITION_PENALTY}")
 
 # Parallel processing settings
 NUM_WORKERS = 4 if HIGH_END_GPU else 2
@@ -115,10 +209,12 @@ def generate_tokens_from_api(prompt: str, voice: str = DEFAULT_VOICE, temperatur
     formatted_prompt = format_prompt(prompt, voice)
     print(f"Generating speech for: {formatted_prompt}")
     
-    # Optimize the token generation for high-end GPUs
+    # Optimize the token generation for GPUs
     if HIGH_END_GPU:
         # Use more aggressive parameters for faster generation on high-end GPUs
         print("Using optimized parameters for high-end GPU")
+    elif torch.cuda.is_available():
+        print("Using optimized parameters for GPU acceleration")
     
     # Create the request payload
     payload = {
@@ -329,6 +425,10 @@ def tokens_decoder_sync(syn_token_gen, output_file=None):
     # Batch processing of tokens for improved throughput
     batch_size = 32 if HIGH_END_GPU else 16
     
+    # Thread synchronization for proper completion detection
+    producer_done_event = threading.Event()
+    producer_started_event = threading.Event()
+    
     # Convert the synchronous token generator into an async generator with batching
     async def async_token_gen():
         batch = []
@@ -338,7 +438,7 @@ def tokens_decoder_sync(syn_token_gen, output_file=None):
                 for t in batch:
                     yield t
                 batch = []
-        # Process any remaining tokens
+        # Process any remaining tokens in the final batch
         for t in batch:
             yield t
 
@@ -348,82 +448,120 @@ def tokens_decoder_sync(syn_token_gen, output_file=None):
         chunk_count = 0
         last_log_time = start_time
         
-        async for audio_chunk in tokens_decoder(async_token_gen()):
-            audio_queue.put(audio_chunk)
-            chunk_count += 1
+        try:
+            # Signal that producer has started processing
+            producer_started_event.set()
             
-            # Log performance periodically
-            current_time = time.time()
-            if current_time - last_log_time >= 3.0:  # Every 3 seconds
-                elapsed = current_time - start_time
-                if elapsed > 0:
-                    chunks_per_sec = chunk_count / elapsed
-                    print(f"Audio generation rate: {chunks_per_sec:.2f} chunks/second")
-                last_log_time = current_time
-                
-        # Signal completion
-        audio_queue.put(None)
+            async for audio_chunk in tokens_decoder(async_token_gen()):
+                # Process each audio chunk from the decoder
+                if audio_chunk:
+                    audio_queue.put(audio_chunk)
+                    chunk_count += 1
+                    
+                    # Log performance periodically
+                    current_time = time.time()
+                    if current_time - last_log_time >= 3.0:  # Every 3 seconds
+                        elapsed = current_time - last_log_time
+                        if elapsed > 0:
+                            recent_chunks = chunk_count
+                            chunks_per_sec = recent_chunks / elapsed
+                            print(f"Audio generation rate: {chunks_per_sec:.2f} chunks/second")
+                        last_log_time = current_time
+                        # Reset chunk counter for next interval
+                        chunk_count = 0
+        except Exception as e:
+            print(f"Error in token processing: {str(e)}")
+            import traceback
+            traceback.print_exc()
+        finally:
+            # Always signal completion, even if there was an error
+            print("Producer completed - setting done event")
+            producer_done_event.set()
+            # Add sentinel to queue to signal end of stream
+            audio_queue.put(None)
 
     def run_async():
+        """Run the async producer in its own thread"""
         asyncio.run(async_producer())
 
     # Use a separate thread with higher priority for producer
-    thread = threading.Thread(target=run_async)
+    thread = threading.Thread(target=run_async, name="TokenProcessor")
     thread.daemon = True  # Allow thread to be terminated when main thread exits
     thread.start()
     
-    # For high-end GPUs, use a ThreadPoolExecutor for parallel file I/O
-    if HIGH_END_GPU and wav_file:
-        # Buffer for collecting chunks before writing
-        write_buffer = []
-        buffer_size = 10  # Write every 10 chunks
-        
-        def write_chunks_to_file(chunks, file):
-            for chunk in chunks:
-                file.writeframes(chunk)
-        
-        with ThreadPoolExecutor(max_workers=2) as executor:
-            future = None
+    # Wait for producer to actually start before proceeding
+    # This avoids race conditions where we might try to read from an empty queue
+    # before the producer has had a chance to add anything
+    producer_started_event.wait(timeout=5.0)
+    
+    # Optimized I/O approach for all systems
+    # This approach is simpler and more reliable than separate code paths
+    write_buffer = bytearray()
+    buffer_max_size = 1024 * 1024  # 1MB max buffer size (adjustable)
+    
+    # Keep track of the last time we checked for completion
+    last_check_time = time.time()
+    check_interval = 1.0  # Check producer status every second
+    
+    # Process audio chunks until we're done
+    while True:
+        try:
+            # Get the next audio chunk with a short timeout
+            # This allows us to periodically check status and handle other events
+            audio = audio_queue.get(timeout=0.1)
             
-            while True:
-                audio = audio_queue.get()
-                if audio is None:
-                    # Write any remaining buffered chunks
-                    if write_buffer and wav_file:
-                        if future:
-                            future.result()  # Wait for previous write to complete
-                        write_chunks_to_file(write_buffer, wav_file)
-                    break
-                
-                audio_segments.append(audio)
-                
-                if wav_file:
-                    write_buffer.append(audio)
-                    if len(write_buffer) >= buffer_size:
-                        if future:
-                            future.result()  # Wait for previous write to complete
-                        # Write in a separate thread to avoid blocking
-                        chunks_to_write = write_buffer
-                        write_buffer = []
-                        future = executor.submit(write_chunks_to_file, chunks_to_write, wav_file)
-    else:
-        # Simpler direct approach for lower-end systems
-        while True:
-            audio = audio_queue.get()
+            # None marker indicates end of stream
             if audio is None:
+                print("Received end-of-stream marker")
                 break
             
+            # Store the audio segment for return value
             audio_segments.append(audio)
             
-            # Write to WAV file if provided
+            # Write to file if needed
             if wav_file:
-                wav_file.writeframes(audio)
+                write_buffer.extend(audio)
+                
+                # Flush buffer if it's large enough
+                if len(write_buffer) >= buffer_max_size:
+                    wav_file.writeframes(write_buffer)
+                    write_buffer = bytearray()  # Reset buffer
+        
+        except queue.Empty:
+            # No data available right now
+            current_time = time.time()
+            
+            # Periodically check if producer is done
+            if current_time - last_check_time > check_interval:
+                last_check_time = current_time
+                
+                # If producer is done and queue is empty, we're finished
+                if producer_done_event.is_set() and audio_queue.empty():
+                    print("Producer done and queue empty - finishing consumer")
+                    break
+                
+                # Flush buffer periodically even if not full
+                if wav_file and len(write_buffer) > 0:
+                    wav_file.writeframes(write_buffer)
+                    write_buffer = bytearray()  # Reset buffer
+    
+    # Extra safety check - ensure thread is done
+    if thread.is_alive():
+        print("Waiting for token processor thread to complete...")
+        thread.join(timeout=10.0)
+        if thread.is_alive():
+            print("WARNING: Token processor thread did not complete within timeout")
+    
+    # Final flush of any remaining data
+    if wav_file and len(write_buffer) > 0:
+        print(f"Final buffer flush: {len(write_buffer)} bytes")
+        wav_file.writeframes(write_buffer)
     
     # Close WAV file if opened
     if wav_file:
         wav_file.close()
-    
-    thread.join()
+        if output_file:
+            print(f"Audio saved to {output_file}")
     
     # Calculate and print detailed performance metrics
     if audio_segments:
@@ -461,8 +599,59 @@ def stream_audio(audio_buffer):
     except Exception as e:
         print(f"Audio playback error: {e}")
 
+import re
+import numpy as np
+from io import BytesIO
+import wave
+
+def split_text_into_sentences(text):
+    """Split text into sentences with a more reliable approach."""
+    # We'll use a simple approach that doesn't rely on variable-width lookbehinds
+    # which aren't supported in Python's regex engine
+    
+    # First, split on common sentence ending punctuation
+    # This isn't perfect but works for most cases and avoids the regex error
+    parts = []
+    current_sentence = ""
+    
+    for char in text:
+        current_sentence += char
+        
+        # If we hit a sentence ending followed by a space, consider this a potential sentence end
+        if char in (' ', '\n', '\t') and len(current_sentence) > 1:
+            prev_char = current_sentence[-2]
+            if prev_char in ('.', '!', '?'):
+                # Check if this is likely a real sentence end and not an abbreviation
+                # (Simple heuristic: if there's a space before the period, it's likely a real sentence end)
+                if len(current_sentence) > 3 and current_sentence[-3] not in ('.', ' '):
+                    parts.append(current_sentence.strip())
+                    current_sentence = ""
+    
+    # Add any remaining text
+    if current_sentence.strip():
+        parts.append(current_sentence.strip())
+    
+    # Combine very short segments to avoid tiny audio files
+    min_chars = 20  # Minimum reasonable sentence length
+    combined_sentences = []
+    i = 0
+    
+    while i < len(parts):
+        current = parts[i]
+        
+        # If this is a short sentence and not the last one, combine with next
+        while i < len(parts) - 1 and len(current) < min_chars:
+            i += 1
+            current += " " + parts[i]
+            
+        combined_sentences.append(current)
+        i += 1
+    
+    return combined_sentences
+
 def generate_speech_from_api(prompt, voice=DEFAULT_VOICE, output_file=None, temperature=TEMPERATURE, 
-                     top_p=TOP_P, max_tokens=MAX_TOKENS, repetition_penalty=REPETITION_PENALTY):
+                     top_p=TOP_P, max_tokens=MAX_TOKENS, repetition_penalty=None, 
+                     use_batching=True, max_batch_chars=1000):
     """Generate speech from text using Orpheus model with performance optimizations."""
     print(f"Starting speech generation for '{prompt[:50]}{'...' if len(prompt) > 50 else ''}'")
     print(f"Using voice: {voice}, GPU acceleration: {'Yes (High-end)' if HIGH_END_GPU else 'Yes' if torch.cuda.is_available() else 'No'}")
@@ -473,25 +662,186 @@ def generate_speech_from_api(prompt, voice=DEFAULT_VOICE, output_file=None, temp
     
     start_time = time.time()
     
-    # Generate speech with optimized settings
-    result = tokens_decoder_sync(
-        generate_tokens_from_api(
-            prompt=prompt, 
-            voice=voice,
-            temperature=temperature,
-            top_p=top_p,
-            max_tokens=max_tokens,
-            repetition_penalty=repetition_penalty
-        ),
-        output_file=output_file
-    )
+    # For shorter text, use the standard non-batched approach
+    if not use_batching or len(prompt) < max_batch_chars:
+        # Note: we ignore any provided repetition_penalty and always use the hardcoded value
+        # This ensures consistent quality regardless of what might be passed in
+        result = tokens_decoder_sync(
+            generate_tokens_from_api(
+                prompt=prompt, 
+                voice=voice,
+                temperature=temperature,
+                top_p=top_p,
+                max_tokens=max_tokens,
+                repetition_penalty=REPETITION_PENALTY  # Always use hardcoded value
+            ),
+            output_file=output_file
+        )
+        
+        # Report final performance metrics
+        end_time = time.time()
+        total_time = end_time - start_time
+        print(f"Total speech generation completed in {total_time:.2f} seconds")
+        
+        return result
+    
+    # For longer text, use sentence-based batching
+    print(f"Using sentence-based batching for text with {len(prompt)} characters")
+    
+    # Split the text into sentences
+    sentences = split_text_into_sentences(prompt)
+    print(f"Split text into {len(sentences)} segments")
+    
+    # Create batches by combining sentences up to max_batch_chars
+    batches = []
+    current_batch = ""
+    
+    for sentence in sentences:
+        # If adding this sentence would exceed the batch size, start a new batch
+        if len(current_batch) + len(sentence) > max_batch_chars and current_batch:
+            batches.append(current_batch)
+            current_batch = sentence
+        else:
+            # Add separator space if needed
+            if current_batch:
+                current_batch += " "
+            current_batch += sentence
+    
+    # Add the last batch if it's not empty
+    if current_batch:
+        batches.append(current_batch)
+    
+    print(f"Created {len(batches)} batches for processing")
+    
+    # Process each batch and collect audio segments
+    all_audio_segments = []
+    batch_temp_files = []
+    
+    for i, batch in enumerate(batches):
+        print(f"Processing batch {i+1}/{len(batches)} ({len(batch)} characters)")
+        
+        # Create a temporary file for this batch if an output file is requested
+        temp_output_file = None
+        if output_file:
+            temp_output_file = f"outputs/temp_batch_{i}_{int(time.time())}.wav"
+            batch_temp_files.append(temp_output_file)
+        
+        # Generate speech for this batch
+        batch_segments = tokens_decoder_sync(
+            generate_tokens_from_api(
+                prompt=batch,
+                voice=voice,
+                temperature=temperature,
+                top_p=top_p,
+                max_tokens=max_tokens,
+                repetition_penalty=REPETITION_PENALTY
+            ),
+            output_file=temp_output_file
+        )
+        
+        # Add to our collection
+        all_audio_segments.extend(batch_segments)
+    
+    # If an output file was requested, stitch together the temporary files
+    if output_file and batch_temp_files:
+        # Stitch together WAV files
+        stitch_wav_files(batch_temp_files, output_file)
+        
+        # Clean up temporary files
+        for temp_file in batch_temp_files:
+            try:
+                os.remove(temp_file)
+            except Exception as e:
+                print(f"Warning: Could not remove temporary file {temp_file}: {e}")
     
     # Report final performance metrics
     end_time = time.time()
     total_time = end_time - start_time
+    
+    # Calculate combined duration
+    if all_audio_segments:
+        total_bytes = sum(len(segment) for segment in all_audio_segments)
+        duration = total_bytes / (2 * SAMPLE_RATE)  # 2 bytes per sample at 24kHz
+        print(f"Generated {len(all_audio_segments)} audio segments")
+        print(f"Generated {duration:.2f} seconds of audio in {total_time:.2f} seconds")
+        print(f"Realtime factor: {duration/total_time:.2f}x")
+        
     print(f"Total speech generation completed in {total_time:.2f} seconds")
     
-    return result
+    return all_audio_segments
+
+def stitch_wav_files(input_files, output_file, crossfade_ms=50):
+    """Stitch multiple WAV files together with crossfading for smooth transitions."""
+    if not input_files:
+        return
+        
+    print(f"Stitching {len(input_files)} WAV files together with {crossfade_ms}ms crossfade")
+    
+    # If only one file, just copy it
+    if len(input_files) == 1:
+        import shutil
+        shutil.copy(input_files[0], output_file)
+        return
+    
+    # Convert crossfade_ms to samples
+    crossfade_samples = int(SAMPLE_RATE * crossfade_ms / 1000)
+    print(f"Using {crossfade_samples} samples for crossfade at {SAMPLE_RATE}Hz")
+    
+    # Build the final audio in memory with crossfades
+    final_audio = np.array([], dtype=np.int16)
+    first_params = None
+    
+    for i, input_file in enumerate(input_files):
+        try:
+            with wave.open(input_file, 'rb') as wav:
+                if first_params is None:
+                    first_params = wav.getparams()
+                elif wav.getparams() != first_params:
+                    print(f"Warning: WAV file {input_file} has different parameters")
+                    
+                frames = wav.readframes(wav.getnframes())
+                audio = np.frombuffer(frames, dtype=np.int16)
+                
+                if i == 0:
+                    # First segment - use as is
+                    final_audio = audio
+                else:
+                    # Apply crossfade with previous segment
+                    if len(final_audio) >= crossfade_samples and len(audio) >= crossfade_samples:
+                        # Create crossfade weights
+                        fade_out = np.linspace(1.0, 0.0, crossfade_samples)
+                        fade_in = np.linspace(0.0, 1.0, crossfade_samples)
+                        
+                        # Apply crossfade
+                        crossfade_region = (final_audio[-crossfade_samples:] * fade_out + 
+                                           audio[:crossfade_samples] * fade_in).astype(np.int16)
+                        
+                        # Combine: original without last crossfade_samples + crossfade + new without first crossfade_samples
+                        final_audio = np.concatenate([final_audio[:-crossfade_samples], 
+                                                    crossfade_region, 
+                                                    audio[crossfade_samples:]])
+                    else:
+                        # One segment too short for crossfade, just append
+                        print(f"Segment {i} too short for crossfade, concatenating directly")
+                        final_audio = np.concatenate([final_audio, audio])
+        except Exception as e:
+            print(f"Error processing file {input_file}: {e}")
+            if i == 0:
+                raise  # Critical failure if first file fails
+    
+    # Write the final audio data to the output file
+    try:
+        with wave.open(output_file, 'wb') as output_wav:
+            if first_params is None:
+                raise ValueError("No valid WAV files were processed")
+                
+            output_wav.setparams(first_params)
+            output_wav.writeframes(final_audio.tobytes())
+        
+        print(f"Successfully stitched audio to {output_file} with crossfading")
+    except Exception as e:
+        print(f"Error writing output file {output_file}: {e}")
+        raise
 
 def list_available_voices():
     """List all available voices with the recommended one marked."""
@@ -514,7 +864,7 @@ def main():
     parser.add_argument("--temperature", type=float, default=TEMPERATURE, help="Temperature for generation")
     parser.add_argument("--top_p", type=float, default=TOP_P, help="Top-p sampling parameter")
     parser.add_argument("--repetition_penalty", type=float, default=REPETITION_PENALTY, 
-                       help="Repetition penalty (>=1.1 required for stable generation)")
+                       help="Repetition penalty (fixed at 1.1 for stable generation - parameter kept for compatibility)")
     
     args = parser.parse_args()
     

tts_engine/speechpipe.py

@@ -145,15 +145,17 @@ token_cache = {}
 MAX_CACHE_SIZE = 1000  # Limit cache size to prevent memory bloat
 
 async def tokens_decoder(token_gen):
-    """Optimized token decoder with caching and conservative batch processing to ensure correct output"""
+    """Optimized token decoder with reliable end-of-buffer handling for complete audio generation"""
     buffer = []
     count = 0
-    # Start with conservative parameters to ensure we get audio output
-    min_frames_required = 28  # Default minimum frames (4 chunks of 7)
-    process_every_n = 7  # Process every 7 tokens
+    # Use a smaller minimum frame requirement to allow more flexible processing
+    min_frames_required = 28  # Lower requirement (4 chunks of 7 tokens)
+    ideal_frames = 49  # Ideal standard frame size (7×7 window)
+    process_every_n = 7  # Process every 7 tokens (standard for Orpheus model)
     
     start_time = time.time()
     token_count = 0
+    last_log_time = start_time
     
     async for token_sim in token_gen:
         token_count += 1
@@ -172,18 +174,70 @@ async def tokens_decoder(token_gen):
             buffer.append(token)
             count += 1
 
-            # Process in larger batches for better GPU utilization
-            if count % process_every_n == 0 and count >= min_frames_required:
-                buffer_to_proc = buffer[-min_frames_required:]
+            # Log throughput periodically
+            current_time = time.time()
+            if current_time - last_log_time > 5.0:  # Every 5 seconds
+                elapsed = current_time - last_log_time
+                if elapsed > 0:
+                    recent_tokens = token_count
+                    tokens_per_sec = recent_tokens / elapsed
+                    print(f"Token processing rate: {tokens_per_sec:.1f} tokens/second")
+                last_log_time = current_time
+                token_count = 0
+            
+            # Process standard batches when we have enough tokens
+            if count % process_every_n == 0:
+                # Best case: we have enough for the ideal frame size
+                if len(buffer) >= ideal_frames:
+                    buffer_to_proc = buffer[-ideal_frames:]
+                # Fallback: we have enough for the minimum requirement
+                elif len(buffer) >= min_frames_required:
+                    buffer_to_proc = buffer[-min_frames_required:]
+                # For the first few frames, we may not have enough yet
+                else:
+                    continue
+                
+                # Debug output to help diagnose issues
+                if count % 28 == 0:
+                    print(f"Processing buffer with {len(buffer_to_proc)} tokens, total collected: {len(buffer)}")
+                
+                # Process the tokens
                 audio_samples = convert_to_audio(buffer_to_proc, count)
                 if audio_samples is not None:
-                    # Log processing rate occasionally
-                    if count % 140 == 0:  # Log every 20 chunks (assuming process_every_n=7)
-                        elapsed = time.time() - start_time
-                        tokens_per_sec = token_count / elapsed if elapsed > 0 else 0
-                        print(f"Processing speed: {tokens_per_sec:.1f} tokens/sec, buffer size: {len(buffer)}")
-                    
                     yield audio_samples
+    
+    # CRITICAL: End-of-generation handling - process all remaining frames
+    # Process remaining complete frames (ideal size)
+    if len(buffer) >= ideal_frames:
+        buffer_to_proc = buffer[-ideal_frames:]
+        audio_samples = convert_to_audio(buffer_to_proc, count)
+        if audio_samples is not None:
+            yield audio_samples
+            
+    # Process any additional complete frames (minimum size)
+    elif len(buffer) >= min_frames_required:
+        buffer_to_proc = buffer[-min_frames_required:]
+        audio_samples = convert_to_audio(buffer_to_proc, count)
+        if audio_samples is not None:
+            yield audio_samples
+            
+    # Final special case: even if we don't have minimum frames, try to process
+    # what we have by padding with silence tokens that won't affect the audio
+    elif len(buffer) >= process_every_n:
+        # Pad to minimum frame requirement with copies of the final token
+        # This is more continuous than using unrelated tokens from the beginning
+        last_token = buffer[-1]
+        padding_needed = min_frames_required - len(buffer)
+        
+        # Create a padding array of copies of the last token
+        # This maintains continuity much better than circular buffering
+        padding = [last_token] * padding_needed
+        padded_buffer = buffer + padding
+        
+        print(f"Processing final partial frame: {len(buffer)} tokens + {padding_needed} repeated-token padding")
+        audio_samples = convert_to_audio(padded_buffer, count)
+        if audio_samples is not None:
+            yield audio_samples
 # ------------------ Synchronous Tokens Decoder Wrapper ------------------ #
 def tokens_decoder_sync(syn_token_gen):
     """Optimized synchronous decoder with larger queue and parallel processing"""
@@ -213,18 +267,25 @@ def tokens_decoder_sync(syn_token_gen):
         start_time = time.time()
         chunk_count = 0
         
-        # Process audio chunks from the token decoder
-        async for audio_chunk in tokens_decoder(async_token_gen()):
-            audio_queue.put(audio_chunk)
-            chunk_count += 1
-            
-            # Log performance stats periodically
-            if chunk_count % 10 == 0:
-                elapsed = time.time() - start_time
-                print(f"Generated {chunk_count} chunks in {elapsed:.2f}s ({chunk_count/elapsed:.2f} chunks/sec)")
-                
-        # Signal completion
-        audio_queue.put(None)  # Sentinel
+        try:
+            # Process audio chunks from the token decoder
+            async for audio_chunk in tokens_decoder(async_token_gen()):
+                if audio_chunk:  # Validate audio chunk before adding to queue
+                    audio_queue.put(audio_chunk)
+                    chunk_count += 1
+                    
+                    # Log performance stats periodically
+                    if chunk_count % 10 == 0:
+                        elapsed = time.time() - start_time
+                        print(f"Generated {chunk_count} chunks in {elapsed:.2f}s ({chunk_count/elapsed:.2f} chunks/sec)")
+        except Exception as e:
+            print(f"Error in audio producer: {e}")
+            import traceback
+            traceback.print_exc()
+        finally:    
+            # Signal completion
+            print("Audio producer completed - finalizing all chunks")
+            audio_queue.put(None)  # Sentinel
 
     def run_async():
         asyncio.run(async_producer())