LLM 추론 최적화 Part 4 — 프로덕션 서빙

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    dtype="float16",

    # === 메모리 관리 ===
    gpu_memory_utilization=0.90,   # GPU 메모리의 90% 사용
    max_model_len=32768,            # 최대 컨텍스트 길이

    # === KV Cache 최적화 ===
    kv_cache_dtype="auto",          # "auto", "fp8_e5m2", "fp8_e4m3"
    # kv_cache_dtype="fp8_e5m2",    # FP8 KV Cache → 메모리 2x 절감

    # === 양자화 ===
    # quantization="awq",           # 모델 가중치 양자화

    # === 병렬화 ===
    tensor_parallel_size=1,         # GPU 수
)

params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=1024,
    stop=["<|eot_id|>"],
)

output = llm.generate("Explain quantum computing.", params)
print(output[0].outputs[0].text)

# vLLM 서버 시작
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dtype float16 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 32768 \
    --kv-cache-dtype fp8_e5m2 \
    --port 8000

# 클라이언트에서 사용 (OpenAI SDK 호환)
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing."},
    ],
    temperature=0.7,
    max_tokens=1024,
)

print(response.choices[0].message.content)

[Static Batching]
Request A: "Hello" → 50 tokens (빨리 끝남)
Request B: "Write an essay..." → 500 tokens (느림)
Request C: "Hi" → 30 tokens (빨리 끝남)

→ 배치가 모두 끝날 때까지 기다려야 함
→ A, C는 B를 기다리며 GPU 놀림
→ Total time: ~B의 시간 (비효율)

[Continuous Batching]
Step 1: A, B, C 동시 처리
Step 30: C 완료 → 즉시 새 Request D 투입
Step 50: A 완료 → 즉시 새 Request E 투입
Step 500: B 완료

→ GPU가 항상 최대 부하로 동작
→ Throughput 2~5x 향상

# vLLM은 기본적으로 Continuous Batching 사용
# 별도 설정 없이 여러 요청을 동시에 보내면 자동 배치

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

async def send_request(prompt: str):
    response = await client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=256,
    )
    return response.choices[0].message.content

async def benchmark_continuous_batching():
    """동시 요청으로 Continuous Batching 테스트"""
    prompts = [
        "What is Python?",
        "Write a haiku about AI.",
        "Explain gravity in one sentence.",
        "List 5 programming languages.",
        "What is the speed of light?",
        "Describe photosynthesis briefly.",
        "What is machine learning?",
        "Name 3 planets.",
    ]

    import time
    start = time.perf_counter()

    # 모든 요청을 동시에 전송
    results = await asyncio.gather(*[send_request(p) for p in prompts])

    elapsed = time.perf_counter() - start
    print(f"Total time for {len(prompts)} requests: {elapsed:.1f}s")
    print(f"Average per request: {elapsed/len(prompts):.1f}s")
    print(f"Throughput: {len(prompts)/elapsed:.1f} req/s")

asyncio.run(benchmark_continuous_batching())

[기존 Decode]
Step 1: Large model → token 1
Step 2: Large model → token 2
Step 3: Large model → token 3
→ 3번의 Large model forward pass

[Speculative Decoding]
Step 1: Small model → [token 1, token 2, token 3] (빠름)
Step 2: Large model → 3개 토큰 한 번에 검증
→ 1번의 Large model forward pass로 ~3 토큰 생성

# vLLM에서 Speculative Decoding 설정
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --speculative-model meta-llama/Llama-3.2-1B-Instruct \
    --num-speculative-tokens 5 \
    --dtype float16 \
    --port 8000

def design_memory_budget(
    model_name: str,
    model_params_b: float,
    num_layers: int,
    kv_heads: int,
    head_dim: int,
    gpu_vram_gb: float,
    model_quant: str = "fp16",       # "fp16", "int8", "int4"
    kv_quant: str = "fp16",          # "fp16", "fp8", "int8"
    target_context: int = 8192,
):
    """실전 VRAM 버짓 설계"""

    quant_bytes = {"fp16": 2, "fp8": 1, "int8": 1, "int4": 0.5}
    model_bytes = quant_bytes[model_quant]
    kv_bytes = quant_bytes[kv_quant]

    # 1. 모델 가중치
    model_gb = model_params_b * 1e9 * model_bytes / 1024**3

    # 2. CUDA Context + Overhead (~1.5 GB)
    overhead_gb = 1.5

    # 3. 사용 가능한 KV Cache 공간
    available_gb = gpu_vram_gb - model_gb - overhead_gb

    # 4. 토큰당 KV Cache 크기
    kv_per_token = 2 * num_layers * kv_heads * head_dim * kv_bytes

    # 5. 최대 동시 토큰 수 (모든 배치의 합)
    max_total_tokens = int(available_gb * 1024**3 / kv_per_token)

    # 6. 목표 컨텍스트에서 최대 동시 요청 수
    max_concurrent = max_total_tokens // target_context

    print(f"=== {model_name} on {gpu_vram_gb}GB GPU ===")
    print(f"Model ({model_quant}): {model_gb:.1f} GB")
    print(f"Overhead: {overhead_gb:.1f} GB")
    print(f"Available for KV: {available_gb:.1f} GB")
    print(f"KV per token ({kv_quant}): {kv_per_token} bytes")
    print(f"Max total tokens: {max_total_tokens:,}")
    print(f"@ {target_context} context → Max concurrent: {max_concurrent}")
    print()

    return {
        'model_gb': model_gb,
        'available_kv_gb': available_gb,
        'max_total_tokens': max_total_tokens,
        'max_concurrent': max_concurrent,
    }

# === 시나리오별 설계 ===

# 시나리오 1: RTX 4090 (24GB) — 개인 서버
print("=== Scenario 1: Personal Server (RTX 4090 24GB) ===\n")
design_memory_budget("Llama 3.1 8B", 8, 32, 8, 128, 24,
                     model_quant="int4", kv_quant="fp16", target_context=8192)

# 시나리오 2: A100 (80GB) — 프로덕션
print("=== Scenario 2: Production (A100 80GB) ===\n")
design_memory_budget("Llama 3.1 8B", 8, 32, 8, 128, 80,
                     model_quant="fp16", kv_quant="fp8", target_context=16384)

# 시나리오 3: A100 (80GB) — 긴 컨텍스트 에이전트
print("=== Scenario 3: Long Context Agent (A100 80GB) ===\n")
design_memory_budget("Llama 3.1 8B", 8, 32, 8, 128, 80,
                     model_quant="int4", kv_quant="fp8", target_context=65536)

# 시나리오 4: H100 (80GB) — 70B 모델
print("=== Scenario 4: Large Model (H100 80GB) ===\n")
design_memory_budget("Llama 3.1 70B", 70, 80, 8, 128, 80,
                     model_quant="int4", kv_quant="fp8", target_context=8192)

# TGI 서버 시작 (Docker)
docker run --gpus all -p 8080:80 \
    -v $HOME/.cache/huggingface:/data \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id meta-llama/Llama-3.1-8B-Instruct \
    --dtype float16 \
    --max-input-tokens 4096 \
    --max-total-tokens 8192 \
    --max-batch-prefill-tokens 16384 \
    --max-concurrent-requests 64

# TGI 클라이언트
from huggingface_hub import InferenceClient

client = InferenceClient("http://localhost:8080")

response = client.chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the KV cache."},
    ],
    max_tokens=512,
    temperature=0.7,
)

print(response.choices[0].message.content)

import time
import asyncio
from openai import AsyncOpenAI

async def benchmark_throughput(
    base_url: str,
    model: str,
    num_requests: int = 100,
    max_concurrent: int = 16,
    input_tokens: int = 128,
    output_tokens: int = 256,
):
    """서빙 엔진 처리량 벤치마크"""
    client = AsyncOpenAI(base_url=base_url, api_key="dummy")

    # 고정 길이 프롬프트 생성
    prompt = "Write a detailed explanation about: " + "token " * input_tokens

    semaphore = asyncio.Semaphore(max_concurrent)
    results = []

    async def single_request():
        async with semaphore:
            start = time.perf_counter()
            try:
                response = await client.chat.completions.create(
                    model=model,
                    messages=[{"role": "user", "content": prompt}],
                    max_tokens=output_tokens,
                    temperature=0.7,
                )
                elapsed = time.perf_counter() - start
                tokens = response.usage.completion_tokens
                return {"elapsed": elapsed, "tokens": tokens, "success": True}
            except Exception as e:
                return {"elapsed": 0, "tokens": 0, "success": False, "error": str(e)}

    start_total = time.perf_counter()
    results = await asyncio.gather(*[single_request() for _ in range(num_requests)])
    total_time = time.perf_counter() - start_total

    successful = [r for r in results if r["success"]]
    total_tokens = sum(r["tokens"] for r in successful)

    print(f"=== Throughput Benchmark ===")
    print(f"Requests: {len(successful)}/{num_requests} successful")
    print(f"Total time: {total_time:.1f}s")
    print(f"Throughput: {len(successful)/total_time:.1f} req/s")
    print(f"Token throughput: {total_tokens/total_time:.0f} tok/s")
    print(f"Avg latency: {sum(r['elapsed'] for r in successful)/len(successful)*1000:.0f} ms")
    print(f"P50 latency: {sorted(r['elapsed'] for r in successful)[len(successful)//2]*1000:.0f} ms")
    print(f"P99 latency: {sorted(r['elapsed'] for r in successful)[int(len(successful)*0.99)]*1000:.0f} ms")

# 벤치마크 실행
asyncio.run(benchmark_throughput(
    base_url="http://localhost:8000/v1",
    model="meta-llama/Llama-3.1-8B-Instruct",
    num_requests=100,
    max_concurrent=16,
))

[개인/소규모]
모델: Qwen 2.5 7B (GQA-4, 적은 KV Cache)
양자화: GPTQ int4
서빙: vLLM + Flash Attention 2
KV: fp16 (7B 모델은 KV가 작아서 양자화 불필요)

[프로덕션]
모델: Llama 3.1 8B 또는 70B
양자화: AWQ int4 (8B) 또는 fp16 Multi-GPU (70B)
서빙: vLLM + Continuous Batching + FP8 KV Cache
추가: Speculative Decoding (코드 생성 등)

[긴 컨텍스트 에이전트]
모델: DeepSeek-V3.2 (DSA 내장) 또는 Llama 3.1 + DMS
양자화: AWQ int4
서빙: vLLM + FP8 KV Cache
추가: IndexCache (DSA 모델), 또는 KVTC 스타일 압축