AI LLM Gateway Billing Options
Use Case: Using the platform as an enterprise-internal AI LLM gateway, providing unified access to self-hosted model inference instances (private inference) and external AI channels (OpenAI / Zhipu / Tongyi and other providers), with differentiated cost calculation and allocation schemes for different departments (workspaces / projects).
This guide is intended for customers/users and explains the platform's metering and billing capabilities and recommended implementation configurations.
1. Overview
The platform is a complete AI Model Gateway + Metering and Billing system that natively covers the three key requirements:
| Requirement | Platform Capability | Readiness |
|---|---|---|
| Access and billing for self-hosted model inference instances | Token Governance (private inference billing: source_type=system, model billing strategy) | ✅ Implemented |
| External AI channel integration and billing | API Channel Billing (channel cost: source_type=custom, channel cost strategy) | ✅ Implemented |
| Cost calculation across departments | Workspace / Namespace billing and allocation | ✅ Implemented |
Core design: For each LLM call, the platform simultaneously answers two questions:
- How much did the user spend (payable amount): billed to their department / individual account based on the model alias invoked by the user.
- How much did the platform pay (channel cost): records the platform's procurement cost based on the actual external channel matched, used for reconciliation with providers.
The two pricing chains are independently configured and coexist in the same call log entry, without interfering with each other.
2. Billing System Overview
2.1 Dual Billing Dimensions
| Dimension | Meaning | Audience | Frontend Entry |
|---|---|---|---|
| Payable Amount | Cost of one LLM invocation for the user | End users / department heads / finance | Token Governance pages |
| Channel Cost | Actual procurement cost paid by the platform to external channels | Platform administrators / operations / channel reconciliation | API Channel pages |
Typical relationships:
- For self-hosted inference: Both dimensions may be identical (platform-operated, no procurement cost), charged per the same model billing strategy.
- For external channels: Payable = channel cost × profit margin (typical resale), or payable = 0 (internal free team), but costs still need to be tracked.
- The same virtual model can use multi-channel canary routing (A/B / fallback); the user unit price remains stable while the actual channel cost may vary per call.
2.2 Billing Pipeline
User Request → Higress AI Gateway → (matches AiRoute) → Self-hosted Inference Instance or External AI Channel
│
▼
ClickHouse Call Log (one row per request, recording token/latency/identity)
│
▼ ⏰ Scheduled task backfill
Calculates two prices: Payable Amount (by model alias) + Channel Cost (by upstream model)
│
▼
PostgreSQL Billing Strategy Rules (billings table, pricing by level/model/channel)Costs are not calculated in real time: requests first land in logs, and a scheduled task batch-backfills prices (default interval of a few minutes, can be triggered manually). Failed calls (HTTP ≥ 400) are not billed.
3. Billing Strategy Configuration
3.1 Four Types of Billing Strategies
| Strategy | Location | Billing Target | Unit |
|---|---|---|---|
| Model Billing Strategy | /admin/billing/governance/strategy | LLM calls (Token / count / duration / characters) | ¥/1M Token, ¥/call, ¥/1k seconds, ¥/1k characters |
| Channel Cost Strategy | /admin/billing/channel/strategy | External API channel procurement cost | Same as above (multi-currency CNY / USD) |
| Physical Resource Billing Strategy | /admin/billing/strategy | CPU / memory / GPU memory | ¥/core, ¥/GiB (per billing cycle) |
| Machine Resource Billing Strategy | /admin/billing/machine/strategy | Physical machine CPU / memory / GPU memory / GPU compute | ¥/core, ¥/GiB |
3.2 Model Billing Strategy (Token Governance — User Payable)
Each model can be configured with the following pricing tiers:
| Tier | Description | Unit |
|---|---|---|
| Input (prompt) | Input tokens in the user request | ¥/1M Token (required) |
| Output (completion) | Output tokens generated by the model | ¥/1M Token |
| Cache Hit (cached read) | Tokens matching the context cache | ¥/1M Token |
| Cache Write 5 Minutes | Anthropic-style short TTL cache write | ¥/1M Token |
| Cache Write 1 Hour | Anthropic-style long TTL cache write | ¥/1M Token |
In addition to token-based pricing, non-text billing units are also supported:
| Unit | Applicable Scenario | Description |
|---|---|---|
| Per Call | Image generation / video generation, etc. | Fixed price per invocation; supports tiered pricing tables (e.g., by resolution 512×512 / 1024×1792, by duration, by quality tier) |
| Per 1k Seconds | ASR speech recognition | Billed by audio duration |
| Per 1k Characters | TTS speech synthesis | Billed by text character count |
Supports multimodal metering: images by count/resolution tier, audio by seconds, text by token/character.
3.3 Channel Cost Strategy (API Channel — Platform Reconciliation)
| Configuration Item | Description |
|---|---|
| Pricing Dimension | (Channel, upstream model name); supports * wildcard for "channel base price" |
| Price Mode | "Specific Model" (exact model pricing) or "Channel Base Price" (uniform price for all models on the channel) |
| Currency | CNY (default) or USD; exchange rate snapshot locked |
| Unit | Same as model billing strategy (Token five-tier / per call / per 1k seconds / per 1k characters) |
4. Differentiated Billing Across Departments (Workspace / Namespace)
4.1 Billing Strategy Scope
Billing strategies support multi-level scope overrides, from platform-wide defaults to workspace-level overrides:
| Level | Meaning | Description |
|---|---|---|
system | Platform default price | Baseline price effective across the entire platform |
workspace | Workspace override price | Custom pricing for a specific department/tenant, overrides the platform default |
inference | Private inference exclusive price | Dedicated pricing for self-hosted NovaInference inference services |
api_channel | Channel cost rule | Pricing for external channel procurement costs |
4.2 Price Hit Priority (5 Levels + Wildcard)
1. Channel + Workspace + Model ← Most specific
2. Channel + Workspace + "*"
3. Channel + Model
4. Channel + "*" ← Channel base price
5. Workspace + Model ← Department exclusive price
6. Platform + Model ← Platform default price
7. FallbackConclusion: You can configure different billing strategies for different departments (workspaces). For example: R&D department pays ¥0.8/1M Token for internal allocation, marketing department pays ¥2/1M Token, independently; departments without custom configuration automatically inherit the platform default price.
4.3 Cost Allocation Dimensions
| Dimension | Description |
|---|---|
| Workspace | Department-level cost summary (billing strategies are configured at this level) |
| Project / Namespace | Project-level cost drill-down (billing reports support aggregation by namespace) |
| User (creator) | Individual-level costs |
| Model × User | Combined analysis by model and user |
Physical resource billing records (BillRecord) support four-level drill-down:
system → workspace → namespace → creator; model token billing records support multi-level aggregation:system → workspace → model → user → API Key.
5. Cost Query and Reports
5.1 Token Governance (User Perspective)
| Page | Capabilities |
|---|---|
| Token Governance Overview | Total requests / total tokens / payable amount / error rate / average latency KPIs |
| Call Records | Per-call detail (with five pricing tier breakdown) |
| Consumer Analysis | Aggregated by user / workspace; payable amount ranking |
| Key Analysis | Usage, quota, and rate limiting by API Key |
| Model Ranking | Token consumption and cost by model |
5.2 API Channels (Platform Perspective)
| Page | Capabilities |
|---|---|
| Channel Overview | Channel cost KPIs, per-channel cost trends |
| Channel Analysis | Cost aggregated by channel (AiProvider) |
| Upstream Models | Aggregated by (channel, upstream model) |
| Channel Quality | Latency / error rate monitoring |
| Channel Calls | Call records with source_type=custom forced |
5.3 Physical Resource Cost (Billing Management)
- Three-level drill-down: node cost / GPU card type cost / per-card cost
- Cost pie charts, trend line charts, distribution charts, Top 5 ranking
- Call log CSV export (up to 10,000 rows per export), resource cost detail export
5.4 End-User Self-Service
- My Usage (
/admin/model-x/my-usage): Personal usage overview / trends / top models / recent calls - My Calls (
/admin/model-x/my-calls): Personal call records - End-user view automatically hides administrative dimensions such as channels, upstream models, and routing
6. Recommended Implementation
6.1 Onboarding Steps for New Scenarios
- Connect self-hosted inference: Deploy model inference instances (NovaInference) → system automatically registers as a "system channel".
- Connect external channels: Configure API channels (AiProvider — OpenAI / Anthropic / Zhipu protocol) → configure routing (AiRoute) and model mapping.
- Set prices:
systemlevel: Configure platform-wide default model billing strategy (external charging).workspacelevel: Configure department-exclusive pricing (differentiated charging).api_channellevel: Configure channel cost strategy (procurement cost, multi-currency).
- Create accounts: Create workspaces + projects for departments, assign API Keys.
- Review reports: Operations use channel analysis for reconciliation; departments use consumer analysis for allocation; individuals use My Usage for self-service queries.
6.2 Recommended Pricing Model Combinations
| Business Scenario | Recommended Configuration |
|---|---|
| Internal general-purpose self-hosted models | system level model billing strategy, five-tier token pricing |
| External LLM resale | api_channel channel cost strategy + system/workspace payable strategy (margin = markup) |
| Cross-department differential pricing | Per-department workspace level model billing strategy (overrides platform default) |
| Image / video generation services | per_call per invocation + tiered pricing table (resolution / quality) |
| ASR speech services | per_1k_seconds per 1k seconds |
| TTS speech services | per_1k_chars per 1k characters |
7. Current Limitations and Notes
| Capability | Status | Notes |
|---|---|---|
| GPU / CPU / memory metering and billing | ✅ | Supported, per core / GiB / billing cycle |
| Storage (disk / PVC) metering and billing | ❌ Not yet supported | Physical resource billing dimensions do not include standalone storage; planned as an extension |
| Budget alerts / quota enforcement | ⏳ Planned | Quota configuration and usage queries are currently supported; budget-triggered actions are planned for a future phase |
| Multi-currency | ⚠️ Partial | Channel cost side supports CNY / USD; payable side defaults to CNY |
| Billing strategy version management | ✅ | Each strategy supports viewing historical versions (revision) for audit trail |
8. Summary
The platform is ready with AI Model Gateway + Dual-Dimension Metering and Billing capabilities and can directly support the integrated requirements of "self-hosted inference + external channel integration + department-level cost allocation":
- Internal: Self-hosted inference is transparently billed by token/count, with department-level allocation by workspace/namespace.
- External: External channels are reconciled at actual cost, with clear resale margin visibility.
- Management: Multi-level strategy scopes + five-layer price priority + version auditing provide flexible configuration and auditable accounting.
For detailed configuration steps, pricing examples, or custom reports, see Billing > Token Governance and Billing > API Channels, or contact the platform administrator.