Snowglobe
A sealed world you can shake.
One binary that generates a whole distributed system's telemetry: logs, traces and metrics from a topology of up to 28 services with the shapes real systems actually make. The default (-complexity normal) runs 20 traditional services; -complexity heavy adds the 8 AI services for 28. Diamond dependencies, scatter-gather fan-out, sagas that compensate, timeouts that cascade. Point it at anything that speaks OTLP and watch it arrive.
No Docker Compose. No microservices to deploy. No 6GB stack to babysit. One executable, or a 6.4 MB container.
docker run --rm immersivefusion/snowglobe -insecure -endpoint host.docker.internal:4317
A public good for the OpenTelemetry community. Apache-2.0, no account, no sign-up, no lock-in. The output is plain OTLP, so it works in whatever you already run: Jaeger, Tempo, Grafana, SigNoz, an OpenTelemetry Collector, or spatial tools such as DeepCube, which are one consumer among many.
How this differs from telemetrygen. telemetrygen generates spans, and it is the official tool and good at that. Snowglobe generates a system: a topology with real dependency shapes, injectable failures, and AI agent spans under the OTel GenAI semantic conventions. If you need load, use telemetrygen. If you need something that looks like a production incident, that is what this is for.
Most trace generators die. Their repos go quiet, the SDKs move, and the thing stops building. This one is maintained, and that is the point of it.
It has a sibling. Snowglobe makes realistic telemetry out of nothing: synthetic OTel. sos-beacon is the organic counterpart, a real workload doing a real job that emits real telemetry as a byproduct. Between them you can fill a backend with either kind.
Why This Exists
Every existing trace generator falls into one of two categories:
- Flat span generators (telemetrygen, tracepusher) - produce uniform, identical spans with no service topology
- Full demo apps (OTel Astronomy Shop, Jaeger HotROD) - require Docker Compose with 15+ containers and ~6GB RAM
And none of them generate AI agentic traces. The LLM observability market has no standalone tool that combines traditional APM with LLM observability. Every specialized LLM tool (Langfuse, LangSmith, Helicone, Arize, Traceloop, Portkey, Galileo) tracks token usage, model costs, and agent tool calls - but none of them provide traditional distributed tracing.
This tool generates topology-rich, failure-injectable traces from a single binary - covering both traditional microservice flows AND AI agentic patterns with OTel GenAI semantic conventions. One binary proves that a platform can visualize both. The AI services and their GenAI spans are in the heavy tier, so run -complexity heavy to get them; the default emits traditional traffic only.
Quick Start
# Download the latest release (or build from source)
go install github.com/ImmersiveFusion/snowglobe/cmd/snowglobe@latest
# Send to a local OTLP collector (Jaeger, Tempo, etc.)
snowglobe -insecure
# Send to a remote endpoint with auth headers
snowglobe -endpoint your-otlp-endpoint:443 -headers "api-key=YOUR_KEY"
# Or set headers via the standard OTel environment variable
export OTEL_EXPORTER_OTLP_HEADERS="api-key=YOUR_KEY"
snowglobe -endpoint your-otlp-endpoint:443
Installed from an old path? Releases up to v0.7.7 were published as github.com/ImmersiveFusion/if-opentelemetry-tracegen, and releases up to v0.8.0 as github.com/ImmersiveFusion/opentelemetry-tracegen. Pinned installs of those versions keep working, but @latest on either old path stops resolving from the next release onward, so update your command to the path above. The installed binary is now named snowglobe rather than tracegen.
See it in 3D - Send traces to DeepCube (
snowglobe -endpoint otlp.deepcube.ai:443 -headers "api-key=YOUR_KEY", how to get a key) to explore them as a 3D force-directed graph, drill into conventional trace waterfalls for detailed analysis, and get AI-assisted insights from Tessa. For a ready-made example without any setup, try Shoebox at shoebox.deepcube.ai - paste a diagram of a system, break a call in it, and fire one request through.
Live demo grids: see it running
Seven always-on demo grids stream live OpenTelemetry traces, logs and metrics into DeepCube's 3D player right now: a clean baseline, an AI-native app, a blended environment, phantom-service detection, an AI-outage, and a full incident. Each grid is this container, deployed declaratively via GitOps (Argo CD) in the Immersive Fusion cloud, multi-arch and distroless, one matrix row per grid, shipping to otlp.deepcube.ai:443.
Just want to look? The grids are streamed live on Twitch at twitch.tv/deepcubelive, around the clock. No account, no install, nothing to sign up for. It is running whether or not you do anything, which is the easiest way to see what this generator's output actually looks like at the other end.
See them in 3D: the full experience is DeepCube, the immersive 3D client: install it and open a grid to walk the live traces. On mobile or can't install right now? DeepCube Web runs the same grids in your browser at portal.deepcube.ai.
Where else does Snowglobe run?: a community board of deployments. Add yours.
Features
28 Microservices
Traditional Services (20)
| Service | Pods | Role |
|---|---|---|
| web-frontend | 2 | Browser client, SPA |
| api-gateway | 3 | HTTP routing, auth |
| order-service | 3 | Order lifecycle |
| payment-service | 2 | Stripe integration |
| inventory-service | 2 | Stock management |
| notification-service | 2 | Event-driven notifications |
| user-service | 2 | Auth, profiles |
| cache-service | 3 | Redis cluster |
| search-service | 2 | Elasticsearch queries |
| scheduler-service | 1 | Cron jobs (singleton) |
| auth-service | 3 | JWT, webhook verification |
| recommendation-service | 2 | ML-based recommendations |
| cart-service | 2 | Shopping cart |
| product-service | 3 | Product catalog |
| shipping-service | 2 | Rates, labels, tracking |
| fraud-service | 2 | ML fraud scoring |
| email-service | 2 | SMTP relay (SendGrid) |
| tax-service | 1 | Tax calculation |
| analytics-service | 3 | Event tracking (Kafka) |
| config-service | 1 | Feature flags |
AI Services (8)
| Service | Pods | Role |
|---|---|---|
| llm-gateway | 3 | OpenAI API routing, token tracking |
| embedding-service | 2 | Text-to-vector operations |
| vector-db-service | 2 | Qdrant similarity search |
| ai-agent-service | 2 | Agent orchestration (plan/act/reflect) |
| content-moderation-service | 2 | Safety classifiers, PII detection |
| model-registry-service | 1 | Model versioning (singleton) |
| feature-store-service | 2 | ML feature serving |
| data-pipeline-service | 2 | Batch embedding, retraining |
All 59 pods are distributed across 5 AKS VMSS nodes (2 node pools) with realistic service.instance.id and host.name resource attributes.
40 Scenario Flows
Traditional Scenarios (15 original + 13 new)
| Scenario | Graph Shape | Key Pattern |
|---|---|---|
| Create Order | Long chain (8 services, 14+ spans) | Producer/consumer with queue delays |
| Search & Browse | Linear with cache | Elasticsearch + Redis |
| User Login | Branching (success/failure) | Auth with session creation |
| Failed Payment | Error chain | Stripe 402 + error propagation |
| Bulk Notifications | Fan-out (3-5 parallel) | Batch email processing |
| Health Check | Star topology (6 parallel) | Concurrent health pings |
| Inventory Sync | Fan-out + reindex | Parallel cache warming |
| Scheduled Report | Headless chain (no UI) | Cron job entry point |
| Stripe Webhook | Headless chain (no gateway) | External callback entry |
| Recommendations | Scatter-gather / bowtie | Fan-out to 3, gather, cache |
| Add to Cart | Cross-service with feature flags | Config service + analytics |
| Full Checkout | Monster chain (15 services) | Tax+shipping parallel, fraud ML |
| Shipping Update | Carrier webhook (headless) | External webhook + email relay |
| Saga Compensation | Forward chain + 4-way compensation fan-out | Payment retries + rollback |
| Timeout Cascade | Branching with circuit breaker | Stale cache fallback |
| User Registration | Linear with async branch | Email verification token, duplicate detection |
| Product Review | Write + async moderation | Optimistic write + background processing |
| Return/Refund | Parallel reverse flow (16-18 spans) | Parallel refund + restock, reverse money flow |
| Wishlist + Price Alert | Write-through with async | Write-through cache, async price monitoring |
| Coupon Application | Validation chain | Cart recalculation, validation branch |
| Gift Card Purchase | Payment splitting | Balance check, payment splitting |
| Subscription Management | Webhook-driven lifecycle | Stripe subscription, renewal webhook |
| A/B Test Exposure | Feature flag branch | Variant assignment, sticky session |
| Rate Limiting | Early termination (4-6 spans) | Redis sliding window, 429 response |
| Admin Product CRUD | Write-amplification fan-out | Cache + search reindex on write |
| Order History | Paginated read | Keyset pagination, cursor-based |
| Support Ticket | Cross-domain trace | SLA assignment, team routing |
| Multi-Currency Checkout | External API chain | FX rate API, cache hit ratio |
AI Agentic Scenarios (12)
| Scenario | Graph Shape | Key Pattern |
|---|---|---|
| Semantic Search (RAG) | Linear with 2 LLM calls (14-16 spans) | Embedding + vector search + LLM reranking |
| AI Chatbot with Tool Use | Double bowtie (18-22 spans) | Plan -> fan-out tool calls -> synthesize |
| AI Content Moderation | Parallel classifiers + 3-way branch (12-16 spans) | Safety/spam scoring, guardrail decisions |
| Multi-Step Agent | Iterative loop (28-40 spans) | Plan -> act -> reflect cycle (3-5 iterations) |
| AI Customer Support | Branching with escalation (16-20 spans) | Sentiment classification, intent detection |
| AI Content Generation | Linear with safety filter (12-15 spans) | Temperature-controlled generation, content safety |
| Embedding Pipeline | High fan-out batch (25-40 spans) | Batch chunking, parallel embedding, vector upsert |
| Dynamic Pricing Agent | Headless agent (14-18 spans) | Feature store lookup, autonomous price updates |
| Fraud with Explainability | Linear with LLM explanation (10-12 spans) | SHAP-style feature attribution via LLM |
| Inventory Reorder Agent | Autonomous agent (16-20 spans) | Demand forecast, autonomous purchase orders |
| Model Retraining Pipeline | Batch pipeline (14-18 spans) | ML training spans, model registry, quality gate |
| Conversational Commerce | Multi-turn session (10-14 spans/turn) | Growing context tokens, session continuity |
Note: Failed Payment, Saga Compensation, Timeout Cascade, lost messages, and retry storms only activate when
-errors > 0. AI error scenarios (rate limits, hallucinated tool calls, token budget exceeded, content filter blocks) also require-errors > 0.
Correlated Logs
Every service emits OTel log records via OTLP alongside traces. Logs are automatically correlated with the active span context (trace_id, span_id), so your APM platform can link logs to the exact span that produced them.
- ERROR logs are emitted alongside every exception event (cache failures, DB errors, payment declines, LLM rate limits, agent failures)
- WARN logs fire on auth failures, content moderation flags, payment retries, and LLM fallbacks
- INFO logs cover request entry points, payment processing, fraud analysis, agent invocations, and iteration progress
Disable with -no-logs to emit traces only.
Metrics
Every service emits OTel metrics via OTLP alongside traces and logs. They are derived from the spans the scenarios already produce, so metrics and traces agree by construction: the p99 in a histogram is the same span you can click into.
Metrics aggregate in process and flush on an interval, so unlike traces their volume is decoupled from -level. A -level 10 run emits roughly 350 traces/second and the same metric load as -level 1.
| Metric | Type | What it shows |
|---|---|---|
http.server.request.duration |
Histogram (s) | RED metrics per route, method and status. Semconv names, seconds, spec bucket boundaries |
http.server.active_requests |
UpDownCounter | In-flight requests, rising and falling with real scenario concurrency |
system.cpu.utilization |
Gauge | Driven by each service's actual span volume, not a random walk |
system.memory.usage |
UpDownCounter | Resident memory, drifting with load. Carries system.memory.state |
db.client.connection.count |
UpDownCounter | Pool occupancy split used/idle, tracking real database spans |
tracegen.messaging.queue.depth |
Gauge | Published minus consumed, per destination |
gen_ai.client.token.usage |
Histogram | Input and output tokens by model, system and operation |
gen_ai.client.operation.duration |
Histogram (s) | LLM call latency |
The queue depth is the one to watch. Producer spans increment it and consumer spans decrement it, so running with -no-consumers makes the backlog climb without bound while the trace graph still looks busy. That is a story traces alone can only imply.
Naming. Where OpenTelemetry defines a metric, Snowglobe uses its name and conforms to its contract, including seconds as the unit. Everything else lives under a tracegen. prefix rather than extending a semconv namespace. That prefix keeps the old name on purpose: a metric name is a wire contract, and renaming it to snowglobe. would break every dashboard and alert already keyed on it. Do not sweep it.
Cardinality. service.instance.id is deliberately off by default: it multiplies every series by the pod count (up to 59 at -complexity heavy), and metric backends commonly bill per active series with no cardinality control on an OTLP push path. Turn it on with -metrics-instance-id when per-pod resolution is the thing you are demonstrating. Metric attributes are an explicit allowlist, never a copy of the span's attributes, so user, session and order ids never reach a metric.
Temporality. -metrics-temporality delta switches from the default cumulative. Prometheus-backed stores are cumulative-native, while some OTLP ingest paths ask producers for delta, and which behavior a given endpoint implements is often undocumented. A producer that can emit either shape on demand is the cheapest way to settle that by experiment.
Disable with -no-metrics to emit traces and logs only.
OTel GenAI Semantic Conventions
All AI scenarios emit spans following OTel GenAI Semantic Conventions and matching the exact span shapes produced by Microsoft Semantic Kernel and Microsoft Agent Framework.
Span types:
| Span Name Pattern | SpanKind | Example |
|---|---|---|
chat {model} |
CLIENT | chat gpt-4o |
embedding {model} |
CLIENT | embedding text-embedding-3-small |
invoke_agent {name} |
CLIENT | invoke_agent CustomerSupportAgent |
execute_tool {name} |
INTERNAL | execute_tool get_order_status |
{operation} {collection} |
CLIENT | query product-embeddings |
Attributes on every LLM span:
gen_ai.provider.name- LLM provider (e.g.,openai)gen_ai.request.model/gen_ai.response.model- model requested and usedgen_ai.usage.input_tokens/gen_ai.usage.output_tokens- token consumptiongen_ai.response.finish_reasons- completion reason (stop,tool_calls,length,content_filter)gen_ai.response.id- unique response identifiergen_ai.request.temperature,gen_ai.request.max_tokens- request parameters
Agent-specific attributes:
gen_ai.agent.id/gen_ai.agent.name/gen_ai.agent.description- agent identitygen_ai.conversation.id- session linking for multi-turn interactionsgen_ai.tool.name/gen_ai.tool.type/gen_ai.tool.call.id- tool call trackinggen_ai.data_source.id- RAG data source identifiergen_ai.request.embedding.dimensions- embedding dimensions
These attributes match what every LLM observability tool on the market tracks - enabling direct comparison of visualization capabilities.
Chaos & Failure Injection
| Feature | Description |
|---|---|
| Lost messages | 5% chance per queue hop that the consumer never fires - trace ends abruptly |
| Dead consumer mode | -no-consumers flag: producers fire, consumers never pick up |
| Retry storms | Payment retries 3x with exponential backoff before saga compensation |
| Timeout cascades | Search service times out, gateway returns 504, circuit breaker serves stale cache |
| Saga compensation | Payment fails after order+inventory committed - triggers 4-way parallel rollback |
| LLM rate limits | OpenAI 429 with token budget details, fallback to text search |
| Hallucinated tool calls | Agent requests non-existent tool, triggers error handling |
| Token budget exceeded | Agent exceeds iteration token limit, graceful degradation |
| Content filter blocks | Safety classifier blocks content, alternate flow triggered |
| Tunable error rate | -errors 0 (none) to -errors 10 (chaos) with realistic .NET stack traces |
Realistic Details
The generated traces simulate a .NET-based e-commerce platform with AI capabilities. Stack traces and library names reflect the .NET ecosystem by design.
- Stack traces: Npgsql, StackExchange.Redis, Stripe SDK, Elasticsearch.Net, System.Net.Http, OpenAI SDK, Qdrant client
- Database operations: PostgreSQL INSERT/SELECT/UPDATE with semantic conventions
- Cache operations: Redis GET/SET/HSET/MSET/DEL with TTL and key attributes
- Messaging: RabbitMQ and Kafka with producer/consumer span kinds and queue delays
- External APIs: Stripe charges, SendGrid email, UPS shipping, OpenAI chat/embeddings
- LLM operations: Chat completions, embeddings, agent tool calls with token tracking
- Vector search: Qdrant similarity search with cosine distance, dimension validation
- Agent orchestration: Plan/act/reflect loops, tool dispatch, session management
- Content moderation: Safety classifiers, PII detection, guardrail enforcement
- ML inference: Fraud detection model scoring with feature counts
- Feature flags: Config service checks that gate behavior
Usage
snowglobe [flags]
Flags:
-endpoint string OTLP gRPC endpoint host:port (default "localhost:4317")
-headers string OTLP headers as