Engine:Observability

Engine:Observability configures the observability capability — where telemetry goes, how much of it ships, what’s attached to every signal. Always configure observability in production, even if you skip other capabilities.

Full schema

1
{
2
  "Engine": {
3
    "Observability": {
4
      "Provider": "OpenTelemetry",                  // "OpenTelemetry" | "AzureMonitor" | "Serilog" | "AWS" | "GCP" | "AlibabaCloud" | "HuaweiCloud" | …
5
      "Endpoint": "http://localhost:4317",          // OTLP exporter endpoint
6
      "Protocol": "Grpc",                           // "Grpc" | "HttpProtobuf"
7
      "Headers":  "Authorization=Basic <BASE64>",   // OTLP headers (auth, tenancy)
8
      "Sampling": {
9
        "Strategy": "ParentBased",                  // "AlwaysOn" | "AlwaysOff" | "TraceIdRatio" | "ParentBased"
10
        "Ratio":    1.0                             // 0.0 - 1.0 (used by TraceIdRatio / ParentBased fallback)
11
      },
12
      "Resource": {
13
        "ServiceName":      null,                   // defaults to Engine:Id
14
        "ServiceVersion":   null,                   // defaults to assembly version
15
        "DeploymentEnvironment": null,              // defaults to ASPNETCORE_ENVIRONMENT
16
        "Attributes": {
17
          "team":   "platform",
18
          "region": "eu-west-1"
19
        }
20
      },
21
      "Logs":    { "Enabled": true,  "IncludeScopes": true,  "IncludeFormattedMessage": true },
22
      "Metrics": { "Enabled": true,  "ExportIntervalMs": 30000 },
23
      "Traces":  { "Enabled": true,  "RecordExceptions": true },
24
      "Sources": {
25
        "Otel":            true,                    // capture System.Diagnostics ActivitySources
26
        "AspNetCore":      true,                    // ASP.NET Core diagnostic source
27
        "HttpClient":      true,                    // outbound HTTP
28
        "EntityFramework": true,                    // EF Core spans
29
        "Wolverine":       true                     // eventing spans
30
      },
31
      "DependencyHealth": {
32
        "Enabled":        true,
33
        "IntervalSeconds": 15,
34
        "TimeoutSeconds":  5
35
      },
36
      "Provider:AzureMonitor": {                    // when Provider="AzureMonitor"
37
        "ConnectionString": "InstrumentationKey=…"
38
      },
39
      "Provider:GCP": {                             // when Provider="GCP"
40
        "ProjectId": "acme-prod"
41
      }
42
    }
43
  }
44
}

Each option in detail

`Provider`

Type	Default	Allowed values
enum string	`"OpenTelemetry"`	`"OpenTelemetry"`, `"AzureMonitor"`, `"AlibabaCloud"`, `"AWS"`, `"GCP"`, `"HuaweiCloud"`, `"DigitalOcean"`, `"GrafanaCloud"`, `"Kubernetes"`, `"NewRelic"`, `"OpenShift"`, `"OracleCloud"`, `"Tanzu"`, `"Serilog"`

The observability provider. Each maps to a Cephalon.Observability.* package:

Value	Package	Use when
`"OpenTelemetry"`	`Cephalon.Observability.OpenTelemetry`	Default. OTLP exporter to any collector.
`"AzureMonitor"`	`Cephalon.Observability.AzureMonitor`	Native Application Insights / Azure Monitor.
`"AWS"`	`Cephalon.Observability.Aws`	X-Ray-compatible OTLP.
`"GCP"`	`Cephalon.Observability.Gcp`	Google Cloud managed traces/metrics.
`"GrafanaCloud"`	`Cephalon.Observability.GrafanaCloud`	Grafana Cloud OTLP gateway.
`"NewRelic"`	`Cephalon.Observability.NewRelic`	New Relic OTLP + api-key.
`"Serilog"`	`Cephalon.Observability.Serilog`	Serilog-based logging (no OTLP).

Cloud-provider adapters (AlibabaCloud, HuaweiCloud, OracleCloud, Tanzu, DigitalOcean, Kubernetes, OpenShift) wire to their respective managed APM services.

`Endpoint`

Type	Default
URL string	`"http://localhost:4317"`

The OTLP collector endpoint. Format depends on Protocol:

Protocol	Default port	Path
`"Grpc"`	`4317`	/ (no path)
`"HttpProtobuf"`	`4318`	`/v1/traces`, `/v1/metrics`, `/v1/logs`

Examples:

1
{ "Endpoint": "http://otel-collector:4317" }                       // local cluster collector
2
{ "Endpoint": "https://otlp-gateway-prod-eu-west-2.grafana.net/otlp" }  // Grafana Cloud
3
{ "Endpoint": "https://api.honeycomb.io" }                          // Honeycomb

`Protocol`

Type	Default	Allowed values
enum string	`"Grpc"`	`"Grpc"`, `"HttpProtobuf"`

Wire format for OTLP.

Value	When to use
`"Grpc"`	Default. Most efficient. Required by most managed collectors.
`"HttpProtobuf"`	When the network blocks gRPC (some corporate firewalls). Slightly higher overhead.

`Headers`

Type	Default
comma-separated string	`null`

Headers sent with every OTLP export. Used for authentication and tenancy headers.

1
{ "Headers": "Authorization=Basic <BASE64_TOKEN>" }
2
{ "Headers": "Authorization=Bearer eyJ…,X-Org=acme" }
3
{ "Headers": "api-key=<NEW_RELIC_KEY>" }                          // New Relic
4
{ "Headers": "x-honeycomb-team=<KEY>,x-honeycomb-dataset=prod" }   // Honeycomb

Security: Put secrets in env vars / Key Vault, not appsettings.json.

`Sampling.Strategy`

Type	Default	Allowed values
enum string	`"ParentBased"`	`"AlwaysOn"`, `"AlwaysOff"`, `"TraceIdRatio"`, `"ParentBased"`

How traces are sampled.

Value	Behaviour
`"AlwaysOn"`	Sample every trace. Highest cost; useful in dev.
`"AlwaysOff"`	Sample nothing. Disable tracing entirely.
`"TraceIdRatio"`	Sample a fraction of traces based on a hash of trace ID. Stateless, consistent across services.
`"ParentBased"`	Default. Honour upstream sampling decision; fall back to `Ratio` for root spans. Best for distributed tracing.

`Sampling.Ratio`

Type	Default
float (0.0 – 1.0)	`1.0`

For TraceIdRatio and ParentBased, the fraction of root spans to sample.

Value	Effect
`1.0`	Sample everything (default — fine for low-volume apps)
`0.1`	Sample 10% — typical for high-volume production
`0.01`	Sample 1% — for very high-volume APIs
`0.0`	Sample nothing (same as `AlwaysOff`)

Guideline: Start at 1.0. Drop to 0.1 when trace volume becomes expensive. Keep 1.0 for critical-path services so issues are always captured.

`Resource`

`Resource.ServiceName`

Type	Default
string	`Engine:Id`

OTEL service.name resource attribute. Defaults to Engine:Id if not set.

`Resource.ServiceVersion`

Type	Default
string	assembly informational version

OTEL service.version. Defaults to the host assembly’s [AssemblyInformationalVersion]. Set explicitly to override (e.g. include a commit SHA: "1.2.0+a1b2c3d").

`Resource.DeploymentEnvironment`

Type	Default
string	`ASPNETCORE_ENVIRONMENT` value

OTEL deployment.environment ("production", "staging", etc.).

`Resource.Attributes`

Type	Default
object (string → string)	`{}`

Custom resource attributes attached to every signal. Use for team/region/SLO labels.

1
{
2
  "Resource": {
3
    "Attributes": {
4
      "team":     "platform",
5
      "region":   "eu-west-1",
6
      "tier":     "production",
7
      "sla":      "99.9"
8
    }
9
  }
10
}

`Logs`

`Logs.Enabled` / `Metrics.Enabled` / `Traces.Enabled`

Type	Default
boolean	`true` (all three)

Disable individual signal types. Useful for cost control (e.g. disable logs to OTLP if you already ship logs via a separate agent).

`Logs.IncludeScopes`

Type	Default
boolean	`true`

Include ILogger.BeginScope(...) data in exported log records. Adds context but increases payload size.

`Logs.IncludeFormattedMessage`

Type	Default
boolean	`true`

Include the formatted log text ("User 42 logged in") alongside the structured fields. Disable to save bytes when the backend reconstructs from fields.

`Metrics.ExportIntervalMs`

Type	Default
integer (ms)	`30000` (30 sec)

How often metrics are exported. Trade-off:

Lower (5–10s) — faster anomaly detection, higher backend cost.
Higher (60s+) — cheaper, slower to react.

`Traces.RecordExceptions`

Type	Default
boolean	`true`

Attach exception stack traces to spans on errors. Disable in extreme-PII scenarios.

`Sources`

Granular control of which diagnostic sources are captured.

Source	Default	What it includes
`Otel`	`true`	All explicit `ActivitySource` spans. Disable to drop your own custom spans.
`AspNetCore`	`true`	`Microsoft.AspNetCore` activity source — request spans, middleware.
`HttpClient`	`true`	Outbound `HttpClient` calls.
`EntityFramework`	`true`	EF Core command-execution spans.
`Wolverine`	`true`	Eventing publish/handle spans.

Examples:

1
// Disable EF spans (cost reduction; you can reproduce via DB logs)
2
{ "Sources": { "EntityFramework": false } }
3

4
// Disable HttpClient (you already trace outbound elsewhere)
5
{ "Sources": { "HttpClient": false } }

`DependencyHealth`

Configure per-backend health probes (Postgres, Redis, RabbitMQ, etc.). Wired via Cephalon.Observability.*Dependencies packages.

`DependencyHealth.Enabled`

Type	Default
boolean	`true`

Globally enable / disable dependency probes. When false, /health only reports lifecycle health.

`DependencyHealth.IntervalSeconds`

Type	Default
integer	`15`

How often each probe runs. Lower = fresher data, more probe traffic.

`DependencyHealth.TimeoutSeconds`

Type	Default
integer	`5`

Per-probe timeout. After this, the probe reports Unhealthy with "timeout" reason.

Common scenarios

Scenario 1: local development — OTLP collector + Grafana

1
{
2
  "Engine": {
3
    "Observability": {
4
      "Provider": "OpenTelemetry",
5
      "Endpoint": "http://localhost:4317",
6
      "Sampling": { "Strategy": "AlwaysOn", "Ratio": 1.0 },
7
      "Resource": {
8
        "Attributes": { "team": "platform", "tier": "dev" }
9
      }
10
    }
11
  }
12
}

Pair with the docker-compose otel-collector-config.yaml generated by cephalon new.

Scenario 2: production — Grafana Cloud with sampling

1
{
2
  "Engine": {
3
    "Observability": {
4
      "Provider": "OpenTelemetry",
5
      "Endpoint": "https://otlp-gateway-prod-eu-west-2.grafana.net/otlp",
6
      "Headers":  "Authorization=Basic <BASE64>",     // from env var in practice
7
      "Sampling": { "Strategy": "ParentBased", "Ratio": 0.1 },
8
      "Resource": {
9
        "Attributes": { "team": "platform", "region": "eu-west-1", "tier": "production" }
10
      },
11
      "Metrics": { "ExportIntervalMs": 60000 }
12
    }
13
  }
14
}

Scenario 3: Azure-native — Azure Monitor

1
{
2
  "Engine": {
3
    "Observability": {
4
      "Provider": "AzureMonitor",
5
      "Provider:AzureMonitor": {
6
        "ConnectionString": "InstrumentationKey=…;IngestionEndpoint=…"
7
      },
8
      "Sampling": { "Strategy": "ParentBased", "Ratio": 0.1 }
9
    }
10
  }
11
}

Scenario 4: New Relic OTLP

1
{
2
  "Engine": {
3
    "Observability": {
4
      "Provider": "NewRelic",
5
      "Endpoint": "https://otlp.nr-data.net",
6
      "Headers":  "api-key=<NR_LICENSE_KEY>",
7
      "Sampling": { "Strategy": "ParentBased", "Ratio": 1.0 }
8
    }
9
  }
10
}

Scenario 5: cost-conscious production (logs only, no traces/metrics)

1
{
2
  "Engine": {
3
    "Observability": {
4
      "Provider": "OpenTelemetry",
5
      "Endpoint": "https://otlp.example/",
6
      "Logs":     { "Enabled": true },
7
      "Metrics":  { "Enabled": false },
8
      "Traces":   { "Enabled": false }
9
    }
10
  }
11
}

Scenario 6: Serilog (no OTLP)

1
{
2
  "Engine": {
3
    "Observability": {
4
      "Provider": "Serilog"
5
    }
6
  }
7
}

Configure Serilog sinks via standard Serilog appsettings patterns.

Environment-variable equivalents

Engine__Observability__Provider=OpenTelemetry
Engine__Observability__Endpoint=https://otlp.example/
Engine__Observability__Headers=Authorization=Bearer eyJ…
Engine__Observability__Sampling__Strategy=ParentBased
Engine__Observability__Sampling__Ratio=0.1
Engine__Observability__Resource__Attributes__team=platform
Engine__Observability__Sources__EntityFramework=false
Engine__Observability__DependencyHealth__IntervalSeconds=15

Limits & gotchas

Sampling.Ratio=0.1 doesn’t mean exactly 10% of traces. It’s probabilistic — over time it converges, but small samples vary.
ParentBased requires upstream services to propagate trace context. Without W3C traceparent headers, every span looks like a root span.
Headers is a single string, not an array. Multiple headers are comma-separated key=value pairs.
Provider:AzureMonitor / Provider:GCP sections are flat keys with :. In env vars use Engine__Observability__Provider_AzureMonitor__ConnectionString (underscore stays).
Reducing Metrics.ExportIntervalMs below 5000ms has diminishing returns and can overwhelm the backend.
Logs.IncludeScopes=true can leak sensitive data if scopes contain PII. Audit your scope usage before enabling in production.
Sources.EntityFramework captures the SQL command text by default. This includes parameter values — sensitive in healthcare / finance contexts. Disable if compliance requires.
DependencyHealth probes run from the host’s perspective. They don’t validate that other services can reach the dependency.

Engine:Observability

Full schema

Each option in detail

Provider

Endpoint

Protocol

Headers

Sampling.Strategy

Sampling.Ratio

Resource

Resource.ServiceName

Resource.ServiceVersion

Resource.DeploymentEnvironment

Resource.Attributes

Logs

Logs.Enabled / Metrics.Enabled / Traces.Enabled

Logs.IncludeScopes

Logs.IncludeFormattedMessage

Metrics.ExportIntervalMs

Traces.RecordExceptions

Sources

DependencyHealth

DependencyHealth.Enabled

DependencyHealth.IntervalSeconds

DependencyHealth.TimeoutSeconds