Trace Customization

Orchestrion offers several ways to control the traces produced by instrumented applications.

Prevent instrumentation of a section of code

By default, orchestrion injects instrumentation everywhere possible. This ensures users get the maximum possible coverage from their applications, as it removes the possibility of someone forgetting to instrument a particular call.

There are however cases where you may want specific sections of your application to not be instrumented, either because they result in excessively verbose traces, or because those trace spans would be duplicated.

The //orchestrion:ignore directive can be added anywhere in your application’s code, and will disable all orchestrion instrumentation in the annotated syntax tree.

For example:

package demo

import "net/http"

//orchestrion:ignore I don't want any of this to be instrumented, ever.
func noInstrumentationThere() {
  // Orchestrion will never add or modify any code in this function
  // ... etc ...
}

func definitelyInstrumented() {
  // Orchestrion may add or modify code in this function
  // ... etc ...

  //orchestrion:ignore This particular database connection will NOT be instrumented
  db, err := db.Open("driver-name", "database=example")

  // Orchestrion may add or modify code further down in this function
  // ... etc ...
}

Creating custom trace spans

Any function annotated with the //dd:span directive will result in a trace span being created around the function’s execution. The directive can optionally provide custom span tags as key:value pairs (all parsed as literal strings):

//dd:span tag-name:foo other-tag:bar
func tracedFunction() {
  // This function will be represented as a span named "tracedFunction"
}

Result Capture

Functions annotated with //dd:span which return an error value will automatically annotate spans with the returned error information if that is non-nil.

package demo

import "errors"

//dd:span
func failableFunction() (any, error) {
  // This span will have error information attached automatically.
  return nil, errors.ErrUnsupported
}

Operation Name

The name of the operation (span name) is determined using the following precedence list (first non-empty is selected):

• The span.name tag specified as a directive argument

  //dd:span span.name:operationName
  func tracedFunction() {
    // This function will be represented as a span named "operationName"
  }
• The name of the function (closures do not have a name)

  //dd:span tag-name:for other-tag:bar
  func tracedFunction() {
    // This function will be represented as a span named "tracedFunction"
  }
• The value of the very first tag from the directive arguments list

  //dd:span tag-name:spanName other-tag:bar
  tracedFunction := func() {
    // This function will be represented as a span named "spanName"
  }

Trace Context Propagation

If the annotated function accepts a context.Context argument, that context will be used for trace propagation. Otherwise, if the function accepts a *http.Request argument, the request’s context will be used for trace propagation.

Functions that accept neither solely rely on goroutine local storage for trace propagation. This means that traces may be split on goroutine boundaries unless a context.Context or *http.Request value carrying trace context is passed across.

Trace context carrying context.Context values are those that:

• have been received by a //dd:span annotated function, as instrumentation will create a new trace root span if if did not already carry trace context
• are returned by:
  • Tracer v1: tracer.StartSpanFromContext / Tracer v2: tracer.StartSpanFromContext
  • Tracer v1: tracer.ContextWithSpan / Tracer v2: tracer.ContextWithSpan

package demo

//dd:span
func caller(ctx context.Context) {
  wait := make(chan struct{}, 1)
  defer close(wait)

  // Weaving the span context into the child goroutine
  go callee(ctx, wait)
  <-wait
}

//dd:span
func callee(ctx context.Context, done chan<- struct{}) {
  done <- struct{}{}
}

Manual Instrumentation

The gopkg.in/DataDog/dd-trace-go.v1 or github.com/DataDog/dd-trace-go/v2 library can be used to manually instrument sections of your code even when building with orchestrion.

You can use APIs such as tracer.StartSpanFromContext (in v2: tracer.StartSpanFromContext) to create spans in any section of your code. This can be useful to delimit a specific section of your code with a span without having to refactor it in a separate function (which would allow the use of the //dd:span directive), or when you need to customize the span more than the //dd:span directive allows.

OpenTelemetry API

Set the environment variable DD_TRACE_OTEL_ENABLED to true to turn on this support. When it is not set, orchestrion does not change trace.SpanFromContext or the OpenTelemetry global tracer.

Code that uses the OpenTelemetry API works in an application built with orchestrion. The code does not need to be specific to Datadog.

• trace.SpanFromContext returns the span that orchestrion instrumentation started, for example the net/http server span. Attributes, events, errors, and status that you set on this span go to the Datadog span. If the context already contains a span from another OpenTelemetry implementation, or a remote span context, that span is returned instead.
• If the application does not call otel.SetTracerProvider, the spans that otel.Tracer(...).Start creates are Datadog spans. They are children of the Datadog span in the context.Context that you give to Start.
• If the application calls otel.SetTracerProvider (for example with the OpenTelemetry SDK), the OpenTelemetry API uses that TracerProvider. orchestrion does not replace it. The parent of a span comes from the context.Context that you give to Start.

In both cases, pass the context that contains the span, for example r.Context(). Without SetTracerProvider, a span started with context.Background() can take the active Datadog span of the goroutine as parent, as Datadog spans do. With the OpenTelemetry SDK, it starts a new trace.

package demo

import (
  "net/http"

  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/trace"
)

func handler(w http.ResponseWriter, r *http.Request) {
  ctx := r.Context()

  // The net/http server span that orchestrion started.
  span := trace.SpanFromContext(ctx)
  span.SetAttributes(attribute.String("user.id", "42"))

  // A child span of the net/http server span.
  ctx, child := otel.Tracer("demo").Start(ctx, "load-cart")
  defer child.End()

  // ... etc ...
}

Earlier dd-trace-go versions

With github.com/DataDog/dd-trace-go/v2 versions before v2.13.0, trace.SpanFromContext returns a no-op span for the spans that orchestrion started. Spans that your code starts with the OpenTelemetry API work as usual. To make otel.Tracer(...).Start create Datadog child spans, import opentelemetry as ddotel (import ddotel "github.com/DataDog/dd-trace-go/v2/ddtrace/opentelemetry") and call otel.SetTracerProvider(ddotel.NewTracerProvider()). With this workaround, trace.SpanFromContext still returns a no-op span for the spans that orchestrion started. For more information, refer to issue 936.