Spring Cloud Sleuth 3.0 Migration Guide

** This document is a work in progress **

As OpenTelemetry continues to evolve, we have decided to move OpenTelemetry support from Spring Cloud Sleuth and to an incubator project . To continue using OpenTelemetry with Spring Cloud Sleuth you will need to add the Spring repositories, the spring-cloud-sleuth-otel-dependencies BOM and spring-cloud-sleuth-otel-autoconfigure dependency.

We have released a 1.0.0-M1 of Spring Cloud Sleuth OTel that is compatible with the 2020.0.0-RC1 release train.

.Maven

 <properties>
   <spring-cloud-sleuth-otel.version>1.0.0-M1</spring-cloud-sleuth-otel.version>
 </properties>

 <dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-dependencies</artifactId>
				<!-- Provide the latest stable Spring Cloud release train version (e.g. 2020.0.0) -->
				<version>${release.train.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-sleuth-otel-dependencies</artifactId>
				<!-- Provide the version of the Spring Cloud Sleuth OpenTelemetry project -->
				<version>${spring-cloud-sleuth-otel.version}</version>
				<scope>import</scope>
				<type>pom</type>
			</dependency>
		</dependencies>
	</dependencyManagement>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-sleuth</artifactId>
        <exclusions>
            <exclusion>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-sleuth-brave</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-sleuth-otel-autoconfigure</artifactId>
    </dependency>

		<!-- You 'll need those to add OTel support -->
		<repositories>
			<repository>
				<id>spring-snapshots</id>
				<url>https://repo.spring.io/snapshot</url>
				<snapshots><enabled>true</enabled></snapshots>
			</repository>
			<repository>
				<id>spring-milestones</id>
				<url>https://repo.spring.io/milestone</url>
			</repository>
		</repositories>
		<pluginRepositories>
			<pluginRepository>
				<id>spring-snapshots</id>
				<url>https://repo.spring.io/snapshot</url>
			</pluginRepository>
			<pluginRepository>
				<id>spring-milestones</id>
				<url>https://repo.spring.io/milestone</url>
			</pluginRepository>
		</pluginRepositories>

.Gradle

ext {
  springCloudSleuthOtelVersion = "1.0.0-M1"
}

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
        mavenBom "org.springframework.cloud:spring-cloud-sleuth-otel-dependencies:${springCloudSleuthOtelVersion}"
    }
}

dependencies {
    implementation("org.springframework.cloud:spring-cloud-starter-sleuth") {
        exclude group: 'org.springframework.cloud', module: 'spring-cloud-sleuth-brave'
    }
    implementation "org.springframework.cloud:spring-cloud-sleuth-otel-autoconfigure"
}

repositories {
    mavenCentral()
    maven {
            url "https://repo.spring.io/snapshot"
    }
    maven {
            url "https://repo.spring.io/milestone"
    }
    maven {
            url "https://repo.spring.io/release"
    }
}

• New modules: spring-cloud-sleuth-autoconfigure, spring-cloud-sleuth-api, spring-cloud-sleuth-instrumentation spring-cloud-sleuth-core removed and changed to spring-cloud-sleuth-instrumentation & spring-cloud-sleuth-api
• Removed spring-cloud-starter-sleuth-otel To add OpenTelemetry support you need to add spring-cloud-starter-sleuth (adds Brave by default), exclude Brave and add spring-cloud-sleuth-otel dependency
• Except for the tests, spring-cloud-sleuth-autoconfigure is the only module that can have access to @Configuration, @ConfiguraationProperties classes. Tests have been added to ensure such separation.

• org.springframework.cloud.sleuth.brave.autoconfig -> org.springframework.cloud.sleuth.autoconfig.brave
• org.springframework.cloud.sleuth.otel.autoconfig -> org.springframework.cloud.sleuth.autoconfig.otel
• org.springframework.cloud.sleuth -> org.springframework.cloud.sleuth
• Instrumentation: org.springframework.cloud.sleuth.annotation -> org.springframework.cloud.sleuth.instrument.annotation

• Any class registered as a bean is now public

• RateLimitingSampler constructor changed

With Spring Cloud Gateway we're by default setting the manual reactor tracing instrumentation. That means that in order for your manual retrieval of spans or logging to work properly you should use the WebFluxSleuthOperators. Example below:

@Bean
		RouterFunction<ServerResponse> handlers(org.springframework.cloud.sleuth.Tracer tracing,
				CurrentTraceContext currentTraceContext, ManualRequestSender requestSender) {
			return route(GET("/noFlatMap"), request -> {
				ServerWebExchange exchange = request.exchange();
				WebFluxSleuthOperators.withSpanInScope(tracing, currentTraceContext, exchange,
						() -> LOGGER.info("noFlatMap"));
				Flux<Integer> one = requestSender.getAll().map(String::length);
				return ServerResponse.ok().body(one, Integer.class);
			}).andRoute(GET("/withFlatMap"), request -> {
				ServerWebExchange exchange = request.exchange();
				WebFluxSleuthOperators.withSpanInScope(tracing, currentTraceContext, exchange,
						() -> LOGGER.info("withFlatMap"));
				Flux<Integer> one = requestSender.getAll().map(String::length);
				Flux<Integer> response = one
						.flatMap(size -> requestSender.getAll().doOnEach(sig -> WebFluxSleuthOperators
								.withSpanInScope(sig.getContext(), () -> LOGGER.info(sig.getContext().toString()))))
						.map(string -> {
							WebFluxSleuthOperators.withSpanInScope(tracing, currentTraceContext, exchange,
									() -> LOGGER.info("WHATEVER YEAH"));
							return string.length();
						});
				return ServerResponse.ok().body(response, Integer.class);
			}).andRoute(GET("/foo"), request -> {
				ServerWebExchange exchange = request.exchange();
				WebFluxSleuthOperators.withSpanInScope(tracing, currentTraceContext, exchange, () -> {
					LOGGER.info("foo");
					this.spanInFoo = tracing.currentSpan();
				});
				return ServerResponse.ok().body(Flux.just(1), Integer.class);
			});
		}

In order to use the legacy mode where each operator is decorated please use spring.sleuth.reactor.instrumentation-type=decorate-on-each. Remember that this feature is deprecated and we're considering to remove it in the future.

Most new features in Sleuth 3.0 are around Baggage, described in Brave 5.11 Release Notes.

For example, there's now a nicer way to manipulate baggage values, or add them as tags:

// assuming country-code is setup in sleuth.baggage.remote-keys
BaggageField COUNTRY_CODE = BaggageField.create("country-code");

COUNTRY_CODE.updateValue(span.context(), "FO");
String countryCode = COUNTRY_CODE.get(span.context());
Tags.BAGGAGE_FIELD.tag(COUNTRY_CODE, span);

You can also change baggage configuration with Java config.

For example, you can now configure fields to flush immediately to MDC. This is helpful for functions or message processors who learn a value late.

BaggageField BUSINESS_PROCESS = BaggageField.create("bp");

@SendTo(SourceChannels.OUTPUT)
public void timerMessageSource() {
  BUSINESS_PROCESS.updateValue("accounting");
  // You want the expression %{bp} to show "accounting" in businessCode()
  businessCode();
}

Using Java config, you can setup very smart handling without affecting other properties.

@Configuration
class BusinessProcessBaggageConfiguration {
  BaggageField BUSINESS_PROCESS = BaggageField.create("bp");

  /** {@link #BUSINESS_PROCESS} will not be sent as a header */
  @Bean
  BaggagePropagationCustomizer propagateBusinessProcessLocally() {
    return fb -> fb.add(SingleBaggageField.local(BUSINESS_PROCESS));
  }

  /** {@link BaggageField#updateValue(TraceContext, String)} now flushes to MDC */
  @Bean
  CorrelationScopeCustomizer flushBusinessProcessToMDCOnUpdate() {
    return b -> b.add(
        SingleCorrelationField.newBuilder(BUSINESS_PROCESS).flushOnUpdate().build()
    );
  }

  /** {@link #BUSINESS_PROCESS} is added as a tag only in the first span. */
  @Bean
  FinishedSpanHandler tagBusinessProcessOncePerProcess() {
    return new FinishedSpanHandler() {
      @Override public boolean handle(TraceContext context, MutableSpan span) {
        if (context.isLocalRoot()) {
          Tags.BAGGAGE_FIELD.tag(BUSINESS_PROCESS, context, span);
        }
        return true;
      }
    };
  }
}

Please use Spring Cloud LoadBalancer.

Please use Spring Cloud CircuitBreaker.

The notion of SpanReporters has changed to SpanHandlers. ArrayListSpanReporter has been used in tests and its replacement is now TestSpanHandler. Example of usage:

public class SpringCloudSleuthDocTests {

	TestSpanHandler spans = new TestSpanHandler();

	StrictCurrentTraceContext currentTraceContext = StrictCurrentTraceContext.create();

	Tracing tracing = Tracing.newBuilder().currentTraceContext(this.currentTraceContext)
			.sampler(Sampler.ALWAYS_SAMPLE).addSpanHandler(this.spans).build();

	Tracer tracer = this.tracing.tracer();

	@BeforeEach
	public void setup() {
		this.spans.clear();
	}

	@AfterEach
	public void close() {
		this.tracing.close();
		this.currentTraceContext.close();
	}

	@Test
	public void should_set_runnable_name_to_annotated_value()
			throws ExecutionException, InterruptedException {
		ExecutorService executorService = Executors.newSingleThreadExecutor();
		SpanNamer spanNamer = new DefaultSpanNamer();

		// tag::span_name_annotated_runnable_execution[]
		Runnable runnable = new TraceRunnable(this.tracing, spanNamer,
				new TaxCountingRunnable());
		Future<?> future = executorService.submit(runnable);
		// ... some additional logic ...
		future.get();
		// end::span_name_annotated_runnable_execution[]

		then(this.spans).hasSize(1);
		then(this.spans.get(0).name()).isEqualTo("calculateTax");
	}
}

If you were overriding ExtraFieldPropagation.Factory to control the underlying trace format, do the same with BaggagePropagation.Factory now.

@Bean
BaggagePropagation.FactoryBuilder baggagePropagationFactoryBuilder() {
	return BaggagePropagation.newFactoryBuilder(StackdriverTracePropagation.FACTORY);
}

BaggageField should be used to manipulate current baggage.

Ex.

// assuming country-code is setup in sleuth.baggage.remote-keys
BaggageField COUNTRY_CODE = BaggageField.create("country-code");

COUNTRY_CODE.updateValue(span.context(), "FO");
String countryCode = COUNTRY_CODE.get(span.context());
Tags.BAGGAGE_FIELD.tag(COUNTRY_CODE, span);

https://github.com/openzipkin/brave/tree/master/brave#using-baggagefield

Properties controlling SLF4J are now under "spring.sleuth.baggage", but with the same meaning.

Those coming from 2.x should migrate with the following:

• spring.sleuth.log.slf4j.whitelisted-mdc-keys -> spring.sleuth.baggage.correlation-fields
• spring.sleuth.log.slf4j.enabled -> spring.sleuth.baggage.correlation-enabled

The former property names are still read, but will be removed at some point.

For performance reasons, we no longer set the following fields by default:

• parentId
• spanExportable

Sleuth 3.0 implicitly sets below, if you haven't overridden it with "logging.pattern.level"

logging.pattern.level=%5p [${spring.zipkin.service.name:${spring.application.name:}},%X{traceId:-},%X{spanId:-}]

WARNING: This no longer includes the spanExportable field. If you are using Grok/Logstash, and didn't set "logging.pattern.level" in your 2.2.x application, a change to the implicit pattern can break your parsing.

If this applies to you, before you upgrade, set "logging.pattern.level" in your 2.2.x app explicitly to either the above pattern, or the below including spanExportable if your Grok configuration uses it:

logging.pattern.level=%5p [${spring.zipkin.service.name:${spring.application.name:}},%X{traceId:-},%X{spanId:-},%X{spanExportable:-}]

For those using spanExportable, when upgrading to 3.0, add the following to your Java config to avoid the field always showing up as '-'.

@Bean CorrelationScopeCustomizer addSampled() {
  return b -> b.add(SingleCorrelationField.create(BaggageFields.SAMPLED));
}

The Sleuth 1.x MDC field names are no longer set in correlation config. Their replacements exist since Sleuth 2.0, noted here for completion:

• X-B3-TraceId -> traceId
• X-B3-ParentSpanId -> parentId
• X-B3-SpanId -> spanId
• X-Span-Export -> sampled

If you are still using the deprecated X-B3- names in your logstash configuration, update to the current ones before you upgrade to 3.0.

If for some reason, you need to add the deprecated ones even in 3.0, you can add an additional scope decorator like so, accepting the duplicate overhead of doing so (applicable for Brave):

@Bean ScopeDecorator legacyIds() {
  return MDCScopeDecorator.newBuilder()
                         .clear()
                         .add(SingleCorrelationField.newBuilder(BaggageFields.TRACE_ID)
                                                    .name("X-B3-TraceId").build())
                         .add(SingleCorrelationField.newBuilder(BaggageFields.PARENT_ID)
                                                    .name("X-B3-ParentSpanId").build())
                         .add(SingleCorrelationField.newBuilder(BaggageFields.SPAN_ID)
                                                    .name("X-B3-SpanId").build())
                         .add(SingleCorrelationField.newBuilder(BaggageFields.SAMPLED)
                                                    .name("X-Span-Export").build())
                         .build();
}

Before, all (whitelisted) correlation fields were treated as dirty. Regardless of if values were the same when entering a scope, each value was reverted. This caused extreme overhead in defense of an edge case where someone uses MDC.put() on the same span, and wants it cleaned up for them. We no longer clean up out-of-band changes by default.

If you want to have sleuth clean up ad-hoc use of MDC.put(), you can mark all fields dirty like this, or do similar for the field you want to control:

@Bean
CorrelationScopeCustomizer makeCorrelationFieldsDirty() {
	return b -> {
		Set<CorrelationScopeConfig> configs = b.configs();
		b.clear();

		for (CorrelationScopeConfig config : configs) {
			if (config instanceof SingleCorrelationField) {
				SingleCorrelationField field = (SingleCorrelationField) config;
				if (!field.readOnly()) {
					config = field.toBuilder().dirty().build();
				}
			}
			b.add(config);
		}
	};
}

Properties controlling Baggage are now under "spring.sleuth.baggage", but with the same meaning.

Those coming from 2.x should migrate with the following:

• spring.sleuth.baggage-keys -> custom BaggagePropagationCustomizer described later
• spring.sleuth.local-keys -> spring.sleuth.baggage.local-fields
• spring.sleuth.propagation-keys -> spring.sleuth.baggage.remote-fields
• spring.sleuth.propagation.tag.whitelisted-keys -> spring.sleuth.baggage.tag-fields

The former property names are still read, but will be removed at some point.

The "spring.sleuth.baggage-keys" property assigned two headers for each field. This causes unnecessary overhead and can be accomplished in another way now:

@Configuration
static class CustomBaggageConfiguration {
  @Bean BaggagePropagationCustomizer countryCodeBaggageConfig() {
    return fb -> fb.add(SingleBaggageField.newBuilder(BaggageField.create("country-code"))
                           .addKeyName("baggage-country-code")
                           .addKeyName("baggage_country-code")
                           .build());
  }
}

The preferable approach is to migrate to using spring.sleuth.baggage.remote-keys. The spring.sleuth.baggage-keys would prefix the headers with baggage_ and baggage- so unless all of your applications migrate, to remain backward compatible you would have to add e.g. for spring.sleuth.baggage-keys=foo an entry spring.sleuth.baggage.remote-keys=foo,baggage-foo,baggage_foo and eventually migrate to spring.sleuth.baggage.remote-keys=foo.

The following custom spring-messaging headers added in Sleuth 1.0 are no longer sent, and a log warning is issued once if they are by outside code.

• spanId
• spanSampled
• spanParentSpanId
• spanTraceId
• spanFlags

Sending the above headers actually increases the headers by up to 10 because they are duplicated in the "native" part of messages. This overhead is extreme especially if messages never leave the process.

The solution is to only send b3 single format, which has been in sleuth since 2.0 and is compatible with JMS. The B3 single format is always parsed and takes precedence, even if multiple headers are sent, so this is a safe change.

Note: Unlike RPC, messaging spans never join with their parent. Better performance is achieved by not propagating the producer's parentId downstream.

spring.sleuth.reactor.decorate-on-each and spring.sleuth.reactor.decorate-on-last are deprecated and you should now use spring.sleuth.reactor.instrumentation-type that can take DECORATE_ON_EACH, DECORATE_ON_LAST or MANUAL options. The latter is a new feature.

In order to improve performance, fix automatic reactor instrumentation issues, you can use the new spring.sleuth.reactor.instrumentation-type=MANUAL mode. That will however disable the default option of wrapping every reactor operator in a Sleuth representation. That also means that you have to take care of passing the tracing context yourself. You can use MessagingSleuthOperators or WebFluxSleuthOperators to work with the propagated tracing context for messaging / webflux based apps.