Module runtime 

Runtime system.

This module contains the core components of the runtime system, including supervisors and processes. It’s directly inspired by Erlang/OTP.

To quote the Erlang/OTP documentation:

Workers are processes that perform computations and other actual work. Supervisors are processes that monitor workers. A supervisor can restart a worker if something goes wrong. The supervision tree is a hierarchical arrangement of code into supervisors and workers, which makes it possible to design and program fault-tolerant software.

Processes

An asynchronous system is composed of independent units of computation running concurrently, such as a set of tasks executing on a thread pool. We refer to these as processes. In other systems, these might be called actors, tasks, fibers, virtual threads, goroutines, or something else. Processes are lightweight and able to be (generally) created and destroyed cheaply.

Processes have a few key attributes and invariants:

• every process is a future that runs as an independent asynchronous task on a Tokio runtime
• every process has a unique numerical identifier and a semi-unique name

Unlike Erlang processes, Saluki processes don’t have an inherent mailbox or message passing capabilities. As well, processes can’t run by themselves. They must be supervised.

Supervisors

Supervisors are themselves processes whose only job is to supervise other processes, also called workers. In a supervisor, workers are added and configured through a common convention that allows defining how the worker is created (or recreated on failure), how many times it can be restarted, and more. Supervisors themselves can also be workers, and so nested supervision trees can be constructed.

Supervisors include a number of configurable settings that allow customizing the behavior of how workers are managed, which in turn allows building fault-tolerant systems: we can restart workers for transient failures, give up for permanent failures, and so on.

Spawning children

Usually, you declare children in advance with Supervisor::add_worker. A process can also start new children while it runs. There are three ways to do this. They differ in the owner of the new child:

• spawn, which makes the new child a child of the current process. It is the equivalent of tokio::spawn for supervision. It is the recommended way to spawn for code that runs under supervision.
• spawn_sibling, which gives the new child to the nearest supervisor above the current process. The child is a sibling of the current process only if that supervisor runs the process directly, and not in a scope.
• SupervisorHandle, which names a supervisor explicitly.

For all three ways, a ChildBuilder describes the child. To get a builder, use worker or supervisable. For a named supervisor, use the methods with the same names on SupervisorHandle. The builder is the only way to configure a child. It produces a ChildSpecification, which has no settings of its own. Because of this, you cannot use the specification to make a combination of settings that the builder does not offer.

A child that is itself a Supervisor can be handed to any of the above directly, which is all most callers need. nested_supervisor and SupervisorHandle::nested_supervisor give access to the settings that are not available that way: whether the subtree is needed or hosted, and whether it stops as soon as its owner is told to stop.

All three ways to spawn are synchronous and cannot fail. As with tokio::spawn, an accepted child does not always run. If its owner stops before it reaches the child, the owner never starts the child.

Restart policy

You do not select the restart policy of a child directly. Two properties of the child decide it:

• Whether it can start again. A child reports this through Supervisable::can_restart. A Supervisable builds its work each time it starts, so it can usually start again. A child that consumes something when it starts is one-shot. Examples are a child built from a closure, which consumes its body, and a Supervisable that takes a connection.
• Whether its owner can work without it. The owner of a needed child cannot work without it. The owner of a hosted child only runs it. See ChildBuilder::needed and ChildBuilder::hosted.

Together, these two properties decide what occurs when a child stops without a stop request:

	Needed	Hosted
Can start again	always restarted (Permanent)	restarted if it failed (Transient)
One-shot	its owner fails	removed, and its owner continues

A child that stops because of a stop request is never restarted. A child of a scope that stops after the owner of the scope is told to stop counts as stopped by request. If a child stops more often than the restart limit of its owner allows, the owner fails. This applies to needed and hosted children. The next level of the supervision tree then handles the failure.

A child that can start again, and that you declare in advance with Supervisor::add_worker, is needed by default. This is because it is part of the purpose of its supervisor. Every other child is hosted by default.

Scopes

A supervisor owns its children. Every supervised worker also owns the children that it spawns, and keeps them in its scope.

Each of these children runs as a separate task. The worker does not finish until all of these children finish. When the body of the worker returns, the scope stops them, each according to its own shutdown strategy, as a supervisor does.

A value can also own a scope, so that the background tasks of a primitive stop when the primitive stops. See scope::nested. You can also create a scope before the process that will own it exists, with UnadoptedScope. This lets work spawned during the construction of something belong to the process that later runs that thing.

Where to spawn a child

Spawn most children into the scope of the current process with spawn. Use a different way only when one of these conditions is true:

• A value, not a process, must own the child. Spawn the child into a scope::nested guard.
• A process other than the one that spawns the child must own it. Get the scope of the owner with scope::current, and spawn the child with ChildBuilder::spawn_in.
• The child must continue to run after restarts of the process that spawns it. Make both of them children of a Supervisor, and spawn the child through its SupervisorHandle.
• Children need their own restart strategy, resource group, or runtime. A scope restarts its children at most once in each five-second period, for all of its children together. Put these children under a Supervisor.

If a worker body only waits for a stop request while its children do the work, use a Supervisor instead.

Supervision trees

As supervisors can be nested, this allows building a tree of supervisors (hence supervision trees) where leaf supervisors manage workers specific to a certain area, and parent supervisors manage the leaf supervisors. For example, for a server application serving multiple API endpoints, each endpoint might be managed by a separate supervisor: a worker for accepting connections, a worker for each connection, and so on. Above those supervisors, a parent supervisor manages each leaf supervisor, and potentially other workers that provide necessary services utilized by each endpoint, such as logging, metrics, or other infrastructure services.

As every supervisor can define its own specific restart strategy, and behavior, this allows for more granular grouping and control over which set of workers must be restarted if a related worker fails, and how those failures propagate up and down the supervision tree.

Inspecting the tree

A supervision tree is also a description of what a process is doing, which makes it worth being able to read at runtime. Supervisor::tree_handle returns a read-only handle whose snapshot walks the tree from that supervisor downward, reporting each node’s name, process, restart policy, restart count, lifetime, and resource usage, with its children nested beneath it. A handle can be taken before the supervisor starts and stays valid across every restart of it, which makes it suitable for handing to a worker that serves an operator-facing diagnostics endpoint.

Examples

See the basic_supervisor example which shows how supervisors and workers are composed together, as well as how failed workers and supervisors are restarted.

Re-exports

pub use self::scope::Scope;

pub use self::scope::ScopeError;

pub use self::scope::ScopeGuard;

pub use self::scope::UnadoptedScope;

Modules

scope

Structured concurrency scopes.

state

Runtime state management utilities.

Structs

ChildBuilder

Builder for a yet-to-be-started child task.

ChildId

Identifier for a child managed by a Supervisor.

ChildSpecification

A specification for a process to be added to a Supervisor.

FnWorker

A Supervisable worker built from a plain future.

LoweredChild

A child specification lowered into the supervisor’s internal representation.

NestedSupervisorBuilder

Builder for a yet-to-be-started nested supervisor.

NodeSnapshot

One node – a supervisor or a worker – in a TreeSnapshot.

OneShot

Marks a builder whose worker can only be started once.

ProcessId

Process identifier.

ResourceUsage

Cumulative resource usage for one resource group.

RestartStrategy

Restart strategy for a supervisor.

Restartable

Marks a builder whose worker can usually be started more than once.

RuntimeConfiguration

Configuration for a dedicated Tokio runtime.

SupervisionSettings

How a supervisor supervises its children, and how it has fared.

SupervisionTreeAPIHandler

An API handler for reporting the state of a supervision tree.

SupervisionTreeHandle

A read-only handle for taking snapshots of a supervision tree.

SupervisionTreeState

State used for the supervision tree API handler.

SupervisionTreeWorker

A worker that exposes a supervision tree over the control plane.

Supervisor

Supervises a set of workers.

SupervisorHandle

A handle for spawning dynamic children on a running Supervisor.

SupervisorSpec

Child specification state for a supervisor.

TreeSnapshot

A point-in-time view of a supervision tree.

TreeTotals

Aggregate counts across a whole TreeSnapshot.

UnixMillis

A point in time, as milliseconds since the Unix epoch.

WorkerSpec

Child specification state for a worker.

Enums

AutoShutdown

Policy for automatically shutting a supervisor down based on the termination of its significant children.

InitializationError

Initialization errors.

NodeKind

Whether a node supervises other nodes or performs work itself.

NodeState

Where a node is in its lifecycle.

RestartMode

Restart mode for child processes.

RestartType

Restart policy for an individual child process.

RuntimeMode

Controls which runtime a supervisor runs on.

ShutdownStrategy

Strategy for shutting down a process.

SupervisorError

Supervisor errors.

Constants

SUPERVISION_TREE_ROUTE

API route serving a snapshot of a supervision tree.

Traits

BuilderState

The kind of worker a ChildBuilder is describing.

ChildState

Child specification state.

IntoWorkerResult

An output type that a function-based worker can produce.

Supervisable

A supervisable process.

Functions

get_sanitized_name

Sanitizes name into a single process-name segment.

nested_supervisor

Creates a builder for a nested supervisor spawned into the current process.

spawn

Spawns a child of the current process.

spawn_sibling

Spawns a child onto the ambient supervisor, the nearest supervisor above the current process.

supervisable

Creates a builder for a supervisable child task of the current process.

worker

Creates a builder for a child task of the current process.

worker_with_shutdown

Creates a builder for a child task of the current process, with a body that receives the shutdown signal.

Type Aliases

SupervisorFuture

A Future that represents the execution of a supervised process.