Skip to main content

Converters and encryption - Go SDK

By default, Temporal payloads are stored unencrypted inside of its data store. Consequently, this means that string payloads can be read from the Temporal Web UI and CLI in plain text. When working with sensitive data, Temporal implementers may need to adopt encryption algorithms, manage encryption keys, or restrict a subset of their users from viewing payload output.

A Custom Codec allows a developer to transform the payload of a message sent or received by a Temporal Client. This ensures that the data is encrypted as it travels across the network and when it is stored in the Event History, readable only by those with access to the key.

Use a custom Payload Codec in Go​

How to use a custom Payload Codec using the Go SDK.

Step 1: Create a custom Payload Codec

Create a custom PayloadCodec implementation and define your encryption/compression and decryption/decompression logic in the Encode and Decode functions.

The Payload Codec converts bytes to bytes. It must be used in an instance of CodecDataConverter that wraps a Data Converter to do the Payload conversions, and applies the custom encoding and decoding in PayloadCodec to the converted Payloads.

The following example from the Data Converter sample shows how to create a custom NewCodecDataConverter that wraps an instance of a Data Converter with a custom PayloadCodec.

// Create an instance of Data Converter with your codec.
var DataConverter = converter.NewCodecDataConverter(
	converter.GetDefaultDataConverter(),
	NewPayloadCodec(),
)
//...
// Create an instance of PaylodCodec.
func NewPayloadCodec() converter.PayloadCodec {
	return &Codec{}
}

Implement your encryption/compression logic in the Encode function and the decryption/decompression logic in the Decode function in your custom PayloadCodec, as shown in the following example.

// Codec implements converter.PayloadEncoder for snappy compression.
type Codec struct{}

// Encode implements converter.PayloadCodec.Encode.
func (Codec) Encode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
	result := make([]*commonpb.Payload, len(payloads))
	for i, p := range payloads {
		// Marshal proto
		origBytes, err := p.Marshal()
		if err != nil {
			return payloads, err
		}
		// Compress
		b := snappy.Encode(nil, origBytes)
		result[i] = &commonpb.Payload{
			Metadata: map[string][]byte{converter.MetadataEncoding: []byte("binary/snappy")},
			Data:     b,
		}
	}

	return result, nil
}

// Decode implements converter.PayloadCodec.Decode.
func (Codec) Decode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
	result := make([]*commonpb.Payload, len(payloads))
	for i, p := range payloads {
		// Decode only if it's our encoding
		if string(p.Metadata[converter.MetadataEncoding]) != "binary/snappy" {
			result[i] = p
			continue
		}
		// Uncompress
		b, err := snappy.Decode(nil, p.Data)
		if err != nil {
			return payloads, err
		}
		// Unmarshal proto
		result[i] = &commonpb.Payload{}
		err = result[i].Unmarshal(b)
		if err != nil {
			return payloads, err
		}
	}

	return result, nil
}

Step 2: Set Data Converter to use custom Payload Codec.

Set your custom PayloadCodec with an instance of DataConverter in your Dial client options that you use to create the client.

The following example shows how to set your custom Data Converter from a package called mycodecpackage.

//...
c, err := client.Dial(client.Options{
		// Set DataConverter here to ensure that Workflow inputs and results are
		// encoded as required.
		DataConverter: mycodecpackage.DataConverter,
	})
//...
  • Data encoding is performed by the client using the default converter provided by Temporal or your custom Data Converter when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted.
  • Data decoding may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the temporal workflow show command or when browsing the Web UI to view output.

For reference, see the following sample:

Using a Codec Server​

A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.

For reference, see the following sample:

Use custom Payload conversion​

How to customize the conversion of a payload using the Go SDK.

Temporal SDKs provide a default Payload Converter that can be customized to convert a custom data type to Payload and back.

The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion.

Payload Converters can be customized independently of a Payload Codec. Temporal's Converter architecture looks like this:

Billing page showing Credits tab

How to use a custom Payload Converter in Go​

How to use a custom Payload Converter using the Go SDK.

Use a Composite Data Converter to apply custom, type-specific Payload Converters in a specified order. Defining a new Composite Data Converter is not always necessary to implement custom data handling. You can override the default Converter with a custom Codec, but a Composite Data Converter may be necessary for complex Workflow logic.

NewCompositeDataConverter creates a new instance of CompositeDataConverter from an ordered list of type-specific Payload Converters. The following type-specific Payload Converters are available in the Go SDK, listed in the order that they are applied by the default Data Converter:

The order in which the Payload Converters are applied is important because during serialization the Data Converter tries the Payload Converters in that specific order until a Payload Converter returns a non-nil Payload.

To set your custom Payload Converter, use NewCompositeDataConverter and set it as the Data Converter in the Client options.

  • To replace the default Data Converter with a custom NewCompositeDataConverter, use the following.

    dataConverter := converter.NewCompositeDataConverter(YourCustomPayloadConverter())

  • To add your custom type conversion to the default Data Converter, use the following to keep the defaults but set yours just before the default JSON fall through.

    dataConverter := converter.NewCompositeDataConverter(
      converter.NewNilPayloadConverter(),
      converter.NewByteSlicePayloadConverter(),
      converter.NewProtoJSONPayloadConverter(),
      converter.NewProtoPayloadConverter(),
      YourCustomPayloadConverter(),
      converter.NewJSONPayloadConverter(),
    )