Experiments

Experiments are a form of A/B testing used by Discord in the client- and server-side of their applications to serve different experiences or behaviours to different users randomly and/or based on location, client version, etc.

Currently, Discord uses two different systems for experiments: the legacy system, which encapsulates user and guild experiments, and the new Apex system, which is more flexible but provides clients with less data. Both systems are still in use, but most new experiments are created in the Apex system, so the legacy system will eventually be deprecated.

At their core, rollouts are simply YAML inputted into the Discord admin panel; however, not all of the data is required by clients. Therefore, the actual experiments clients see are minified and complex to decipher.

Fingerprints

Even the marketing website uses A/B tests to hook users into the app. Therefore, the app needs a unique way to identify the person using the website without using authentication (for users initially visiting the landing page), so it resorts to using "fingerprints".

These aren't the usual fingerprints generated by collecting information about the browser; instead, they are snowflakes generated by unauthenticated requests to Get Experiment Assignments. It is expected that fingerprints are sent in the X-Fingerprint header in all subsequent requests to the API until authentication, in order to track A/B tests and allow access to API-locked portions of experiments.

A fingerprint is comprised of a snowflake and a hashed cryptographic value. It looks like this: 1084179945133187083.JQddgNMmwJPghoBtFmaH7jTmdsw.

When registering a new account, the fingerprint is passed, and (if valid) is used as the created user's ID. This is done in order to preserve experiments across to the registered user. Therefore, a user account's creation time theoretically represents the first time they visited Discord's marketing website.

Installations

The new Apex experiment system introduces the concept of installations, which are a unique identifier for a specific installation of a Discord client. This allows experiments to be targeted not just by user or guild, but by specific app installations, enabling more granular testing and rollouts.

Installations use an ID system very similar to fingerprints. The biggest difference is that they persist even after authentication. If a client already has an installation ID, it should be sent in the X-Installation-ID header in all requests to the API, and provided in the installation_id field when identifying with the Gateway. If an installation ID is invalid or not provided, a new one will be provided in the installation field of the response body of the Get Apex Experiment Assignments endpoint, or the apex_experiments field in the Ready event.

Rollouts

Legacy experiment rollouts are defined in populations based on the user or guild's rollout position and can have filters to narrow each population's availability.

Example Rollout

Note that this example is editorial and does not exactly represent how experiments are represented internally.

2023-02_stage_boosting (1816004721)

### Treatment 1

Filters

Guild Features: [COMMUNITY]
Member Count Range: 1000 - null
Hash Range: Hash Key: 1816004721, Target: 10000

Position Ranges

5000-9500, 9500-10000

### Control

Filters

Guild Features: [COMMUNITY]
Member Count Range: 1000 - null

Position Ranges

0 - 10000

### None

Position Ranges

0 - 10000

Treatments

Control and None are represented in rollouts as the integer values 0 and -1, respectively. Note that there are some instances of experiments having specific unnecessary treatments labelled as Control with different treatment values.

Rollout Positions

A rollout position is calculated using the following pseudo code, where exp_name is the human readable name for the experiment and resource_id is the user, fingerprint or guild's ID. This position is used in conjunction with the rollout populations and filters to figure out what the assigned bucket for the experiments are by simply checking which treatment the population is included in.

result = mmh3.hash('exp_name:resource_id', signed=False) % 10000

Data Structures

There are two types of experiments, user experiments and guild experiments. Metadata and human-readable experiment names are available in the user-facing clients. However, API objects do not contain such data, except for guild experiments which may have a human-readable name provided for hash calculations, overriding the one in clients.

Most of the below objects are represented as arrays following the order the fields are documented in.

User Experiments

User experiment data returned by the API is very limited. In contrast to the wide range of data the API provides for guild rollouts, the only values we can programmatically retrieve from the API are the user or fingerprint's assigned bucket for the experiment.

User Experiment Structure

This object is represented as an array of the following fields:

¹ The bucket for A/A tested experiments should always be None (-1) unless an override is present for the resource.

² Holdout information is only present if the user or fingerprint has an assigned bucket for the holdout experiment. Therefore, if holdout experiment information is present and the population bucket is set to None (-1), the experiment has been disabled by the holdout. As user experiments are opaque, no client handling is required for this field. Just follow the population field as usual.

Example User Experiment

[826493636, 3, -1, -1, 0, 203, 0, 1, "2025-02_user_profile_editing", 2, 0]

Guild Experiments

The data provided here is more detailed, because the client has to figure out itself the assigned bucket for each guild. It may seem daunting to parse this given the sheer amount of arrays, but it's really quite simple.

Guild Experiment Structure

This object is represented as an array of the following fields:

¹ Used to categorize multiple experiments together for coordinated rollouts.

² The population bucket for A/A tested experiments should always be None (-1) unless an override is present for the resource.

³ If a holdout experiment is present and the guild is in the holdout bucket, the population bucket will be set to None (-1), disabling the experiment unless an override is present.

Example Guild Experiment

[
  1405831955,
  "2021-06_guild_role_subscriptions",
  0,
  [
    [
      [
        [
          -1,
          [
            {
              "s": 7200,
              "e": 10000
            }
          ]
        ],
        [
          1,
          [
            {
              "s": 0,
              "e": 7200
            }
          ]
        ]
      ],
      [
        [
          2294888943,
          [
            [2690752156, 1405831955],
            [1982804121, 10000]
          ]
        ]
      ]
    ]
  ],
  [],
  [
    [
      [
        [
          [
            1,
            [
              {
                "s": 0,
                "e": 10000
              }
            ]
          ]
        ],
        [[1604612045, [[1183251248, ["GUILD_ROLE_SUBSCRIPTIONS"]]]]]
      ]
    ]
  ],
  null,
  null,
  0,
  0
]

Experiment Population Object

The population object defines a set of filters and position ranges required to meet specific buckets.

Experiment Population Structure

This object is represented as an array of the following fields:

Example Experiment Population

[
  [
    [
      -1,
      [
        {
          "s": 7200,
          "e": 10000
        }
      ]
    ]
  ],
  [
    [
      2294888943,
      [
        [2690752156, 1405831955],
        [1982804121, 10000]
      ]
    ]
  ]
]

Experiment Population Range Object

If the filters in a given population are satisfied and a range includes the resource's rollout position, the resource is then eligible for the given bucket.

Experiment Population Range Structure

This object is represented as an array of the following fields:

Experiment Population Rollout Structure

Example Experiment Population Range

[
  1,
  [
    {
      "s": 0,
      "e": 4750
    }
  ]
]

Experiment Population Filters Object

This object defines the filters required to be eligible for the ranges. All provided filters must be satisfied for the resource to be eligible for the given bucket.

The filters are an object represented as an array of arrays. The first item in the nested array is a 32-bit unsigned Murmur3 hashed representation of the key, and the second item is the value, with the value being another array-represented object. All structures below are represented in this way.

Experiment Population Filters Structure

¹ The guild age is determined from the guild's ID. See the snowflake documentation for more information. The age can be calculated using the following pseudocode, where resource_id is the guild's ID:

timestamp = ((resource_id >> 22) + 1420070400000) / 1000
guild_age = (time.time() - timestamp) / 86400

Experiment Population Guild Feature Filter Structure

Experiment Population Range Filter Structure

Experiment Population ID Filter Structure

Experiment Population Hub Type Filter Structure

Experiment Population Vanity URL Filter Structure

Experiment Population Range By Hash Filter Structure

Example Experiment Population Filters

[
  [
    1604612045, // guild_has_feature
    [
      [
        1183251248, // guild_features
        ["ROLE_SUBSCRIPTIONS_ENABLED"]
      ]
    ]
  ]
]

Experiment Bucket Override Object

An override represents a manual setting by Discord employees to grant a guild early or specific access to an experiment.

Experiment Bucket Override Structure

Experiment Bucket Override Example

{
  "b": 1,
  "k": ["882680660588904448", "882703776794959873", "859533785225494528", "859533828754505741"]
}

Apex Experiments

Apex experiments are a new type of experiment that is far more flexible and allows for more complex targeting and assignment logic. While they are far more powerful, they also provide a lot less data to clients and allow for more opaque experimentation, preventing feature disclosure.

Notably, Apex experiments use the concept of variants instead of buckets, and are assigned to specific units (i.e. users, guilds, etc.) instead of being calculated on the client side based on filters and rollout positions.

Eligibility

Apex experiments use flags to determine evaluation precedence, especially for GUILD unit type experiments which evaluate both a guild-level and user-level assignment. Clients should determine final variant using the following priority:

1. User Override: If a user assignment is present and has the IS_OVERRIDE flag, it should be used as the assigned variant for all guilds
2. Guild Override: If a guild assignment is present and has the IS_OVERRIDE flag, it should be used as the assigned variant for the guild
3. Eligibility Gate: If a guild assignment is present, if and only if a user assignment is present with the USE_AS_ELIGIBILITY flag, should the guild assignment be used as the assigned variant for the guild

If none of the above are satisfied, the guild does not have an assigned variant for the experiment. Additionally, if the final evaluated assignment ever possesses the USE_AS_ELIGIBILITY flag, it should be discarded and the resource should be considered ineligible for the experiment, as this flag is only meant to be used as an eligibility gate and should not be used as an actual assignment.

Apex Experiments Structure

¹ This field is omitted if a valid installation ID is provided in request headers or when identifying with the Gateway. See the installations section for more information.

Apex Experiment Unit Type

Apex Experiment Assignments Structure

Apex Experiment Assignment Structure

This object is represented as an array of the following fields:

Apex Experiment Flags

Example Apex Experiments

{
  "assignments": {
    "1": {
      "852892297661906993": {
        "evaluation_id": "cd659a5d",
        "assignments": [
          [3759954850, 1, 0, 2],
          [4240376458, 0, 0, 8],
          [956087042, 1, 0, 2],
          [1605213660, 0, 2, 0],
          [3164674236, 1, 0, 4],
          [3436651650, 1, 1, 0],
          [1581149260, 0, 0, 3],
          [13763179, 1, 2, 0],
          [721000765, 1, 2, 0],
          [2657741306, 0, 2, 3],
          [341733491, 0, 2, 3],
          [76075180, 0, 2, 3],
          [2722069991, 0, 2, 3],
          [196982166, 1, 0, 0],
          [1228731397, 1, 2, 0],
          [1132732657, 1, 2, 0]
        ]
      }
    }
  }
}

Endpoints

Returns the user experiment assignments and optionally guild experiment rollouts for the requesting user or fingerprint.

Query String Parameters

¹ Including this parameter requires valid authentication.

Experiment Platform

Response Body

Generates a new fingerprint.

Response Body

Returns an apex experiments object for the requesting user and installation.

Query String Parameters

Apex Experiment Surface

Returns metadata for apex experiments.

Query String Parameters

Response Body

Apex Experiment Metadata Structure

Apex Experiment Variant Structure

Apex Experiment Variant Type