Shadow tests

A shadow test is an online experiment that runs in the background and compares the results of two instances: baseline vs. candidate. When the shadow test has started, any run made on the baseline instance will trigger a run on the candidate instance using the same input and options. The results of the shadow test are often used to determine if a new version of a model is ready to be promoted to production.

To be concise, the shadow test deploys the candidate instance as a shadow. Any time a run is made to the app, using the baseline instance, the input is also sent to the candidate instance. Once the test is completed, the results are compared with the same framework as in a batch experiment.

A shadow test is defined by the following parameters:

• start_events (optional): a list of events that will trigger the start of the shadow test data collection. Supported start events are:
  • time: the shadow test will start at a specific time. The test can be started manually before the start time, but it will not begin collecting data until this start time is reached. If the start time is omitted, the test will be executed immediately when it is started.
• termination_events (required): a list of events that will trigger the termination of the shadow test. One or all termination events may be defined, and the first condition met will cause the shadow test to stop. Supported termination events are:
  • maximum_runs (required): the shadow test will terminate after a specific number of runs.
  • time: the shadow test will terminate at a specific time.

These are the steps to start a shadow test:

1. Create the shadow test. This step is like creating a draft of the test.
2. Start the shadow test. A test does not start automatically even if a criteria part of start_events is met. You must start the test manually.

A shadow test may be manually completed or canceled at any time, if you don't want to wait until the termination_events are fulfilled. Once a shadow test starts, you can view partial results in the Console web interface, without having to wait for the test to complete. Results will be updated as the test progresses and finishes.

Shadow tests are designed to be visualized in the Console web interface. Go to the app, Experiments > Shadow tab.

There are several interfaces for creating shadow tests:

• Console: use the web interface.
• Nextmv CLI: use your terminal.
• Cloud API: use the HTTP endpoints.

Console

Go to the Console web interface, and open your app. Go to the Experiments > Shadow tab. Click on New Shadow Test. Fill in the fields.

You can either create the shadow test in draft mode and start it later, or start it right away. Once the shadow test is started, you can view partial results, without having to wait for the test to complete. You can also cancel or complete the test at any time.

Nextmv CLI

Define the desired shadow test ID and name.

1. Create the shadow test.

   • Bash
   • Output (JSON)

   nextmv experiment shadow create \
       --app-id $APP_ID \
       --shadow-test-id $SHADOW_TEST_ID \
       --name $SHADOW_TEST_ID \
       --baseline-instance-id "latest" \
       --candidate-instance-id "latest-2" \
       --start-time "2024-01-12T21:00:00Z" \
       --end-time "2024-01-13T21:00:00Z" \
       --maximum-runs 2 \
       --description "An optional description"
   

   Copy

2. Start the shadow test. A blank output means the shadow test has started.

   • Bash

   nextmv experiment shadow start \
       --app-id $APP_ID \
       --shadow-test-id $SHADOW_TEST_ID
   

   Copy

You can stop the shadow test at any time. The intent can be complete or cancel, depending on which label you want to attribute to the test. The results will be kept in both cases. A blank output means the shadow test has stopped.

nextmv experiment shadow stop \
    --app-id $APP_ID \
    --shadow-test-id $SHADOW_TEST_ID \
    --intent "cancel"

Cloud API

Define the desired shadow test ID and name.

1. Create the shadow test.

   POSThttps://api.cloud.nextmv.io/v1/applications/{application_id}/experiments/shadow

   Create a shadow test in draft mode.

   Create a shadow test in draft mode.

   View API details
   • Bash
   • Output (JSON)

   curl -L -X POST \
       "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/shadow" \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer $NEXTMV_API_KEY" \
       -d "{
           \"app_id\": \"$APP_ID\",
           \"id\": \"$SHADOW_TEST_ID\",
           \"name\": \"$SHADOW_TEST_ID\",
           \"comparison\": {
               \"baseline_instance_id\": \"latest\",
               \"candidate_instance_id\": \"latest-2\"
           },
           \"start_events\": {
               \"time\": \"2024-01-12T21:00:00Z\"
           },
           \"termination_events\": {
               \"maximum_runs\": 2,
               \"time\": \"2024-01-13T21:00:00Z\"
           },
           \"description\": \"An optional description\"
       }"
   

   Copy

2. Start the shadow test. A blank output means the shadow test has started.

   PUThttps://api.cloud.nextmv.io/v1/applications/{application_id}/experiments/shadow/{shadow_id}/start

   Start shadow test.

   Start shadow test.

   View API details
   • Bash

   curl -L -X PUT \
       "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/shadow/$SHADOW_TEST_ID/start" \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer $NEXTMV_API_KEY"
   

   Copy

You can stop the shadow test at any time. The intent can be complete or cancel, depending on which label you want to attribute to the test. The results will be kept in both cases. A blank output means the shadow test has stopped.

curl -L -X PUT \
    "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/shadow/$SHADOW_TEST_ID/stop" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $NEXTMV_API_KEY" \
    -d "{
        \"intent\": \"cancel\"
    }"