Experiments Tests

Input sets

A tutorial for creating and managing input sets on Nextmv Cloud.

Input sets are defined sets of input files to use for offline tests, such as batch experiments and acceptance tests. You can visualize the input sets of an app in the Console web interface. Go to the app, Experiments > Input Sets tab.

Input set

There are several interfaces for creating input sets:

Inputs for the input sets can be defined using any of the following creation types:

  1. Uploading files.
  2. Creating inputs.
  3. Making runs.

Note that the maximum number of inputs allowed in an input set is 20.

Uploading files

If your app takes a single file as input (e.g. csv, json, or other utf-8), you can upload your input set files directly.

Via Console:

Uploading input sets

Creating Inputs

If you'd like to store individual inputs for easier reference in input set creation, you can create a stored input.

Via Console:

Uploading input sets

Making a run

To use past runs to define an input set you can chose from one of the following options:

  • Specifying the run IDs of the runs that will make up the input set.
  • Specifying a date range and an instance ID where runs were made to gather inputs for an input set.

Via Console:

Go to the Console web interface, and open your app. Go to the Experiments > Input Sets tab. Click on New Input Set. Fill in the fields.

Input sets

Via Nextmv CLI:

Define the desired input set ID and name. After, create the input set by:

  • Referencing the run IDs.

    nextmv experiment input-set create \
       --app-id $APP_ID \
       --input-set-id $INPUT_SET_ID \
       --name $INPUT_SET_ID \
       --run-ids "latest-RNBs7AKSg,latest-UlulZAKSR,latest-fK_wZ0FSg" \
       --description "An optional description"
    
    Copy
  • Referencing a date range and an instance ID.

    nextmv experiment input-set create \
       --app-id $APP_ID \
       --input-set-id $INPUT_SET_ID \
       --name $INPUT_SET_ID \
       --instance-id $INSTANCE_ID \
       --start-time "2024-01-05T00:00:00Z" \
       --end-time "2024-01-05T23:59:00Z" \
       --description "An optional description"
    
    Copy

Via Python SDK:

Define the desired input set ID and name. After, create the input set by:

  • Referencing the run IDs.

    import json
    import os
    
    from nextmv.cloud import Application, Client
    
    client = Client(api_key=os.getenv("NEXTMV_API_KEY"))
    app = Application(client=client, id=os.getenv("APP_ID"))
    input_set = app.new_input_set(
       id=os.getenv("INPUT_SET_ID"),
       name=os.getenv("INPUT_SET_ID"),
       description="An optional description",
       run_ids=["latest-RNBs7AKSg", "latest-UlulZAKSR", "latest-fK_wZ0FSg"],
    )
    print(json.dumps(input_set.to_dict(), indent=2))  # Pretty print.
    
    Copy
  • Referencing a date range and an instance ID.

    import json
    import os
    from datetime import datetime, timezone  
    
    from nextmv.cloud import Application, Client
    
    client = Client(api_key=os.getenv("NEXTMV_API_KEY"))
    app = Application(client=client, id=os.getenv("APP_ID"))
    input_set = app.new_input_set(
       id=os.getenv("INPUT_SET_ID"),
       name=os.getenv("INPUT_SET_ID"),
       description="An optional description",
       instance_id=os.getenv("INSTANCE_ID"),
       start_time=datetime(2024, 1, 5, 0, 0, 0, tzinfo=timezone.utc),
       end_time=datetime(2024, 1, 5, 23, 59, 59, tzinfo=timezone.utc),
    )
    print(json.dumps(input_set.to_dict(), indent=2))  # Pretty print.
    
    Copy

Via Cloud API:

POSThttps://api.cloud.nextmv.io/v1/applications/{application_id}/experiments/inputsets

Create new input set.

Create new application input set for experiments.

Define the desired input set ID and name. After, create the input set by:

  • Referencing the run IDs.
curl -sS -L -X POST \
    "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/inputsets" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $NEXTMV_API_KEY" \
    -d "{
      \"id\": \"$INPUT_SET_ID\",
      \"name\": \"$INPUT_SET_ID\",
      \"run_ids\": [\"$RUN_ID\"],
      \"description\": \"An optional description\"
    }" | jq
Copy
  • Referencing a date range and an instance ID.

    curl -L -X POST \
       "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/inputsets" \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer $NEXTMV_API_KEY" \
       -d "{
          \"app_id\": \"$APP_ID\",
          \"id\": \"$INPUT_SET_ID\",
          \"name\": \"$INPUT_SET_ID\",
          \"instance_id\": \"$INSTANCE_ID\",
          \"start_time\": \"2024-01-05T00:00:00Z\",
          \"end_time\": \"2024-01-05T23:59:00Z\",
          \"description\": \"An optional description\"
       }"
    
    Copy

Update an input set

Update an input set (change its name, description, etc...) using the following interfaces:

  • Console. Go to the Console web interface, and open your app. Go to the Experiments > Input Sets tab. Click on the input set you want to update. Click on Edit. Fill in the fields.

  • Nextmv CLI. Use the nextmv experiment input-set update subcommand.

  • Cloud API. Make sure your NEXTMV_API_KEY is exported as an environment variable. The API uses Bearer Authentication.

    PUThttps://api.cloud.nextmv.io/v1/applications/{application_id}/experiments/inputsets/{input_set_id}

    Update input set metadata.

    Update input set metadata with defined data, specified by application and input set ID.

curl -sS -L -X PUT \
      "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/inputsets/$INPUT_SET_ID" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $NEXTMV_API_KEY" \
      -d "{
         \"name\": \"Another name\",
         \"description\": \"Another description\"
      }" | jq
Copy

Page last updated

Go to on-page nav menu