Deepracer model-generated artifacts

From AWS DeepRacer Community Wiki
Jump to navigation Jump to search

In addition to the DeepRacer Console and Kinesis feed, more data about model training and evaluation performance is available in two locations: the S3 DeepRacer bucket, and the CloudWatch logstream. The CloudWatch logstream contains traces of the intermediate training state, which makes it particularly helpful in assessing the quality of the reward function and convergence of the model. Stackdumps of errors that occur in the reward function can also be identified in the logstream.

The S3 bucket contains file dumps with final high level metrics gathered from the training and evaluation, the robomaker environment settings, the trained model weight files, and the python reward function.

S3 file manifest[edit]

S3 Bucket Key (Path) File Name Content Summary
/DeepRacer-Metrics EvaluationMetrics-[1].json Summary results of evaluation. Data is presented at "Trial" granularity.
/DeepRacer-Metrics TrainingMetrics-[1].json Summary results of training. Data is presented at "Episode" granularity.
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/ip done Contains the word "done". Appears to be used as a semaphore.
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/ip hyperparameters.json Model hyperparameters
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/ip ip.json An IP address
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/model .coach_checkpoint
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/model .ready
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/model [1]_Step-[2].ckpt.data-[3]-of-[4]
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/model [1]_Step-[2].ckpt.index
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/model [1]_Step-[2].ckpt.meta
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/model model_[1].pb Tensorflow savedModel protobuf file
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL]/model model_metadata.json Vehicle configuration and action space
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL] eval_params_[1].yaml Contains an index of services and S3 files attached to the evaluation instance
/DeepRacer-SageMaker-RoboMaker-comm-[SERIAL] Contians an index of services and S3 files attached to the training instance
/DeepRacer-SageMaker-rlmdl-[SERIAL]/dr-sm-rltj--[SERIAL]/output model.tar.gz Trained model weights file
/model-metadata/[MODEL_NAME] model_metadata.json Describes the configuration of the deepracer used to train the model
/reward-functions/[MODEL_NAME] reward_function.py The python reward function entered into the deepracer console


EvaluationMetrics-[1].json[edit]

Field
metric_time
elapsed_time_in_milliseconds
start_time
completion_percentage
trial
episode_status

TrainingMetrics-[1].json[edit]

Field
completion_percentage
episode
start_time
metric_time
episode_status
reward_score
elapsed_time_in_milliseconds

hyperparameters.json[edit]

Field
batch_size
beta_entropy
discount_factor
e_greedy_value
epsilon_steps
exploration_type
loss_type
lr
num_episodes_between_training
num_epochs
stack_size
term_cond_avg_score
term_cond_max_episodes


eval_params_[1].yaml[edit]

Field
METRICS_S3_OBJECT_KEY
AWS_REGION
NUMBER_OF_TRIALS
CAR_NAME
KINESIS_VIDEO_STREAM_NAME
WORLD_NAME
ROBOMAKER_SIMULATION_JOB_ACCOUNT_ID
METRICS_S3_BUCKET
MODEL_S3_PREFIX
RACE_TYPE
JOB_TYPE
MODEL_S3_BUCKET
CAR_COLOR

training_params_[1].yaml[edit]

Field
SAGEMAKER_SHARED_S3_PREFIX
CHANGE_START_POSITION
METRICS_S3_OBJECT_KEY
AWS_REGION
NUMBER_OF_EPISODES
KINESIS_VIDEO_STREAM_NAME
TRAINING_JOB_ARN
REWARD_FILE_S3_KEY
METRIC_NAMESPACE
ALTERNATE_DRIVING_DIRECTION
WORLD_NAME
ROBOMAKER_SIMULATION_JOB_ACCOUNT_ID
METRICS_S3_BUCKET
SAGEMAKER_SHARED_S3_BUCKET
RACE_TYPE
JOB_TYPE
MODEL_METADATA_FILE_S3_KEY
METRIC_NAME
TARGET_REWARD_SCORE

model_metadata.json[edit]

Field
action_space
sensor
neural_network
version