In this post I will share how to add an AWS service to AWS CLI utility and for use with boto3.
One of the missing features AWS DeepRacer Community have been regularly raising has been lack of CLI access to AWS DeepRacer Console. Thanks to Don Barber I have finally understood how AWS CLI and Boto3 library work and would like to share it with you.
Icon image used shared under Creative Commons here
What is AWS CLI
AWS Command Line Interface is a utility that you can use to interact with AWS services through a terminal. It supports many of the services that Amazon Web Services Provide:
accessanalyzer | acm
acm-pca | alexaforbusiness
amp | amplify
amplifybackend | apigateway
apigatewaymanagementapi | apigatewayv2
appconfig | appflow
appintegrations | application-autoscaling
application-insights | appmesh
appstream | appsync
athena | auditmanager
autoscaling | autoscaling-plans
backup | batch
braket | budgets
ce | chime
cloud9 | clouddirectory
cloudformation | cloudfront
cloudhsm | cloudhsmv2
cloudsearch | cloudsearchdomain
cloudtrail | cloudwatch
codeartifact | codebuild
codecommit | codeguru-reviewer
codeguruprofiler | codepipeline
codestar | codestar-connections
codestar-notifications | cognito-identity
cognito-idp | cognito-sync
comprehend | comprehendmedical
compute-optimizer | connect
connect-contact-lens | connectparticipant
cur | customer-profiles
databrew | dataexchange
datapipeline | datasync
dax | deepracer
detective | devicefarm
devops-guru | directconnect
discovery | dlm
dms | docdb
ds | dynamodb
dynamodbstreams | ebs
ec2 | ec2-instance-connect
ecr | ecr-public
ecs | efs
eks | elastic-inference
elasticache | elasticbeanstalk
elastictranscoder | elb
elbv2 | emr
emr-containers | es
events | firehose
fms | forecast
forecastquery | frauddetector
fsx | gamelift
glacier | globalaccelerator
glue | greengrass
greengrassv2 | groundstation
guardduty | health
healthlake | honeycode
iam | identitystore
imagebuilder | importexport
inspector | iot
iot-data | iot-jobs-data
iot1click-devices | iot1click-projects
iotanalytics | iotdeviceadvisor
iotevents | iotevents-data
iotfleethub | iotsecuretunneling
iotsitewise | iotthingsgraph
iotwireless | ivs
kafka | kendra
kinesis | kinesis-video-archived-media
kinesis-video-media | kinesis-video-signaling
kinesisanalytics | kinesisanalyticsv2
kinesisvideo | kms
lakeformation | lambda
lex-models | lex-runtime
license-manager | lightsail
location | logs
lookoutvision | machinelearning
macie | macie2
managedblockchain | marketplace-catalog
marketplace-entitlement | marketplacecommerceanalytics
mediaconnect | mediaconvert
medialive | mediapackage
mediapackage-vod | mediastore
mediastore-data | mediatailor
meteringmarketplace | mgh
migrationhub-config | mobile
mq | mturk
mwaa | neptune
network-firewall | networkmanager
opsworks | opsworkscm
organizations | outposts
personalize | personalize-events
personalize-runtime | pi
pinpoint | pinpoint-email
pinpoint-sms-voice | polly
pricing | qldb
qldb-session | quicksight
ram | rds
rds-data | redshift
redshift-data | rekognition
resource-groups | resourcegroupstaggingapi
robomaker | route53
route53domains | route53resolver
s3control | s3outposts
sagemaker | sagemaker-a2i-runtime
sagemaker-edge | sagemaker-featurestore-runtime
sagemaker-runtime | savingsplans
schemas | sdb
secretsmanager | securityhub
serverlessrepo | service-quotas
servicecatalog | servicecatalog-appregistry
servicediscovery | ses
sesv2 | shield
signer | sms
sms-voice | snowball
sns | sqs
ssm | sso
sso-admin | sso-oidc
stepfunctions | storagegateway
sts | support
swf | synthetics
textract | timestream-query
timestream-write | transcribe
transfer | translate
waf | waf-regional
wafv2 | wellarchitected
workdocs | worklink
workmail | workmailmessageflow
workspaces | xray
s3api | s3
configure | deploy
configservice | opsworks-cm
runtime.sagemaker | history
The tool uses Amazon's boto3 library which allows for a programatic access to those services in Python. Boto3 in turn uses botocore as a dependency that contains definitions of the services.
How are service endpoints defined in AWS CLI?
Each service has at least a Service Definition File which is written in JSON and contains a few pieces of information:
- version - version of the service definition file
- metadata - additional information about interacting with the service
- operations - what requests can be sent to the service
- shapes - operations inputs and outputs
- documentation - pieces of data that are used to form documentation in
aws <command> help
This is not the only file available and you can read about more of them in loaders.py docstring. They are not needed in my case.
The SDF file needs to be present in one of the following locations top be used by AWS CLI or boto3:
<botocore root>/data/~/.aws/models- A path added to a list in the
AWS_DATA_PATHenv var
If you're using boto3, you can also use code to add your location to the default loader.
Can I create my own service definition?
I think not. I was actually planning to do that myself but then I have realised that you need to know the exact service API endpoint and version to be able to work with it. Once you do you can observe what your service console page is calling on the api gateway and do the same yourself.
But I guess that's not the use case AWS are after. When the service authors decide their API is ready they can expose it together with the version, they can provide the service definition files for people with access to preview and they can then use the API. If you're in contact with the dev team you can try some pretty-please approach and maybe they will share it with you.
AWS DeepRacer CLI
If you haven't yet seen Don's reInvent 2020 talk "Shift your ML model into overdrive with AWS DeepRacer analysis tools" I recommend that you do. As part of preparation for the talk Don has prepared some updates to log analysis tool provided by AWS. The tool has been there from day one together with DeepRacer workshops. I've looked at this new line:
region = "us-east-1"
dr_client = boto3.client('deepracer', region_name=region,
endpoint_url="https://deepracer-prod.{}.amazonaws.com".format(region))
models = dr_client.list_models(ModelType="REINFORCEMENT_LEARNING",MaxResults=100)["Models"]
And thought: "cool, we can now use aws cli!". Lesson learned: don't skip half of the code provided when trying to get something to work. Right above this code Done does this:
envroot = os.getcwd()
aws_data_path = set(os.environ.get('AWS_DATA_PATH', '').split(os.pathsep))
aws_data_path.add(os.path.join(envroot, 'models'))
os.environ.update({'AWS_DATA_PATH': os.pathsep.join(aws_data_path)})
And there it was: The service definition file for DeepRacer.
I am pleased to say that I have added the service definition file to deepracer-utils 0.8. If you would like to use it with boto3, simply run:
from deepracer.boto3_enhancer import add_deepracer
client = deepracer_client()
dir(client)
I'm leaving it out there since you can already start playing with it, but expect updates to install it for use in AWS CLI and for some handy use cases, examples and perhaps even docs :)