How to add service X to AWS CLI and boto3

in #engrave4 years ago

In this post I will share how to add an AWS service to AWS CLI utility and for use with boto3.

One of the missing features AWS DeepRacer Community have been regularly raising has been lack of CLI access to AWS DeepRacer Console. Thanks to Don Barber I have finally understood how AWS CLI and Boto3 library work and would like to share it with you.


Icon image used shared under Creative Commons here

What is AWS CLI

AWS Command Line Interface is a utility that you can use to interact with AWS services through a terminal. It supports many of the services that Amazon Web Services Provide:

accessanalyzer                           | acm                                     
acm-pca                                  | alexaforbusiness                        
amp                                      | amplify                                 
amplifybackend                           | apigateway                              
apigatewaymanagementapi                  | apigatewayv2                            
appconfig                                | appflow                                 
appintegrations                          | application-autoscaling                 
application-insights                     | appmesh                                 
appstream                                | appsync                                 
athena                                   | auditmanager                            
autoscaling                              | autoscaling-plans                       
backup                                   | batch                                   
braket                                   | budgets                                 
ce                                       | chime                                   
cloud9                                   | clouddirectory                          
cloudformation                           | cloudfront                              
cloudhsm                                 | cloudhsmv2                              
cloudsearch                              | cloudsearchdomain                       
cloudtrail                               | cloudwatch                              
codeartifact                             | codebuild                               
codecommit                               | codeguru-reviewer                       
codeguruprofiler                         | codepipeline                            
codestar                                 | codestar-connections                    
codestar-notifications                   | cognito-identity                        
cognito-idp                              | cognito-sync                            
comprehend                               | comprehendmedical                       
compute-optimizer                        | connect                                 
connect-contact-lens                     | connectparticipant                      
cur                                      | customer-profiles                       
databrew                                 | dataexchange                            
datapipeline                             | datasync                                
dax                                      | deepracer                               
detective                                | devicefarm                              
devops-guru                              | directconnect                           
discovery                                | dlm                                     
dms                                      | docdb                                   
ds                                       | dynamodb                                
dynamodbstreams                          | ebs                                     
ec2                                      | ec2-instance-connect                    
ecr                                      | ecr-public                              
ecs                                      | efs                                     
eks                                      | elastic-inference                       
elasticache                              | elasticbeanstalk                        
elastictranscoder                        | elb                                     
elbv2                                    | emr                                     
emr-containers                           | es                                      
events                                   | firehose                                
fms                                      | forecast                                
forecastquery                            | frauddetector                           
fsx                                      | gamelift                                
glacier                                  | globalaccelerator                       
glue                                     | greengrass                              
greengrassv2                             | groundstation                           
guardduty                                | health                                  
healthlake                               | honeycode                               
iam                                      | identitystore                           
imagebuilder                             | importexport                            
inspector                                | iot                                     
iot-data                                 | iot-jobs-data                           
iot1click-devices                        | iot1click-projects                      
iotanalytics                             | iotdeviceadvisor                        
iotevents                                | iotevents-data                          
iotfleethub                              | iotsecuretunneling                      
iotsitewise                              | iotthingsgraph                          
iotwireless                              | ivs                                     
kafka                                    | kendra                                  
kinesis                                  | kinesis-video-archived-media            
kinesis-video-media                      | kinesis-video-signaling                 
kinesisanalytics                         | kinesisanalyticsv2                      
kinesisvideo                             | kms                                     
lakeformation                            | lambda                                  
lex-models                               | lex-runtime                             
license-manager                          | lightsail                               
location                                 | logs                                    
lookoutvision                            | machinelearning                         
macie                                    | macie2                                  
managedblockchain                        | marketplace-catalog                     
marketplace-entitlement                  | marketplacecommerceanalytics            
mediaconnect                             | mediaconvert                            
medialive                                | mediapackage                            
mediapackage-vod                         | mediastore                              
mediastore-data                          | mediatailor                             
meteringmarketplace                      | mgh                                     
migrationhub-config                      | mobile                                  
mq                                       | mturk                                   
mwaa                                     | neptune                                 
network-firewall                         | networkmanager                          
opsworks                                 | opsworkscm                              
organizations                            | outposts                                
personalize                              | personalize-events                      
personalize-runtime                      | pi                                      
pinpoint                                 | pinpoint-email                          
pinpoint-sms-voice                       | polly                                   
pricing                                  | qldb                                    
qldb-session                             | quicksight                              
ram                                      | rds                                     
rds-data                                 | redshift                                
redshift-data                            | rekognition                             
resource-groups                          | resourcegroupstaggingapi                
robomaker                                | route53                                 
route53domains                           | route53resolver                         
s3control                                | s3outposts                              
sagemaker                                | sagemaker-a2i-runtime                   
sagemaker-edge                           | sagemaker-featurestore-runtime          
sagemaker-runtime                        | savingsplans                            
schemas                                  | sdb                                     
secretsmanager                           | securityhub                             
serverlessrepo                           | service-quotas                          
servicecatalog                           | servicecatalog-appregistry              
servicediscovery                         | ses                                     
sesv2                                    | shield                                  
signer                                   | sms                                     
sms-voice                                | snowball                                
sns                                      | sqs                                     
ssm                                      | sso                                     
sso-admin                                | sso-oidc                                
stepfunctions                            | storagegateway                          
sts                                      | support                                 
swf                                      | synthetics                              
textract                                 | timestream-query                        
timestream-write                         | transcribe                              
transfer                                 | translate                               
waf                                      | waf-regional                            
wafv2                                    | wellarchitected                         
workdocs                                 | worklink                                
workmail                                 | workmailmessageflow                     
workspaces                               | xray                                    
s3api                                    | s3                                      
configure                                | deploy                                  
configservice                            | opsworks-cm                             
runtime.sagemaker                        | history

The tool uses Amazon's boto3 library which allows for a programatic access to those services in Python. Boto3 in turn uses botocore as a dependency that contains definitions of the services.

How are service endpoints defined in AWS CLI?

Each service has at least a Service Definition File which is written in JSON and contains a few pieces of information:

  • version - version of the service definition file
  • metadata - additional information about interacting with the service
  • operations - what requests can be sent to the service
  • shapes - operations inputs and outputs
  • documentation - pieces of data that are used to form documentation in aws <command> help

This is not the only file available and you can read about more of them in loaders.py docstring. They are not needed in my case.

The SDF file needs to be present in one of the following locations top be used by AWS CLI or boto3:

  • <botocore root>/data/
  • ~/.aws/models
  • A path added to a list in the AWS_DATA_PATH env var

If you're using boto3, you can also use code to add your location to the default loader.

Can I create my own service definition?

I think not. I was actually planning to do that myself but then I have realised that you need to know the exact service API endpoint and version to be able to work with it. Once you do you can observe what your service console page is calling on the api gateway and do the same yourself.

But I guess that's not the use case AWS are after. When the service authors decide their API is ready they can expose it together with the version, they can provide the service definition files for people with access to preview and they can then use the API. If you're in contact with the dev team you can try some pretty-please approach and maybe they will share it with you.

AWS DeepRacer CLI

If you haven't yet seen Don's reInvent 2020 talk "Shift your ML model into overdrive with AWS DeepRacer analysis tools" I recommend that you do. As part of preparation for the talk Don has prepared some updates to log analysis tool provided by AWS. The tool has been there from day one together with DeepRacer workshops. I've looked at this new line:

region = "us-east-1"
dr_client = boto3.client('deepracer', region_name=region,
        endpoint_url="https://deepracer-prod.{}.amazonaws.com".format(region))
models = dr_client.list_models(ModelType="REINFORCEMENT_LEARNING",MaxResults=100)["Models"]

And thought: "cool, we can now use aws cli!". Lesson learned: don't skip half of the code provided when trying to get something to work. Right above this code Done does this:

envroot = os.getcwd()
aws_data_path = set(os.environ.get('AWS_DATA_PATH', '').split(os.pathsep))
aws_data_path.add(os.path.join(envroot, 'models'))
os.environ.update({'AWS_DATA_PATH': os.pathsep.join(aws_data_path)})

And there it was: The service definition file for DeepRacer.
I am pleased to say that I have added the service definition file to deepracer-utils 0.8. If you would like to use it with boto3, simply run:

from deepracer.boto3_enhancer import add_deepracer

client = deepracer_client()

dir(client)

I'm leaving it out there since you can already start playing with it, but expect updates to install it for use in AWS CLI and for some handy use cases, examples and perhaps even docs :)


Originally posted on Code Like A Mother. Hive blog powered by ENGRAVE.