Creating and testing your own Python domain-API with aioflureedb and fsst

In this blog post we want to explore a new feature the Fluree Schema Scenario Tool (fsst) that works closely with a recently new addition to the aiuflureedb Python library for FlureeDB. The features we are going to look at are the aioflureedb domain-API interface and the Fluree Schema Scenario Tool domain-API testing hooks.

As you may know, FlureeDB is a blockchain-based graph database. The aioflureedb Python library is an asynchonous python library for communicating with FlureeDB and the Fluree Schema Scenario Tool is a Python script plus a collection of docker images meant primaraly for compiling and testing a FlureeDB schema and its embedded smart functions.

There can sometimes be a bit of a disconnect between the FlureeQL queries and transactions used in the Fluree Schema Scenario Tool schema and smart-function unit-tests and the queries used in actual client code. One way out of this disconnect is moving the actual FlureeQL transactions and queries out of the Python (and at some later point in time, JavaScript) code. The moving queries out of Python code part is where the aioflureedb feature comes in.

About roles in FlureeDB

With FlureeDB, access control is governed by signing keys. A client communicates with FlureeDB using signed queries and signed transactions. Like in any blockchain, the signing keys are linked to addresses. In FlureeDB these addresses reside in a special collection within the graph database named _auth. The database also knows users, that it stores in its _user collection. A user will usually have one or more _auth records bound to it. And then there are roles. Roles are basically a collection of rules that determine what is permitted and or what is accessible to the client. Both the _user record and the _auth record can have one or more _role record bound to it.

Because a role is a collection of rules that determine what a client can do or see, the role is the most logical place to devide any domain-API up into sub-APIs. After all, it's not all that useful to expose parts of an API to a role only to deny them on access.

Roles to APIs

So here is where things start. We define a mapping from roles to domain API calls, and we distinguish between queries and transactions. We do this by defining a number of JSON files in a roles directory. The names of the files are the names of the roles as defined in our schema.

Lets say we have a role device_master defined in our schema, and we create a file device_master.json in our roles directory.

{
    "transactions": [
        "create_device"
    ],
    "queries": [
        "check_device"
    ]
}

So in this case we have one query and one transaction.

Query and transaction templates

We declared two API methods for the role device_master, but so far there is no definition yet. So let's start off with the query. For that we first define a query template in the query sub directory:

{
  "select": [ 
      {"?device": ["type", "name"]}
  ],
  "where": [
      ["?device", "device/identifier", "::device_id"]
  ]
}

This query template has one template parameter device_id, that will need to be supplied as named API-call argument later on.

The transaction, that we shall define as a JSON file in the transaction subdir works in a similar way:

[
  {
    "_id" : "device",
    "identifier": ["::device_id"],
    "types": [["device_types/id","::device_type"]], 
    "name": "::device_name"
  }
]

Response transformations with jsonata

Sometimes the result we get from FlureeDB on a query isn't quite formatted the way we would like it for a domain-API. In such cases, we can make use of jsonata transforms . To use such transforms, we create an extra file in our query directory, and place a file with the file extention xform to accompany the json template:

{"device_type": $[0].type, "device_name": $[0].name}

Domain-API in aioflureedb

Now that we have our role, our two templates and our transform defined, how do we use the newly defined minimal domain API from python with aioflureedb?

The first thing we need to do is to point aioflureedb to our api map directory and create a domain API object from our database context:

async with db(privkey, addr) as database:
        full_api = aioflureedb.FlureeDomainAPI("./api_maps", database)
        role_api = full_api.get_api_by_role("device_master")

Now we can do queries:

   my_result = await role_api.check_device(device_id="the_main_device")

And transactions:

    transaction = role_api.create_device(device_id="another_device",
                                         device_type="misc",
                                         device_name="Uncle Bob")
    await transaction()

A semi-domain API for testing

So far the directory structure for our api map looks something like this:

• roles
  • <role1>.json
• query
  • <method_1>.json
  • <method_2>.json
  • <method_2>.xform
• transaction
  • <method_3>.json

For the sake of fsst and testing , we are going to extend this a little bit.

• roles
  • <role1>.json
• query
  • <method_1>.json
  • <method_2>.json
  • <method_2>.xform
• transaction
  • <method_3>.json
• fsst
  • roles
    • root.json
    • test.json
  • query
    • <helper_1>.json
    • <helper_2>.json
    • <helper_2>.xform
  • transaction
    • <helper_3>.json
  • tests
    • <role_1>.py

As you can see, the structure of the fsst directory is basicaly the same as for the top level api map directory. There are two predefined roles: test, and root. This structure allows us to define templates and transformations foe a little helper API for use within our unit tests later on.

Testing modules

One directory in the above structure we didn't discuss yet is the fsst/tests directory. This directory contains python files. These python files are role unit test modules for fsst. Our module could maybe look something like this:

class DomainApiTest:
    """Domain-API test class"""
    def __init__(self, test_env):
        """Constructor"""
        self.current = test_env["current"]

    async def run_test_create_device(self, domain_api, test_api):
        """The actual test"""
        transaction = domain_api.create_device(device_id="another_device",
                                         device_type="misc",
                                         device_name="Uncle Bob")
        transaction()
        return True

    async def run_test_check_device(self, domain_api, test_api):
        """The actual test"""
        transaction = domain_api.create_device(device_id="yet_another_device",
                                         device_type="misc",
                                         device_name="Aunt Juanita")
        transaction()
        result = await domain_api.check_device(device_id="yet_another_device")
        if "device_name" in result and result["device_name"] == "Aunt Juanita":
            return True
        else:
            return False, "What happened to aunt Juanita?"

Note that the names of the methods start with run_test_ followed by the name of the API method. There are some other things we can add to the unit test module like pre-tests initialization with root role credentials, but that falls outside of the scope of this overview blog post.

Linking it up with fsst

So far everything was in the API-map directory. But for use in stage tests with fsst, we need to link things up with the fsst fluree_parts directory structure. As you may remember from our previous blog post , every stage or step of a build target in a fluree_parts dir has its own sub directory with a main.json, possibly some clojure files, and optionally a test.json with accompanying subdirs.

Starting with version 0.3 of fsst, you can now also define a file named domain.json in a stage directory, that will link one or more test descriptions between the fluree_parts stage directory and the apimap dir we thus far discussed.

[
  {
      "auth": "$auth00",
      "user": "test_user",
      "auth_roles": ["device_master"],
      "api_role": "device_master",
      "minimal_coverage": 100
  }
]

This file tells fsst to run all of the tests defined for the API role device_master as a user test_user that has an _auth that is linked to the actual device_master role. Again, there are more advanced ways to fill this file, including negative tests, warn-only mode for tests, and running tests with different roles that don't actually match the api map, but again, these fall outside of the scope of this quick overview of the domain-API features.

Closing

I hope the description of the new domain-API feature for both aioflureedb and fsst demonstrates at least part of its potential within a FlureeDB based development environment. The feature would be even more powerful if the apimap directory structure was usable from other languages as well, most importantly JavaScript, but even now, it makes the writing of unit tests quite a bit more powerful.