TUTORIAL: Using the Splinterlands API and KNIME to determine what cards you require to complete your untamed collection

About me

I've been playing Splinterlands since 2018.

I love data. I love crunching it. I love looking at it, storing it, and making decisions based on it. In my day job, I use a tool called KNIME which is an analytics tool that can transform and crunch data in any way shape or form. KNIME is freely downloadable, and available on all major operating systems.

About this tutorial

This tutorial is lengthy and advanced in nature. If you're a developer, you likely won't gain anything from this. If, however, you are an Excel junkie and have not yet graduated into the realm of data science and analytics, KNIME is a tool that can certainly take you to the next level. You'll learn about KNIME basics and apply it to something fun - Splinterlands cards.

Using Knime

Let's begin by downloading knime.

Once it is downloaded, install the software.

Run the software, set a directory to use as Knime's "scratch", and you're ready to go.

Knime has a variety of Windows. The top left is your explorer, where different KNIME flows live in your working directory. Here is also where you will find example analytical models from the community to learn how to do various things in KNIME.

The next window down is a workflow coach which helps you out with what sort of nodes can be linked together.

What are nodes? Nodes are blocks that perform operations on your data. They may modify it, search for something, filter, or do any of the other common operations you may want to do when working with large, small, or unfamiliar data sets. You can also perform a number of standard SQL functions like joins, which are incredibly useful.

The Node Library has hundreds of plug ins, and you can even drop R, Python, or JavaScript snippets into KNIME to perform operations for which there may not be a node. You've then got the convas in the middle of the screen, which is where you will do most of the work in KNIME.

KNIME sees you link a series of nodes to perform operations or analysis on data, and the processed result is then accessible at the end of the flow. For the rest of this tutorial, I will explain what I have created which can be built upon further in the future. The creation I've produced is indeed a minimum viable product, and just one of the many applications you could perform on Splinterlands card data.

Building the analytical flow

In order to determine the answer to the query: "How many regular foil cards do I require to max out my Untamed deck?" we need to know a number things.

1. What cards do I as a player have?
2. What cards exist in Untamed?
3. What is the number of cards required to max out my deck?
4. What is the difference between what I have and the max level cards?

It is important to know everything you need before starting any sort of analysis work - or you may have to start from scratch all over again. In my day job I make sure that parameters and scope is set before I start my research in order to avoid complexities in future iterations of reporting or analysis which means prior data sets are invalidated by iterative improvement (or changing) business requirements.

Here, we've got a fairly flat set of requirements - so let's get started.

Thanks to @kiokizz's excellent Splinterlands API documentation, a lot of the information can be extracted direct from the game's API.

What cards do I have as a player?

This is the first part of the puzzle. The Splinterlands API responds using a data format called JSON. Thankfully, Knime can import JSON data structures as well as CSV, Excel files, or even connect directly to a SQL database!

I create a JSON Reader Node by searching for JSON reader in the node repository window, and dragging it onto the canvas. I then right click to configure the node.

Right clicking the node gets me:

The next step is decoding and tabulating the JSON data to individual rows that can then be analysed. Otherwise, the response is a lengthy single table cell which lists all the individual cards I have:

Now, I don't need all the information about the card in order to answer my query - so I create a list of JSON paths (ie the variable structures) that I want to focus on. Think of this as a filter to only grab the variables and information that are relevant to the question I'm asking - which is "How many cards do I need to max out my Untamed regular foil collection?"

I need to know the following:

• uid

This is the unique ID for a card.

• card_detail_ids

This is the "serial number" for the card - it will tell me what number card it is - more on this later.

• golds

a boolean value that identifies if the card is a regular, or gold foil

• xps

The XP (how many cards) have been combined under the uid (or, as we know it, bcx)

• players

Who owns the card

• levels

What level the card is at currently

• editions

What edition the card is - Alpha, Beta, Untamed, Reward, Dice, Orbs, etc

The next operation in Knime is to "Ungroup" the JSON data of my card collection, so I get a list of each and every card that I have.

This gives me a table structure with each and every card I have. The UID is the card's unique "serial" - or the underlying NFT number. The card_detail_ids tells us what card it is - then if its gold, how much xp, who owns it, what level it is, and from which edition the card came...

But - it isn't very human readable. For that, we're going to need to use a really cool Knime feature called the Rule engine. Because I can go build a reference table to find out what edition a card is, I can make that "Editions" Column something that is very human readable.

This results in the previous table having a new column added:

However, we still don't know what card is what just by looking. This is where the card_id field will come in handy - but I certainly don't want to go and build a massive table listing each and every card's name, splinter, and other variables, when that already exists in the Splinterlands game API - I just need to pull it using a different part of the API documentation.

This is everything we've achieved so far:

Now - we'll look at the second part - getting the details of the cards from the API, which is the second line of the screenshot above.

What are the card details?

To get the card details, we need to make a call to the Splinterlands API.

Using the JSON Reader node as we did to get a user's card details, we simply make a different call to the game API:

The subsequent nodes transform the data into a table that looks a bit more useful...

For some reason, some of the number based variables came through as Strings - so I need to convert them to a number in order to further categorise the output, and make the table a bit more human readable.

Determining Rarity:

Determining the Splinter:

Next is where the major magic happens - a JOIN.

Joining is a function of SQL and data science / analytics where you take two disparate sets of data with a field that allows an association. In a list of people - this might be their full name. So, you might have a list of full names, and a list of drivers licences and full names, along with a date of birth. You might have another table that has the driver's licence number and the person's traffic infringement history.

By joining the driver's licence, and full name from the people table to the "fines" table, using the driver's licence number - you can enrich that data to learn about Bob's speeding tickets from the past.

Except here, we're wanting to take all the lovely details we have about Goblin Shamans, Hydras and Tyrus into our own card collection.

The top table is my card collection, as pulled from the API. The bottom table is the card details, as pulled from the API. By joining, or matching on the card details, I get an enriched data set of the cards I have.

You can see here that there's now card names, edition details, and a whole bunch more. However, this only works for the cards that I DO HAVE. If I'm missing anything from Untamed, it won't appear in this list. In order to figure this out, I need to calculate the difference - or, the cards that DIDNT match - where there's a card I don't have in my collection at all.

This doesn't really have a name for it, but I like how KNIME calls it an "Anti-Join" - this is the data from the secondary table that wasn't matched. I sort of build a cache to store this infomration in temporarily.

I then split out any gold / non gold cards,after only looking for untamed cards in the collection. That is the "Rule based Row Filter node"

The next step is to COUNT the individual cards I have. This is to determine how many of EACH Untamed card I have, and, if they're combined, how much XP (BCX) they have - to determine further calculations which will enable us to work out how many cards I need to buy to max out my particular card.

This is achieved by the GROUP BY node, which works much like a Pivot Table in Microsoft Excel, except it has dozens of powerful functions and features, and has the added advantage of processing in a fraction of the time.

I want to group by the individual card.

Then, I want to injet some meaning into the card...

I want a SUM of the xps - and the rarity and splinter details, so that I can sort it all later, that's the next tab in the Group By configuration.

The resulting table is as such:

A list of cards, and the XP, which splinter they belong to, and their rarity. Though these may be a combination of individual (1 BCX) and some combined cards - that doesn't matter, as the only thing I'm interested in is how many more BCX do I need to complete a max instance of an individual card.

The next step is to bring in some figures about what it takes to get a card to max level in the Untamed Edition.

I do this by using a subsequent Rule Engine, which explains the max levels:

Now of course, as I indiated earlier, this will only work for cards that are in my collection. For those that are not (those in the so called Anti-Join) - I can make the assumption that my owned quantity is zero, and that I therefore need the maximum amount to get them to the max level.

To find what cards I'm missing, I then concatenate the joined cards and the "Anti-Joined" cards (ie, the missing ones, into one tabulated list)

The output is a bit messy, so I use a few nodes to clean up the data set prior to completing my final calculations...

You can also see at the bottom of this output exactly which UNTAMED cards I am missing in my collection!

The final clean up is as follows - and I've documented it in my KNIME flow itself, as these are rather simple, explanatory nodes with basic configuration - up until the Math Formula Node. \

The Math Formula node is where this all comes together, to generate the final tables and calculations of what will be required to complete the collection at Max Level.

Here's the result of that formula:

I then tidy up that data to make it even more human readable....

From here, I have a list of what I need to complete a Max Level Untamed Collection.

And I'm all done!

If you want me to calculate how many UNTAMED cards you need to MAX OUT your collection, please upvote and leave your username below. When I get some time, I'll generate a report for you!

Future improvements to the above flow can be made by incorporating current market prices, or looking at historical card versions in much the same way - I just need to build the logic. I've got all the cards I (probably) want from prior versions and editions, so I am completely unmotivated to do the work on that currently...

I hope to see you all on the battlefield!

TLDR: Want me to generate a list of cards you need to max out your Untamed REGULAR FOIL collection?

Drop your in-game username below, and please feel free to upvote or leave me a tip!