This is the 5th post in this series on the 0.3 version of the Merg-E language spec. The previous posts are available here.

• part 1 : coding style, files, merging, scoping, name resolution and synchronisation
• part 2 : reverse markdown for documentation
• part 3 : Actors and pools.
• part 4 : Semantic locks, blockers, continuation points and hazardous blockers

For info on the context of the why of Merg-E and how it fits into the bigger picture of the Innuendo Stack :

• My Insane 20-Project Pet Stack : An outline of my blue-sky Innuendo Stack

In this post I want to look at decomposition and composition and how these fit into the DAG oriented design of authority flow in Merg-E.

Semantic lexing

I made a tiny start at a bit of experimental partial implementation for a lexer for Merg-E, the codeberg repo now contains some preliminary work on a Semantic Lexer for Merg-E. A semantic lexer is a bit more than a regular lexer. Because of the design choices to give Merg-E only one true key-word and to make name resolution flexible, the lexer does a number of tasks that would normally be for the parser to handle. Much of the authority propagation and implicit and explicit capturing in Merg-E happens through explicit (authority and mutable vars through explicit captures and app / ns arguments) or implicit (constants) closure capturing, aliasing (through as as or alias) and (as we will discuss next) pruning and enting. So a token that is valid at one level in a Merg-E program, may either not be at a deeper level, or might be reachable through a different identifier.

The resolve_order operation makes this extra flexible for the user, but makes the semantic lexer even more involved.

resolve_order lang.Flow lang.Type lang.Share lang imported export;

The first milestone we hope to achieve is creating a least-authority semantic Lexer, that is, a lexer that combines the explicit delegation and name resolution parts of what would usually be done by the parser. For a simple language, the least authority yet flexible name resolution is probably one of the more challenging part of the Merg-E language design, a part that we aim to put into the lexer for the language in order to lay a solid foundation for a least authority DSL. The semantic lexer needs to do just enough parsing to allow the lexer to keep track of the name resolution deeply enough to not lose track of what lives where, within what execution context, and what is and isn't still reachable in any given execution scope.

Authority/language/data as a DAG

In Merg-E all ambient authority and most of the entire language is defined in a tree adjacent Directional Acyclic Graph or DAG. The available DAG always starts of at the scope anchor. The semantic lexer defined above keeps track of the scope anchored DAG between execution contexts and their direct child, pruning the authority- and/or language- and/or data-tree and in cases grafting pruned sub trees onto an alternate tree living at another level.

The base structure of the scope rooted DAGs is a tree, but it's not a pure tree. The concept of an alias allows some minor deviation from a pure tree shaped structure, but in order to guarantee the directional nature of the DAG, an alias must always point down from its direct parent and is not allowed to directly point to another alias. So no aliases for aliases. There is only one exception to that rule, but that exception lives outside of the tree structure. That exception is operators.

Some examples of aliases in the current 0.3 draft language spec of the lang subtree:

The lang subtree is only a part of the tree that has its root in scope, and it is usually grafted under scope.imported.lang, just as ambient is in source-file scope, application scope and module/namespace scope, but especially for ambient, but also possibly for lang, this can easily be replaced either by a pruned version of lang and ambient, or by a small collection of sub branches.

When constants, functions, and variables are defined or merged or turned into actors, these will populate the scope.export part of the tree. The language definition sets the rule of how different parts of scope.imported and scope.export will end up in the scopes of lower level execution contexts.

Then resolve_order will determine how name tokens used in a Merg-E execution context will be located within the scope tree.

prune, ent and alias

Pruning and grafting aren't just internal to the Merg-E semantic lexer and the Merg-E runtime. The functionality to prune and graft is available to the user.

  mutable dag version = prune scope.imported.lang.Version
  ent scope.export version as Version;
  prune scope.imported.lang.Amplify;

In the above example we prune the Version branch of the lang tree and we assign it to the DAG named version. We than ent (graft) the version branch under scope.export under the name Version. Finally we prune lang.Amplify and don't assign the resulting DAG to anything, so no downstream execution scope should be able to use any authority amplification language features.

Not that pruning and enting happens lazy when it comes to name resolution. The idea is that if you are about to shoot yourself in the foot, you can still fix it in ways, for example dangling aliases will persist and van be made to point to something again before deeper contexts get activated with the reconfigured offspring scope tree.

Note that the grafting operation in Merg-E is called ent, though an alias graft is available for American users and users who prefer graft over ent.

As long as a subtree hasn't been attenuated to disallow it, it can be extended with an alias. Consider scope.export as your default playing ground for things like this :

  alias scope.export.language_version = scope.export.Version.major;

Now that we have seen how you can use graft end ent explicitly, it should prove useful to realize how for example the following line:

app myApp lang ambient.Io as io {

The part 'ambient.Io as io' part of this line basically makes this line mean the same as this would:

shared mutable dag io = prune scope.exported.ambient.Io;
app myApp lang io {

So here ambient won't be available in application scope, only io will.

We can take this further towards the level one deeper:

use io {
    cout as cout;
    cerr as cerr;
    endl as endl;
    };

  mutable main ()::{
    cout as cout;
    cerr as cerr;
    endl as endl;
    }{
      ...

Here we see how in two steps the provided ambient authority is decomposed and delegated down even further. The use operation basically does this:

shared mutable dag cout = prune io.cout;
ent scope.export cout as cout;
shared mutable dag cerr = prune io.cerr;
ent scope.export cerr as cerr;

Then the explicit capture basically gives us:

MAINCONTEXT.scope.inported.cout = deepcopy APPCONTEXT.scope.export.cout;
MAINCONTEXT.scope.inported.cerr = deepcopy APPCONTEXT.scope.export.cerr;
MAINCONTEXT.scope.inported.endl = deepcopy APPCONTEXT.scope.export.endl;

where the explicit capture of non static non authority-free parts of scope.export to the scope.imported of the child context is implemented by deep copies.

Coming up

In this post we looked at the importance of DAGs in the language design of Merg-E. How semantic lexing is needed for Merg-E because of its single keyword + DAG design and its flexible and powerful name resolution. We learned about prune, ent (aka graft) and alias and how these concepts work in both user space as well as in the abstractions that Merg-E provides. I'll need a few more posts to talk about things like data-frames, freezing, attenuation, parallelism models, operators, capability patterns, and a few more. In the next post I hope to write about how Merg-E uses the dataframe as primary container for immutable structured data, and how dataframes fit into Merg-E's parallel asynchronous processing design, and why this justifies using the parquet data and file format as a prime citizen in the Merg-E language.