# Its a Team Effort [platforms]
We noted in the introduction that developing software is a team sport. This is true at multiple levels -- from building libraries to constructing platforms that support thousands of applications and users. In this chapter we turn our attention to the issues involved in the latter and see how **Star**'s features can support this goal.

One of the most robust form of relationships is that between vendor and customer. The reason for this is that it is easy to be clear about the different responsibilities of the parties: the vendor supplies the solution and the customer uses it; and pays for it. This relationship is key to understanding the reason why platforms are compelling and also why it can be hard to execute.

It is easy to see that an applications platform can be viewed as a marketplace where clients can access solutions and providers can offer technologies:

**Applications Platform**
: An _applications platform_ is a system which allows clients and providers to share technologies and to gain efficiencies of scale.

What distinguishes a good platform from a poor one is a combination of the ease with which clients and providers can interact -- and choose alternatives -- and the richness of the offerings: both in terms of the number of potential clients (the size of the market) and the variety of the technologies on offer (the size of the market).

We cannot mandate a large platform here; but we can explore the technical aspects of what makes a platform great.

## Responding to Change
One of the sources of inefficiencies in software development is the effort needed to adopt a technology: be it a library, a service or a programming technique. This becomes especially obvious when some change is necessary; for example, when changing technologies or when responding to a change in requirements.

There are three common sources of change at the macro-level: a need to _reuse_ a software artifact, a need to _repurpose_ the artifact and/or a need to _refactor_ the artifact. The first corresponds to using a software artifact in a similar role but in a different context. For example, when a Math library developed for real-time analytics is to be used for a Machine Learning application: it is still a Math library but its context is new.

Repurposing occurs when the artifact is being used to solve a different problem. For example, a messaging platform might be repurposed as an Applications server (because the technology involved can be helpful).

Refactoring occurs when new requirements mean that new properties are desired. For example, a Json parser might be refactored to make it faster when the scale of the data being processed increases.

Another form of refactoring occurs when the implementation technology changes -- for example if the Json parser needs to be reimplemented in C++ because its original Java implementation 'does not fit' any more.

All these kinds of change often expose assumptions that were implicit before. It is these hidden assumptions that can increase the difficulty and friction in responding to change.

Although it is impossible to completely avoid assumptions -- otherwise known as architectural committments -- the best way of preparing for evolution is to minimize all assumptions and to make dependencies explicit through the judicious use of interfaces. This use of interfaces is the hallmark of our approach to building platforms.

## Boxes and Arrows[boxes-and-arrows]

It is a kind of truism that whenever engineers need to explain their systems to each other they tend to resort to drawing pictures with boxes and arrows between them. For example:[^There are other systems of boxes and arrows that have utility. The most well-known example is probably the quite of diagrams that make up UML.]

![A System for Splitting Orders][BoxNArrows]

[BoxNArrows]:images/boxNarrows.png width=300px
could be used to explain the system for managing the way parts are ordered in a car factory that supported a 'build-to-order' model for manufacturing customized vehicles.

The different boxes show the major sub-systems and the arrows between them show the major flows between them: we have an input **Orders** sub-system that accepts incoming orders for cars, we have a **Split**ter that breaks up the orders into orders for specific assemblies and parts. We also have a **Parts** database that 'knows' how different order requirements translate into specific assemblies. The outputs of the system are orders for parts that will go to individual suppliers **S**~0~ through **S**~n~.

There have been a number of attempts to take the boxes-and-arrows intuition and turn it into something that is more actionable. Here, we take an approach that focuses on boxes representing _components_ that denote areas of responsibility and where arrows represent interactions between the components.

In laying out our platform architecture, we tighten up on one rather important aspect of designing systems: all interactions between components are explicitly identified; and, in keeping with the **Star** style, all interactions are strongly statically typed.

For example, if we zoom in on the **Split** component above, and draw it in isolation we might arrive at:

![The Split Component][SplitComponent]

[SplitComponent]:images/splitComponent.png width=300px

The main new feature in this diagram -- compared to the informal version -- are the different _ports_ that represent the entry and exit points to components. The `SplitComponent` above has three different ports on it: a _responding_ port that handles incoming orders, an _originating_ port that will be connected to a database and a _publishing port_ that will be connected to multiple supplier components. Given this diagramming notation, the order processing system would look something like:

![An Order Processing System][OrderProcessing]

[OrderProcessing]:images/orderProcessing.png width=350px

The main difference between this diagram and the original boxes-and-arrows picture is a slight formalization of the notation: we have crystalized the role of boxes (into processing components) and we have formalized the connections in terms of different kinds of ports.

In addition, we emphasize that the key architectural constraint that we adopt is that *all* communication between components should be explicit. Another way of stating this is that

>wiring between components belongs to the containers in which the components are embedded.

Other aspects of platforms which would be necessary to complete the picture include the market place itself: how can clients select components for use in their applications and how can providers make their offerings known? We will come back to these questions below.

It turns out that, with some support from **Star**, we can turn diagrams like these into executable code that can be deployed over distributed systems. Let us see how we might go about this...

### Ports and Speech[ports-and-speech]

First, let us take a slightly deeper look at the anatomy of a port. Ports are intended to represent the points of connectivity of a component: in effect, they form the gateways into and out of the component. By restricting ourselves to components that only interact via their ports we foster re-usability of components and re-purposability (sic) of code.

>This is an important point: unlike most programming languages, and unlike **Star** itself, our diagramming notation does not rely on _scoping_. Instead, all connections and references are explicit. This is a strong constraint that helps to enforce so-called _loose coupling_ between components. This greatly simplifies the kinds of task of assembling applications from components.

Ports denote types as well as data flow: each port is associated with an API _schema_ that determines the type of data that is going through the port. For example, the responding port of the `SplitComponent` has a schema associated with it that shows that the component expects `Order`s coming in:

![An Order Port][OrderPort]

[OrderPort]:images/orderPort.png width=350px

The originating port will be connected to a data source component. It's type schema declares what kind of data it is looking for:

![The Parts DB Port][dbPort]

[dbPort]:images/dbPort.png width=350px

It should come as no surprise at this point that we also declare that ports are intimately associated with speech actions: communication between components is mediated by speech actions and ports codify both the sender/receiver relationship and the type of communication.

Notice that we are speaking in terms of _originating_ and _responding_ here. This reflects the fact that in any given speech action we have an originating speaker and a responding listener. We have deliberately avoided using terms such as input or output here because connections between components can involve _both_ flows of information but only one side ever initiates the action. For example, the `SplitComponent` will raise a query to the `PartsDB` component: the query itself involves sending the query in one direction and receiving the data from the `PartsDB` component in the other.

Our final kind of port is a specialization of the originating port: the _publishing port_ as in:

![The Supplier Port][supplierPort]

[supplierPort]:images/supplierPort.png width=360px

The difference between a normal originating port and a publishing port is two-fold: it represents a one-to-many fan-out -- i.e., it may be connected to an arbitrary number of other components -- and it also has an additional feature: the _discrimination function_ that will allow the `SplitComponent` internally to 'select' the right supplier to send parts orders to.

The net effect of this is that we have a diagramming notation that supports a high-level modeling of applications that is intuitive to many software engineers. Furthermore, we can support type safety of communication between elements of the application.

>This notation is not quite a complete 'spanning set' if the end goal is to construct a platform for building a variety of distributed applications. However, it is the kernel of such a set and can easily form the basis of a complete distributed applications platform.

### A Component in Star[a-component-in-Star]

One of the intentions behind boxes-and-arrows diagrams is to call out the major functional pieces of an application.[^By _functional_ we mean _important to the solution_ not necessarily as in functional programming.] The intuition is that individual components have a specific role in the application; but that they are typically 'quite large'. We have already seen that a component may have multiple ports but we have not exposed what kind of computation may be going on inside.

One of the non-goals of boxes-and-arrows diagrams is to be a complete programming language. Instead, the idea is to capture the large scale granularity in a picture but to use text for the actual programming of components. Actually building components is best left to 'real' code; in our case, we denote the code of a component as a **Star** `component`; which is a special form of `package`.

>The one exception to the strategy of using written code to build components is with _composite components_ -- components built by assembling and wiring other components. However, this is outside the scope of this book.

The code for a component must implement the various ports that the component has on the diagram and must also implement the functionality of the component. For example, our `SplitComponent` may start it's implementation with:

```
import boxesNarrows
splitComponent is component{
  port incoming is respond{
    on O on order do
      processOrder(O)
  }
  port suppliers is publish{
    discriminator:(string) => port of {
      order:occurrence of Order
    }
  }
  port parts is originate{
    assembly:set of (part,list[part])
    supplier:set of (part,list[supplier])
  }
  ...
}
```

The ports that are surfaced in the order processing diagram are also represented in the code for the `splitComponent`. Responding ports have a body that essentially the same as for an actor (or concurrent actor); typically most of the activity within a component is initiated by code that is present in or referred from responding ports.

Notice that, unlike regular actors, outgoing speech actions are also strongly indicated. Any outgoing speech actions must be 'applied' to one of the originating ports. For example, part of processing a parts order will be a query to the `partsDB` component; but the `partsDB` component is not explicitly identified in the `splitComponent`. Instead, the query is directed to the `parts` originating port:

```
fun assemblySuppliers(A) is
  query parts with list of { unique S where
      (A,Ps) in assembly and P in Ps and
      (P,Ss) in supplier and
      S in Ss
    }
```

This function can be used by the order processing code to query the parts database for all the suppliers involved in a given assembly. It works by formulating the appropriate `query` speech action to the originating `parts` port.

>The `unique` keyword here implies that the result will have duplicates eliminated from it.

Using a publishing port is slightly more complex than using a regular originating port. Since there may be any number of components attached to a publishing port we must first of all select which one we want to address our speech action to. We do this using the embedded _discriminator function_ that is part of the publishing port. For example, to place a parts order for the supplier "Alpha Wheels Inc."" we perform:

```
prc placeOrder(S,parts) do
  notify suppliers.discriminator(S.name) with
    order{ content=parts } on order
```

Notice that this is a regular speech action; the primary difference is that instead of a fixed recipient we compute who the recipient will be -- based on the name of the supplier.

### Re-purposing Components[re-purposing-components]

One of the distinguishing features of the boxes-and-arrows diagram is that it highlights the major sources of input and output in the system. This is in marked contrast with most programming languages where the I/O functionality is buried deep within the code itself. For example, our `splitComponent` does not directly communicate with a database; instead it poses a `query` to the `partsDB` component. Similarly, the components for recovering incoming orders and sending out orders to suppliers are mostly about implementing the appropriate I/O operations.

The `partsDB` component is interesting for another reason: it can be modeled as an adapter to a normal database engine. In fact, the `partsDB` is quite an interesting component.

Consider the problem of the `partsDB` component: if we are to build this component in a robust fashion then it must respond to arbitrary queries about the two `relation`s that are exposed in its responding port. However, we assume that the data it uses to answer those queries is not 'in' the component itself but is stored in some actual database; perhaps an SQL database.

In order to see how we can achieve this we need to look a little deeper into how speech actions are actually represented. Recall that we stated that a speech action requires an entity that implements the `speech` contract. That contract, in simplified form, is:

```
contract speech over t determines a is {
  _query:for all s ~~
    (t,(a)=>s,()=>quoted,()=>dictionary of (string,quoted))=>s
  _request has type
    (t,(a)=>(),()=>quoted,()=>dictionary of (string,quoted))=>()
  _notify:(t,(a)=>())=>()
}
```

>The actual contract is somewhat more involved and involves the use of the `execution` contract -- which is part of the support for **Star**'s concurrency features. However, this variant is sufficient for us to expose the required issues.

The salient element here is the entry for `_query`. A speech action like:

```
query parts with list of { unique S where
      (A,Ps) in assembly and P in Ps and
      (P,Ss) in supplier and
      S in Ss
    }
```

is translated -- by a standard built-in macro processor -- into a call to `_query` of the form:

```
_query(parts,
  (Ax)=>list of { unique S where
    (A,Ps) in Ax.assembly and P in Ps and
    (P,Ss) in Ax.supplier and S in Ss },
  ()=><|list of { unique S where
      (A,Ps) in assembly and P in Ps and
      (P,Ss) in supplier and
      S in Ss
    }|>,
  ()=>dictionary of ["A"->A as quoted]
)
```

This is kind of complicated to follow at first; but is quite straightforward if taken one step at a time.

* The first argument to `_query` is the entity being queried with the `query` speech action. The structure of the `speech` contract allows a responder to a speech action to 'pass on' the action to another responder.

* The second argument is the expression that must be evaluated by the actual responding entity. It is encapsulated in a single argument function -- the argument being a record that implements the speech action API. Evaluating this function in the appropriate context has the effect of computing the response to the `query` speech action -- and the result of the function is returned as the value of the `query` speech action itself.

* The third and final arguments are used when the responder cannot or does not wish to use the 'compiled query'. The third argument is a function that returns the original 'text' of the query -- as a `quoted` value. The fourth argument is also a function, one that returns the values of any _free_ variables in the `query` -- variables whose values are determined by the context of the speech action itself. In this case there is only one free variable -- `A` -- which was the part that our `splitComponent` needed to find suppliers for.

This arrangement allows for two kinds of speech action processing: if the respondent trusts the originator of the speech action then a very rapid response to the `query` is possible -- by evaluating the embedded function. However, in the case where the respondent does not want to simply execute the query function, or it cannot, then the respondent has access to the original text of the query -- together with the values of any free variables that appear in the `query`.

In the case of the `partsDB` component it cannot simply trust the `query`; even if it wanted to. This is because the query must be mapped into a form that the attached SQL database can understand. In effect, the query must be translated from **Star**'s query language to SQL.

![The PartsDB Component][partsDB]

[partsDB]:images/dbComponent.png width=330px

I.e., the query expression is translated by the `partsDB` component into SQL:

```
select S from table assembly as As, supplier as Sp, Ps, Ss where
  As.part="Alloy Wheel" and Ps.partno=As.partno and
  Ps.supplier = Sp.id and
  Ps.supplier=Ss.id and Ss.supplier=S
```

The details of how this translation are achieved are beyond the scope of this book; but it involves similar techniques to those we saw in [our chapter on DSLs][application-policy-mechanism].

One important point to note here is that the machinery for translating queries into SQL is quite general; and not at all restricted to our parts database. In fact, if we provide the actual database component with a URL of the database we are interested in, the DSL processor is able to dynamically inspect the actual database, construct the appropriate responder port for the `partsDB` component as well as being able to answer queries by mapping them to SQL.

As a result, we can construct a general purpose adapter component that is able to be used for _any_ database, not just the one we are using in this application. The result of which is that the application is populated with a mixture of standard components that are configured and specifically written components that implement the specific functionality of the application.

### Wiring up Boxes and Arrows[wiring-up-boxes-and-arrows]

We started this section with a graphical depiction of an application as boxes and arrows between them. However, not many computers can execute boxes, and so if we want to run the application we have to construct a complete _written_ program that represents the _drawn_ diagram.

The written form of our car part sourcing application is not that hard to follow, given the material we have covered so far:

```
import boxesNarrows
partSource is application{
  def ordersIn is import ordersInComponent
  def split is import splitComponent
  def db is import dbComponent
  import supplier

  ordersIn.out connect to split.incoming
  split.parts connect to db.queryIn
  split.suppliers publish to {
    def megaWheel is supplier("MegaWheel")
    ...
    def discriminator("Mega") is megaWheel
    ...
  }
}
```

This sketch shows how we can construct a written version of the boxes-and-arrows diagram in a way that lends itself to executable code.

### There is more to a platform than this

We have actually just scratched the surface of the potential of this kind of programming platform. In truth, like many of the chapters in this book, a full treatment of a boxes-and-arrows platform would justify a book in its own right.

Other aspects that we have omitted include composite components, component templates, wiring diagrams as **Star** programs, the dynamic behavior or components and so on.
