INTERNAL API

TBD

TBD

TBD

TBD

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API A GraphAssembly represents a small stream processing graph to be executed by the interpreter. Instances of this class **must not** be mutated after construction. The array OriginalAttributes may contain the attribute information of the original atomic module, otherwise it must contain a none (otherwise the enclosing module could not overwrite attributes defined in this array). The arrays Inlets and Outlets correspond to the notion of a *connection* in the GraphInterpreter. Each slot *i* contains the input and output port corresponding to connection *i*. Slots where the graph is not closed (i.e. ports are exposed to the external world) are marked with null values. For example if an input port p is exposed, then Outlets[p] will contain a null. The arrays InletOwners and OutletOwners are lookup tables from a connection id(the index of the slot) to a slot in the Stages array, indicating which stage is the owner of the given input or output port. Slots which would correspond to non-existent stages(where the corresponding port is null since it represents the currently unknown external context) contain the value Boundary. The current assumption by the infrastructure is that the layout of these arrays looks like this: +---------------------------------------+-----------------+ inOwners: | index to stages array | Boundary(-1) | +----------------+----------------------+-----------------+ ins: | exposed inputs | internal connections | nulls | +----------------+----------------------+-----------------+ outs: | nulls | internal connections | exposed outputs | +----------------+----------------------+-----------------+ outOwners: | Boundary(-1) | index to stages array | +----------------+----------------------------------------+ In addition, it is also assumed by the infrastructure that the order of exposed inputs and outputs in the corresponding segments of these arrays matches the exact same order of the ports in the Shape.

INTERNAL API From an external viewpoint, the GraphInterpreter takes an assembly of graph processing stages encoded as a Assembly object and provides facilities to execute and interact with this assembly.

The lifecycle of the Interpreter is roughly the following:

- Boundary logics are attached via AttachDownstreamBoundary(GraphInterpreterConnection, GraphInterpreterDownstreamBoundaryStageLogic) and AttachUpstreamBoundary(GraphInterpreterConnection, GraphInterpreterUpstreamBoundaryStageLogic)

- Init(IMaterializer) is called

- Execute(Int32) is called whenever there is need for execution, providing an upper limit on the processed events

- Finish is called before the interpreter is disposed, preferably after IsCompleted returned true, although in abort cases this is not strictly necessary The Execute(Int32) method of the interpreter accepts an upper bound on the events it will process. After this limit is reached or there are no more pending events to be processed, the call returns. It is possible to inspect if there are unprocessed events left via the IsSuspended method. IsCompleted returns true once all stages reported completion inside the interpreter. The internal architecture of the interpreter is based on the usage of arrays and optimized for reducing allocations on the hot paths. One of the basic abstractions inside the interpreter is the GraphInterpreterConnection. A connection represents an output-input port pair (an analogue for a connected RS Publisher-Subscriber pair). The Connection object contains all the necessary data for the interpreter to pass elements, demand, completion or errors across the Connection.

In particular

- portStates contains a bitfield that tracks the states of the ports (output-input) corresponding to this connection. This bitfield is used to decode the event that is in-flight.

- connectionSlot contains a potential element or exception that accompanies the event encoded in the portStates bitfield

- inHandler contains the InHandler instance that handles the events corresponding to the input port of the connection

- outHandler contains the OutHandler instance that handles the events corresponding to the output port of the connection On top of the Connection table there is an eventQueue, represented as a circular buffer of Connections. The queue contains the Connections that have pending events to be processed. The pending event itself is encoded in the portState bitfield of the Connection. This implies that there can be only one event in flight for a given Connection, which is true in almost all cases, except a complete-after-push or fail-after-push which has to be decoded accordingly. The layout of the portState bitfield is the following: |- state machn.-| Only one bit is hot among these bits 64 32 16 | 8 4 2 1 | +---+---+---|---+---+---+---| | | | | | | | | | | | | | | From the following flags only one is active in any given time. These bits encode | | | | | | | state machine states, and they are "moved" around using XOR masks to keep other bits | | | | | | | intact. | | | | | | | | | | | | | +- InReady: The input port is ready to be pulled | | | | | +----- Pulling: A pull is active, but have not arrived yet (queued) | | | | +--------- Pushing: A push is active, but have not arrived yet (queued) | | | +------------- OutReady: The output port is ready to be pushed | | | | | +----------------- InClosed: The input port is closed and will not receive any events. | | A push might be still in flight which will be then processed first. | +--------------------- OutClosed: The output port is closed and will not receive any events. +------------------------- InFailed: Always set in conjunction with InClosed. Indicates that the close event is a failure Sending an event is usually the following sequence: - An action is requested by a stage logic (push, pull, complete, etc.) - the state machine in portStates is transitioned from a ready state to a pending event - the affected Connection is enqueued Receiving an event is usually the following sequence: - the connection to be processed is dequeued - the type of the event is determined from the bits set on portStates - the state machine in portStates is transitioned to a ready state - using the inHandlers/outHandlers table the corresponding callback is called on the stage logic. Because of the FIFO construction of the queue the interpreter is fair, i.e. a pending event is always executed after a bounded number of other events. This property, together with suspendability means that even infinite cycles can be modeled, or even dissolved (if preempted and a "stealing" external event is injected; for example the non-cycle edge of a balance is pulled, dissolving the original cycle).

INTERNAL API Contains all the necessary information for the GraphInterpreter to be able to implement a connection between an output and input ports.

TBD

Marker object that indicates that a port holds no element since it was already grabbed. The port is still pullable, but there is no more element to grab.

TBD

TBD

INTERNAL API

INTERNAL API

INTERNAL API

TBD

INTERNAL API

INTERNAL API

TBD

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API This source is not reusable, it is only created internally.

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

TBD

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

INTERNAL API

TBD

TBD

INTERNAL API

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD