An overview of Tangence ======================= Tangence is all of the following: 1. A single server/multiple client protocol for sharing information about objects. 2. A data model - it defines the types of values that are transmitted between the server and clients. 3. An object model - it defines the abstract look-and-feel of objects that are visible in the server end, and the proxies to them that exist in the client ends. 4. A wire protocol - it defines the bits down the wire of some stream. 5. A collection of Perl modules (a Perl distribution) which implements all of the above. These writings may sometimes suffer the "Java problem"; the problem of the same name being applied to too many different concepts. I'll try to make the context or wording clear to minimise confusions. 1. Server/Client ---------------- In a Tangence system, one program is distinct in being the server. It is the program that hosts the actual objects being considered. It is also the program that holds the networking socket to which the clients connect. The other programs are all clients, which connect to the server. While each client is notionally distinct, they all share access to the same objects within the server. The clients are not directly aware of each other's existence, though each one's effects on the system may be visible to the others as a result of calling methods or altering properties on the objects. Internally, the clients will use proxy objects through which to access the objects in the server. There will be a one-to-one correspondance between server objects and client proxies. Not every server object needs to have a corresponding proxy in every client - proxies are created lazily when they are required. 2. Data Model ------------- Whenever a value is sent across the connection between the server and a client, that value has a fixed type. The underlying streaming layer recognises the following fundamental types of values. Each type has a string to identify call it, called the signature. These are used by introspection data; see later. * Booleans Uses the type signature "bool". * Integers, both signed and unsigned, in 8, 16, 32 and 64bit lengths An integer of unspecified size uses the type signature "int". Specific sized integers use the type signatures "s8", "s16", "s32", "s64", "u8", "u16", "u32", "u64" * Floating-point numbers, in 16, 32 and 64bit lengths A float of unspecified size uses the type signature "float". Specific sized floats use the type signatures "float16", "float32", "float64" Note that the Intel-specific 80bit "extended double" format is not supported * Unicode strings Uses the type signature "str". * References to Tangence objects Uses the type signature "obj". * Lists of values Uses the type signature "list(T)" where T is the type signature of its element type. * Dictionaries of (string) named keys to values Uses the type signature "dict(T)" where T is the type signature of its element type. * Structured records of values Uses a type signature giving the name of the structure type. * For type signatures, there is also the type of "any", which allows any type. As Tangence is primarily an interprocess-communication layer, its main focus is that of communication. The Data Model applies transiently, to data as it is in transit between the server and a client. A consequence here is that it only considers the surface value of the types of data, rather than any deeper significance. It does not preserve self-referential data, nor can it cope with cyclic structures. More complex shaped data should be represented by real Tangence objects. 3. Object Model --------------- In Tangence, the primary item of interaction is an object. Tangence objects exist in the server, most likely bearing at least some relationship to some native objects in the server implementation (though if and when the occasion ever arises that a C program can host a Tangence server, obviously this association will be somewhat looser). In the server, two special objects exist - one is the Root object, the other is the Repository. These are the only two well-known objects that the client knows always exist. All the other objects are initially accessed via these. The client(s) interact with the server almost entirely by performing operations on objects. When the client connects to the server, two special object proxies are constructed in the client, to represent the Root and Repository objects. These are the base through which all the other interactions are performed. Other object proxies may only be obtained by the return values of methods on existing objects, arguments passed in events from them, or retrieved as the value of properties on objects. Each object is an instance of some particular class. The class provides all of the typing information for that instance. Principly, that class defines a name, and the collection of methods, events, and properties that exist on instances of that class. Each class may also name other classes as parents; recursively merging the interface of all those named. Tangence concerns itself with the interface of and ways to interact with the objects in the server, and not with any ways in which the objects themselves are actually implemented. The class inheritance therefore only applies to the interface, and does not directly relate to any implementation behaviour the server might implement. 3.1. Methods Each object class may define named methods that clients can invoke on objects in the server. Each method has: + a name + argument types + a return type The arguments to a method are positional. The return is a single value (not a list of values, such as Perl could represent). Methods on objects in the server may be invoked by clients. Once a method is invoked by a client, the client must wait until it returns before it can send any other request to the server. 3.2 Events Each object class may define named events that objects may emit. Each method has: + a name + argument types Like methods, the arguments to an event are positional. Events do not have return types, as they are simple notifications from the server to the client, to inform them that some event happened. Clients are not automatically informed of every event on every object. Instead, the client must specifically register interest in specific events on specific objects. 3.3 Properties Each object class may define named properties that the object has. Each object in the class will have a value for the property. Each property has: + a name + a dimension - scalar, queue, array, hash or object set + a type + a boolean indicating if it is "smashed" Properties do not have arguments. A client can request the current value of a property on an object, or set a new value. It can also register an interest in the property, where the server will inform the client of changes to the value. Each property has a dimension; one of scalar, queue, array, hash, or object set. The behaviour of each type of property is: 3.3.1 Scalar Properties The property is a single atomic scalar. It is set atomically by the server, and may be queried. 3.3.2 Queue and Array Properties The property is a contiguous array of individual elements. Each element is indexed by a non-negative integer. The property type gives the type of each element in the array. These properties differ in the types of operations they can support. Queues do not support splice or move operations, arrays do. 3.3.3 Hash Properties The property is an association between string and values. Each element is uniquely indexed by a null-terminated string. The property type gives the type of each element in the hash. The elements do not have an inherent ordering and are indexed by unique strings. 3.3.4 Object Set Properties The property is an unordered collection of Tangence objects. Scalar properties have a single atomic value. If it changes, the client is informed of the entire new value, even if its type indicates it to be a list or dictionary type. For non-scalar properties, the value of each element in the collection is set individually by the server. Elements can be changed, added or removed. Changes to individual elements can be sent to the clients independently of the others. Certain properties may be deemed by the application to be important enough for all clients to be aware of all of the time (such as a name or other key item of information). These properties are called "smashed properties". When the server first sends a new object to a client, the object construction message will also contain initial values of these properties. The client will be automatically informed of any changes to these properties when they change, as if the client had specifically requested to be informed. When the object is sent to a new client, it is said to be "smashed"; the initial values of these automatic properties are called "smash values". [There are issues here that need resolving to move Tangence out from being Perl-specific into a more general-purpose layer - more on this in a later email]. 4. Wire Protocol ---------------- The wire protocol used by Tangence operates over a reliable stream. This stream may be provided by a TCP socket, UNIX local socket, or even the STDIN/STDOUT pipe pair of an SSH connection. The following message descriptions all use the symbolic constant names from the Tangence::Constants perl module, to be more readable. 4.1. Messages At its lowest level, the wire protocol consists of a pair of endpoints to the stream, each sending and receiving messages to its peer. The protocol at this level is symmetric between the client and the server. It consists of messages that are either reqests or responses. An endpoint sends a request, which the peer must then respond to. Each request has exactly one response. The requests and responses are paired sequentially in a pipeline fashion. The two endpoints are distinct from each other, in that there is no requirement for a peer to respond to an outstanding request it has received before sending a new request of its own. There is also no requirement to wait on the response to a request it has sent, before sending another. The basic message format is a binary exchange of messages in the following format: Code: 1 byte integer Length: 4 bytes integer, big-endian Payload: n bytes The code is a single byte which defines the message type. The collection of message types is given below. The length is a big-endian 4 byte integer which gives the size of the message payload, excluding this header. Thus, the length of the entire message will always be 5 bytes more. The data payload of the message is encoded in the data serialisation scheme given below. Each argument to the message is encoded as a single serialisation item. For message types with a variable number of arguments, the length of the message itself defines the number of arguments given. The stream protocol is designed to be used in situations where the CPU power of each endpoint is high, but the connection in between may have high latency, or low bandwidth. It is therefore optimised in favour of roundtrips and byte count overhead, at the expense of processing power needed to encode or decode it. One consequence here is that no attempt is made to align multi-byte values. 4.2. Data Serialisation The data serialisation format applies recursively down a data structure tree. Each node in structure is either a string, an object reference, or a list or dictionary of other values. The serialised bytes encode the tree structure recursively. Other types of entry also exist in the serialised stream, which carry metadata about the types, such as object classes and instances. The encoding of each node in the data structure consists of a type, a size, and the actual data payload. The type and size of a node are encoded in its leader byte (or bytes). The top three bits of the first byte determines the type: Type Bits Description DATA_NUMBER 0 0 0 t t t t t numeric where 'ttttt' gives the number subtype DATA_STRING 0 0 1 s s s s s string DATA_LIST 0 1 0 s s s s s list of values DATA_DICT 0 1 1 s s s s s dictionary of string->value DATA_OBJECT 1 0 0 s s s s s Tangence object reference DATA_RECORD 1 0 1 s s s s s structured record where 'sssss' gives the size DATA_META 1 1 1 n n n n n where 'nnnnn' gives the metadata type For numbers, the lower five bits encode the numeric type, which defines how many more bytes will be used Subtype Subtype bits Extra bytes Description DATANUM_BOOLFALSE 0 0 0 0 0 0 Boolean false DATANUM_BOOLTRUE 0 0 0 0 1 0 Boolean true DATANUM_UINT8 0 0 0 1 0 1 Unsigned 8bit DATANUM_SINT8 0 0 0 1 1 1 Signed 8bit DATANUM_UINT16 0 0 1 0 0 2 Unsigned 16bit DATANUM_SINT16 0 0 1 0 1 2 Signed 16bit DATANUM_UINT32 0 0 1 1 0 4 Unsigned 32bit DATANUM_SINT32 0 0 1 1 1 4 Signed 32bit DATANUM_UINT64 0 1 0 0 0 8 Unsigned 64bit DATANUM_SINT64 0 1 0 0 1 8 Signed 64bit DATANUM_FLOAT16 1 0 0 0 0 2 Floating 16bit DATANUM_FLOAT32 1 0 0 0 1 4 Floating 32bit DATANUM_FLOAT64 1 0 0 1 0 8 Floating 64bit All multi-byte integers are always stored in big-endian form. Floating-point values are stored in IEEE 754 form, as three bitfields containing sign, exponent and mantissa. The sign always has one bit, clear for positive, set for negative. The exponent and mantissa have the following sizes and bias. Subtype Exponent Bias Mantissa DATANUM_FLOAT16 5 bits +15 10 bits DATANUM_FLOAT32 8 bits +127 23 bits DATANUM_FLOAT64 11 bits +1023 52 bits Infinities and Not-a-Number values are represented by the exponent having its maximum allowed value. If the mantissa is zero this represents an infinity of the given sign, and if the mantissa is non-zero, it is a not-a-number value. For canonical identity, the non-zero mantissa should have only its top bit set, and the sign bit should be clear. Subtype Exponent Mantissa DATANUM_FLOAT16 31 0 Inf DATANUM_FLOAT16 31 1 << 9 NaN DATANUM_FLOAT32 255 0 Inf DATANUM_FLOAT32 255 1 << 22 NaN DATANUM_FLOAT64 1023 0 Inf DATANUM_FLOAT64 1023 1 << 51 NaN For string, list, dict and object types, the lower five bits give a number, 0 to 31, which helps encode the size. For items of size 30 or below, this size is encoded directly. Where the size is 31 or more, the number 31 is encoded, and the actual size follows this leading byte. For sizes 31 to 127, the next byte encodes it. For sizes 128 or above, the next 4 bytes encode it in big-endian format, with the top bit set. Sizes above 2^31 cannot be encoded. Following the leader are bytes encoding the data. The exact meaning of the size depends on the type of the node. For strings, the size gives the number of bytes in the string. These bytes then follow the leader. For lists, the size gives the number of elements in the list. Following the leader will be this number of data serialisations, one per list element. For dictionaries, this size gives the number of key/value pairs. Following the leader will be this number of key/value pairs. Each pair consists of a string for the key name, then a data serialisation for the value. For objects, the size gives the number of bytes in the object's ID number, followed by a big-endian encoding of the object's ID number. Currently, this will always be a 4 byte number. For structured records, the size gives the count of serialied data members for the record. Following the leader will be the ID number of the structure type as an int, followed by the given number of data members, in the order that the structure type declares. The field names are not serialised, as they can be inferred from the structure type's definition. Meta-data items may be embedded within a data stream in order to create the object classes and instances which it contains. These metadata items do not count towards the overall size of a collection value. Meta-data operations encode a subtype number, rather than a size, in the bottom five bits. Metadata type Bits Description DATAMETA_CONSTRUCT 1 1 1 0 0 0 0 1 Construct an object DATAMETA_CLASS 1 1 1 0 0 0 1 0 Create a new object class DATAMETA_STRUCT 1 1 1 0 0 0 1 1 Create a new record struct type Following each metadata item is an encoding of its arguments. DATAMETA_CONSTRUCT: Object ID: int Class ID: int Smash values: 0 or more bytes, encoded per type (in a list container) If the object class defines smash properties, the construct message will also contain the values for the smash properties. These will be sent in a list, one value per property, in the same order as the object class's schema defines the smash keys. Each will be encoded as per its declared type. DATAMETA_CLASS: Class name: string Class ID: int Class: struct of type Tangence.Class Smash keys: data encoded (list) The class definition itself will be encoded as a Tangence.Class structure, containing nested Tangence.Method, Tangence.Event and Tangence.Property elements. If the class declares any superclasses, these will be sent in other DATAMETA_CLASS metadata items before this one. The smash keys will be encoded as a possibly-empty list of strings. DATAMETA_STRUCT: Struct name: string Struct ID: int Field names: list of strings Field types: list of strings 4.3. Message Types Each of the messages defines the layout of its data payload. Some messages pass a fixed number of items, some have a variable number of items in the last position. For these messages, no explicit encoding of the size is given. Instead, the data payload area is packed with as many data encodings as are required. The receiver should use the size of the containing message to know when all the items have been unpacked. The following request types are defined. Any message may be responded to by MSG_ERROR in case of an error, so this response type is not listed. Some of these messages are sent from the client to the server (C->S), others are sent from the server to client (S->C) MSG_CALL (C->S) (0x01) INT object ID STRING method name data... arguments Responses: MSG_RESULT Calls the named method on the given object. MSG_SUBSCRIBE (C->S) (0x02) INT object ID STRING event name Responses: MSG_SUBSCRIBED Subscribes the client to be informed of the event on given object. MSG_UNSUBSCRIBE (C->S) (0x03) INT object ID STRING event name Responses: MSG_OK Cancels an event subscription. MSG_EVENT (S->C) (0x04) INT object ID STRING event name data... arguments Responses: MSG_OK Informs the client that the event has occured. MSG_GETPROP (C->S) (0x05) INT object ID STRING property name Responses: MSG_RESULT Requests the current value of the property MSG_SETPROP (C->S) (0x06) INT object ID STRING property name data new value Responses: MSG_OK Sets the new value of the property MSG_WATCH (C->S) (0x07) INT object ID STRING property name BOOL want initial? Responses: MSG_WATCHING Requests to be informed of changes to the property value. If the boolean 'want initial' value is true, the client will be sent an initial MSG_CHANGE message for the current value of the property. MSG_UNWATCH (C->S) (0x08) INT object ID STRING property name Responses: MSG_OK Cancels a request to watch a property MSG_UPDATE (S->C) (0x09) INT object ID STRING property name U8 change type data... change value Responses: MSG_OK Informs the client that the property value has now changed. The type of change is given by the change type argument, and defines the data layout in the value arguments. The exact meaning of the operation depends on the dimension of the property it acts on. For DIM_SCALAR: CHANGE_SET: data new value Sets the new value of the property. For DIM_HASH: CHANGE_SET: DICT new value Sets the new value of the property. CHANGE_ADD: STRING key data value Adds a new element to the hash. CHANGE_DEL: STRING key Deletes an element from the hash. For DIM_QUEUE: CHANGE_SET: LIST new value Sets the new value of the property. CHANGE_PUSH: data... additional values Appends the additional values to the end of the queue. CHANGE_SHIFT: INT number of elements Removes a number of leading elements from the beginning of the queue. For DIM_ARRAY: CHANGE_SET: LIST new value Sets the new value of the property. CHANGE_PUSH: data... additional values Appends the additional values to the end of the array. CHANGE_SHIFT: INT number of elements Removes a number of leading elements from the beginning of the array. CHANGE_SPLICE: INT start INT count data... new elements Replaces the given range of the array with the new elements given. The new list of values may be a different length to the replaced section - in this case, subsequent elements will be shifted up or down accordingly. CHANGE_MOVE: INT index INT delta Moves the item currently at the index forward a (signed) delta amount, such that its new index becomes index+delta. The items inbetween the old and new index will be moved up or down as appropriate. For DIM_OBJSET: CHANGE_SET: LIST objects Sets the new value for the property. Will be given a list of Tangence object references. CHANGE_ADD: OBJECT new object Adds the given object to the set CHANGE_DEL: STRING object ID Removes the object of the given ID from the set. MSG_DESTROY (S->C) (0x0a) INT object ID Responses: MSG_OK Informs the client that the object is due for destruction in the server. Upon receipt of this message the client should destroy any remaining references it has to the object. After it has sent the MSG_OK response, it will not be allowed to invoke any methods, subscribe to any events, nor interact with any properties on the object. Any existing event subscriptions or property watches will have been removed by the server before this message is sent. MSG_GETPROPELEM (C->S) (0x0b) INT object ID STRING property name INT|STRING element index or key Responses: MSG_RESULT Requests the current value of a single element in a queue or array (by element index), or hash (by key name). Cannot be applied to scalar or objset properties. MSG_WATCH_CUSR (C->S) (0x0c) INT object ID STRING property name INT from Responses: MSG_WATCHING_CUSR Similar to MSG_WATCH, requests to be informed of changes to the property value, which must be a queue property. Creates a new cursor for the property, beginning at the first index (if from == 1) or the last (if from == 2). MSG_CUSR_NEXT (C->S) (0x0d) INT cursor ID INT direction INT count Responses: MSG_CUSR_RESULT Requests the next few items from a property cursor. It will yield a MSG_RESULT message containing up to the given number of items, by moving forwards (if direction == 1) or backwards (if direction == 2). If the cursor is already at the edge of the queue then the MSG_RESULT will contain no extra items. MSG_CUSR_DESTROY (C->S) (0x0e) INT cursor ID Informs the server that the client has finished using the cursor, and it can release any resources attached to it. MSG_GETROOT (C->S) (0x40) data identity Responses: MSG_RESULT Initial message to be sent by the client to obtain the root object. The identity may be used to identify this particular client, as part of its login procedure. The result will contain a single object reference, being the root object. MSG_GETREGISTRY (C->S) (0x41) [no arguments] Responses: MSG_RESULT Requests the registry object from the server. The result will contain a single object reference, being the registry object. MSG_INIT (C->S) (0x7f) INT major version INT maximal minor version INT minimal minor version Responses: MSG_INITED Requests the start of the Tangence stream. This must be the first message sent by the client. If the server is unwilling to provide a suitable version it can return MSG_ERROR. Otherwise, the accepted minor is returned in the MSG_INITED message. The version specified by this document is major 0, minor 4. The following responses may be sent to a request: MSG_OK (0x80) [no arguments] A simple OK message, informing the requester that the operation was successful, an no error occured. MSG_ERROR (0x81) STRING error message An error occured; the text of the message is included. MSG_RESULT (0x82) data... values Contains the return value from a method call, a property value, or the initial root or registry object. MSG_SUBSCRIBED (0x83) [no arguments] Informs the client that a MSG_SUBSCRIBE was successful. MSG_WATCHING (0x84) [no arguments] Informs the client that a MSG_WATCH was successful. MSG_WATCHING_CUSR (0x85) INT cursor ID INT first index (inclusive) INT last index (inclusive) Informs the client that a MSG_WATCH_CUSR was successful, and returns the new cursor ID and the first and last indices inclusive of the queue it will iterate over. ((The reason for using first and last indices inclusively, rather than yielding the total size of the queue, is that this makes it easier to support iterating over hashes in a future version)) MSG_CUSR_RESULT (0x86) INT first item index data... values Contains the return value from a MSG_CUSR_NEXT call. Gives the index of the first item in the returned result, and the requested items. There may fewer items than requested, if the edge of the property value was reached. MSG_INITED (0xff) INT major version INT minor version Informs the client that the initial MSG_INIT was successful, and what minor version was accepted. 4.4 Built-in Structure Types The following structure types are built-in, with the given structure ID numbers. They can be assumed pre-knowledge by both ends of the stream and do not need serialising by DATAMETA_STRUCT records. 4.4.1 Tangence.Class Structure ID: 1 Fields: methods : dict(any) events : dict(any) properties : dict(any) superclasses : list(str) 4.4.2 Tangence.Method Structure ID: 2 Fields: arguments : list(str) returns : str 4.4.3 Tangence.Event Structure ID: 3 Fields: arguments : list(str) 4.4.4 Tangence.Property Structure ID: 4 Fields: dimension : int type : str smashed : bool -- Paul "LeoNerd" Evans leonerd@leonerd.org.uk http://www.leonerd.org.uk/