Symbian OS Developers

Kernelside architecture

4.1.5.1 Server - queue management

The kernel-side architecture reflects the structure of the user-side clientserver architecture it is designed to support. The first kernel-side object I'll discuss is the server object, corresponding to the RServer2 handle held by the server process. The kernel-side server object has two purposes:

• To ensure the uniqueness of the user-mode server within the system

• To provide a FIFO queue of messages to be delivered to the server thread. A client may deliver messages to the server at any time, but the server thread only receives the messages one at a time as it sequentially requests them from the kernel.

The first requirement is easy to achieve: during server creation the kernel adds the server object being added to a global object container for servers. As I will show in the next chapter, object containers mandate the uniqueness of names within them. Also, another check is performed at server creation: the server may only specify a name beginning with ''!'' if the server has the ProtServ capability. This allows the client to be certain that servers with names beginning with ''!'' have not been spoofed by some other malicious code.

To fulfill the second requirement, server objects use the state machine shown in Figure 4.3 to manage their FIFO queue of messages.

Figure 4.3 Server object state machine

The labeled state transitions listed below are executed under the protection of the system lock, to maintain state integrity. Each of these transitions is designed to hold the system lock for constant execution time. This maintains the real-time characteristics of the kernel as there will always be a maximum, constant time for which the system lock will be held before being released:

2. DSession::Detach()

3. DServer::Receive()

4. DServer::Cancel(), DServer::Close() [when closing last reference]

5. DServer::Deliver()

The states in the above transition diagram are defined as follows:

	Session queue	Message queue	[User] Server request status
IDLE	Empty	Empty	NULL
SESSION ATTACHED	Non-empty	Empty	NULL
AWAITING MESSAGE	Don't care	Empty	Non-NULL
MESSAGE PENDING	Don't care	Non-empty	NULL

The implementation of this state machine can be seen in the methods and members of the DServer class:

class DServer : public DObject {

public:

DServer();

virtual ^DServer();

virtual TInt Close(TAny*);

virtual TInt RequestUserHandle(DThread* aThread, TOwnerType aType);

// aMessage bit 0 = 0 -> RMessage, bit 0 = 1 -> RMessage2 void Receive(TRequestStatus& aStatus, TAny* aMessage); void Cancel();

void Accept(RMessageK* aMsg); void Deliver(RMessageK* aMsg); public:

inline TBool IsClosing(); public:

DThread* iOwningThread; // thread which receives messages TAny* iMsgDest; // where to deliver messages

TRequestStatus* iStatus; // completed to signal message arrival SDblQue iSessionQ; // list of sessions

SDblQue iDeliveredQ; // messages delivered but not yet accepted

TUint8 iSessionType; // TIpcSessionType };

The server object itself is created by ExecHandler::ServerCreate(), called (indirectly via RServer2) from CServer2::Start(). Once ExecHandler::ServerCreate() has created the kernel-side server object, it opens a handle on the current (server) thread, so the pointer to the server thread (iOwningThread) is always valid.

This is required because a process-relative handle may be created to the server object by any thread in the server's process, using Duplicated and hence the server objects may be held open after the server thread terminates. The first handle to the server object that is created and subsequently vested within the RServer2 will be a process-relative handle if the server is anonymous (that is, has a zero-length name) and thread-relative otherwise. To exercise control over the use of Dupli-cate(), DServer over-rides DObject's RequestUserHandle() method, which is called whenever a user-mode thread wishes to create a handle to an object. DServer enforces the policy that handles to it may only be created within the server's process. The server object then closes the reference to the server thread in its destructor, so the thread object will only ever be safely destroyed after the server object has finished using it.

DServer::Receive() and DServer::Cancel() provide the kernel-side implementation of the private RServer2 API used to retrieve messages for the server. These receive and cancel functions provide an API for an asynchronous request to de-queue the message at the head of the FIFO queue. After de-queuing a message, the request is completed on the server thread. If a message is present in the server's queue when the request is made, this operation is performed immediately. Otherwise no action is taken and the next message delivered to the DServer object is used to immediately complete this request, rather than being placed on the server's queue.

The server thread may choose to block until a message is available (for example, using User::WaitForRequest()) or may use another method to wait for the request completion. In the case of a standard Symbian OS server, the CServer2-derived class is an active object and uses the active scheduler as a mechanism to wait for the request for the next message to be completed.

The procedure of writing a message to the server process's address space and signaling it to notify it of the completion of its request for a message is known as ''accepting'' a message. DServer::Deliver() is the client thread's API to deliver a message to the server's message queue. Both DServer::Receive() and DServer::Deliver() will accept a message immediately, where appropriate. These methods both call a common subroutine DServer::Accept(), which contains the code to accept a message. It updates the message's state to reflect the fact the server has accepted its delivery before writing the message to the server's address space and finally signaling the server's request status to indicate completion of its request.

The kernel-side message object (RMessageK) is converted into the correct format for user-side message object by using a utility classes whose structure mirrors RMessage2:

class RMessageU2 {

public:

inline RMessageU2(const RMessageK& a); public:

TInt iHandle; TInt iFunction;

TInt iArgs[KMaxMessageArguments]; TUint32 iSpare1;

const TAny* iSessionPtr; };

Note that the iSpare1 member of RMessageU2 is simply zeroed by the kernel when writing the message to the user (that is, the server thread), but the other ''unused'' members of RMessage2 will not be overwritten by the kernel when it writes a message to user-space. A separate structure is used here as the format of RMessageK is private to the kernel itself and this class therefore provides translation between the internal message format used by the kernel and the public message format of RMessage2 used by user-side code.

Most of the methods mentioned above can be written to hold the system lock for a constant time with relative ease as they encompass tasks such as adding to or removing from a doubly linked list, updating state and performing a fast write of a small amount of data. However, when the DServer object is closed for the last time in DServer::Close(), the session list has to be iterated to detach all the sessions still attached to the server. This must be done under the protection of the system lock so that the server's state is updated in a consistent manner. As there are an arbitrary number of sessions to detach, this operation has an execution time linearly proportional to the number of sessions, as opposed to a constant execution time.

This operation is therefore carefully split up into n separate operations, each of which only hold the system lock for a constant time and each of which leave the data structures in a consistent state. DServ-er::Close() acquires the system lock before detaching each session by calling DSession::Detach(), which will release the lock before returning. DSession::Detach() is an operation made up of freeing an arbitrary number of uncompleted messages that have been sent by that session to the server. Again, this operation is split up by acquiring the system lock before freeing each message and then releasing it again afterwards, so the system lock is never held for more than a constant, bounded time whilst one message is freed or one session is removed from the server's queue.

To achieve the consistency required whilst splitting up these operations, the fact that a server or session is closing (iAccessCount is 0) is used to restrict what actions may occur. For example, new sessions cannot be attached to a server whilst it is closing and messages cannot be completed whilst a session is closing.

4.1.5.2 Sessions - delivery and message pool management

In the previous section, I described how sessions provide the context for communication between the client and server. Specifically, the kernel session objects manage the delivery of messages to the server and ensure message completion, even under out-of-memory conditions. They also manage user-mode access to a session, as specified by the session's ''sharability''.

To ensure message completion, the session object maintains a queue of message objects distinct from that of the server. This queue also includes messages sent by the session that have not yet been completed by the server. The interaction of this queue with both the lifetime of the client and the server is controlled via the state machine shown in Figure 4.4.

Figure 4.4 Session object state machine

Again, the labeled state transitions listed below are executed under the protection of the system lock to maintain state integrity. These transitions are also designed to hold the system lock for a constant execution time, in order to maintain the real-time characteristics of the kernel:

2. ExecHandler::MessageComplete()

3. DSession::Detach()

4. DSession::Close() [when closing last reference]

[no connect message pending or not accepted by server]

5. DSession::CloseFromDisconnect()

6. ExecHandler::SetSessionPtr()

7. DSession::Close() [when closing last reference]

[connect message pending and accepted by server]

The states in Figure 4.4 are defined like this:

	Server queue (of sessions)	Client references	Message queue
ATTACHED	Queued	Open	Empty
PROCESSING MESSAGE	Queued	Open	Non-empty
CLOSING	Queued	Closed	Non-empty, contains connect msg.
DISCONNECT PENDING	Queued	Closed	Non-empty
UNATTACHED	De-queued	Open	Empty
IDLE	De-queued	Closed	Empty

The implementation of DSession to support this is as follows:

The implementation of DSession to support this is as follows:

class DSession : public DObject {

public:

DSession();

virtual ^DSession();

virtual Tint Close(TAny*);

virtual Tint RequestUserHandle(DThread* aThread, TOwnerType aType);

void Detach(TInt aReason); RMessageK* GetNextFreeMessage(); RMessageK* ExpandGlobalPool(); void CloseFromDisconnect();

static TInt New(DSession*& aS, TInt aMsgSlots, TInt aMode); TInt Add(DServer* aSvr, const TSecurityPolicy* aSecurityPolicy); TInt MakeHandle();

TInt Send(TInt aFunction, const TInt* aPtr, TRequestStatus* aStatus); TInt SendSync(TInt aFunction, const TInt* aPtr,

TRequestStatus* aStatus); TInt Send(RMessageK* aMsg, TInt aFunction, const TInt* aPtr,

TRequestStatus* aStatus);

public:

inline TBool IsClosing(); public:

DServer* iServer; // pointer to kernel-side server object SDblQueLink iServerLink; // link to attach session to server const TAny* iSessionPtr;

// pointer to server-side CSession2 (user cookie)

TUint16 iTotalAccessCount;

TUint8 iSessionType; // TIpcSessionType

TUint8 iSvrSessionType;// TIpcSessionType

TInt iMsgCount;

// total number of outstanding messages on this session TInt iMsgLimit;

// max number of outstanding messages on this session SDblQue iMsgQ;

// q of outstanding msgs on this session (by iSessionLink) RMessageK* iNextFreeMessage; // pointer to next free message in

// per-session message pool, if any RMessageK* iPool; // pointer to per-session message pool, if any RMessageK* iConnectMsg; // pointer to connect msg, if any

RMessageKBase iDisconnectMsg; // vestigial disconnect message };

The session is a standard kernel reference-counted object that user-mode clients hold references to via handles (I'll discuss this mechanism in the next chapter, Kernel Services). However, the lifetime of the session must extend beyond that given by client handles, because it needs to stay in existence whilst the server processes the disconnect message that is sent to the server when a session is closed. To do this, we use iTotalAccessCount, modified underthe protection ofthesystem lock, to keep track of both whether there are any client references and when the session is attached to a server, giving it a count of 1 for either, or 2 for both. The IDLE state is equivalent to iTotalAccessCount reaching 0.

The creation of a session is performed as two distinct operations in two separate executive calls by the client - firstly the creation of the kernelside session object itself and secondly the sending ofthe connect message to the server. The second part of this operation is identical to sending any other message to a server. However, since both a client can connect asynchronously and the server can create a session asynchronously it is now possible for a client to close its handle to the session before the session object has been created in the server.

Normally, closing the last handle to the session would result in a disconnect message being immediately delivered to the server, but if this were done in this case the disconnect message could be accepted by the server, and would contain a null cookie as the session object had not yet been created. There would then be a race between the server setting the cookie after creating the session object and it completing the disconnect request. If the disconnect message is completed first, the kernel-side session object is no longer valid when the server tries to set the cookie using the connect message and it will be panicked. Otherwise, if the session cookie is set first, then the disconnect message will complete as a no-op, since the cookie within it is null, and a session object is leaked in the server.

To avoid either of these situations, we have introduced the CLOSING state. If a connect message has been delivered but not yet completed when the last user handle to the session is closed, the delivery of the disconnect message to the server is delayed until the session cookie is updated to indicate successful creation of a user-side session object. If the connect message completes without the cookie having been updated, there is no user-side session object to clean up but a disconnect message is still sent to ensure the lifetime of the session object extends until the completion of other messages which may have been sent to the unconnected session. If there is an undelivered connect message, this is immediately removed from the queue to avoid the possibility of an orphan session object being created.

Note that the ability to create process-relative session handles and the asynchronous nature of sending a connect message mean that you can send other messages to a server both before and after a connect message has been sent, and before the connect message has been completed, so care must be taken over this possibility. However, when a disconnect message is sent no more messages may be sent to the session by virtue of the fact that it is sent once all user-side handles to the session have been closed. Therefore the disconnect message is always the last message to a session that is completed by the server and the session object may be safely destroyed after completing it, without causing lifetime issues for the server.

There are two methods to send synchronous and asynchronous messages- SendSync() and Send() (first overload above), respectively. These validate certain preconditions, select an appropriate message object to store the new message, and then pass the selected message object to this method:

Send(RMessageK* aMsg, TInt aFunction, const TInt* aPtr,

TRequestStatus* aStatus);

This method populates the message object, increments the current thread's IPC count (DThread::iIpcCount), adds the message to the session's queue and delivers it to the server. If the server has terminated, or is in the process of doing so, sending the message fails immediately.

Neither Send() method permits the sending of a disconnect message, since this is sent automatically by DSession::Close() when the last client session handle is closed.

At the other end of a normal message's life, it is completed in ExecHandler::MessageComplete() (called from RMessage-Ptr2::Complete()). If the message is a disconnect message, then execution is simply transferred to DSession::CloseFromDisconnect(). If not, then what happens next is pretty much the reverse of the send procedure: the message is removed from the session queue, the sending thread's IPC count is decremented and the client thread's request is completed. The only exception to this is if the session is closing; this happens if the client has closed all handles to the session but the disconnect message has not yet been completed by the server. In this case the client thread's request is not completed.

Messages can also be completed from DSession::Detach() ,which is called either when the server terminates (that is, when the last reference to the server is closed in DServer::Close()) or when completing a disconnect message. In this case, the message is again removed from the session queue, the sending thread's IPC count decremented and the client request is completed (if the session is not closing).

We have just seen that it is possible for a message not to be completed - this happens when the session is closing, as we saw above. And yet previously I said that guaranteed message completion is one of the properties of the client-server system. The explanation here is that the client having an outstanding message when calling Close() on a session is considered to be a client-side programming error. There is clearly a race between the server completing such outstanding asynchronous requests and the disconnect request being processed. Those requests processed after the session has been closed cannot be completed whereas those before can, so the behavior of client-server has always been undefined by Symbian in this situation.

The actual behavior of EKA2 differs from EKA1 here. In EKA1, disconnect messages overtake all undelivered messages to the server and these undelivered messages are discarded. Now, in EKA2, the server processes all delivered messages before processing the disconnect message - although, as we've seen, such messages can still not be completed to the client. All this said, we don't advise you to rely on this new behavior of EKA2, because we explicitly state this to be a programming error and may change the behavior of this area in future.

One of the other main functions of the session object is to control the session's ''sharability''. With the advent of a fully message-centric design, there is no requirement for the session to hold a handle to the client thread, and providing different levels of accessibility to a session - restricted to one thread, restricted to one process or sharable with any thread - is now a simple matter of recording the stated intentions of the user-mode server in iSvrSessionType and then creating a handle to the session for any client that is allowed access to it, when requested. That is, whether a given thread can access a session is now determined purely by whether it has a handle to the session or not as the access check is performed at handle-creation time.

This accounts for the introduction of the new method DObj-ect::RequestUserHandle(). Suppose a server only supports non-sharable sessions. Then a user-mode thread with a handle to a session could just duplicate that handle, making a process-relative handle, and over-ride the settings of the server. But DSession's over-ridden Reques-tUserHandle() checks the value in iSvrSessionType to see whether the requested sharing level is allowed by the server, and thereby enforces the user-mode server's requested policy on session sharing.

To maintain backwards compatibility, the user-side APIs for creating sessions default to creating a session that is "unshareable", even if shared sessions are supported by the server. This ''current sharability level'' - specified when creating the initial handle to the session, is stored in iSessionType and is validated against the ''sharability'' level the server supports. To share this session with other threads, the session has either to explicitly create a session that supports the level of sharability required (the preferred method) or subsequently call ShareAuto() (to share within process) or ShareProtected() (to share between processes), as required. If the ShareXxx() method succeeds, it creates a new process-relative handle and closes the old one. The new session creation overloads that allow the ''sharability'' of a session to be specified from session creation are the preferred method, as they avoid the expensive operation of creating a new handle where it is not needed.

These new session creation overloads also support an optional security policy that the client can use to verify the security credentials of the server it is connecting to. Similarly, there are overloads of the new APIs to open a handle to a session shared over client-server IPC or from a creator process which allow the session to be validated against a security policy. This allows you to prevent a handle to a spoof server being passed by these mechanisms, as you may verify the identity of the server whose session you are accepting a handle to. The initial session type and security policy parameters are then marshalled into the call to DSession:: Add(), where they are used to fail session creation if the server does not meet the required policy.

The final responsibility of the session is to find-and allocate if necessary - kernel memory for messages to be stored in. I will discuss these message pools in the next section. At session creation time, you can specify whether the session uses a pool specific to the session or a global kernel pool. The session stores the type of pool it is using in iPool. If it is using a per-session pool, then it maintains a pointer to the next available free message in iNextFreeMessage.

During a send, the session will then use one of the session's disconnect message, the thread's synchronous message or the next free message from the selected pool. If the session is using the global pool and there are no more free messages the system lock is relinquished (to avoid holding it for an unbounded period of time whilst allocating), the message pool is expanded, then the lock is reclaimed and sending proceeds as before.

4.1.5.3 Messages - minimal states and message pool design

Next, I'll consider the design of message pools, that is, the memory used to store messages within the kernel. There is one important constraint on the design of the message pools, namely that there must always be a free message available for a session to send a disconnect message to the server, so that resources may be correctly freed in OOM situations. This disconnect message is naturally associated with the session whose disconnection it is notifying and will always be available if the message is embedded within the session object itself. To minimize the memory used by this disconnect message object, we have designed the message object to have a base class, RMessageKBase, which contains only the data required for the disconnect message, and then derive from it the (larger) message class, RMessageK, which is used for normal messages:

class RMessageKBase : public SDblQueLink {

public:

TBool IsFree() const { return !iNext; } TBool IsDelivered() const

{ return iNext!=0 && (TLinAddr(iNext) & 3)==0; } TBool IsAccepted() const

Tint iFunction; };

class RMessageK : public RMessageKBase {

public:

enum TMsgType {EDisc=0, ESync=1, ESession=2, EGlobal=3}; inline Tint ArgType(TInt aParam) const; inline TInt Arg(TInt aParam) const; void Free();

static RMessageK* NewMsgBlock(TInt aCount, TInt aType); IMPORT_C DThread* Thread() const;

static RMessageK* MessageK(TInt aHandle, DThread* aThread); IMPORT_C static RMessageK* MessageK(TInt aHandle); public:

TInt iArgs[4];

TUint16 iArgFlags; // describes which arguments are descriptors/handles TUint8 iPool; // 0=disconnect msg, 1=thread sync message, // 2=from session pool, 3=from global pool

TUint8 iPad;

DSession* iSession; // pointer to session

SDblQueLink iSessionLink; // attaches message to session

DThread* iClient; // pointer to client thread (not reference counted)

TRequestStatus* iStatus; // pointer to user side TRequestStatus };

After we have ensured that session cleanup works properly, the next most important concern in designing the message allocation strategy is to minimize the memory that the kernel uses for sending messages. In the original client-server implementation of Symbian OS v5, a fixed number of message objects were allocated for each session, resulting in poor message object utilization, considering that most IPC calls were synchronous and hence only one of the message objects was in use at any one time!

Analysis of the client-server system, by instrumenting EUSER and EKERN, has shown that as many as 99% of the calls to RSession-Base::SendReceive() are to the synchronous overload. Obviously, a thread cannot send a synchronous message to more than one server at a time, because it waits for the synchronous message's completion inside EUSER immediately after dispatching it. This means that we can use a per-thread message object to avoid having to allocate message objects for all the synchronous messages that are sent. This message object (RMes-sageK) is embedded within the DThread object, and therefore avoids allocation issues by being allocated as part of the thread object itself.

Thus all that remains to be determined is the allocation strategy for message objects used by asynchronous IPC (such messages are typically used for remote I/O such as sockets or for event notification). These message objects are allocated on a per-session basis, or dynamically from a global pool. As we only need a small number of them, the overhead for non-utilization of these message objects is not large.

You may wonder why we do not insist on a global pool, and cut memory requirements further. This is because for real-time code a guaranteed response time is important, and a global, dynamic pool does not provide those guarantees (as it may require memory allocation in the kernel). This means that we must provide the option of creating a per-session pool, which allows the real-time code to manage the time it takes to process an asynchronous request precisely. That said, it is more common for servers to use asynchronous message completion for event notification, in which case using the dynamic global pool becomes more attractive due to its smaller memory footprint. This is therefore the recommended option where real-time guarantees are not required for any asynchronous IPC calls to the server.

This allocation scheme allows any number of threads to invoke synchronous IPC on a server using the same session without having to increase the session's message pool and it also provides guaranteed message sending for synchronous IPC.

We have achieved further memory savings by minimizing the state that is required within the message objects themselves. There are only three states that a message can have, as shown in Figure 4.5.

To assure a message's cleanup when a session closes, we attach it to a session queue whilst it is not free. We also have to ensure no message can persist beyond the lifetime of the thread that sent it, as such a message can no longer be completed. By assuming a strategy that messages should never be discarded prematurely (for example, when the thread Exit() s), but only at the last possible moment (just before the thread object is destroyed), we can avoid the complication of maintaining an open reference on the thread and the associated state required for this. When a thread exits, therefore, we need not iterate through a queue to discard DELIVERED messages - they are simply allowed to propagate through the server as usual. Instead of an open reference on the thread in each message, we need only maintain a count of the outstanding messages for each thread (in DThread::iIpcCount).

When a thread exits it checks this count and if it is non-zero it increments its own reference count so it is not destroyed and sets a flag (the top bit of the message count) to indicate this has been done. Completing a message decrements the outstanding message count for the client thread and if its value reaches 0x80000000, this means the

ACCEPTED

FREE

DELIVERED

The message is not currently in use

The message has been sent but the server hasn't seen it yet

The server has received the message but not yet completed it.

Figure 4.5 Message object state machine

message count for the thread has reached zero and the flag has been set. The extra reference on the thread is then closed, allowing it to be safely destroyed.

Closing a session does not unilaterally discard DELIVERED messages either-again they are simply allowed to propagate through the server. This means that session close needs only to send the disconnect message to the server and has no need to iterate its queue of messages.

So, we only need to perform message queue iteration either when the server itself terminates or when the server completes a disconnect message. In the latter case, the iteration is needed to free any remaining ACCEPTED messages (no DELIVERED messages can remain since the disconnect message is guaranteed to be the last message received from that session). The messages concerned are not on any other queue so there is no contention when we iterate over the session queue. When the server itself terminates, the complete system of server and sessions is frozen, because clients are not allowed to send a message to the server whilst it is terminating, so again there is no contention on the session queue when a message is being completed and hence no need for an intermediate COMPLETED state to deal with delayed removal of messages from the session queue.

The three states are then encoded in a minimal way into the doubly linked list fields:

	iLink.iNext	iLink.iPrev
FREE	NULL	N/A
DELIVERED	Valid address (multiple of 4) [bottom bits == 0 0b]	N/A
[bottom bits == lib]	(server DProcess)

4.1.5.4 Message handles

Another key design decision we took for message objects was not to derive them from DObject. This means that they do not have standard object containers and user handles. Rather, the user-side handle for an RMessageK is in fact its address on the kernel heap. To verify the handles that user-mode operations give, the kernel uses the fact that a user-side component only ever has a valid handle to a message when it is in the ACCEPTED state and does the following:

• Checks the address and size of the object are within the kernel heap

• If so, reads the memory under an exception trap (the machine coded versions of these functions use the magic exception immunity mechanism, see Section 5.4.3.1)

• Check that msg.iPrev == ~ (requestingThread->

iOwning Process).

If these tests pass, then the message object is taken to be valid and the requested operation can proceed.

0 0