| <chapter id="ole"> |
| <title>COM in Wine</title> |
| |
| <sect1 id="com-writing"> |
| <title>Writing COM Components for Wine</title> |
| |
| <para> |
| This section describes how to create your own natively |
| compiled COM components. |
| </para> |
| |
| <sect2> |
| <title>Macros to define a COM interface</title> |
| |
| <para> |
| The goal of the following set of definitions is to provide a |
| way to use the same header file definitions to provide both |
| a C interface and a C++ object oriented interface to COM |
| interfaces. The type of interface is selected automatically |
| depending on the language but it is always possible to get |
| the C interface in C++ by defining CINTERFACE. |
| </para> |
| <para> |
| It is based on the following assumptions: |
| </para> |
| <itemizedlist> |
| <listitem> |
| <para> |
| all COM interfaces derive from IUnknown, this should not |
| be a problem. |
| </para> |
| </listitem> |
| <listitem> |
| <para> |
| the header file only defines the interface, the actual |
| fields are defined separately in the C file implementing |
| the interface. |
| </para> |
| </listitem> |
| </itemizedlist> |
| <para> |
| The natural approach to this problem would be to make sure |
| we get a C++ class and virtual methods in C++ and a |
| structure with a table of pointer to functions in C. |
| Unfortunately the layout of the virtual table is compiler |
| specific, the layout of g++ virtual tables is not the same |
| as that of an egcs virtual table which is not the same as |
| that generated by Visual C++. There are work arounds to make |
| the virtual tables compatible via padding but unfortunately |
| the one which is imposed to the Wine emulator by the Windows |
| binaries, i.e. the Visual C++ one, is the most compact of |
| all. |
| </para> |
| <para> |
| So the solution I finally adopted does not use virtual |
| tables. Instead I use in-line non virtual methods that |
| dereference the method pointer themselves and perform the |
| call. |
| </para> |
| <para> |
| Let's take Direct3D as an example: |
| </para> |
| <programlisting>#define ICOM_INTERFACE IDirect3D |
| #define IDirect3D_METHODS \ |
| ICOM_METHOD1(HRESULT,Initialize, REFIID,) \ |
| ICOM_METHOD2(HRESULT,EnumDevices, LPD3DENUMDEVICESCALLBACK,, LPVOID,) \ |
| ICOM_METHOD2(HRESULT,CreateLight, LPDIRECT3DLIGHT*,, IUnknown*,) \ |
| ICOM_METHOD2(HRESULT,CreateMaterial,LPDIRECT3DMATERIAL*,, IUnknown*,) \ |
| ICOM_METHOD2(HRESULT,CreateViewport,LPDIRECT3DVIEWPORT*,, IUnknown*,) \ |
| ICOM_METHOD2(HRESULT,FindDevice, LPD3DFINDDEVICESEARCH,, LPD3DFINDDEVICERESULT,) |
| #define IDirect3D_IMETHODS \ |
| IUnknown_IMETHODS \ |
| IDirect3D_METHODS |
| ICOM_DEFINE(IDirect3D,IUnknown) |
| #undef ICOM_INTERFACE |
| |
| #ifdef ICOM_CINTERFACE |
| // *** IUnknown methods *** // |
| #define IDirect3D_QueryInterface(p,a,b) ICOM_CALL2(QueryInterface,p,a,b) |
| #define IDirect3D_AddRef(p) ICOM_CALL (AddRef,p) |
| #define IDirect3D_Release(p) ICOM_CALL (Release,p) |
| // *** IDirect3D methods *** // |
| #define IDirect3D_Initialize(p,a) ICOM_CALL1(Initialize,p,a) |
| #define IDirect3D_EnumDevices(p,a,b) ICOM_CALL2(EnumDevice,p,a,b) |
| #define IDirect3D_CreateLight(p,a,b) ICOM_CALL2(CreateLight,p,a,b) |
| #define IDirect3D_CreateMaterial(p,a,b) ICOM_CALL2(CreateMaterial,p,a,b) |
| #define IDirect3D_CreateViewport(p,a,b) ICOM_CALL2(CreateViewport,p,a,b) |
| #define IDirect3D_FindDevice(p,a,b) ICOM_CALL2(FindDevice,p,a,b) |
| #endif</programlisting> |
| <para> |
| Comments: |
| </para> |
| <para> |
| The ICOM_INTERFACE macro is used in the ICOM_METHOD macros |
| to define the type of the 'this' pointer. Defining this |
| macro here saves us the trouble of having to repeat the |
| interface name everywhere. Note however that because of the |
| way macros work, a macro like ICOM_METHOD1 cannot use |
| 'ICOM_INTERFACE##_VTABLE' because this would give |
| 'ICOM_INTERFACE_VTABLE' and not 'IDirect3D_VTABLE'. |
| </para> |
| <para> |
| ICOM_METHODS defines the methods specific to this |
| interface. It is then aggregated with the inherited methods |
| to form ICOM_IMETHODS. |
| </para> |
| <para> |
| ICOM_IMETHODS defines the list of methods that are |
| inheritable from this interface. It must be written manually |
| (rather than using a macro to generate the equivalent code) |
| to avoid macro recursion (which compilers don't like). |
| </para> |
| <para> |
| The ICOM_DEFINE finally declares all the structures |
| necessary for the interface. We have to explicitly use the |
| interface name for macro expansion reasons again. Inherited |
| methods are inherited in C by using the IDirect3D_METHODS |
| macro and the parent's Xxx_IMETHODS macro. In C++ we need |
| only use the IDirect3D_METHODS since method inheritance is |
| taken care of by the language. |
| </para> |
| <para> |
| In C++ the ICOM_METHOD macros generate a function prototype |
| and a call to a function pointer method. This means using |
| once 't1 p1, t2 p2, ...' and once 'p1, p2' without the |
| types. The only way I found to handle this is to have one |
| ICOM_METHOD macro per number of parameters and to have it |
| take only the type information (with const if necessary) as |
| parameters. The 'undef ICOM_INTERFACE' is here to remind |
| you that using ICOM_INTERFACE in the following macros will |
| not work. This time it's because the ICOM_CALL macro |
| expansion is done only once the 'IDirect3D_Xxx' macro is |
| expanded. And by that time ICOM_INTERFACE will be long gone |
| anyway. |
| </para> |
| <para> |
| You may have noticed the double commas after each parameter |
| type. This allows you to put the name of that parameter |
| which I think is good for documentation. It is not required |
| and since I did not know what to put there for this example |
| (I could only find doc about IDirect3D2), I left them blank. |
| </para> |
| <para> |
| Finally the set of 'IDirect3D_Xxx' macros is a standard set |
| of macros defined to ease access to the interface methods in |
| C. Unfortunately I don't see any way to avoid having to |
| duplicate the inherited method definitions there. This time |
| I could have used a trick to use only one macro whatever the |
| number of parameters but I preferred to have it work the same |
| way as above. |
| </para> |
| <para> |
| You probably have noticed that we don't define the fields we |
| need to actually implement this interface: reference count, |
| pointer to other resources and miscellaneous fields. That's |
| because these interfaces are just that: interfaces. They may |
| be implemented more than once, in different contexts and |
| sometimes not even in Wine. Thus it would not make sense to |
| impose that the interface contains some specific fields. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Bindings in C</title> |
| |
| <para> |
| In C this gives: |
| </para> |
| <programlisting>typedef struct IDirect3DVtbl IDirect3DVtbl; |
| struct IDirect3D { |
| IDirect3DVtbl* lpVtbl; |
| }; |
| struct IDirect3DVtbl { |
| HRESULT (*fnQueryInterface)(IDirect3D* me, REFIID riid, LPVOID* ppvObj); |
| ULONG (*fnAddRef)(IDirect3D* me); |
| ULONG (*fnRelease)(IDirect3D* me); |
| HRESULT (*fnInitialize)(IDirect3D* me, REFIID a); |
| HRESULT (*fnEnumDevices)(IDirect3D* me, LPD3DENUMDEVICESCALLBACK a, LPVOID b); |
| HRESULT (*fnCreateLight)(IDirect3D* me, LPDIRECT3DLIGHT* a, IUnknown* b); |
| HRESULT (*fnCreateMaterial)(IDirect3D* me, LPDIRECT3DMATERIAL* a, IUnknown* b); |
| HRESULT (*fnCreateViewport)(IDirect3D* me, LPDIRECT3DVIEWPORT* a, IUnknown* b); |
| HRESULT (*fnFindDevice)(IDirect3D* me, LPD3DFINDDEVICESEARCH a, LPD3DFINDDEVICERESULT b); |
| }; |
| |
| #ifdef ICOM_CINTERFACE |
| // *** IUnknown methods *** // |
| #define IDirect3D_QueryInterface(p,a,b) (p)->lpVtbl->fnQueryInterface(p,a,b) |
| #define IDirect3D_AddRef(p) (p)->lpVtbl->fnAddRef(p) |
| #define IDirect3D_Release(p) (p)->lpVtbl->fnRelease(p) |
| // *** IDirect3D methods *** // |
| #define IDirect3D_Initialize(p,a) (p)->lpVtbl->fnInitialize(p,a) |
| #define IDirect3D_EnumDevices(p,a,b) (p)->lpVtbl->fnEnumDevice(p,a,b) |
| #define IDirect3D_CreateLight(p,a,b) (p)->lpVtbl->fnCreateLight(p,a,b) |
| #define IDirect3D_CreateMaterial(p,a,b) (p)->lpVtbl->fnCreateMaterial(p,a,b) |
| #define IDirect3D_CreateViewport(p,a,b) (p)->lpVtbl->fnCreateViewport(p,a,b) |
| #define IDirect3D_FindDevice(p,a,b) (p)->lpVtbl->fnFindDevice(p,a,b) |
| #endif</programlisting> |
| <para> |
| Comments: |
| </para> |
| <para> |
| IDirect3D only contains a pointer to the IDirect3D |
| virtual/jump table. This is the only thing the user needs to |
| know to use the interface. Of course the structure we will |
| define to implement this interface will have more fields but |
| the first one will match this pointer. |
| </para> |
| <para> |
| The code generated by ICOM_DEFINE defines both the structure |
| representing the interface and the structure for the jump |
| table. ICOM_DEFINE uses the parent's Xxx_IMETHODS macro to |
| automatically repeat the prototypes of all the inherited |
| methods and then uses IDirect3D_METHODS to define the |
| IDirect3D methods. |
| </para> |
| <para> |
| Each method is declared as a pointer to function field in |
| the jump table. The implementation will fill this jump table |
| with appropriate values, probably using a static variable, |
| and initialize the lpVtbl field to point to this variable. |
| </para> |
| <para> |
| The IDirect3D_Xxx macros then just dereference the lpVtbl |
| pointer and use the function pointer corresponding to the |
| macro name. This emulates the behavior of a virtual table |
| and should be just as fast. |
| </para> |
| <para> |
| This C code should be quite compatible with the Windows |
| headers both for code that uses COM interfaces and for code |
| implementing a COM interface. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Bindings in C++</title> |
| <para> |
| And in C++ (with gcc's g++): |
| </para> |
| <programlisting>typedef struct IDirect3D: public IUnknown { |
| private: HRESULT (*fnInitialize)(IDirect3D* me, REFIID a); |
| public: inline HRESULT Initialize(REFIID a) { return ((IDirect3D*)t.lpVtbl)->fnInitialize(this,a); }; |
| private: HRESULT (*fnEnumDevices)(IDirect3D* me, LPD3DENUMDEVICESCALLBACK a, LPVOID b); |
| public: inline HRESULT EnumDevices(LPD3DENUMDEVICESCALLBACK a, LPVOID b) |
| { return ((IDirect3D*)t.lpVtbl)->fnEnumDevices(this,a,b); }; |
| private: HRESULT (*fnCreateLight)(IDirect3D* me, LPDIRECT3DLIGHT* a, IUnknown* b); |
| public: inline HRESULT CreateLight(LPDIRECT3DLIGHT* a, IUnknown* b) |
| { return ((IDirect3D*)t.lpVtbl)->fnCreateLight(this,a,b); }; |
| private: HRESULT (*fnCreateMaterial)(IDirect3D* me, LPDIRECT3DMATERIAL* a, IUnknown* b); |
| public: inline HRESULT CreateMaterial(LPDIRECT3DMATERIAL* a, IUnknown* b) |
| { return ((IDirect3D*)t.lpVtbl)->fnCreateMaterial(this,a,b); }; |
| private: HRESULT (*fnCreateViewport)(IDirect3D* me, LPDIRECT3DVIEWPORT* a, IUnknown* b); |
| public: inline HRESULT CreateViewport(LPDIRECT3DVIEWPORT* a, IUnknown* b) |
| { return ((IDirect3D*)t.lpVtbl)->fnCreateViewport(this,a,b); }; |
| private: HRESULT (*fnFindDevice)(IDirect3D* me, LPD3DFINDDEVICESEARCH a, LPD3DFINDDEVICERESULT b); |
| public: inline HRESULT FindDevice(LPD3DFINDDEVICESEARCH a, LPD3DFINDDEVICERESULT b) |
| { return ((IDirect3D*)t.lpVtbl)->fnFindDevice(this,a,b); }; |
| };</programlisting> |
| <para> |
| Comments: |
| </para> |
| <para> |
| In C++ IDirect3D does double duty as both the virtual/jump |
| table and as the interface definition. The reason for this |
| is to avoid having to duplicate the method definitions: once |
| to have the function pointers in the jump table and once to |
| have the methods in the interface class. Here one macro can |
| generate both. This means though that the first pointer, |
| t.lpVtbl defined in IUnknown, must be interpreted as the |
| jump table pointer if we interpret the structure as the |
| interface class, and as the function pointer to the |
| QueryInterface method, t.fnQueryInterface, if we interpret |
| the structure as the jump table. Fortunately this gymnastic |
| is entirely taken care of in the header of IUnknown. |
| </para> |
| <para> |
| Of course in C++ we use inheritance so that we don't have to |
| duplicate the method definitions. |
| </para> |
| <para> |
| Since IDirect3D does double duty, each ICOM_METHOD macro |
| defines both a function pointer and a non-virtual inline |
| method which dereferences it and calls it. This way this |
| method behaves just like a virtual method but does not |
| create a true C++ virtual table which would break the |
| structure layout. If you look at the implementation of these |
| methods you'll notice that they would not work for void |
| functions. We have to return something and fortunately this |
| seems to be what all the COM methods do (otherwise we would |
| need another set of macros). |
| </para> |
| <para> |
| Note how the ICOM_METHOD generates both function prototypes |
| mixing types and formal parameter names and the method |
| invocation using only the formal parameter name. This is the |
| reason why we need different macros to handle different |
| numbers of parameters. |
| </para> |
| <para> |
| Finally there is no IDirect3D_Xxx macro. These are not |
| needed in C++ unless the CINTERFACE macro is defined in |
| which case we would not be here. |
| </para> |
| <para> |
| This C++ code works well for code that just uses COM |
| interfaces. But it will not work with C++ code implement a |
| COM interface. That's because such code assumes the |
| interface methods are declared as virtual C++ methods which |
| is not the case here. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Implementing a COM interface.</title> |
| |
| <para> |
| This continues the above example. This example assumes that |
| the implementation is in C. |
| </para> |
| <programlisting>typedef struct _IDirect3D { |
| void* lpVtbl; |
| // ... |
| } _IDirect3D; |
| |
| static ICOM_VTABLE(IDirect3D) d3dvt; |
| |
| // implement the IDirect3D methods here |
| |
| int IDirect3D_fnQueryInterface(IDirect3D* me) |
| { |
| ICOM_THIS(IDirect3D,me); |
| // ... |
| } |
| |
| // ... |
| |
| static ICOM_VTABLE(IDirect3D) d3dvt = { |
| ICOM_MSVTABLE_COMPAT_DummyRTTIVALUE |
| IDirect3D_fnQueryInterface, |
| IDirect3D_fnAdd, |
| IDirect3D_fnAdd2, |
| IDirect3D_fnInitialize, |
| IDirect3D_fnSetWidth |
| };</programlisting> |
| <para> |
| Comments: |
| </para> |
| <para> |
| We first define what the interface really contains. This is |
| the _IDirect3D structure. The first field must of course be |
| the virtual table pointer. Everything else is free. |
| </para> |
| <para> |
| Then we predeclare our static virtual table variable, we |
| will need its address in some methods to initialize the |
| virtual table pointer of the returned interface objects. |
| </para> |
| <para> |
| Then we implement the interface methods. To match what has |
| been declared in the header file they must take a pointer to |
| an IDirect3D structure and we must cast it to an _IDirect3D |
| so that we can manipulate the fields. This is performed by |
| the ICOM_THIS macro. |
| </para> |
| <para> |
| Finally we initialize the virtual table. |
| </para> |
| </sect2> |
| </sect1> |
| |
| <sect1 id="dcom-1"> |
| <title>A brief introduction to DCOM in Wine</title> |
| |
| <para> |
| This section explains the basic principles behind DCOM remoting as used by InstallShield and others. |
| </para> |
| |
| <sect2> |
| <title>Basics</title> |
| |
| <para> |
| The basic idea behind DCOM is to take a COM object and make it location |
| transparent. That means you can use it from other threads, processes and |
| machines without having to worry about the fact that you can't just |
| dereference the interface vtable pointer to call methods on it. |
| </para> |
| |
| <para> |
| You might be wondering about putting threads next to processes and |
| machines in that last paragraph. You can access thread safe objects from |
| multiple threads without DCOM normally, right? Why would you need RPC |
| magic to do that? |
| </para> |
| |
| <para> |
| The answer is of course that COM doesn't assume that objects actually |
| are thread-safe. Most real-world objects aren't, in fact, for various |
| reasons. What these reasons are isn't too important here, though; it's |
| just important to realize that the problem of thread-unsafe objects is |
| what COM tries hard to solve with its apartment model. There are also |
| ways to tell COM that your object is truly thread-safe (namely the |
| free-threaded marshaller). In general, no object is truly thread-safe if |
| it could potentially use another not so thread-safe object, though, so |
| the free-threaded marshaller is less used than you'd think. |
| </para> |
| |
| <para> |
| For now, suffice it to say that COM lets you "marshal" interfaces into |
| other "apartments". An apartment (you may see it referred to as a |
| context in modern versions of COM) can be thought of as a location, and |
| contains objects. |
| </para> |
| |
| <para> |
| Every thread in a program that uses COM exists in an apartment. If a |
| thread wishes to use an object from another apartment, marshalling and |
| the whole DCOM infrastructure gets involved to make that happen behind |
| the scenes. |
| </para> |
| |
| <para> |
| So. Each COM object resides in an apartment, and each apartment |
| resides in a process, and each process resides in a machine, and each |
| machine resides in a network. Allowing those objects to be used |
| from <emphasis>any</emphasis> of these different places is what DCOM |
| is all about. |
| </para> |
| |
| <para> |
| The process of marshalling refers to taking a function call in an |
| apartment and actually performing it in another apartment. Let's say you |
| have two machines, A and B, and on machine B there is an object sitting |
| in a DLL on the hard disk. You want to create an instance of that object |
| (activate it) and use it as if you had compiled it into your own |
| program. This is hard, because the remote object is expecting to be |
| called by code in its own address space - it may do things like accept |
| pointers to linked lists and even return other objects. |
| </para> |
| |
| <para> |
| Very basic marshalling is easy enough to understand. You take a method |
| on a remote interface (that is a COM interface that is |
| implemented on the remote computer), copy each of its |
| parameters into a buffer, and |
| send it to the remote computer. On the other end, the remote server |
| reads each parameter from the buffer, calls the method, writes the |
| result into another buffer and sends it back. |
| </para> |
| |
| <para> |
| The tricky part is exactly how to encode those parameters in the buffer, |
| and how to convert standard stdcall/cdecl method calls to network |
| packets and back again. This is the job of the RPCRT4.DLL file - or the |
| Remote Procedure Call Runtime. |
| </para> |
| |
| <para> |
| The backbone of DCOM is this RPC runtime, which is an implementation |
| of <ulink |
| url="http://www.opengroup.org/onlinepubs/009629399/toc.htm">DCE |
| RPC</ulink>. DCE RPC is not naturally object oriented, so this |
| protocol is extended with some new constructs and by assigning new |
| meanings to some of the packet fields, to produce ORPC or Object |
| RPC. You might see it called MS-RPC as well. |
| </para> |
| |
| <para> |
| RPC packets contain a buffer containing marshalled data in NDR format. |
| NDR is short for "Network Data Representation" and is similar |
| to the XDR |
| format used in SunRPC (the closest native equivalent on Linux to DCE |
| RPC). NDR/XDR are all based on the idea of graph serialization and were |
| worked out during the 80s, meaning they are very powerful and can do |
| things like marshal doubly linked lists and other rather tricky |
| structures. |
| </para> |
| |
| <para> |
| In Wine, our DCOM implementation is <emphasis>not</emphasis> |
| currently based on the |
| RPC runtime, as while few programs use DCOM even fewer use |
| RPC directly so it was developed some time after |
| OLE32/OLEAUT32 were. Eventually this will have to be fixed, |
| otherwise our DCOM will never be compatible with |
| Microsoft's. Bear this in mind as you read through the code |
| however. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Proxies and Stubs</title> |
| |
| <para> |
| Manually marshalling and unmarshalling each method call using the NDR |
| APIs (NdrConformantArrayMarshall etc) is very tedious work, so the |
| Platform SDK ships with a tool called "midl" which is an IDL compiler. |
| IDL or the "Interface Definition Language" is a tool designed |
| specifically for describing interfaces in a reasonably language neutral |
| fashion, though in reality it bears a close resemblence to C++. |
| </para> |
| |
| <para> |
| By describing the functions you want to expose via RPC in IDL therefore, |
| it becomes possible to pass this file to MIDL which spits out a huge |
| amount of C source code. That code defines functions which have the same |
| prototype as the functions described in your IDL but which internally |
| take each argument, marshal it using Ndr, send the packet, and unmarshal |
| the return. |
| </para> |
| |
| <para> |
| Because this code proxies the code from the client to the server, the |
| functions are called proxies. Easy, right? |
| </para> |
| |
| <para> |
| Of course, in the RPC server process at the other end, you need some way |
| to unmarshal the RPCs, so you have functions also generated by MIDL |
| which are the inverse of the proxies; they accept an NDR buffer, extract |
| the parameters, call the real function and then marshal the result back. |
| They are called stubs, and stand in for the real calling code in the |
| client process. |
| </para> |
| |
| <para> |
| The sort of marshalling/unmarshalling code that MIDL spits out can be |
| seen in dlls/oleaut32/oaidl_p.c - it's not exactly what it would look |
| like as that file contains DCOM proxies/stubs which are different, but |
| you get the idea. Proxy functions take the arguments and feed them to |
| the NDR marshallers (or picklers), invoke an NdrProxySendReceive and |
| then convert the out parameters and return code. There's a ton of goop |
| in there for dealing with buffer allocation, exceptions and so on - it's |
| really ugly code. But, this is the basic concept behind DCE RPC. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Interface Marshalling</title> |
| |
| <para> |
| Standard NDR only knows about C style function calls - they |
| can accept and even return structures, but it has no concept |
| of COM interfaces. Confusingly DCE RPC <emphasis>does</emphasis> have a |
| concept of RPC interfaces which are just convenient ways to |
| bundle function calls together into namespaces, but let's |
| ignore that for now as it just muddies the water. The |
| primary extension made by Microsoft to NDR then was the |
| ability to take a COM interface pointer and marshal that |
| into the NDR stream. |
| </para> |
| |
| <para> |
| The basic theory of proxies and stubs and IDL is still here, but it's |
| been modified slightly. Whereas before you could define a bunch of |
| functions in IDL, now a new "object" keyword has appeared. This tells |
| MIDL that you're describing a COM interface, and as a result the |
| proxies/stubs it generates are also COM objects. |
| </para> |
| |
| <para> |
| That's a very important distinction. When you make a call to a remote |
| COM object you do it via a proxy object that COM has constructed on the |
| fly. Likewise, a stub object on the remote end unpacks the RPC packet |
| and makes the call. |
| </para> |
| |
| <para> |
| Because this is object-oriented RPC, there are a few complications: for |
| instance, a call that goes via the same proxies/stubs may end up at a |
| different object instance, so the RPC runtime keeps track of "this" and |
| "that" in the RPC packets. |
| </para> |
| |
| <para> |
| This leads naturally onto the question of how we got those proxy/stub |
| objects in the first place, and where they came from. You can use the |
| CoCreateInstanceEx API to activate COM objects on a remote machine, this |
| works like CoCreateInstance API. Behind the scenes, a lot of stuff is |
| involved to do this (like IRemoteActivation, IOXIDResolver and so on) |
| but let's gloss over that for now. |
| </para> |
| |
| <para> |
| When DCOM creates an object on a remote machine, the DCOM runtime on |
| that machine activates the object in the usual way (by looking it up in |
| the registry etc) and then marshals the requested interface back to the |
| client. Marshalling an interface takes a pointer, and produces a buffer |
| containing all the information DCOM needs to construct a proxy object in |
| the client, a stub object in the server and link the two together. |
| </para> |
| |
| <para> |
| The structure of a marshalled interface pointer is somewhat complex. |
| Let's ignore that too. The important thing is how COM proxies/stubs are |
| loaded. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>COM Proxy/Stub System</title> |
| |
| <para> |
| COM proxies are objects that implement both the interfaces needing to be |
| proxied and also IRpcProxyBuffer. Likewise, COM stubs implement |
| IRpcStubBuffer and understand how to invoke the methods of the requested |
| interface. |
| </para> |
| |
| <para> |
| You may be wondering what the word "buffer" is doing in those interface |
| names. I'm not sure either, except that a running theme in DCOM is that |
| interfaces which have nothing to do with buffers have the word Buffer |
| appended to them, seemingly at random. Ignore it and <emphasis>don't let it |
| confuse you</emphasis> :) This stuff is convoluted enough ... |
| </para> |
| |
| <para> |
| The IRpc[Proxy/Stub]Buffer interfaces are used to control the proxy/stub |
| objects and are one of the many semi-public interfaces used in DCOM. |
| </para> |
| |
| <para> |
| DCOM is theoretically an internet RFC |
| <ulink url="http://www.grimes.demon.co.uk/DCOM/DCOMSpec.htm">[2]</ulink> and is |
| specced out, but in reality the only implementation of it apart from |
| ours is Microsoft's, and as a result there are lots of interfaces |
| which <emphasis>can</emphasis> be used if you want to customize or |
| control DCOM but in practice are badly documented or not documented at |
| all, or exist mostly as interfaces between MIDL generated code and COM |
| itself. Don't pay too much attention to the MSDN definitions of these |
| interfaces and APIs. |
| </para> |
| |
| <para> |
| COM proxies and stubs are like any other normal COM object - they are |
| registered in the registry, they can be loaded with CoCreateInstance and |
| so on. They have to be in process (in DLLs) however. They aren't |
| activated directly by COM however, instead the process goes something |
| like this: |
| |
| <itemizedlist> |
| <listitem> <para> COM receives a marshalled interface packet, and retrieves the IID of |
| the marshalled interface from it </para> </listitem> |
| |
| |
| <listitem> <para> COM looks in |
| HKEY_CLASSES_ROOT/Interface/{whatever-iid}/ProxyStubClsId32 |
| to retrieve the CLSID of another COM object, which |
| implements IPSFactoryBuffer. </para> </listitem> |
| |
| <listitem> <para> IPSFactoryBuffer has only two methods, CreateProxy and CreateStub. COM |
| calls whichever is appropriate: CreateStub for the server, CreateProxy |
| for the client. MIDL will normally provide an implementation of this |
| object for you in the code it generates. </para></listitem> |
| </itemizedlist> |
| |
| </para> |
| |
| <para> |
| Once CreateProxy has been called, the resultant object is QueryInterfaced to |
| IRpcProxyBuffer, which only has 1 method, IRpcProxyBuffer::Connect. |
| This method only takes one parameter, the IRpcChannelBuffer object which |
| encapsulates the "RPC Channel" between the client and server. |
| </para> |
| |
| <para> |
| On the server side, a similar process is performed - the PSFactoryBuffer |
| is created, CreateStub is called, result is QId to IRpcStubBuffer, and |
| IRpcStubBuffer::Connect is used to link it to the RPC channel. |
| </para> |
| |
| </sect2> |
| |
| <sect2> |
| <title>RPC Channels</title> |
| |
| <para> |
| Remember the RPC runtime? Well, that's not just responsible for |
| marshalling stuff, it also controls the connection and protocols between |
| the client and server. We can ignore the details of this for now, |
| suffice it to say that an RPC Channel is a COM object that implements |
| IRpcChannelBuffer, and it's basically an abstraction of different RPC |
| methods. For instance, in the case of inter-thread marshalling (not |
| covered here) the RPC connection code isn't used, only the NDR |
| marshallers are, so IRpcChannelBuffer in that case isn't actually |
| implemented by RPCRT4 but rather just by the COM/OLE DLLS. |
| </para> |
| |
| <para> |
| On this topic, Ove Kaaven says: It depends on the Windows version, I |
| think. Windows 95 and Windows NT 4 certainly had very different models |
| when I looked. I'm pretty sure the Windows 98 version of RPCRT4 was |
| able to dispatch messages directly to individual apartments. I'd be |
| surprised if some similar functionality was not added to Windows |
| 2000. After all, if an object on machine A wanted to use an object on |
| machine B in an apartment C, wouldn't it be most efficient if the RPC |
| system knew about apartments and could dispatch the message directly |
| to it? And if RPC does know how to efficiently dispatch to apartments, |
| why should COM duplicate this functionality? There were, however, no |
| unified way to tell RPC about them across Windows versions, so in that |
| old patch of mine, I let the COM/OLE dlls do the apartment dispatch, |
| but even then, the RPC runtime was always involved. After all, it |
| could be quite tricky to tell whether the call is merely interthread, |
| without involving the RPC runtime... |
| </para> |
| |
| <para> |
| RPC channels are constructed on the fly by DCOM as part of the |
| marshalling process. So, when you make a call on a COM proxy, it goes |
| like this: |
| </para> |
| |
| <para> |
| Your code -> COM proxy object -> RPC Channel -> COM stub object -> Their code |
| </para> |
| |
| </sect2> |
| |
| <sect2> |
| <title>How this actually works in Wine</title> |
| |
| <para> |
| Right now, Wine does not use the NDR marshallers or RPC to implement its |
| DCOM. When you marshal an interface in Wine, in the server process a |
| _StubMgrThread thread is started. I haven't gone into the stub manager |
| here. The important thing is that eventually a _StubReaderThread is |
| started which accepts marshalled DCOM RPCs, and then passes them to |
| IRpcStubBuffer::Invoke on the correct stub object which in turn |
| demarshals the packet and performs the call. The threads started by our |
| implementation of DCOM are never terminated, they just hang around until |
| the process dies. |
| </para> |
| |
| <para> |
| Remember that I said our DCOM doesn't use RPC? Well, you might be |
| thinking "but we use IRpcStubBuffer like we're supposed to ... isn't |
| that provided by MIDL which generates code that uses the NDR APIs?". If |
| so pat yourself on the back, you're still with me. Go get a cup of |
| coffee. |
| </para> |
| |
| </sect2> |
| |
| <sect2> |
| <title>Typelib Marshaller</title> |
| |
| <para> |
| In fact, the reason for the PSFactoryBuffer layer of indirection is |
| because not all interfaces are marshalled using MIDL generated code. |
| Why not? Well, to understand <emphasis>that</emphasis> |
| you have to see that one of the |
| driving forces behind OLE and by extension DCOM was the development of |
| Visual Basic. Microsoft wanted VB developers to be first class citizens |
| in the COM world, but things like writing IDL and compiling them with a |
| C compiler into DLLs wasn't easy enough. |
| </para> |
| |
| <para> |
| So, type libraries were invented. Actually they were invented as part of |
| a parallel line of COM development known as "OLE Automation", but let's |
| not get into that here. Type libraries are basically binary IDL files, |
| except that despite there being two type library formats neither of them |
| can fully express everything expressable in IDL. Anyway, with a type |
| library (which can be embedded as a resource into a DLL) you have |
| another option beyond compiling MIDL output - you can set the |
| ProxyStubClsId32 registry entry for your interfaces to the CLSID of the |
| "type library marshaller" or "universal marshaller". Both terms are |
| used, but in the Wine source it's called the typelib marshaller. |
| </para> |
| |
| <para> |
| The type library marshaller constructs proxy and stub objects on the |
| fly. It does so by having generic marshalling glue which reads the |
| information from the type libraries, and takes the parameters directly |
| off the stack. The CreateProxy method actually builds a vtable out of |
| blocks of assembly stitched together which pass control to _xCall, which |
| then does the marshalling. You can see all this magic in |
| dlls/oleaut32/tmarshal.c |
| </para> |
| |
| <para> |
| In the case of InstallShield, it actually comes with typelibs for all |
| the interfaces it needs to marshal (fixme: is this right?), but they |
| actually use a mix of MIDL and typelib marshalling. In order to cover up |
| for the fact that we don't really use RPC they're all forced to go via |
| the typelib marshaller - that's what the 1 || hack is for and what the |
| "Registering non-automation type library!" warning is about (I think). |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Apartments</title> |
| |
| <para> |
| Before a thread can use COM it must enter an apartment. Apartments are |
| an abstraction of a COM objects thread safety level. There are many types |
| of apartment but the only two we care about right now are single threaded |
| apartments (STAs) and the multi-threaded apartment (MTA). |
| </para> |
| |
| <para> |
| Any given process may contain at most one MTA and potentially many STAs. |
| This is because all objects in MTAs never care where they are invoked from |
| and hence can all be treated the same. Since objects in STAs do care, they |
| cannot be treated the same. |
| </para> |
| |
| <para> |
| You enter an apartment by calling <function>CoInitializeEx()</function> and |
| passing the desired thread model in as a parameter. The default if you use |
| the deprecated <function>CoInitialize()</function> is a STA, and this is the |
| most common type of apartment used in COM. |
| </para> |
| |
| <para> |
| An object in the multi-threaded apartment may be accessed concurrently by |
| multiple threads: eg, it's supposed to be entirely thread safe. It must also |
| not care about thread-affinity, the object should react the same way no matter |
| which thread is calling it. |
| </para> |
| |
| <para> |
| An object inside a STA does not have to be thread safe, and all calls upon it |
| should come from the same thread - the thread that entered the apartment in |
| the first place. |
| </para> |
| |
| <para> |
| The apartment system was originally designed to deal with the disparity between |
| the Windows NT/C++ world in which threading was given a strong emphasis, and the |
| Visual Basic world in which threading was barely supported and even if it had |
| been fully supported most developers would not have used it. Visual Basic code |
| is not truly multi-threaded, instead if you start a new thread you get an entirely |
| new VM, with separate sets of global variables. Changes made in one thread do |
| <emphasis>not</emphasis> reflect in another, which pretty much violates the |
| expected semantics of multi-threading entirely but this is Visual Basic, so what |
| did you expect? If you access a VB object concurrently from multiple threads, |
| behind the scenes each VM runs in a STA and the calls are marshaled between the |
| threads using DCOM. |
| </para> |
| |
| <para> |
| In the Windows 2000 release of COM, several new types of apartment were added, the |
| most important of which are RTAs (the rental threaded apartment) in which concurrent |
| access are serialised by COM using an apartment-wide lock but thread affinity is |
| not guaranteed. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Structure of a marshaled interface pointer</title> |
| |
| <para> |
| When an interface is marshaled using <function>CoMarshalInterface()</function>, |
| the result is a serialized OBJREF structure. An OBJREF actually contains a union, |
| but we'll be assuming the variant that embeds a STDOBJREF here which is what's |
| used by the system provided standard marshaling. A STDOBJREF (standard object |
| reference) consists of the magic signature 'MEOW', then some flags, then the IID |
| of the marshaled interface. Quite what MEOW stands for is a mystery, but it's |
| definitely not "Microsoft Extended Object Wire". Next comes the STDOBJREF flags, |
| identified by their SORF_ prefix. Most of these are reserved, and their purpose |
| (if any) is unknown, but a few are defined. |
| </para> |
| |
| <para> |
| After the SORF flags comes a count of the references represented by this marshaled |
| interface. Typically this will be 5 in the case of a normal marshal, but may be 0 |
| for table-strong and table-weak marshals (the difference between these is explained below). |
| The reasoning is this: In the general case, we want to know exactly when an object |
| is unmarshaled and released, so we can accurately control the lifetime of the stub |
| object. This is what happens when cPublicRefs is zero. However, in many cases, we |
| only want to unmarshal an object once. Therefore, if we strengthen the rules to say |
| when marshaling that we will only unmarshal once, then we no longer have to know when |
| it is unmarshaled. Therefore, we can give out an arbitrary number of references when |
| marshaling and basically say "don't call me, except when you die." |
| </para> |
| |
| <para> |
| The most interesting part of a STDOBJREF is the OXID, OID, IPID triple. This triple |
| identifies any given marshaled interface pointer in the network. OXIDs are apartment |
| identifiers, and are supposed to be unique network-wide. How this is guaranteed is |
| currently unknown: the original algorithm Windows used was something like the current |
| UNIX time and a local counter. |
| </para> |
| |
| <para> |
| OXIDs are generated and registered with the OXID resolver by performing local RPCs |
| to the RPC subsystem (rpcss.exe). In a fully security-patched Windows system they |
| appear to be randomly generated. This registration is done using the |
| <function>ILocalOxidResolver</function> interface, however the exact structure of |
| this interface is currently unknown. |
| </para> |
| |
| <para> |
| OIDs are object identifiers, and identify a stub manager. The stub manager manages |
| interface stubs. For each exported COM object there are multiple interfaces and |
| therefore multiple interface stubs (<function>IRpcStubBuffer</function> implementations). |
| OIDs are apartment scoped. Each ifstub is identified by an IPID, which identifies |
| a marshaled interface pointer. IPIDs are apartment scoped. |
| </para> |
| |
| <para> |
| Unmarshaling one of these streams therefore means setting up a connection to the |
| object exporter (the apartment holding the marshaled interface pointer) and being |
| able to send RPCs to the right ifstub. Each apartment has its own RPC endpoint and |
| calls can be routed to the correct interface pointer by embedding the IPID into the |
| call using RpcBindingSetObject. IRemUnknown, discussed below, uses a reserved IPID. |
| Please note that this is true only in the current implementation. The native version |
| generates an IPID as per any other object and simply notifies the SCM of this IPID. |
| </para> |
| |
| <para> |
| Both standard and handler marshaled OBJREFs contains an OXID resolver endpoint which |
| is an RPC string binding in a DUALSTRINGARRAY. This is necessary because an OXID |
| alone is not enough to contact the host, as it doesn't contain any network address |
| data. Instead, the combination of the remote OXID resolver RPC endpoint and the OXID |
| itself are passed to the local OXID resolver. It then returns the apartment string binding. |
| </para> |
| |
| <para> |
| This step is an optimisation: technically the OBJREF itself could contain the string |
| binding of the apartment endpoint and the OXID resolver could be bypassed, but by using |
| this DCOM can optimise out a server round-trip by having the local OXID resolver cache |
| the query results. The OXID resolver is a service in the RPC subsystem (rpcss.exe) which |
| implements a raw (non object-oriented) RPC interface called <function>IOXIDResolver</function>. |
| Despite the identical naming convention this is not a COM interface. |
| </para> |
| |
| <para> |
| Unmarshaling an interface pointer stream therefore consists of |
| reading the OXID, OID and IPID from the STDOBJREF, then reading |
| one or more RPC string bindings for the remote OXID resolver. |
| Then <function>RpcBindingFromStringBinding</function> is used |
| to convert this remote string binding into an RPC binding handle |
| which can be passed to the local |
| <function>IOXIDResolver::ResolveOxid</function> implementation |
| along with the OXID. The local OXID resolver consults its list |
| of same-machine OXIDs, then its cache of remote OXIDs, and if |
| not found does an RPC to the remote OXID resolver using the |
| binding handle passed in earlier. The result of the query is |
| stored for future reference in the cache, and finally the |
| unmarshaling application gets back the apartment string binding, |
| the IPID of that apartments <function>IRemUnknown</function> |
| implementation, and a security hint (let's ignore this for now). |
| </para> |
| |
| <para> |
| Once the remote apartments string binding has been located the |
| unmarshalling process constructs an RPC Channel Buffer |
| implementation with the connection handle and the IPID of the |
| needed interface, loads and constructs the |
| <function>IRpcProxyBuffer</function> implementation for that |
| IID and connects it to the channel. Finally the proxy is passed |
| back to the application. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Handling IUnknown</title> |
| |
| <para> |
| There are some subtleties here with respect to IUnknown. IUnknown |
| itself is never marshaled directly: instead a version of it |
| optimised for network usage is used. IRemUnknown is similar in |
| concept to IUnknown except that it allows you to add and release |
| arbitrary numbers of references at once, and it also allows you to |
| query for multiple interfaces at once. |
| </para> |
| |
| <para> |
| IRemUnknown is used for lifecycle management, and for marshaling |
| new interfaces on an object back to the client. Its definition can |
| be seen in dcom.idl - basically the IRemUnknown::RemQueryInterface |
| method takes an IPID and a list of IIDs, then returns STDOBJREFs |
| of each new marshaled interface pointer. |
| </para> |
| |
| <para> |
| There is one IRemUnknown implementation per apartment, not per |
| stub manager as you might expect. This is OK because IPIDs are |
| apartment not object scoped (In fact, according to the DCOM draft |
| spec, they are machine-scoped, but this implies apartment-scoped). |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Table marshaling</title> |
| |
| <para> |
| Normally once you have unmarshaled a marshaled interface pointer |
| that stream is dead, you can't unmarshal it again. Sometimes this |
| isn't what you want. In this case, table marshaling can be used. |
| There are two types: strong and weak. In table-strong marshaling, |
| selected by a specific flag to <function>CoMarshalInterface()</function>, |
| a stream can be unmarshaled as many times as you like. Even if |
| all the proxies are released, the marshaled object reference is |
| still valid. Effectively the stream itself holds a ref on the object. |
| To release the object entirely so its server can shut down, you |
| must use <function>CoReleaseMarshalData()</function> on the stream. |
| </para> |
| |
| <para> |
| In table-weak marshaling the stream can be unmarshaled many times, |
| however the stream does not hold a ref. If you unmarshal the |
| stream twice, once those two proxies have been released remote |
| object will also be released. Attempting to unmarshal the stream |
| at this point will yield <function>CO_E_DISCONNECTED</function>. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>RPC dispatch</title> |
| |
| <para> |
| Exactly how RPC dispatch occurs depends on whether the exported |
| object is in a STA or the MTA. If it's in the MTA then all is |
| simple: the RPC dispatch thread can temporarily enter the MTA, |
| perform the remote call, and then leave it again. If it's in a |
| STA things get more complex, because of the requirement that only |
| one thread can ever access the object. |
| </para> |
| |
| <para> |
| Instead, when entering a STA a hidden window is created implicitly |
| by COM, and the user must manually pump the message loop in order |
| to service incoming RPCs. The RPC dispatch thread performs the |
| context switch into the STA by sending a message to the apartments |
| window, which then proceeds to invoke the remote call in the right |
| thread. |
| </para> |
| |
| <para> |
| RPC dispatch threads are pooled by the RPC runtime. When an incoming |
| RPC needs to be serviced, a thread is pulled from the pool and |
| invokes the call. The main RPC thread then goes back to listening |
| for new calls. It's quite likely for objects in the MTA to therefore |
| be servicing more than one call at once. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Message filtering and re-entrancy</title> |
| |
| <para> |
| When an outgoing call is made from a STA, it's possible that the |
| remote server will re-enter the client, for instance to perform a |
| callback. Because of this potential re-entrancy, when waiting for |
| the reply to an RPC made inside a STA, COM will pump the message loop. |
| That's because while this thread is blocked, the incoming callback |
| will be dispatched by a thread from the RPC dispatch pool, so it |
| must be processing messages. |
| </para> |
| |
| <para> |
| While COM is pumping the message loop, all incoming messages from |
| the operating system are filtered through one or more message filters. |
| These filters are themselves COM objects which can choose to discard, |
| hold or forward window messages. The default message filter drops all |
| input messages and forwards the rest. This is so that if the user |
| chooses a menu option which triggers an RPC, they then cannot choose |
| that menu option *again* and restart the function from the beginning. |
| That type of unexpected re-entrancy is extremely difficult to debug, |
| so it's disallowed. |
| </para> |
| |
| <para> |
| Unfortunately other window messages are allowed through, meaning that |
| it's possible your UI will be required to repaint itself during an |
| outgoing RPC. This makes programming with STAs more complex than it |
| may appear, as you must be prepared to run all kinds of code any time |
| an outgoing call is made. In turn this breaks the idea that COM |
| should abstract object location from the programmer, because an |
| object that was originally free-threaded and is then run from a STA |
| could trigger new and untested codepaths in a program. |
| </para> |
| </sect2> |
| |
| <sect2> |
| <title>Wrapup</title> |
| |
| <para> |
| Theres are still a lot of topics that have not been covered: |
| </para> |
| |
| <itemizedlist> |
| <listitem><para> Format strings/MOPs</para></listitem> |
| |
| <listitem><para> IRemoteActivation</para></listitem> |
| |
| <listitem><para> Complex/simple pings, distributed garbage collection</para></listitem> |
| |
| <listitem><para> Marshalling IDispatch</para></listitem> |
| |
| <listitem><para> ICallFrame</para></listitem> |
| |
| <listitem><para> Interface pointer swizzling</para></listitem> |
| |
| <listitem><para> Runtime class object registration (CoRegisterClassObject), ROT</para></listitem> |
| |
| <listitem><para> Exactly how InstallShield uses DCOM</para></listitem> |
| </itemizedlist> |
| |
| </sect2> |
| |
| <sect2> |
| <title>Further Reading</title> |
| |
| <para> |
| Most of these documents assume you have knowledge only contained in |
| other documents. You may have to reread them a few times for it all to |
| make sense. Don't feel you need to read these to understand DCOM, you |
| don't, you only need to look at them if you're planning to help |
| implement it. |
| </para> |
| |
| <itemizedlist> |
| <listitem><para> |
| <ulink url="http://www-csag.ucsd.edu/individual/achien/cs491-f97/projects/dcom-writeup.ps"> |
| http://www-csag.ucsd.edu/individual/achien/cs491-f97/projects/dcom-writeup.ps</ulink> |
| |
| </para></listitem> |
| |
| <listitem><para> |
| <ulink url="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/com/htm/cmi_n2p_459u.asp"> |
| http://msdn.microsoft.com/library/default.asp?url=/library/en-us/com/htm/cmi_n2p_459u.asp</ulink> |
| |
| </para></listitem> |
| |
| |
| <listitem><para> |
| <ulink url="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/com/htm/cmi_q2z_5ygi.asp"> |
| http://msdn.microsoft.com/library/default.asp?url=/library/en-us/com/htm/cmi_q2z_5ygi.asp</ulink> |
| </para></listitem> |
| |
| |
| <listitem><para> |
| <ulink url="http://www.microsoft.com/msj/0398/dcom.aspx"> |
| http://www.microsoft.com/msj/0398/dcom.aspx</ulink> |
| </para></listitem> |
| |
| <listitem><para> |
| <ulink url="http://www.microsoft.com/ntserver/techresources/appserv/COM/DCOM/4_ConnectionMgmt.asp"> |
| http://www.microsoft.com/ntserver/techresources/appserv/COM/DCOM/4_ConnectionMgmt.asp</ulink> |
| </para></listitem> |
| |
| |
| <listitem><para><ulink url="http://www.idevresource.com/com/library/articles/comonlinux.asp"> |
| http://www.idevresource.com/com/library/articles/comonlinux.asp</ulink> |
| |
| (unfortunately part 2 of this article does not seem to exist anymore, if it was ever written)</para></listitem> |
| </itemizedlist> |
| </sect2> |
| </sect1> |
| </chapter> |
| |
| <!-- Keep this comment at the end of the file |
| Local variables: |
| mode: sgml |
| sgml-parent-document:("wine-devel.sgml" "set" "book" "part" "chapter" "") |
| End: |
| --> |
