OWIN and NGINX

Since a few months I work with non-Microsoft technologies and operating systems. I work with Linux, Puppet, Docker (lightweight Linux containers), Apache, Nginx, Node.js and other. So far, it is fun and I’ve learned a lot. This week I saw a lot of news and buzz around OWIN and Katana project. It seems that OWIN is a hot topic and I decided to give it a try. In this post I will show you how to build OWIN implementation and use Nginx server.

Note: this is proof of concept rather than a production-ready code.

OWIN is a specification that defines a standard interface between .NET web servers and web applications. Its goal is to provide a simple and decoupled way how web frameworks and web servers interact. As the specification states, there is no assembly called OWIN.dll or similar. It is just a way how you can build web applications without dependency on particular web server. Concrete implementations can provide OWIN.dll assembly though.

This is in a contrast with the traditional ASP.NET applications that have a dependency on System.Web.dll assembly. If implemented correctly OWIN eliminates such dependencies. The benefits are that your web application becomes more portable, flexible and lightweight.

Let’s start with the implementation. I modeled my OWIN implementation after the one provided by Microsoft.

public interface IAppBuilder
{
    IDictionary<string, object> Properties { get; }
    object Build(Type returnType);
    IAppBuilder New();
    IAppBuilder Use(object middleware, params object[] args);
}

For the purpose of this post we will implement the Properties property and the Use method. Let’s define our AppFunc application delegate as follows:

delegate Task AppFunc(IDictionary<string, object> environment);

The examples from Katana project provide the following code template for the main function:

static void Main(string[] args)
{
    using (WebApplication.Start<Startup>("http://localhost:5000/"))
    {
        Console.WriteLine("Started");
        Console.ReadKey();
        Console.WriteLine("Stopping");
    }
}

I like it very much so I decided to provide WebApplication class with a single Start method:

public static class WebApplication
{
    public static IDisposable Start(string url)
    {
        return new WebServer(typeof(TStartup));
    }
}

We will provide WebServer implementation later. Let’s see what the implementation of Startup class is:

public class Startup
{
    public void Configuration(IAppBuilder app)
    {
        var myModule = new LightweightModule();
        app.Use(myModule);
    }
}

Let’s summarize it. We have a console application and in the main method it calls Start method and passes two parameters: Startup type and a URL. Start method will start a web server that listens for requests on the specified URL and the server will use Startup class to configure the web application. We don’t have any dependency on System.Web.dll assembly. We have a nice and simple decoupling of the web server and the web application.

So far, so good. Let’s see how the web server configures the web application. In our OWIN implementation we will use reflection to reflect TStartup type and try find Configuration method using naming convention and predefined method signature. The Configuration method instantiates LightweightModule object and passes it to the web server. The web server will inspect the object for its type and will try to find Invoke method compatible with the AppFunc signature. Once Invoke method is found it will be called for every web request. Here is the actual Use method implementation:

public IAppBuilder Use(object middleware, params object[] args)
{
    var type = middleware.GetType();
    var flags = BindingFlags.Instance | BindingFlags.Public;
    var methods = type.GetMethods(flags);

    // TODO: call method "void Initialize(AppFunc next, ...)" with "args"

    var q = from m in methods
            where m.Name == "Invoke"
            let p = m.GetParameters()
            where (p.Length == 1)
                   && (p[0].ParameterType == typeof(IDictionary<string, object>))
                   && (m.ReturnType == typeof(Task))
            select m;

    var candidate = q.FirstOrDefault();

    if (candidate != null)
    {
        var appFunc = Delegate.CreateDelegate(typeof(AppFunc), middleware, candidate) as AppFunc;
        this.registeredMiddlewareObjects.Add(appFunc);
    }

    return this;
}

Finally we come to WebServer implementation. This is where Nginx comes. For the purpose of this post we will assume that Nginx server is started and configured. You can easily extend this code to start Nginx via System.Diagnostics.Process class. I built and tested this example with Nginx version 1.4.2. Let’s see how we have to configure Nginx server. Open nginx.conf file and find the following settings:

    server {
        listen       80;
        server_name  localhost;

and change the port to 5000 (this is the port we use in the example). A few lines below you should see the following settings:

        location / {
            root   html;
            index  index.html index.htm;
        }

You should modify it as follows:

        location / {
            root   html;
            index  index.html index.htm;
            fastcgi_index Default.aspx;
            fastcgi_pass 127.0.0.1:9000;
            include fastcgi_params;
        }

That’s all. In short, we configured Nginx to listen on port 5000 and configured fastcgi settings. With these settings Nginx will pass every request to a FastCGI server at 127.0.0.1:9000 using FastCGI protocol. FastCGI is a protocol for interfacing programs with a web server.

So, now we need a FastCGI server. Implementing FastCGI server is not hard but for the sake of this post we will use SharpCGI implementation. We are going to use SharpCGI library in WebServer implementation. First, we have to start listening on port 9000:

private void Start()
{
    var config = new Options();
    config.Bind = BindMode.CreateSocket;
    var addr = IPAddress.Parse("127.0.0.1");
    config.EndPoint = new IPEndPoint(addr, 9000);
    config.OnError = Console.WriteLine;
    Server.Start(this.HandleRequest, config);
}

The code is straightforward and the only piece we haven’t look at is HandleRequest method. This is where web requests are processed:

private void HandleRequest(Request req, Response res)
{
    var outputBuff = new byte[1000];

    // TODO: use middleware chaining instead a loop

    foreach (var appFunc in this.appBuilder.RegisteredMiddlewareObjects)
    {
        using (var ms = new MemoryStream(outputBuff))
        {
            this.appBuilder.Properties["owin.RequestPath"] = req.ScriptName.Value;
            this.appBuilder.Properties["owin.RequestQueryString"] = req.QueryString.Value;

            this.appBuilder.Properties["owin.ResponseBody"] = ms;
            this.appBuilder.Properties["owin.ResponseStatusCode"] = 0;

            var task = appFunc(this.appBuilder.Properties);

            // TODO: don't task.Wait() and use res.AsyncPut(outputBuff);

            task.Wait();
            res.Put(outputBuff);
        }
    }
}

This was the last piece from our OWIN implementaion. This is where we call the web application specific method via AppFunc delegate.

In closing I think OWIN helps the developers to build better web applications. Please note that my implementation is neither complete neither production-ready. There is a lot of room for improvement. You can find the source code here:

OWINDemo.zip

 

 

Software Razzie Awards

This is not a new idea. Every now and then someone suggests it. Apparently I hear about it more often than, say, 5 years ago. Sure, today software is more spread than 5 years ago but somehow I don’t think this is the only reason.

I guess the main reason for people dissatisfaction is that nowadays people have higher expectation of the product’s quality. Let’s take for example the mobile apps. Everyone expects mobile apps to work fast and smooth and to provide good user experience. These expectations are transferred to the PCs as well. What was acceptable 5 years ago is not anymore.

Also I don’t think software became worse with time. Sure, there are products affected from software bloat. Some notorious examples are Nero Burning ROM and iTunes. Although software bloat manifests in APIs (e.g. Win32 API and Linux Kernel API) and frameworks (e.g. JDK and .NET) there is a tendency that most developers try to minimize and control their code. As a result a new wave of lightweight software (e.g. Google Chrome, Node.js, nginx, etc.) becomes popular.

So, what if there are Software Razzie Awards? There are Pwnie Awards but they are very security focused. I still cannot decide if such awards can be stimulating for the IT industry or not. I guess it won’t harm anyone.

Software agents from the past

Nowadays, we are all familiar with Siri, Google Now and TellMe but are these intelligent personal assistants something new? Probably you remember Clippy or Rover that Microsoft introduced in the late 1990s and early 2000s. Or probably you remember the intelligent personal assistants for Mac OS or OS/2.

Today, I would like to remind you about the agents built by SRI International back in 1990s. Check these short videos out.

You can find the original videos at http://www.ai.sri.com/~oaa/distrib.html#videos.

Dispose Pattern

There are a lot of blog posts and articles about the proper IDisposable implementation, so why writing another one? I guess I am writing it because the topic is quite subjective as there are many different scenarios for IDisposable usage.

I am not going to explore all the details about IDisposable implementation. I would recommend the following readings though I would advise you to be a selective reader so don’t take everything for granted truth:

These are valuable articles because they contain a lot of different opinions. I would also recommend the following two MSDN links which you may find contradicting:

While the most people talk about Dispose and Finalize patterns as different things I would recommend to be careful and to think about them as a single pattern. It is always hard, even impossible, to predict how your code will be used from other software developers. So, having a safe strategy and implementing Dispose and Finalize patterns together might be the right choice.

Let’s see the following class definition:

public class ProcessInfo : IDisposable
{
    [DllImport("kernel32")]
    static extern IntPtr OpenProcess(int dwDesiredAccess, bool bInheritHandle, int dwProcessId);

    [DllImport("kernel32")]
    static extern int GetCurrentProcessId();

    [DllImport("kernel32")]
    static extern bool CloseHandle(IntPtr hHandle);

    private readonly IntPtr hProcess;

    public ProcessInfo()
    {
        const int PROCESS_VM_READ = 0x10;

        this.hProcess = OpenProcess(PROCESS_VM_READ, false, GetCurrentProcessId());
    }

    public void Dispose()
    {
        this.Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (this.hProcess != IntPtr.Zero)
        {
            CloseHandle(hProcess);
        }
    }
}

It is not the best example but let’s pretend this class has some useful methods and properties for the current process. Because the class implements IDisposable interface it is reasonable to expect something like the following code:

using (var procInfo = new ProcessInfo())
{
    // call some methods and properties
}

The code fragment above seems all good and fine. The tricky part is that the C# compiler will actually emit the following equivalent code:

ProcessInfo procInfo = null;
try
{
    procInfo = new ProcessInfo();
    // call some methods and properties
}
finally
{
    if (procInfo != null)
    {
        procInfo.Dispose();
    }
}

Let’s focus on the constructor. The main purpose of the constructor is to construct an object. This means that the constructed object should be in a usable state. This concept is deeply embraced in .NET base class library. Let’s see the following code fragment:

var fs = new FileStream("", FileMode.Open);

As you can guess, it will throw ArgumentException. Let’s see another canonical example:

public class Person
{
    public Person(string name, uint age)
    {
        if (string.IsNullOrWhiteSpace(name))
        {
            throw new ArgumentException("invalid name", "name");
        }

        if (age < MIN_AGE)
        {
            throw new ArgumentException("invalid age", "age");
        }

        // store "name" and "age"
    }
    // Some useful methods and properties
}

Most of us have written such code. Often it is perfectly legal to throw an exception when we cannot construct a usable object. Back to our example:

using (var procInfo = new ProcessInfo())
{
    // call some methods and properties
}

If an exception is thrown in the constructor then procInfo variable won’t be assigned and therefore Dispose() method won’t be called. Let’s modify ProcInfo constructor a bit:

public ProcessInfo()
{
    const int PROCESS_VM_READ = 0x10;

    this.hProcess = OpenProcess(PROCESS_VM_READ, false, GetCurrentProcessId());

    // something wrong
    throw new Exception();
}

Oops, we have Win32 handle leak. One option is to use try-catch clause inside the constructor and call CloseHandle(…) method. Sometimes this is the best option.

Another option is to implement the Finalize pattern. It is easy and I find it more clear:

~ProcessInfo()
{
    this.Dispose(false);
}

Now, there are clear roles for the constructor and Dispose/Finalize methods and the Win32 handle leak is fixed.

Introduction to CLR metadata

If you are a .NET developer then you’ve probably heard or read about CLR metadata. In this blog post I will try to give you a pragmatic view of what CLR metadata is and how to work with it. You are going to learn how to use unmanaged metadata API from C# code. I will try to keep the blog post brief and clear, so I am going to skip some details where is possible. For more information, please refer to the links at the end.

A Simple Case Example

To make things real we will try to solve the following problem. Suppose we have the following interface definition in C#:

interface IFoo
{
    string Bar { get; }
    string Baz { set; }
}

Our goal is to get the property names (think of System.Reflection.PropertyInfo) with a pretty and fluent syntax in C#. For property Bar the task is trivial:

IFoo foo = null;
Expression<Func<string>> expr = () => foo.Bar;
var pi = (expr.Body as MemberExpression).Member as PropertyInfo;

In the case when the property has a getter we can leverage the power of C# compiler and use C# expression trees. The things for property Baz are more complicated. Usually, we end up something like this:

IFoo foo = null;
Action act = () => foo.Baz = "";

In this case the compiler emits a new type and the things get ugly because we have to decompile IL code in order to get the property name. As we are going to see, we will use the metadata API for a very simple form of decompilation.

Background

The term metadata refers to “data about data”.
— wikipedia

When it comes to CLR metadata the term metadata refers to what is known as descriptive metadata. I promised that I will be pragmatic, so let’s see a real example. I guess that you, as a .NET developer, know that Microsoft promotes .NET as a platform that solve the problem known as DLL hell. The main argument is that .NET provides a feature called Strong-Name Signing that assigns a unique identity to an assembly. Suppose we have two assemblies LibraryA.DLL and LibraryB.DLL and the former depends on the latter.

libalibb

Suppose that during deployment we accidentally deploy LibraryB.DLL version 1.0.0.0 instead of version 2.5.0.0. The next time CLR tries to load the assembly LibraryA.DLL it will detect that we have deployed a wrong version of LibraryB.DLL and will fail with assembly load exception. Let’s see another scenario. You find and fix a bug in assembly LibraryB.DLL and replace the older file with the new one. Your existing assembly LibraryA.DLL continues to work as expected.

From these two scenarios, it is easy to guess that assembly LibraryA.DLL has enough information about LibraryB.DLL so that the CLR can spot the difference in case of improper deployment. This information comes from the CLR metadata. Assembly LibraryB.DLL has metadata that stores information about it (assembly name, assembly version, public types, etc.). When the compiler generates assembly LibraryA.DLL it stores in its metadata information about assembly LibraryB.DLL.

The CLR metadata is organized as a normalized relational database. This means that CLR metadata consists of rectangular tables with foreign keys between each other. In case you have previous experience with databases you should be familiar with this concept. Since .NET 2.0 there are more than 40 tables that define the CLR metadata. Here is a sample list of some of the tables:

  • Module (0x00) – contains information about the current module
  • TypeRef (0x01) – contains information about the types that are referenced from other modules
  • TypeDef (0x02) – contains information about the types defined in the current module
  • Field (0x04) – contains information about the fields defined in the current module
  • Method (0x06) – contains information about the methods defined in the current module
  • Property (0x17) – contains information about the properties defined in the current module
  • Assembly (0x20) – contains information about the current assembly
  • AssemblyRef (0x23) – contains information about the referenced assemblies

We can see that most of the information is organized around the current module. As we will see it soon, this organizational model is reflected in the metadata API. For now, it is important to say that an assembly has one or more modules. The same model is used in System.Reflection API as well. You can read more about it here.

Internally, the CLR metadata is organized in a very optimized format. Instead of using human readable table names it uses numbers (table identifiers, TIDs). In the table list above the numbers are shown in hexadecimal format. To understand the CLR metadata, we have to know another important concept – Record Index (RID). Let’s have a look at TypeDef table. Each row (record) in TypeDef table describes a type that is defined in the current module. The record indexes are continuously growing starting from 1. The tuple (TID, RID) is enough to unambiguously identify every metadata record in the current module. This tuple is called metadata token. In the current CLR implementation metadata tokens are 4 byte integers where the highest byte is TID and the rest 3 bytes are used for RID. For example, the metadata token 0x02000006 identifies the 6th record from TypeDef table.

Let’s see the content of metadata tables. The easiest way is to use ILDASM tool. For the sake of this article I will load System.DLL assembly.

ildasm0

Once the assembly is loaded you can see the CLR metadata via View->MetaInfo->Show menu item (you can use CTRL+M shortcut as well). It takes some time to load all the metadata. Let’s first navigate to AssemblyRef (0x23) table.

ildasm2

As we can see the assembly System.DLL refers to 3 other assemblies (mscorlib, System.Configuration and System.Xml). For each row (record), we can see what the actual column (Name, Version, Major, Minor, etc.) values are. Let’s have a look at TypeRef (0x01) table. The first record describes a reference to System.Object type from assembly with token 0x23000001, which is mscorlib (see the screenshot above). The next referred type is System.Runtime.Serialization.ISerializable and so on.

ildasm3ildasm4
By now, I hope you understand at high level what CLR metadata is and how it is organized. There are a lot of tricky technical details about the actual implementation, but they are not important for the sake of this article. I encourage you to play with ILDASM and try the different views under View->MetaInfo menu. For more information, please refer to the links at the end.

Let’s write some code

Microsoft provides unmanaged (COM based) API to work with metadata. This API is used by debuggers, profilers, compilers, decompilers and other tools. First, we have to obtain an instance of CorMetaDataDispenser which implements IMetaDataDispenserEx interface. Probably, the easiest way is to use the following type definition:

[ComImport, GuidAttribute("E5CB7A31-7512-11D2-89CE-0080C792E5D8")]
class CorMetaDataDispenserExClass
{
}
//
var dispenser = new CorMetaDataDispenserExClass();

The value “E5CB7A31-7512-11D2-89CE-0080C792E5D8” passed to the GuidAttribute constructor is the string representation of the CLSID_CorMetaDataDispenser value defined in cor.h file from the Windows SDK. Another way to obtain an instance of the metadata dispenser is through Type/Activator collaboration:

Guid clsid = new Guid("E5CB7A31-7512-11D2-89CE-0080C792E5D8");
Type dispenserType = Type.GetTypeFromCLSID(clsid);
object dispenser = Activator.CreateInstance(dispenserType);

Either way, we cannot do much with the dispenser instance right now. As we said earlier, we need an instance of IMetaDataDispenserEx interface, so let’s cast it to IMetaDataDispenserEx interface:

var dispenserEx = dispenser as IMetaDataDispenserEx;

The CLR will do QueryInterface for us and return the right interface type. Let’s see IMetaDataDispenserEx definition:

[ComImport, GuidAttribute("31BCFCE2-DAFB-11D2-9F81-00C04F79A0A3"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IMetaDataDispenserEx
{
    uint DefineScope(ref Guid rclsid, uint dwCreateFlags, ref Guid riid, out object ppIUnk);

    uint OpenScope(string szScope, uint dwOpenFlags, ref Guid riid, out object ppIUnk);

    uint OpenScopeOnMemory(IntPtr pData, uint cbData, uint dwOpenFlags, ref Guid riid, out object ppIUnk);

    uint SetOption(ref Guid optionid, object value);

    uint GetOption(ref Guid optionid, out object pvalue);

    uint OpenScopeOnITypeInfo(ITypeInfo pITI, uint dwOpenFlags, ref Guid riid, out object ppIUnk);

    uint GetCORSystemDirectory(char[] szBuffer, uint cchBuffer, out uint pchBuffer);

    uint FindAssembly(string szAppBase, string szPrivateBin, string szGlobalBin, string szAssemblyName, char[] szName, uint cchName, out uint pcName);

    uint FindAssemblyModule(string szAppBase, string szPrivateBin,string szGlobalBin, string szAssemblyName, string szModuleName, char[] szName, uint cchName, out uint pcName);
}

For the sake of readability, I removed all the marshaling hints that the CLR needs when doing the COM interop. Let’s see some of the details in this interface definition. The value “31BCFCE2-DAFB-11D2-9F81-00C04F79A0A3” is the string representation of IID_IMetaDataDispenserEx value defined in cor.h file from the Windows SDK. For the purpose of this article we are going to call the following method only:

uint OpenScope(string szScope, uint dwOpenFlags, ref Guid riid, out object ppIUnk);

Here goes the method description from the MSDN documentation:

  • szScope – The name of the file to be opened. The file must contain common language runtime (CLR) metadata
  • dwOpenFlags – A value of the CorOpenFlags enumeration to specify the mode (read, write, and so on) for opening
  • riid – The IID of the desired metadata interface to be returned; the caller will use the interface to import (read) or emit (write) metadata. The value of riid must specify one of the “import” or “emit” interfaces. Valid values are IID_IMetaDataEmit, IID_IMetaDataImport, IID_IMetaDataAssemblyEmit, IID_IMetaDataAssemblyImport, IID_IMetaDataEmit2, or IID_IMetaDataImport2
  • ppIUnk – The pointer to the returned interface

The most interesting parameter is the third one (riid). We have to pass the GUID of the interface we want to get. In our case this is IMetaDataImport  interface (import interfaces are used to read metadata while emit interfaces are used to write metadata).

So, to obtain an instance of IMetaDataImport  interface we have to find its GUID from cor.h file and call OpenScope method:

Guid metaDataImportGuid = new Guid("7DAC8207-D3AE-4c75-9B67-92801A497D44");
object rawScope = null;
var hr = dispenserEx.OpenScope("myAssembly.dll", 0, ref metaDataImportGuid, out rawScope);
var import = rawScope as IMetaDataImport;

Finally, we have an instance of IMetaDataImport interface. Because the interface has more than 60 methods I present here only the methods we are going to use:

[ComImport, GuidAttribute("7DAC8207-D3AE-4C75-9B67-92801A497D44"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IMetaDataImport
{
    void CloseEnum(IntPtr hEnum);

    uint GetScopeProps(char[] szName, uint cchName, out uint pchName, ref Guid pmvid);

    uint GetTypeRefProps(uint tr, out uint ptkResolutionScope, char[] szName, uint cchName, out uint pchName);

    uint ResolveTypeRef(uint tr, ref Guid riid, out object ppIScope, out uint ptd);

    uint EnumMethods(ref IntPtr phEnum, uint cl, uint[] rMethods, uint cMax, out uint pcTokens);

    uint GetMethodProps(uint mb, out uint pClass, char[] szMethod, uint cchMethod, out uint pchMethod, out uint pdwAttr, out IntPtr ppvSigBlob, out uint pcbSigBlob, out uint pulCodeRVA, out uint pdwImplFlags);

    uint GetMemberRefProps(uint mr, out uint ptk, char[] szMember, uint cchMember, out uint pchMember, out IntPtr ppvSigBlob, out uint pbSigBlob);

    // for brevity, the rest of the methods are omitted
}

Probably the most interesting methods are GetMethodProps and GetMemberRefProps. The first method takes MethodDef token as a first parameter while the second method takes MemberRef token. As we know, MethodDef tokens are stored in Method (0x02) table and MemberRef tokens are stored in MemberRef (0x0a) table. The first ones describe methods defined in the current module while the second ones describe methods (and members) referenced from other modules.

At first sight there is not much difference between MethodDef and MemberRef tokens, but as we are going to see the difference is crucial. Let’s have a look at the following code fragment:

Expression<Func<bool>> expr = () => string.IsNullOrEmpty("");
MethodInfo mi = (expr.Body as MethodCallExpression).Method;
int methodDef = mi.MetadataToken;

The sole purpose of this code is to obtain the MethodDef token of IsNullOrEmpty method. At first it may not be obvious but MetadataToken property returns the MethodDef token of that method. On my machine the actual token value is 0x06000303. So, the table name is MethodDef (0x06) and the RID is 0x303. A quick check with ILDASM tool confirms it:

ildasm5

Let’s experiment and execute the following code fragment:

Type[] mscorlibTypes = typeof(string).Assembly.GetTypes();
var q = from type in mscorlibTypes
        from method in type.GetMethods()
        where method.MetadataToken == 0x06000303
        select method;

MethodInfo mi2 = q.SingleOrDefault();
// "same" is true
bool same = object.ReferenceEquals(mi, mi2);

So, our strategy is to get the MethodDef token of the property setter and then run a LINQ query similar to the one shown above. In reality the code is more complex:

private static MethodInfo GetMethodInfoFromScope(uint token, Assembly assemblyScope, Guid mvid)
{
    MethodInfo mi = null;

    if ((token != mdTokenNil) && (assemblyScope != null) && (mvid != Guid.Empty))
    {
        const BindingFlags flags = BindingFlags.Static | BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic;

        var q = from module in assemblyScope.Modules
                from type in module.GetTypes()
                from method in type.GetMethods(flags)
                where (method.MetadataToken == token)
                       && (method.Module.ModuleVersionId == mvid)
                select method;

        mi = q.SingleOrDefault();
    }

    return mi;
}

Because tokens are unique inside a module only, we have to iterate over all assembly modules and find the right one. Fortunately, each module has version id (module version ID, MVID) which is a unique GUID generated during compilation. For example, the module CommonLanguageRuntimeLibrary from mscorlib assembly installed on my machine has the following MVID “AD977517-F6DA-4BDA-BEEA-FFC184D5CD3F”.

ildasm6

We can obtain MVID via GetScopeProps method from IMetaDataImport interface.

So far we saw how to get a MethodInfo object for any method (including property setters) if we know the MethodDef token and the MVID of the containing module. However, we still don’t know how to proceed in case of MemberRef token. The solution is to find the original MethodDef token.

Here is the high level algorithm we have to follow:

  1. Call GetMemberRefProps to get the method name and the TypeRef token of the containing type (this is the second method parameter ptk)
  2. If TypeRef is a nested type, call GetTypeRefProps as many times as needed to get the most outer TypeRef
  3. Call ResolveTypeRef to get IMetaDataImport interface for the module that contains the corresponding TypeDef token
  4. Call EnumMethods for the TypeDef token from previous step
  5. For each MethodDef from previous step, call GetMethodProps to get its name, compare it with the name from step 1 and if they match return the current MethodDef token

This algorithm can be used for any method MemberRef, not just property setters. The only potential pitfall is the string comparison in the last step. As you know, the CLR supports method overloading. Fortunately, the current .NET compilers with property support don’t allow property overloading. Even if that was the case, we can use some of the flags returned by GetMemberRefProps for unambiguous method resolution.

Here is a high level skeleton of the algorithm:

uint methodRef = 0;
IMetaDataImport import = null;

uint typeRef;
uint memberRefNameLen = 0;
uint memberRefSigLen;
var memberRefName = new char[1024];
IntPtr ptrMemberRefSig;
var hr = import.GetMemberRefProps(methodRef, out typeRef, memberRefName, (uint)memberRefName.Length, out memberRefNameLen, out ptrMemberRefSig, out memberRefSigLen);

var name = new string(memberRefName, 0, (int)memberRefNameLen - 1);

uint resScope;
uint parentNameLen;
hr = import.GetTypeRefProps(typeRef, out resScope, null, 0, out parentNameLen);

uint typeDef = mdTokenNil;
object newRawScope = null;
Guid metaDataImportGuid = new Guid(IID_IMetaDataImport);
hr = import.ResolveTypeRef(typeRef, ref metaDataImportGuid, out newRawScope, out typeDef);

var newImport = newRawScope as IMetaDataImport;

var hEnum = IntPtr.Zero;
var methodTokens = new uint[1024];
var methodName = new char[1024];
uint len;
hr = newImport.EnumMethods(ref hEnum, typeDef, methodTokens, (uint)methodTokens.Length, out len);

for (var i = 0; i < len; i++)
{
    uint tmpTypeDef;
    uint attrs;
    var sig = IntPtr.Zero;
    uint sigLen;
    uint rva;
    uint flags;
    uint methodNameLen;
    hr = newImport.GetMethodProps(methodTokens[i], out tmpTypeDef, methodName, (uint)methodName.Length, out methodNameLen, out attrs, out sig, out sigLen, out rva, out flags);

    var curName = new string(methodName, 0, (int)methodNameLen - 1);

    if (name == curName)
    {
        // found methodDef
        break;
    }
}

The actual implementation is more complex. There are two tricky details:

  1. TypeRef resolution from step 2, in case TypeRef token is a token of a nested type
  2. The assemblyScope parameter in GetMethodInfoFromScope method

To solve the first issue we can simply use a stack data structure. We push all TypeRef tokens from GetTypeRefProps method. Then we pop the tokens and resolve them with ResolveTypeRef method. The second issue may not be obvious. We have to pass an assembly instance to GetMethodInfoFromScope method. However the metadata API does not require the CLR loader to load the assembly (in an AppDomain). In a matter of fact the metadata API does not require CLR to be loaded at all. So we have to take care and load the assembly if needed:

newAssembly = AppDomain.CurrentDomain.GetAssemblies().FirstOrDefault(a => a.GetName().FullName.Equals(asmName.FullName));
if (newAssembly == null)
{
    newAssembly = Assembly.Load(asmName);
}
// pass newAssembly to GetMethodInfoFromScope(...) method

So far, we worked mostly with modules but right now we have to find an assembly by name. This means that we have to obtain the assembly name from the current module. For this purpose we are going to use IMetaDataAssemblyImport interface. This interface has more than a dozen methods, but for the sake of this article we are going to use two methods only:

[ComImport, GuidAttribute("EE62470B-E94B-424E-9B7C-2F00C9249F93"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IMetaDataAssemblyImport
{
    uint GetAssemblyFromScope(out uint ptkAssembly);

    uint GetAssemblyProps(uint mda,
                            out IntPtr ppbPublicKey,
                            out uint pcbPublicKey,
                            out uint pulHashAlgId,
                            char[] szName,
                            uint cchName,
                            out uint pchName,
                            out ASSEMBLYMETADATA pMetaData,
                            out uint pdwAssemblyFlags);

    // for brevity, the rest of the methods are omitted
}

To obtain an instance of IMetaDataAssemblyImport interface we have to cast it from existing an instance of IMetaDataImport interface:

var asmImport = import as IMetaDataAssemblyImport;

Again, the CLR will do QueryInterface for us and return the proper interface. Once we have an instance of IMetaDataAssemblyImport interface we use it as follows:

  1. Call GetAssemblyFromScope to obtain the current assemblyId
  2. Call GetAssemblyProps with the assemblyId from the previous step

Until now, we saw how to use the metadata API from C#. One question left unanswered though. How can we get MethodDef or MemberRef token? At the beginning we started with a simple delegate, so we can use it to get the byte array of IL instructions that represent the method body:

Action act = () => foo.Baz = "";
byte[] il = act.Method.GetMethodBody().GetILAsByteArray();

It is important to recall that our goal is to decompile the method body of the method pointed by this delegate. This method has a few instructions, it simply calls the property setter with some value. We are not going to do any kind of control flow analysis. All we are going to do is to find the last method invocation (in our case the property setter) inside the method body.

Almost every method call in the CLR is done through call or callvirt instruction. Both instructions take a method token as a single parameter. All we have to do is parse the IL byte array for call or callvirt instruction and interpret the next 4 bytes as a method token. We then check whether the token is MethodDef or MemberRef and if this is the case we pass it to the metadata API.

I hope this article gave you an idea what metadata API is and how to use it from C#. I tried to focus on the interfaces needed for solving our specific scenario. The metadata API has much more functionality than what we saw here. It allows you to emit new methods, new types and new assemblies as well.

Source Code

MetadataDemo.zip

References

 

 

Analyzing crash dumps with ClrMD

Microsoft recently released the first beta version of Microsoft.Diagnostics.Runtime component. Lee Culver describes ClrMD as follows:

ClrMD is a set of advanced APIs for programmatically inspecting a crash dump of a .NET program much in the same way as the SOS Debugging Extensions (SOS). It allows you to write automated crash analysis for your applications and automate many common debugger tasks.

Lee Culver also showed a nice LINQ query over heap objects. This example reminds me the SharpDevelop approach to Profiler Query Language (PQL). Here is the short description of PQL:

  •  PQL v1.1 (SharpDevelop 4.x)
    •  Extend PQL capabilities (Views and Categories)
    •  PQL code completion and editing improvements
    •  PQL performance improvements (optimized LINQ implementation LINQ-to-Profiler)

I am very keen on PQL concept and the ClrMD component makes it possible. I decided to have a high-level overview of ClrMD implementation so I loaded the assembly in JustDecompile.

clrmd1

I was not surprised to see that Microsoft reused a lot of code from PerfView which is built around IXCLRDataProcess interface. Actually the latest PerfView version (1.4.1.0) already comes with ClrMD component instead of ClrMemDiag assembly. Something else caught my eye though. This is the Redhawk namespace. And this is a very nice surprise indeed.

For those who are not familiar with Redhawk I would recommend the following Channel 9 page. As far as I know, there is nothing official about Redhawk yet and the ClrMD component might be the first clue. What is coming next? Only time will tell.

Parsing “Unsafe” Method Signatures

If you’ve ever used JustMock then you should be familiar with code fragments like this one:

public interface IFoo
{
    byte Bar(int i, string s, long l);
}

var foo = Mock.Create<IFoo>();
Mock.Arrange(() => foo.Bar(123, "test", 321)).Returns(5);

The interesting thing in this code is in the Arrange method. Its signature is as follows:

public static FuncExpectation<TResult> Arrange<TResult>(Expression<Func<TResult>> expression)

The method accepts a single parameter of Expression<TDelegate> type. This allows JustMock to parse the actual method name and the parameter values. In this blog post I am going to show you a simple and yet effective approach for parsing methods that contain pointer parameters.

Lets make our example more interesting.

public unsafe interface IFoo
{
    byte** Bar(int* i, string s, ref long* l);
}

Because Expression<TDelegate> expects a delegate type as a type parameter we can define a new delegate type and use it to construct a proper expression object.

public unsafe delegate byte** BarDel();
//..
int* pi = (int*)IntPtr.Zero;
long* pl = (long*)IntPtr.Zero;
Expression<BarDel> expr = () => Bar(pi, "test", ref pl);

This is all good and nice. Now we can change Arrange method signature as follows:

// to keep it simple, we changed the return type to void
public static void Arrange<TDelegate>(Expression<TDelegate> expr) { }
//..
int* pi = (int*)IntPtr.Zero;
long* pl = (long*)IntPtr.Zero;
Mock.Arrange<BarDel>(() => Bar(pi, "test", ref pl));

The only thing that left is to make the return type independent from the return type of the actual Bar method. This is easy. A pointer to a pointer to a byte (byte**) is just… a pointer! The only tricky thing is that JustMock doesn’t know what actual type is pointed. So, it is a sane decision to leave this knowledge to the unit test author. We need a simple generalization over IntPtr type and that’s all.

namespace Telerik.JustMock
{
    public class PtrBase
    {
        private readonly IntPtr addr;

        protected PtrBase(IntPtr addr)
        {
            this.addr = addr;
        }

        public IntPtr Addr { get { return this.addr; } }
    }

    public delegate T PtrDel<T>() where T : PtrBase;
}

Now, we have to change Arrange method as follows:

public static void Arrange<TPtr>(Expression<PtrDel<TPtr>> expr, params TPtr[] arr)
    where TPtr : PtrBase
{
    var methodCallExpr = (expr.Body as UnaryExpression).Operand as MethodCallExpression;

    // process methodCallExpr
}

Lets see how we have to change our test code so we can call the new Arrange method. Now, we have to take care for all pointers and convert them. This is a trivial task:

public class Ptr : PtrBase
{
    private Ptr(IntPtr addr) : base(addr) { }

    public unsafe static implicit operator Ptr(int* ptr)
    {
        return new Ptr(new IntPtr(ptr));
    }

    public unsafe static implicit operator Ptr(long* ptr)
    {
        return new Ptr(new IntPtr(ptr));
    }

    public unsafe static implicit operator Ptr(byte** ptr)
    {
        return new Ptr(new IntPtr(ptr));
    }
}

Finally, we have to change our Arrange method invocation:

int* pi = (int*)IntPtr.Zero;
long* pl = (long*)IntPtr.Zero;
Mock.Arrange<Ptr>(() => Foo(pi, "test", ref pl), pi, pl);

The code can be improved in many ways. One way to improve it is to replace PtrDel<T> with Func<TResult> though I prefer the explicit constraint on PtrBase. Another way to improved the code is by making PtrBase abstract type and providing a template method that will be called inside the Arrange method, so the actual value can be retrieved. Here is a simple implementation:

namespace Telerik.JustMock
{
    public abstract class PtrBase
    {
        private readonly IntPtr addr;
        private readonly int cookie;

        protected PtrBase(IntPtr addr, int cookie)
        {
            this.addr = addr;
            this.cookie = cookie;
        }

        public IntPtr Addr { get { return this.addr; } }
        public int Cookie { get { return this.cookie; } }

        public abstract object ReadValue();
    }
}

When you create a new type which inherits from PtrBase type you should provide a unique cookie for each pointer type (e.g. 1 for int*, 2 for long*, 3 for byte**, etc.). When Arrange method is called, it will call in turn your implementation of ReadValue method and use the return value. In a matter of fact we don’t need this cookie mechanism because we can get the actual pointer type from methodCallExpr variable. The real purpose of ReadValue is to allow execution of user defined code in a lazy fashion. In case you don’t want this feature you can read the actual pointed value inside Ptr implicit conversion operator implementation and pass it to PtrBase constructor.

JustMock Design Choices

JustMock is a small product. It is around 10,000 lines of code. Despite its small size there are a lot of decisions that were made and many others that have to be made. In this post I would like to shed some light on our decision making process and I’ll try to answer the question why JustMock is built the way it is. The topic is quite large for a single blog post so I am going to focus on small part from JustMock – namely design choices for private method resolution.

First, I would like to emphasize that JustMock was built around C#. Every time when we had to decide how particular feature should be implemented we designed it from C# perspective. As a result JustMock works with concepts that are better expressed in C# rather than VB.NET for example. The reason for this decision is that it seems C# developers tend to use more often mocking tools/frameworks than VB.NET ones. So far, it seems we did the right choice.

As a consequence JustMock is more or less tightly coupled to the C# compiler. So are other mocking libraries. Lets consider the following example with Moq:

    public interface IFoo
    {
        int Bar(string s);
        int Bar(StringBuilder s);
    }

Often, it makes sense to define overloading methods in your code. The way IFoo is defined is perfectly legal. So, lets see a corner case with this interface. Suppose we want to mock Bar(string s) method when the argument is null:

    var mock = new Mock<IFoo>();
    mock.Setup(foo => foo.Bar(null)).Returns(1);

You will quickly find that the this code fragment does not compile. Here goes the error:

error CS0121: The call is ambiguous between the following methods or properties:
'IFoo.Bar(string)' and 'IFoo.Bar(System.Text.StringBuilder)'

This error has nothing to do with Moq. It is related to the C# compiler which in this case does not have enough information how to do method resolution. So, it is our fault and we have to fix it.

Here comes the second guide line we follow when design JustMock – there are no “right” or “wrong” choices. It is not white or black; rather it is like different gray shades. We could employ C# syntax in JustMock API to provide single method resolution (in most cases) but we didn’t. The reason is that different C# developers have different strategies to correct this issue. Here is a sample list with the most common fixes:

    // option 1
    mock.Setup(foo => foo.Bar((string)null)).Returns(1);

    // option 2
    mock.Setup(foo => foo.Bar(null as string)).Returns(1);

    // option 3
    mock.Setup(foo => foo.Bar(default(string))).Returns(1);

    // option 4
    string s = null;
    mock.Setup(foo => foo.Bar(s)).Returns(1);

After all, it’s matter of taste. Please notice that while I provided the sample above with Moq the issue is valid for JustMock. Lets make the things more interesting. Suppose we have the following concrete type:

    public sealed class Foo
    {
        public Foo() { ... }
        // ...
        private int Bar(string s) { ... }
        private int Bar(StringBuilder s) { ... }
    }

This time Bar(string s) and Bar(StringBuilder s) are private methods in a sealed type. Now we have to design an API for mocking private methods.

Note: Mocking private methods is a controversial thing. There are two equally large camps and a hot discussion between them. For good or bad, JustMock offers this feature.

We can provide the following method:

    public static void Arrange(object instance, string methodName, params object[] args) 
    { ... }

And then use it as follows:

    var foo = Mock.Create<Foo>();
    Mock.Arrange(foo, "Bar", null);

    string s = null;
    int actual = foo.Bar(s);

The problem with this approach is that the last argument passed to Arrange method is null and thus we cannot resolve the correct Bar(…) method. This time options 1 to 4 provided above don’t solve the issue. Lets see what can we do.

Approach 1: no method resolution

The problem is all about method resolution so if we don’t have to do it then we are good. Lets provide the API:

    public static void Arrange(object instance, MethodInfo method, params object[] args)
    { ... }

We have to change our test code:

    Type[] argTypes = { typeof(string) };
    var bar = typeof(Foo).GetMethod("Bar", argTypes);

    var foo = Mock.Create<Foo>();
    Mock.Arrange(foo, bar, null);

The test code is more messy. Now we have to deal with System.Reflection stuff which is not related to our code. Some developers are fine with this approach though. While this approach is suitable for API-to-API scenarios a lot of people prefer something more “friendly”.

Approach 2: pass method parameter types

We can pass the actual parameter types along with the parameter values. Lets change the Arrange method signature:

    public static void Arrange(object instance, string methodName, Type[] types, params object[] args) 
    { ... }

We added Type[] argument to Arrange method so we have to change our test code:

    var foo = Mock.Create<Foo>();
    Type[] argTypes = { typeof(string) };
    Mock.Arrange(foo, "Bar", argTypes, null);

    // use foo instance

This solution is easy to understand. It can also handle ref and out parameters and other more complex scenarios.

Approach 3: pass method parameter types as generic parameters

In case we decide to support private methods with simple signatures (no ref and out parameters and other fancy stuff) we can modify our previous approach by providing generic version of Arrange method (we can use T4 template for code generation; for the sake of simplicity we add a single generic parameter):

    public static void Arrange<T>(object instance, string methodName, params object[] args) 
    { ... }

We change our test code accordingly to:

    var foo = Mock.Create<Foo>();
    Mock.Arrange<string>(foo, "Bar", null);

    // use foo instance

This all looks good and nice but now we have to provide generic parameter types even when we don’t need them.

    var foo = Mock.Create<Foo>();
    Mock.Arrange<string>(foo, "Bar", "test");

    // use foo instance

It is not that bad but this can be annoying if Bar(…) method accepts, lets say, 5 integer parameters.

Approach 4: use typed references

The CLR and .NET framework already provide support for typed references but for our need we can come up with something much simpler. We have to provide support for our corner case when values are null. So we can provide a simple Null<T> type as follows:

    public sealed class Null<T>
        where T : class
    {
    }

Every time when we have to pass null value we are going to replace it with an instance of Null<T> type:

    var foo = Mock.Create<Foo>();
    Mock.Arrange(foo, "Bar", new Null<string>());

    // use foo instance

As the previous approach this one does not support methods with ref and out parameters. We can try to fix it by providing another constructor for Null<T> type:

    public sealed class Null<T>
        where T : class
    {
        public Null() {}

        public Null(System.Reflection.ParameterAttributes attrs)
        {
            this.Attrs = attrs;
        }

        public ParameterAttributes Attrs { get; private set; }
    }

Suppose we add the following method to Foo:

    private int Bar(ref string s) { ... }

If we want to mock it then our test code should like something like:

    var foo = Mock.Create<Foo>();
    Mock.Arrange(foo, "Bar", new Null<string>(ParameterAttributes.Out));

    // use foo instance

It doesn’t look nice to me. The syntax becomes more verbose and doesn’t flow naturally. We can mitigate it by adding new helper type as follows:

    public static class Arg
    {
        public static Null<T> Null<T>()
            where T : class
        {
            return new Null<T>();
        }
    }

Now we can compare the two options

    var foo = Mock.Create<Foo>();

    //option 1
    Mock.Arrange(foo, "Bar", new Null<string>());

    //option 2
    Mock.Arrange(foo, "Bar", Arg.Null<string>());

    // use foo instance

The readability seems a little bit better but many developers find the syntax verbose.

Approach 5: using dynamic (C# 4 and higher)

This is a good approach for a lot of developers. There is a caveat though. There are a plenty of customers that are still using .NET 3.5 and earlier. Shall we abandon them? A tough question.

Which one to use?

First, it is a wrong question. There is no silver bullet and the best thing we can do is to decide on a case-by-case basis. So far, we saw a few possible options; there are other alternatives as well. The options are not mutually exclusive which makes it even harder. If you are curious how we implemented it you can download a trial JustMock version.

NUI or GUI or … not

Today there are a lot of devices (smartphones, tablets, handheld game consoles, etc.) with support for gestural interface. We often call such interfaces natural user interfaces (NUI). These interfaces are in contrast with the traditional graphical user interfaces (GUI). In this post I am going to share my thoughts and experience with NUI.

Every software engineer knows that one should not make definitive assumptions how IT systems will be used. Often IT systems are used in unexpected and unpredicted ways. However such assumptions are rarely considered when it comes to user interfaces.

Natural user interfaces strive to offer more intuitive and easy human-technology interaction. This sounds all good and nice but lets focus on the word natural. In my understanding, natural means that the user doesn’t have to use artificial input/interaction devices such as keyboard and mouse. However, this does not mean that NUI is easy and intuitive for everyone. Users have still to learn it. While NUI is crucial for fast adoption of a new product it doesn’t mean it is easy to achieve it.

Today’s NUIs are not natural. Every company provides its own NUI standard and its per se artificial gestural language. Trying forcefully to apply NUI standard is not a solution. Cultural aspects should be preserved and considered. For example, almost every smartphone does not have support for left handed people.

My experience with different Android, iOS and Windows (Phone) 8 devices shows me that there is inconsistency between all of them. Often different applications for a particular platform use different gestures for the same command. Sometimes the companies use different NUI vocabularies for the same gesture. This could be confusing for the users.

In closing, I think that natural interfaces should allow us to interact with devices in the manner we interact with objects in everyday life. The devices should be able to learn user’s natural gestures/language and adapt to it. The companies should provide their natural interfaces as a fallback option.

 

Thoughts on C# Compiler

In 2010 I wrote the blog post Fun with pointers in C#. Back then, I thought it was fun. Today, I am not so sure. Lets take a look at the following code fragment:

using System;

namespace ClassLibrary1
{
    public class Class1
    {
        unsafe public void Method1(ref object* obj)
        {
            Console.WriteLine(obj->ToString());
        }
    }
}

If you try to compile this code with C# compiler distributed with .NET 1.1 you will get the following error:

error CS1005: Indirection to managed type is not valid

This is all good and nice because error CS1005 is aligned with the C# language specification which states:

Unlike references (values of reference types), pointers are not tracked by the garbage collector—the garbage collector has no knowledge of pointers and the data to which they point. For this reason a pointer is not permitted to point to a reference or to a struct that contains references, and the referent type of a pointer must be an unmanaged-type.

An unmanaged-type is any type that isn’t a reference-type or constructed type, and doesn’t contain reference-type or constructed type fields at any level of nesting. In other words, an unmanaged-type is one of the following:

sbyte, byte, short, ushort, int, uint, long, ulong, char, float, double, decimal, or bool.

Any enum-type.

Any pointer-type.

Any user-defined struct-type that is not a constructed type and contains fields of unmanaged-types only.

However since .NET 2.0 Microsoft introduced a bug that allows you to compile the code fragment above. Currently, I use C# compiler version 4.0.30319.17929 and I can still compile the code. If you run peverify.exe tool on the produced assembly you will get the following error:

[IL]: Error: [ClassLibrary1.dll : ClassLibrary1.Class1::Method1][offset 0x00000001]
Unmanaged pointers are not a verifiable type.
1 Error(s) Verifying ClassLibrary1.dll

Mono C# compiler (version 3.0.6) does it right. It fails, as expected, with error CS0208:

Class1.cs(7,40): error CS0208: Cannot take the address of, 
get the size of, or declare a pointer to a managed type `object'

Considering the fact that .NET 2.0 was released in 2005 it is hard to believe that Microsoft has not fixed the bug for the last 7 years. The only explanation I have here is that the bug is quite esoteric.