Autogenerate Docs from Source code. Hello mindtouch.doc; Goodbye NDoc, Javadoc…

November 30th, 2011 | | Uncategorized

Recently I’ve had several users and customers ask me about auto-generating documentation from source code comments. I talked to the some of the engineers here at MindTouch and I was informed of a tool that they developed a while back named mindtouch.doc. Indeed, our DReAM framework documentation was created by this tool.

We’re making mindtouch.doc available for free as open source. It’s basically a much better alternative to Javadoc or similar doc generation tools. The auto-generated doc output can be pushed directly into your MindTouch or you can produce a MindTouch Archive (mtarc) and import this to your MindTouch site. I’m told it supports all the C# compiler supported tags for documentation comments.

Read more…

Dynamically modifying Dream Feature access

October 28th, 2008 | | Dream

Features on MindTouch Dream use the C# native protection levels to specify who can access them:

  • Public – everyone
  • Internal – only Parent, Child or services that were provided the Internal or Private key as a service-key cookie
  • Protected – same as Internal
  • Private – only Parent or services that were provided the Private key as a service-key cookie

Initial grants are according to relationships at service creation, but if you need to give someone special access you can provide them the appropriate service-key cookie and they are able to access the features matching the key provided.

But sometimes you want more granular access than “you there, feel free to call all my internal features” or you actually want to lock down access on more liberal features according to a configuration value.  Luckily, the access protection defined by the C# protection levels is completely customizable at run time.

A Quick Diversion: Dream access & cookies

When a request comes in to Dream and the Feature called is not public, Dream will check for a cookie called service-key.  This cookie is compared to the service’s InternalAccessKey and PrivateAccessKey to determine what access level should be granted.  By default the parent of a service (i.e. the service that created it) is automatically provided the PrivateAccessKey, as is the host service.  In addition, the InternalAccessKey is provided to child services.  By manually providing these cookies via set-cookie to callers, a service can easily provide safe access to other services.

Just like in the browser use-case, cookies are used in Dream to attach path specific data that allows service calls to be made without having to worry about all sorts of ancillary information to be passed in the query or custom headers. In the browser, cookie management and persistence is an automatic core feature.  Often times REST based systems skip cookies, relying more on custom headers, since the calling system might be a simple curl wrapper that doesn’t worry about persisting cookie jars. However most http request systems offer some capability to handle cookies rather than leaving the busywork of dealing with the headers to the caller, making cookies simpler than X-* headers in the general case.  When writing Dream Services, cookies are managed by the currently active DreamContext and therefore are mostly invisible.  When calling services from code outside of a DreamService, you can leverage the Plug class for your service calls and have it manage the cookie jar for you.

With cookies granting access to an internal Feature becomes as simple as setting the appropriate service-key in a cookie.  But the use of the service-key cookie is not restricted to InternalAccessKey or PrivateAccessKey, nor is authorization limited to the use of cookies in general.  By hooking into the access system of Dream, virtually any type of access security can be added to a Feature.  It should be kept in mind that violating the access levels published in the service blueprints should be done with care, as it can make for user unfriendly services.

Granting granular access

Most of the time the type of access modification needed is that of opening up access to some features that are by default locked down. To intercept the authorization for these features, you can override DetermineAccess.

public override DreamAccess DetermineAccess(DreamMessage request) {
  return base.DetermineAccess(request);
}

In order to grant access to a Feature marked internal, DetermineAccess must return that level of DreamAccess or higher. If you simply have a custom header, that, if set, allows the caller to access any internal feature, then the code could be as simple as this:

public override DreamAccess DetermineAccess(DreamMessage request) {
    if(request.Headers["X-Custom-Internal-Auth"] == _customInternalAuthKey) {
        return DreamAccess.Internal;
    }
    return base.DetermineAccess(request);
}

However, if the grant is more granular, say specific to a particular feature, you need to inspect the context. DetermineAccess isn’t passed a context like it is for a feature, but it is still accessible via DreamContext.Current (note: with Dream 1.6 the context will be passed to DetermineAccess). In particular, you can inspect the current Feature via DreamContext.Current.Feature. There are a number of ways to determine whether the called feature is the appropriate one, from looking at the fully qualified method name in DreamContext.Current.Feature.MainStage to looking at the signature components. For this example, we’re going to match on signature, since it is more likely to survive code refactoring than a method name and we will only use our custom header to grant access on posts to foo:

public override DreamAccess DetermineAccess(DreamMessage request) {
    if(DreamContext.Current.Feature.VerbSignature == "POST:foo" &&
        request.Headers["X-Custom-Internal-Auth"] == _customInternalAuthKey) {
        return DreamAccess.Internal;
    }
    return base.DetermineAccess(request);
}

Locking down default access

Locking down access from the default should be considered an edge case, since you are now going back on what you promised in your blueprint.  I.e. a service at a known location really should not restrict features previously advertised as public.

One scenario in which locking down is appropriate is a service that may be instantiated as public and discoverable in some cases but as a private child service for exclusive use by the parent in other cases.  In this scenario, previously public features may need to be locked down to add security beyond the obscurity of the path at which the parent created the service. An example of this scenario is the StorageService in Dream 1.6. By default it is a public service, but if instantiated via the private SID, it turns into a private service, generally used as a child service for private service storage.  This scenario cannot be accommodated by an override to DetermineAccess, since it only allows the easing of restrictions.  Instead, the Prologue system of Dream comes into play.

Dream contains feature intercepts for both pre- and post-feature invocation. These are named Prologues and Epilogues, respectively and are exposed as properties on DreamService. These interceptors are implemented as feature stages, i.e. they have the same signature as a normal feature. At feature invocation time, the list of prologues are each called before the feature is called, while the list of epilogues are called after the feature. The default implementation in a service returns null for either, although there are also host specified prologues and epilogues that wrap feature invocations.

In order to add a prologue, we first must override Prologues to return the interceptor we want called. The prologue/feature/epilogue chain is built up at service start, which means that the override of Prologues has to generate a static list without knowledge of the invocation. That moves responsibility of whether the prologue is appropriate for the current feature to a runtime responsibility of the prologue. The same is true for epilogues. The most basic implementation of a prologue/epilogue must at pass the request through, i.e.

private Yield DummyPrologue(
 DreamContext context,
 DreamMessage request,
 Result response
) {
   response.Return(request);
   yield break;
}

For illustration, we will consider the use case of the Storage Service as implemented in Dream 1.6, in which we have set a _private flag in the service at Start time depending on the SID that was used to create the service. First we add our prologue to every Feature invocation by overriding Prologues:

public override DreamFeatureStage[] Prologues {
    get {
        return new DreamFeatureStage[] {
            new DreamFeatureStage(
              "check-private-storage-access",
              this.ProloguePrivateStorage,
              DreamAccess.Public
            )
        };
    }
}

This created an interceptor named “check-private-storage-access” (the name is purely informational in this scenario) that calls ProloguePrivateStorage on any call (DreamAccess.Public indicates that it applies to public, internal and private invocations). Now for the implementation of ProloguePrivateStorage:

private Yield ProloguePrivateStorage(
  DreamContext context,
  DreamMessage request,
  Result response
) {

    //check if this services is private
    if(_private) {
        Cookie cookie = request.HasCookies
          ? HttpUtil.GetCookie(request.Cookies, "service-key")
          : null;
        if(cookie == null || cookie.Value != PrivateAccessKey) {
            throw new DreamForbiddenException("insufficient access privileges");
        }
    }
    response.Return(request);
    yield break;
}

ProloguePrivateStorage checks if the service is marked as private and checks for the service-key and whether it matches our PrivateAccessKey before allowing the call to continue to the next DreamFeatureStage.

Between cookies, DetermineAccess and the Prologue/Epilogue systems, Dream provides very simple and versatile methods for access control without having to front load the business logic of the Features themselves with authorization concerns. This keeps Features focused on their task and allows for Service wide access control independent of Feature development and refactoring.

Note: For reference this article along with a number of other Dream related Tutorials can also be found here. If you have any problem with the examples, check the wiki version, as it may reflect changes in Dream since this writing.

Dream Asynchronicity Coroutines

October 21st, 2008 | | Dream

In a previous post, Steve covered the Result class. This post will build on that use of Result by way of Coroutines.

Coroutines

Coroutines are similar to subroutines, functions or methods, except that they have the ability to return multiple times. This means that coroutines have multiple exits and re-entries that allow execution to be suspended and later, resumed.

Coroutines have a long history in more academic programming circles, but have been gaining popularity as of late, in particular for their usefulness with multi-tasking. .NET has supported coroutines since the release of C# 2.0, except that they are called iterators.

Here is simple example of a structure for a linked list of Nodes supporting iteration over the set:

public class Node : IEnumerable<Node> {

    public Node Next;

    public IEnumerator<Node> GetEnumerator() {
        Node current = this;
        while( current != null ) {
            yield return current;
            current = current.Next;
        }
    }
    ...
}

Now all nodes can be traversed with a regular foreach instead of manually calling and examining Next. The magic of Iterators comes from the yield keyword. A side-effect of returning IEnumerator is that once at least one yield exists, the method is considered to be an Iterator by the compiler and can not longer be exited using return. Instead, using yield return, execution of the enumerator is suspended and the returned value placed into IEnumerator.Current. The method is resumed when IEnumerator.MoveNext() is called. This continues until execution reaches the end of the method or yield break is called.

However, there is nothing to keep us from using the yield for our own purposes and manually calling IEnumerator.MoveNext() and retrieving the result from IEnumerator.Current to walk through the multiple entry points of a method:

  public IEnumerable MultipleReturns() {
      Console.WriteLine("** doing some initial work, then yielding execution context **");
      yield return "first result";
      Console.WriteLine("** doing some more work, then yielding execution context again **");
      yield return "second result";
      Console.WriteLine("** finish up the work and exit **");
      yield break;
  }

For simplicity, we’ll illustrate the use of this method assuming we know how many entrypoints exist:

  public void ExecuteMultipleReturns() {
      Console.WriteLine("create coroutine");
      IEnumerator results = MultipleReturns().GetEnumerator();
      Console.WriteLine("start execution");
      bool hasResult = results.MoveNext();
      Console.WriteLine("Is there a result? {0} => {1}", hasResult, results.Current);
      Console.WriteLine("do something then return execution context to coroutine");
      hasResult = results.MoveNext();
      Console.WriteLine("Is there a second result? {0} => {1}", hasResult, results.Current);
      Console.WriteLine("do something else, then return context so corouting can finish");
      hasResult = results.MoveNext();
      Console.WriteLine("is there another result? {0}",hasResult);
      Console.WriteLine("done");
  }

this results in the following output:

create coroutine
start execution
** doing some initial work, then yielding execution context **
Is there a result? True => first result
do something then return execution context to coroutine
** doing some more work, then yielding execution context again **
Is there a second result? True => second result
do something else, then return context so corouting can finish
** finish up the work and exit **
is there another result? False
done

Yield

Now let’s return to our friend Result and see how we can use it along with the Iterator pattern. Suppose you want to use Result to wait for a value via another Result:

  public void UsingResultToReceiveData() {
      Result<int> result = new Result<int>();
      Result<Result<int>> trigger = new Result<Result<int>>();
      trigger.WhenDone(GetResult);
      trigger.Return(result);
      int r = result.Wait();
  }

  public void GetResult(Result<Result<int>> inner) {
      inner.Value.Return(42);
  }

We use a Result object to be able to trigger our request and then block on the our main Result<int> until GetResult returns our result (Note that the order of trigger.WhenDone() and trigger.Return() is irrelevant). Now let’s see how this code can be made simpler by using Dream’s Coroutine helper and yield:

  public void IntroducingCoroutine() {
      Result<int> result = new Result<int>();
      Coroutine.Invoke(GetResult, result);
      int r = result.Wait();
  }

  Yield GetResult(Result<int> result) {
      yield return Async.Sleep(TimeSpan.FromSeconds(2));
      result.Return(42);
      yield break;
  }

Notice how the call has gotten significantly simpler. We got rid of the Result we used to trigger the call and simply pass in our result to Coroutine.Invoke. On the Result generation side, we no longer return void, but instead return a MindTouch Dream construct, called Yield, which is simply an alias for IEnumerator<IYield>. We also added a superflous yield return, just to illustrate that we can have multiple yields, and those yields could be used to kick off asynchronous operations that GetResult will suspend for.

IEnumerator<IYield>

The use of  IEnumerator<IYield>, or the alias Yield is required to make the code compatible with the coroutine framework in MindTouch Dream. Usually code using the coroutine framework will have the following declaration to create the alias after namespace declaration:

    using Yield = IEnumerator<IYield>;

The IYield interface is defined in the MindTouch namespace. It indicates that something can be executed in steps. There are some limitations to what can be described in coroutines. However, these limitations are no different from those imposed by asynchronous programming in general. For instance, methods cannot return values or have out/ref parameters. This limitation is part of the design guidelines for asynchronous code, but for coroutines the compiler will actually enforce it.

Important: There is a very important detail that must be observed. Coroutines will be partially executed when invoked. In order to for the invocation to complete, the call must tie back into the iterative programming framework.

Now that we can invoke a coroutine which can yield to a number of asynchronous processes, another possibility emerges: continuing execution after the call has already returned to the invokee. This is often useful when a task requires some clean-up to be run after the result is already determined. By using the coroutine, the invokee can receive the result and continue on without being blocked by the clean-up still happening in the coroutine.

As a final example, let’s fetch a web page asynchronously, return its content length to the invokee and save the retrieved page to disk in the background:

  Yield GetContentLength(XUri uriToDownload, Result<int> result) {
      Plug p = Plug.New(uriToDownload);
      Result<DreamMessage> res;
      yield return res = p.GetAsync();

      // return the result
      result.Return((int)res.Value.ContentLength);
      yield return result;

      // copy the stream to disk in the background
      yield return Async.CopyStream(
          res.Value.AsStream(),
          File.OpenWrite(Path.GetTempFileName()),
          res.Value.ContentLength,
          new Result<long>());
  }

This can be called with (including timeout):

  Result<int> result = new Result<int>(TimeSpan.FromSeconds(5));
  Coroutine.Invoke(GetContentLength, new XUri("http://www.mindtouch.com"), result);

As you can see coroutines allow for some fairly complex asynchronous constructs with a relatively simple and synchronous looking syntax. They also form the foundation on which Features in Dream Services are built, but more on Dream Services some other time. In the meantime, you can always take a look at the Dream Tutorials, where you can also find this article.

{{Bio.GetIntroduction(“arne”)}}

September 5th, 2008 | | Uncategorized

Hi, I’m a new code slinger at MindTouch. Having used and authored Open Source projects for many years, this is finally my chance to contribute back in an official capacity. I will be working on Dream and DekiScript, helping with daily maintenance and driving it further along the roadmap.

I first came across MindTouch a while back when looking for a simple framework for building RESTful services without the weight of ASP.NET and the unnecessary dependencies it pulls in for things I’d never need. I especially liked the way Dream services were discoverable via the blueprints. But I wasn’t even aware they were looking for someone to help with Dream and Deki. For years, I’ve had automated searches to keep track of what the San Diego .NET scene looked like. Recently I added a new search to see if Erlang was picking up as a skill locally. While I’d used these searches for idle spectating, not really looking for a new gig, when I found the same posting in both my C# and my Erlang feeds, I knew I couldn’t afford not follow up on it. After talking to the crew here, I came away with an opportunity that hit on virtually every one my interests at once. Server side C# infrastructure running on mono or .NET, heterogenous, distributed computing, both in-process and distributed concurrency, functional programming. I could go on…

I’ve been doing server side infrastructure for a number of years now, coming up through scripting languages, moving back and forth between Windows and Unix platforms as needed and settling on C# as my current axe of choice. I’m excited to be working on a product that is deployed on such a broad spectrum and the capabilties of DekiScript and extensibility of Deki are just a lot of fun to work with. I’m currently spending my time digging into the code and trying to get up to speed so that I can start actively contributing and hoping to extend the RESTful approach across other transports for even greater concurrency capabilities.

Web 2.x Blog: “MindTouch will blow your mind away”

May 22nd, 2008 | | Awards and Press

MindTouch will blow your mind away!!! | Web 2.x Blog

So is MindTouch just another entrance into the red ocean plethora of WIKIs lying around? No.

MindTouch is fully based on a SOA architecture. It is written in C#, and uses the MONO framework to allow it to operate in all major UNIX-based operating systems out there (MAC OS X, Linux, Solaris). The SOA architecture means the hooks into the software are all fully based on web or REST services !!!! Here are few reasons why I think MindTouch will stand out compared to other WIKI systems:-

  1. The web services approach allows ease of integration with external systems. E.g. A WIKI page for a technical lead showing the latest snapshot of the statistics of the number of code activity in subversion, as well as testing activity from a online test management system called Mantis.
  2. This ease of integration leads to MindTouch being a powerful platform for Enterprise Mashups
  3. MindTouch still persists the functionality that you would typically find in a WIKI system.
  4. MindTouch requires no configuration (except admin username/password) at all. It’s a simple deployment that’s done through VMWare Workstation. Plug the VM image in the workstation, click play. Dead simple.

MindTouch surely is not the conventional WIKI that we all imagine it to be. It’s much more than a WIKI. It differentiates itself with other key players by having the ability to plug and play services exposed by other information systems.

Thanks for for the write up and kind words. :-)