All posts by ArneC

Erm… sorry for the false alarm.. Misposted on the main blog instead of the dev blog. If still interested, the post has been moved here

The MindTouch RESTful IPA has completed it’s primary fermentation stage at specific gravity of 1.016@72F or 1.017 adjusted. That’s 6.9% Alcohol by Volume, but I fully expect the gravity to drop down a couple more notches by bottling time.

Dry-Hop

It is already a mighty tasty and hoppy brew, and it’s only going to pick up more aroma as the secondary fermentation is being dry-hopped.

  • 2oz. Simcoe plugs

This should give it that over the top flowery hop aroma that SDPA’s are so renowned for. One week of dry-hop secondary fermentation and then it’s time for bottling. So except to see some pictures of the label and bottles in next week’s update.

Some support materialsWhile I generally been posting on geeky programming stuff, today i’m going to switch gears to geeky brewing stuff and write about an IPA I’m currently cooking up.

San Diego is known for their overhopped, high-gravity India Pale Ale’s, usually called a Douple IPA or Imperial IPA, but among my brewing circle it’s been christened the SDPA, since it’s really rather far removed from what a traditional IPA is supposed to be like. So brewing a beer for a San Diego company naturally should represent this local transition. This recipe is one is a variation of one i’ve concocted over a number of iterations, taking inspiration from Alpine‘s Duet, Stone‘s Ruination, Green Flash‘s West Coast IPA, Pizza Port‘s HOP-15 and Ballast Point‘s Sculpin IPA. It’s a extract recipe trying to emulate full grain color and taste by using late addition.

The resultant beer has been named the MindTouch RESTful IPA both for geeky reference to Dream and this high gravity ale’s ability to induce extreme relaxation after only a pint or two.

Ingredients

  • Malt
    • 0.5# Victory Steeping Grains
    • 0.5# 10L Crystal Steeping Grains
    • 1.0# Cara Pils Steeping Grains
    • 7.0# Coopers Light DME
  • Hops
    • 2 oz. Centennial Pellets
    • 2 oz. Chinook Pellets
    • 2 oz. Cascade Pellets
    • 2 oz. Simcoe Plugs
  • Yeast
    • White Labs California Ale Yeast

Steeping

First step was to heat 2 gallons of filtered water to 160F for steeping the grains. The Victory, Cara Pils and Crystal malt were suspended in the water in a steeping bad, so as to not have it burn the bag’s bottom and so I could easily remove them again afterwards. The grains were steeped for 45 minutes.

Boil

Once the steeping grains were removed, the water level was topped to 3 gallons brought to a boil. At the beginning of the 60 minute boil the bittering hops were added:

  • 1 oz. Centennial
  • 1 oz. Chinook
  • 0.5 oz. Cascade

With 15 minutes remaining, I first gradually added the 7 lbs. of Light Dried Malt Extract, then more bittering hope were added:

  • 1 oz. Chinook
  • 1 oz. Cascade

At the end of boil, I added more hops for flavoring and aroma:

  • 1 oz. Centennial
  • 1 oz. Cascade

Pitch

Pre-fermentation

Once the wort had cooled significantly, I transfered it to the a 6.5 gallon carboy (fermenting head space) and topped it off with filtered water for a final volume of 5 gallons. Finally pitched the California Ale Yeast when the wort cooled to about 80F. The original gravity was meaured at 1.064 @ ~80F or 1.067, adjusted. Historically, this recipe ends up at a final gravity of about 1.010, which would give us 7.8% alcohol by volume, or just about right for an SDPA.

Next weekend, I’ll rack it to a 5 gallon carboy it and dry-hop it with 2 oz of Simcoe plugs. Weekend after that it’ll be bottled, so in about 4 weeks bottle conditioning should be complete and the first, if a bit green, tastes will be had.

Sorry, you can't stay at the house, the guest bath is--err, occupied

Sorry, you can't stay at the house, the guest bath is--err, occupied

Features on MindTouch Dream use the C# native protection levels to specify who can access them:

  • Public – everyone
  • Internal – only Parent, Child or services that were provided the Internal or Private key as a service-key cookie
  • Protected – same as Internal
  • Private – only Parent or services that were provided the Private key as a service-key cookie

Initial grants are according to relationships at service creation, but if you need to give someone special access you can provide them the appropriate service-key cookie and they are able to access the features matching the key provided.

But sometimes you want more granular access than “you there, feel free to call all my internal features” or you actually want to lock down access on more liberal features according to a configuration value.  Luckily, the access protection defined by the C# protection levels is completely customizable at run time.

A Quick Diversion: Dream access & cookies

When a request comes in to Dream and the Feature called is not public, Dream will check for a cookie called service-key.  This cookie is compared to the service’s InternalAccessKey and PrivateAccessKey to determine what access level should be granted.  By default the parent of a service (i.e. the service that created it) is automatically provided the PrivateAccessKey, as is the host service.  In addition, the InternalAccessKey is provided to child services.  By manually providing these cookies via set-cookie to callers, a service can easily provide safe access to other services.

Just like in the browser use-case, cookies are used in Dream to attach path specific data that allows service calls to be made without having to worry about all sorts of ancillary information to be passed in the query or custom headers. In the browser, cookie management and persistence is an automatic core feature.  Often times REST based systems skip cookies, relying more on custom headers, since the calling system might be a simple curl wrapper that doesn’t worry about persisting cookie jars. However most http request systems offer some capability to handle cookies rather than leaving the busywork of dealing with the headers to the caller, making cookies simpler than X-* headers in the general case.  When writing Dream Services, cookies are managed by the currently active DreamContext and therefore are mostly invisible.  When calling services from code outside of a DreamService, you can leverage the Plug class for your service calls and have it manage the cookie jar for you.

With cookies granting access to an internal Feature becomes as simple as setting the appropriate service-key in a cookie.  But the use of the service-key cookie is not restricted to InternalAccessKey or PrivateAccessKey, nor is authorization limited to the use of cookies in general.  By hooking into the access system of Dream, virtually any type of access security can be added to a Feature.  It should be kept in mind that violating the access levels published in the service blueprints should be done with care, as it can make for user unfriendly services.

Granting granular access

Most of the time the type of access modification needed is that of opening up access to some features that are by default locked down. To intercept the authorization for these features, you can override DetermineAccess.

public override DreamAccess DetermineAccess(DreamMessage request) {
  return base.DetermineAccess(request);
}

In order to grant access to a Feature marked internal, DetermineAccess must return that level of DreamAccess or higher. If you simply have a custom header, that, if set, allows the caller to access any internal feature, then the code could be as simple as this:

public override DreamAccess DetermineAccess(DreamMessage request) {
    if(request.Headers["X-Custom-Internal-Auth"] == _customInternalAuthKey) {
        return DreamAccess.Internal;
    }
    return base.DetermineAccess(request);
}

However, if the grant is more granular, say specific to a particular feature, you need to inspect the context. DetermineAccess isn’t passed a context like it is for a feature, but it is still accessible via DreamContext.Current (note: with Dream 1.6 the context will be passed to DetermineAccess). In particular, you can inspect the current Feature via DreamContext.Current.Feature. There are a number of ways to determine whether the called feature is the appropriate one, from looking at the fully qualified method name in DreamContext.Current.Feature.MainStage to looking at the signature components. For this example, we’re going to match on signature, since it is more likely to survive code refactoring than a method name and we will only use our custom header to grant access on posts to foo:

public override DreamAccess DetermineAccess(DreamMessage request) {
    if(DreamContext.Current.Feature.VerbSignature == "POST:foo" &&
        request.Headers["X-Custom-Internal-Auth"] == _customInternalAuthKey) {
        return DreamAccess.Internal;
    }
    return base.DetermineAccess(request);
}

Locking down default access

Locking down access from the default should be considered an edge case, since you are now going back on what you promised in your blueprint.  I.e. a service at a known location really should not restrict features previously advertised as public.

One scenario in which locking down is appropriate is a service that may be instantiated as public and discoverable in some cases but as a private child service for exclusive use by the parent in other cases.  In this scenario, previously public features may need to be locked down to add security beyond the obscurity of the path at which the parent created the service. An example of this scenario is the StorageService in Dream 1.6. By default it is a public service, but if instantiated via the private SID, it turns into a private service, generally used as a child service for private service storage.  This scenario cannot be accommodated by an override to DetermineAccess, since it only allows the easing of restrictions.  Instead, the Prologue system of Dream comes into play.

Dream contains feature intercepts for both pre- and post-feature invocation. These are named Prologues and Epilogues, respectively and are exposed as properties on DreamService. These interceptors are implemented as feature stages, i.e. they have the same signature as a normal feature. At feature invocation time, the list of prologues are each called before the feature is called, while the list of epilogues are called after the feature. The default implementation in a service returns null for either, although there are also host specified prologues and epilogues that wrap feature invocations.

In order to add a prologue, we first must override Prologues to return the interceptor we want called. The prologue/feature/epilogue chain is built up at service start, which means that the override of Prologues has to generate a static list without knowledge of the invocation. That moves responsibility of whether the prologue is appropriate for the current feature to a runtime responsibility of the prologue. The same is true for epilogues. The most basic implementation of a prologue/epilogue must at pass the request through, i.e.

private Yield DummyPrologue(
 DreamContext context,
 DreamMessage request,
 Result response
) {
   response.Return(request);
   yield break;
}

For illustration, we will consider the use case of the Storage Service as implemented in Dream 1.6, in which we have set a _private flag in the service at Start time depending on the SID that was used to create the service. First we add our prologue to every Feature invocation by overriding Prologues:

public override DreamFeatureStage[] Prologues {
    get {
        return new DreamFeatureStage[] {
            new DreamFeatureStage(
              "check-private-storage-access",
              this.ProloguePrivateStorage,
              DreamAccess.Public
            )
        };
    }
}

This created an interceptor named “check-private-storage-access” (the name is purely informational in this scenario) that calls ProloguePrivateStorage on any call (DreamAccess.Public indicates that it applies to public, internal and private invocations). Now for the implementation of ProloguePrivateStorage:

private Yield ProloguePrivateStorage(
  DreamContext context,
  DreamMessage request,
  Result response
) {

    //check if this services is private
    if(_private) {
        Cookie cookie = request.HasCookies
          ? HttpUtil.GetCookie(request.Cookies, "service-key")
          : null;
        if(cookie == null || cookie.Value != PrivateAccessKey) {
            throw new DreamForbiddenException("insufficient access privileges");
        }
    }
    response.Return(request);
    yield break;
}

ProloguePrivateStorage checks if the service is marked as private and checks for the service-key and whether it matches our PrivateAccessKey before allowing the call to continue to the next DreamFeatureStage.

Between cookies, DetermineAccess and the Prologue/Epilogue systems, Dream provides very simple and versatile methods for access control without having to front load the business logic of the Features themselves with authorization concerns. This keeps Features focused on their task and allows for Service wide access control independent of Feature development and refactoring.

Note: For reference this article along with a number of other Dream related Tutorials can also be found here. If you have any problem with the examples, check the wiki version, as it may reflect changes in Dream since this writing.

In a previous post, Steve covered the Result class. This post will build on that use of Result by way of Coroutines.

Coroutines

Coroutines are similar to subroutines, functions or methods, except that they have the ability to return multiple times. This means that coroutines have multiple exits and re-entries that allow execution to be suspended and later, resumed.

Coroutines have a long history in more academic programming circles, but have been gaining popularity as of late, in particular for their usefulness with multi-tasking. .NET has supported coroutines since the release of C# 2.0, except that they are called iterators.

Here is simple example of a structure for a linked list of Nodes supporting iteration over the set:

public class Node : IEnumerable<Node> {

    public Node Next;

    public IEnumerator<Node> GetEnumerator() {
        Node current = this;
        while( current != null ) {
            yield return current;
            current = current.Next;
        }
    }
    ...
}

Now all nodes can be traversed with a regular foreach instead of manually calling and examining Next. The magic of Iterators comes from the yield keyword. A side-effect of returning IEnumerator is that once at least one yield exists, the method is considered to be an Iterator by the compiler and can not longer be exited using return. Instead, using yield return, execution of the enumerator is suspended and the returned value placed into IEnumerator.Current. The method is resumed when IEnumerator.MoveNext() is called. This continues until execution reaches the end of the method or yield break is called.

However, there is nothing to keep us from using the yield for our own purposes and manually calling IEnumerator.MoveNext() and retrieving the result from IEnumerator.Current to walk through the multiple entry points of a method:

  public IEnumerable MultipleReturns() {
      Console.WriteLine("** doing some initial work, then yielding execution context **");
      yield return "first result";
      Console.WriteLine("** doing some more work, then yielding execution context again **");
      yield return "second result";
      Console.WriteLine("** finish up the work and exit **");
      yield break;
  }

For simplicity, we’ll illustrate the use of this method assuming we know how many entrypoints exist:

  public void ExecuteMultipleReturns() {
      Console.WriteLine("create coroutine");
      IEnumerator results = MultipleReturns().GetEnumerator();
      Console.WriteLine("start execution");
      bool hasResult = results.MoveNext();
      Console.WriteLine("Is there a result? {0} => {1}", hasResult, results.Current);
      Console.WriteLine("do something then return execution context to coroutine");
      hasResult = results.MoveNext();
      Console.WriteLine("Is there a second result? {0} => {1}", hasResult, results.Current);
      Console.WriteLine("do something else, then return context so corouting can finish");
      hasResult = results.MoveNext();
      Console.WriteLine("is there another result? {0}",hasResult);
      Console.WriteLine("done");
  }

this results in the following output:

create coroutine
start execution
** doing some initial work, then yielding execution context **
Is there a result? True => first result
do something then return execution context to coroutine
** doing some more work, then yielding execution context again **
Is there a second result? True => second result
do something else, then return context so corouting can finish
** finish up the work and exit **
is there another result? False
done

Yield

Now let’s return to our friend Result and see how we can use it along with the Iterator pattern. Suppose you want to use Result to wait for a value via another Result:

  public void UsingResultToReceiveData() {
      Result<int> result = new Result<int>();
      Result<Result<int>> trigger = new Result<Result<int>>();
      trigger.WhenDone(GetResult);
      trigger.Return(result);
      int r = result.Wait();
  }

  public void GetResult(Result<Result<int>> inner) {
      inner.Value.Return(42);
  }

We use a Result object to be able to trigger our request and then block on the our main Result<int> until GetResult returns our result (Note that the order of trigger.WhenDone() and trigger.Return() is irrelevant). Now let’s see how this code can be made simpler by using Dream’s Coroutine helper and yield:

  public void IntroducingCoroutine() {
      Result<int> result = new Result<int>();
      Coroutine.Invoke(GetResult, result);
      int r = result.Wait();
  }

  Yield GetResult(Result<int> result) {
      yield return Async.Sleep(TimeSpan.FromSeconds(2));
      result.Return(42);
      yield break;
  }

Notice how the call has gotten significantly simpler. We got rid of the Result we used to trigger the call and simply pass in our result to Coroutine.Invoke. On the Result generation side, we no longer return void, but instead return a MindTouch Dream construct, called Yield, which is simply an alias for IEnumerator<IYield>. We also added a superflous yield return, just to illustrate that we can have multiple yields, and those yields could be used to kick off asynchronous operations that GetResult will suspend for.

IEnumerator<IYield>

The use of  IEnumerator<IYield>, or the alias Yield is required to make the code compatible with the coroutine framework in MindTouch Dream. Usually code using the coroutine framework will have the following declaration to create the alias after namespace declaration:

    using Yield = IEnumerator<IYield>;

The IYield interface is defined in the MindTouch namespace. It indicates that something can be executed in steps. There are some limitations to what can be described in coroutines. However, these limitations are no different from those imposed by asynchronous programming in general. For instance, methods cannot return values or have out/ref parameters. This limitation is part of the design guidelines for asynchronous code, but for coroutines the compiler will actually enforce it.

Important: There is a very important detail that must be observed. Coroutines will be partially executed when invoked. In order to for the invocation to complete, the call must tie back into the iterative programming framework.

Now that we can invoke a coroutine which can yield to a number of asynchronous processes, another possibility emerges: continuing execution after the call has already returned to the invokee. This is often useful when a task requires some clean-up to be run after the result is already determined. By using the coroutine, the invokee can receive the result and continue on without being blocked by the clean-up still happening in the coroutine.

As a final example, let’s fetch a web page asynchronously, return its content length to the invokee and save the retrieved page to disk in the background:

  Yield GetContentLength(XUri uriToDownload, Result<int> result) {
      Plug p = Plug.New(uriToDownload);
      Result<DreamMessage> res;
      yield return res = p.GetAsync();

      // return the result
      result.Return((int)res.Value.ContentLength);
      yield return result;

      // copy the stream to disk in the background
      yield return Async.CopyStream(
          res.Value.AsStream(),
          File.OpenWrite(Path.GetTempFileName()),
          res.Value.ContentLength,
          new Result<long>());
  }

This can be called with (including timeout):

  Result<int> result = new Result<int>(TimeSpan.FromSeconds(5));
  Coroutine.Invoke(GetContentLength, new XUri("http://www.mindtouch.com"), result);

As you can see coroutines allow for some fairly complex asynchronous constructs with a relatively simple and synchronous looking syntax. They also form the foundation on which Features in Dream Services are built, but more on Dream Services some other time. In the meantime, you can always take a look at the Dream Tutorials, where you can also find this article.

For the upcoming 8.08.1 release we’ve added a bit of new DekiScript syntax to make code easier to write and allow more control over the flow.

Multiple variable declarations and declarations without a value

The old syntax for var required just one variable name and an initial value, like var x = 123. With the new syntax, multiple variables can be declared at once and variables can be declared without a value. This means that all of these are now valid:

var x;
var x,y;
var x = 1, y =2, z=3;
var x = 1, y, z;

Single statement blocks do not need curly braces

Both if/else and foreach required their statements to be enclosed  in { ... }, even if there was only a single statement. A by product of this was a complicated and hard to read syntax for if/else-if/else blocks, since the second if/else effectively was the statement block of the first else, resulting in statements like this:

if( x == 1 ) { "1"; } else { if(x == 2) { "2"; } else { "3"; } }

With the new syntax, curly braces can be omitted if there is only a single statement in the block.

if( x == 1 ) "1"; else if(x == 2) "2"; else "3";

The same simplification works also applies to foreach.

Foreach flow control with break and continue

Another addition for foreach is the introduction of break and continue control flow statements. The break statement will immediately terminate the current foreach loop block and continue executing the code that follows the foreach block. The continue statement on the other hand will terminate the loop block and restart it with the next iteration value.

foreach (var x in [1,2,3,4])
{
  x;
  if( x == 2 ) continue;
  else if(x==3) break;
  ".";
}
"x";

This code with produce the output “1.23x“.

Introducing the switch statement

The final addition to DekiScript for this release is the switch/case statement. So, while if/else-if/else is more concise now, for flow control relying on picking among many paths based on an expression, switch/case offers a much better syntax. The easiest way to illustrate switch/case is with an example:

var x = 2;
switch(x)
{
  case 1:
    "one";
  case 2:
    "two";
  case 3:
  case 4:
    "bigger than 2";
  default:
    "bigger than 4";
}

Here, we take a number and try to emit a string appropriate to the number. The output for this particular bit of code would simply be “two“. The case for 3 illustrates fallthrough, i.e. in our switch implementation there exists implicit fallthrough, but only if the case does not have a block of its own. Therefore the cases for 2 and 3 return the same value. Finally we have the special case of default, which will trigger if no other case matched. default is not required.

We hope these additions make it even easier to create custom dynamic content with dekiscript, so check them out once 8.08.1 is released and let us know how you like them.

With Deki Wiki 1.8.3, we gained the ability to write extensions to DekiScript in XML, as Steve wrote about here and in the the developer wiki article “How do I… Write my first extension in DekiScript?”. But until now testing those extension meant going into the admin console and loading the script and trying it out in a page.

In order to testing easier and to allow some automation of testing for those of us that are creating larger suites of extensions that should be regression tested before being released, we’ve released a utility that can be used both from the command line and programatically to check extension syntax and do test executions of functions.

The tool and some more details about its usage can be found in the developer wiki article “How Do I… Test my DekiScript Extensions?”, but the syntax is really quite simple:

mindtouch.deki.script.exe MyExtension.xml "myextension.SomeFunction()"

This call will load the extension MyExtension.xml, and assuming that the namespace of the extension is myextension, call its function SomeFunction with no arguments. The text, or html, generated by the call is returned to the console. That’s all there is to it.

Hi, I’m a new code slinger at MindTouch. Having used and authored Open Source projects for many years, this is finally my chance to contribute back in an official capacity. I will be working on Dream and DekiScript, helping with daily maintenance and driving it further along the roadmap.

I first came across MindTouch a while back when looking for a simple framework for building RESTful services without the weight of ASP.NET and the unnecessary dependencies it pulls in for things I’d never need. I especially liked the way Dream services were discoverable via the blueprints. But I wasn’t even aware they were looking for someone to help with Dream and Deki. For years, I’ve had automated searches to keep track of what the San Diego .NET scene looked like. Recently I added a new search to see if Erlang was picking up as a skill locally. While I’d used these searches for idle spectating, not really looking for a new gig, when I found the same posting in both my C# and my Erlang feeds, I knew I couldn’t afford not follow up on it. After talking to the crew here, I came away with an opportunity that hit on virtually every one my interests at once. Server side C# infrastructure running on mono or .NET, heterogenous, distributed computing, both in-process and distributed concurrency, functional programming. I could go on…

I’ve been doing server side infrastructure for a number of years now, coming up through scripting languages, moving back and forth between Windows and Unix platforms as needed and settling on C# as my current axe of choice. I’m excited to be working on a product that is deployed on such a broad spectrum and the capabilties of DekiScript and extensibility of Deki are just a lot of fun to work with. I’m currently spending my time digging into the code and trying to get up to speed so that I can start actively contributing and hoping to extend the RESTful approach across other transports for even greater concurrency capabilities.