monogatari

As I wrote in my earlier post, I am working on System.Web.Routing which is new in .NET 3.5 SP1. The code is almost done, and I want to bring it into Mono 2.2 (as opposed to current planned version: 2.4), that is to be branched soon (next month). But I need some dogfooding to mark it as stable. If any of you have or know free sample resources that use this API, let me know.

As part of dogfooding, I am sort of working on System.Web.DynamicData API (some of you might have noticed that I also checked in some implementation), and looks like its use of Sys.Web.Routing is working (since it is okay for me if routing part is working so far, I haven't dug in depth). I'm not sure if I continue DynamicData hacking though (as it is leaving routing part anymore).

And yes, Linq to SQL / dblinq effort continues.

Here I write about some of my recent commits. It does not mean I wrote all of those code. I often commit others' work.

Michael Barker posted a set of patches to implement System.Messaging.dll based on RabbitMQ client C# binding. He once posted a patch for the same purpose but based on different concept. So, it is not a short story, at least for him ;-)

Since MSMQ and JMS/AMQP-based servers have different concept in their designs, the implementation will not cover large part of the functionality that System.Messaging API, but it may be of help for some System.Messaging users.

Since we are not sure if we go with this implementation strategy, it is currently checked in to its own branch. What I personally want to see is some different implementations of its Mono.Messaging SPI, for example, based on Win32 API (I don't love it but probably it is the easiest to test the SPI). We welcome contribution here :)

Originally I was looking for practical use of ASP.NET Routing functionality in .NET 3.5 SP1, and ended up to think that Dynamic Data uses it. For routing part, it looks almost simple, so I sort of decided to hack it.

During generating masterinfos, I quickly noticed that DynamicData depends on LinqDataSource control, which we don't have yet. Fortunately, LinqDataSource seemed a good candidate for dogfooding Linq to SQL effort that we are trying to implement with DBLinq project (as written in my previous post). So I started to hack it.

And now here is a screenshot of working example of our LinqDataSource :)

It is from very simple use of GridView and LinqDataSource, which depends on SQL Server database that Pablo's sample creates (see my previous post for the details).
LinqDataSource-test.zip

It's working here, but it is possible only with my private branch of System.Data.Linq. Also I had to hack DataContext .ctor(string, MappingSource) which is not implemented in DBLinq (there is a discussion on how to implement connectionString support - see this thread in the DBLinq mailing list archive). The change is: : http://monoport.com/37676

This time it is a sshot from Windows as I am running this sample to connect to local SQL Server, but it is not difficult to get it working with other DB engines. For example, SQLite version of the sample would work if you 1) remove "dbo." from every table attribute in the sample code DataClasses1.designer.cs, 2) change connection string for SQLite DB, and 3) change my hack to instantiate SqliteConnection (must be done through reflection; we don't reference Mono.Data.Sqlite types from System.Data.Linq).

Anyways. The whole effort (DBLinq, DBLinq integration and LinqDataSource) is in early stage. We need more code. Especially LinqDataSource is nothing but hack yet ;)

One of the recent project I am sort of involved is an ongoing effort to implement System.Data.Linq.dll, a.k.a Linq to SQL, with DBLinq project. DBLinq is designed to be like Linq to SQL, but opens the door to support more database systems than SQL Server.

The Bootstrap

When I was asked to get involved in Linq to SQL work in this spring, I drew a rough sketch to implement S.D.Linq.dll based on DBLinq. In the middle of May, Jiri Moudry, one of the core DBLinq hacker, was also starting to work on Mono support. Occasionally we were in the middle of Google Summer of Code (SoC) program (beginning phase) and there was a student who was interested in Linq to DB. The student, Pablo Iñigo Blasco, became the core hacker of this effort. In fact, at first I was rather cautious about it: I haven't drawn good-enough picture of how to get every party involved yet, and especially I definitely needed to talk with DBLinq hackers.

Actually, at first I wrote System.Data.Linq API implementation as a wrapper on top of DBLinq. At that time, DBLinq was designed to be dependent on System.Data.Linq.dll. I thought that I had to resolve this issue (as it is a circular dependency), but it turned out that Pascal Craponne, another core DBLinq hacker, explained their plan to implement System.Data.Linq API by themselves(!) I quickly discarded my implementation and started rethinking about the SoC project with Pablo, which now became much hopeful than before. After all, it started off so well.

DBLinq changes for "Mono_STRICT" mode

When the SoC project started in June, the DBLinq team was busy on "sugar" expression tree implementation. So, I asked Pablo to start from writing samples for DBLinq which could be later ported to use S.D.Linq API. He did awesome job.

In the middle of June, as Pascal finished the first sugar work, we started to talk about Mono integration. Including the purpose to clean up the code, they introduced #if MONO_STRICT code.

Our need for the API was however somehow messy: we had to eliminate extra API members from the resulting System.Data.Linq.dll. And DBLinq itself is extensible enough to allow developing third party "vendor" API and hence exposing a lot of extra members. I had messy concern; I was afraid of getting final System.Data.Linq.dll cutting extra database support than SQL Server, since SQL Server is the only public vendor in System.Data.Linq API. (The concern didn't go away until recent talk.)

After a lot of discussion, Pablo came up with his nice Roadmap to build both DBLinq full and Mono's System.Data.Linq from the same code base, and he quickly made it.

Pablo continued and continues contributing to DBLinq project. I'm extraordinary glad to see his achievement and continuous participation to the project. I thank to all those guys who made it happen :)

Trying it out

Currently we still haven't integrated DBLinq source into our mcs tree yet. Though, I often create System.Data.Linq.dll based on their effort. For now, I have an archive: System.Data.Linq-20080930.zip - replace existing mcs/class/System.Data.Linq with the expanded directory, renamed as System.Data.Linq. And then run make (you need make clean).

As the end of the Soc project, we needed a working sample code for review. So I asked Pablo to cook some instruction, and he wrote it. With the latest svn version the test somehow hit NotImplementedException in "sugar" now, but it'd show you current DBLinq / System.Data.Linq implementation status.

A problem is that the current System.Data.Linq.dll contains only support for SQL Server, so you have to use it for now.

Help DBLinq effort

Yes, there is not a few things that need to be implemented. DBLinq project guys are calling for contributions. If you are interested in joining the effort, that'd be awesome. It helps Mono too.

Other than DBLinq (and System.Data.Linq) itself, there are also some interesting stuff to help this effort: ASP.NET LinqDataSource control in .NET 3.5 SP1 uses Linq to SQL, so implementing it will help dogfooding System.Data.Linq API. DynamicData may also help dogfooding, but that needs some extra love (it looks dependent on LinqDataSource). There's a lot of fun stuff here.

During Novell's Hack Week I started a project I was interested around moonlight, but haven't really done it. I'll revisit it once I got one example running (so, no example runs fine yet).

Recently Google Street View has launched in Japan this Summer, and it caused a lot of flamatory and bashing against Google. Details can be read at GlobalVoicesOnline ([1] and [2]). Similar arguments occured in France, England, Canada etc., and the U.S (I think, everywhere).

I'm not to explain all about the arguments, but to just update my recent status. Read those articles above if you are interested.

As one of the board members of MIAU, I was privately busy for preparing a symposium on Google's Street View, to discuss it publicly, for about 60 attendees. (Japanese news links: [1], [2] and [*3])

Apart from the organization which stands at neutral position, I keep (blogging about it (in Japanese) mostly raising caution that people should calm down, point problems precisely in precise context and distinguish issues and nonissues for each subject people raised, so that we do not have to shut down any kind of web service deployments unnecessarily. Even standing on such prudent position, it's been very hard to correct those furious people. I am also defamed by a lot of people including Anonymous Cowards in slashdot.jp for posting fair (in my belief) evaluation on those opinions. (Like "If there's risk of some property rights then it should be evaluated in an evenhanded fashion" => "F you")

It's not an easy bug to get resolved.

Yesterday friends of mine have released Rejaw, a microblogging-like, Comet-based threaded web chat infrastructure. It is with a full set of API. The introduction can be read here. I was one of the alpha testing users.

There are already some articles introducing Rejaw:

• Rejaw is Not Just Another Twitter Clone (mashable)
• Rejaw: Combining Microblogging + Chat (readwriteweb)

This article mostly explains correctly why MySpace and Facebook are not accepted by Japanese.

joining SNS by real name regarded as dangerous

Yes, when Facebook people visited Japan and explained their strategy to expand their market in Japan, by advertising "trusted by real name" network, we found it mostly funky. As the article explained, it is already achieved by mixi. And by that time, mixi is already regarded as dangerous "by exposing real name too widely".

One of the example accidents happened in 2005 August, at the comic market. Comic market place is usually flooded by terrible numbers of otaku people, and they used to be looking bad in general. One of a part time student workers at a hot dog stand wrote an entry in "mixi diary" like: "there was a lot of ugly otaku people there. eek!"

While it is pretty much straightforward, those otaku guys got hurt (at least some of them loudly claimed so), upset and started "profiling" who is that student. It was very easy in mixi, because mixi at that time encouraged to put real-life information with real name. No sooner she was then flooded by a lot of blaming voices, she disappeared from mixi.

OK, she was too careless, or ideally she should not write it (it is always easy to say something ideal). But she was not a geek and does not really understand how the network (mixi) is "open" to others (it is not really "open" by invitation filters, but as mixi grew up to have millions of users, it is of course not "trusted network" anymore). She didn't blame a specific person, and hadn't felt guilty until the company forced her to apologize. This kind of "careless" accidents has kept happening in mixi and it became a social problem.

Nowadays we have the same issue around "zenryaku-prof", where not a few children has faced troubles (for example sexual advances) by the face that the network is "open" to the web by default, while they think it isn't.

Though there must have been similar incidents outside Japan too (for example people fired by his or her blogs), the above (I believe) is the general understanding of the situation in Japan.

Mobile web madness

Another obvious point for Japanese, but would not for else, is that Japanese mobile web support is more important than anything, to get more people joined. Mixi is of course accessible from our cell phones. Even more funky example is "mobage-town", which used to limit access only from cell phones(!). (It is done by sending "contract ID", which is terrible BTW.) Mobage-town is one of the mega hit site in Japanese mobile web. It is mostly for games on the cell phones, but also has a huge SNS inside. It is also funky that the network used to be mostly filled by under-20 children. (Now children grew up to above 20, so the number is not obvious.)

Typically Japanese people spend a lot of boring time, between home and their offices or schools, on trains or buses. They can only do some limited "interesting" stuff. It used to be readings for example, and nowdays it is the mobile web.

Twitter was very successful unlike those failing players. Though I don't think the explanation on the TechCrunch article is right. Twitter had spread by "movatwitter", which is designed as the mobile web UI (and twitter is fully accessible by the API) as well as some additional values such as on-the-fly photo uploader (like Gyazickr for iPhone). It also filled our need (microblogging is a very good way to fill our boring time during our daily move). It lived very well in the mobile web land: no Javascript, no applets, no requirements for huge memory allocation.

When facebook is advertised with its API, what came to my mind was: "Is it even possible to make it for Japanese mobile web? nah"

While we, as a member of "open" world wide web, do not really like this mobile-only web (probably we should read Jonathan Zittrain), it is not a trivial demand that a cell phone is accessible to the mobile-only network. For example, iPhone 3G does not support it (iPhone BTW lacks a lot of features that typical Japanese people expect: for example, camera shake adjuster, it does not provide mobile TV capability, the mobile wallet etc.). It is often referred as "Galapagos network", which is intended as failing to expand their businesses abroad (one of the commentor on the TechCrunch entry mentions it. It is even funny that those iPhone enthusiasts try to claim that their web applications are "open" (as compared to Japanese mobile-only network).

BTW a commentor on the TechCrunch entry tries to object the fact written by the article by quoting google trends worldwide. But (including the graph on top of the article) it is a typical failure on measuring Japanese web access statistics: it does not reflect mobile web access. It is already explained (in Jaanese) very well. The simple fact is that it is becoming less ambient through Alexa, google trends or whatever similar.

SNSes are often domain specific

We would have seen similar phenomena in everywhere else. In China it is QQ. Orkut quickly became SNS for Brazil. There is no universal best.

What other SNSes find business chances in non-mainland countries is some specific purpose. For example, MySpace in Japan is good for producing musicians with its rich UI (many of them would also use mixi as well though).

Are you using XML for general string storage? If yes, most of you are likely wrong. If you do not understand why the following XML is wrong and how you can avoid it, do NOT use XML for your purpose.

<xml version="1.0" encoding="utf-8">
<root>I have to escape \u0007 as &#7;</root>

If you gave up answering by yourself, read what Daniel Veillard wrote 6 years ago.

I read through Scott Guthrie's entry on Silverlight 2.0 beta2 release and noticed that it ships with "Linq to JSON" support. I found it under client SDK. The documentation simply redirects to System.Json namespace page on MSDN.

So, last night I was a bit frustrated and couldn't resist. I'm not sure if MS implementation should work like this yet though.

(You can build it only manually right now; "cd mcs/class/System.Json" and "make PROFILE=net_2_1". The assembly is under mcs/class/lib/net_2_1. You can't do "make PROFILE=net_2_1 install".)

I picked up a sample on another Linq to JSON project to try mine - that is, James Newton-King's JSON.NET. At first, I simply replaced type names and it didn't work. The problematic lines are emphasized below:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Json;

using JsonProperty = System.Collections.Generic.KeyValuePair<
    string, System.Json.JsonValue>;

public class Post
{
  public string Title;
  public string Description;
  public string Link;
  public List<string> Categories = new List<string> ();
}

public class Test
{
  public static void Main ()
  {

List<Post> posts = new List<Post> ();
posts.Add (new Post () { Title = "test", Description = "desc",
    Link = "urn:foo" });

JsonObject rss = 
  new JsonObject(
    new JsonProperty("channel",
      new JsonObject(
        new JsonProperty("title", "Atsushi Eno"),
        new JsonProperty("link", "http://veritas-vos-liberabit.com"),
        new JsonProperty("description", "Atsushi Eno's blog."),
        new JsonProperty("item",
          new JsonArray(
            from p in posts
            orderby p.Title
            select (JsonValue) new JsonObject(
              new JsonProperty("title", p.Title),
              new JsonProperty("description", p.Description),
              new JsonProperty("link", p.Link),
              new JsonProperty("category",
                new JsonArray(
                  from c in p.Categories
                  select (JsonValue) new JsonPrimitive(c)
                  ))))))));

Console.WriteLine ("{0}", rss);
JsonValue parsed = JsonValue.Parse (rss.ToString ());
Console.WriteLine ("{0}", parsed);

  }
}

Unlike Linq to JSON in JSON.NET, JsonObject and JsonArray in System.Json do not accept System.Object as argument. Instead, it requires strictly-typed generic IEnumerable of JsonValue. That is, for example, IEnumerable<JsonPrimitive> is invalid. Thus I was forced to add explicit cast in my linq expression.

It may be improved in SL2 RTM though. Linq to XML classes accept System.Object argument to make it simple (and well, yes, sort of error-prone).

Two years ago I once worked on System.Web.Services complete 2.0 API. That was the worst shamest work I ever had (I hate SOAP). The work was done two years ago, and then there has been no maintainer since the completion. If there is anyone who loves SOAP, he or she can be a hero or heroine.