In short, there is no reason you cannot use RELAX NG in your web services theoretically. That's just the matter of insufficient implementation support of frameworks rather than spec matter.
As I implemented RelaxngDatatypeProvider in Commons.Xml.Relaxng.dll that supports XML Schema datatypes, you can use RELAX NG to represent "typed grammar". And it is possible for some grammars to map its items to runtime types (it is "theoretically" impossible to support runtime-type mapping for all kind of RELAX NG grammars. Read more for details). On the other hand: XML Schema is (also) not always mappable to runtime types. You can try xsd.exe with this simple example schema below and what useful classes it generates:
<![CDATA[
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"></p>
<p><xs:element name="Foo" type="myComplex" />
<xs:complexType name="myComplex">
<xs:choice>
<xs:sequence>
<xs:element name="element_A" type="xs:string" />
<xs:element name="element_B" type="xs:string" />
</xs:sequence>
<xs:sequence>
<xs:element name="element_C" type="xs:string" />
<xs:element name="element_D" type="xs:string" />
</xs:sequence>
</xs:choice>
</xs:complexType></p>
<p></xs:schema>
]]>
(I haven't checked how JAXB 1.0 RI works here.)
The fact is: if you are lucky, your schema can be used for object mapping.
For RELAX NG side, we don't have runtime type mapping tool yet. At least, target grammar must be "deterministic" which is one of the reason why XML Schema is much complex than RELAX NG (XML Schema forces deterministic schema, while RELAX NG allows non-deterministic grammars). To implement it, first we have to create non-deterministic grammar detection utility (like RelaxMeter by Kohsuke Kawaguchi, which is, however, to detect ambiguity).
Once deterministic grammar detection got implemented, we can implement object mapping using RelaxngDatatypeProvider noted above.
But note that, for usual develoers, understanding "how we can create deterministic grammar" might be more difficult than understanding and believing XML Schema blindly (actually it won't be, since XML Schema has some extraneous buggy things like substitution groups). In fact am one of such people who don't fully understand how deterministic model detection will work on RELAX NG.
Am not interested in other discussion such like "New X won't be your solution, since every existing systems are based on old Y". We're those people who believed that .NET Framework makes development easier, while it had abandoned legacy stuff.
