Sharing a resource's schema through REST

I propose that Rails and ActiveResource add support for retrieving a
model's schema via REST. This could make it much easier for REST
clients to auto-figure out attributes, the way that ActiveRecord uses
the database to do the same thing.

The best location for this is, I think, at "/users/new.xml", using a
"User" model as the example. The HTML at "/users/new" provides the
form needed to create a new instance, and the XML version would
provide the information needed to do the same thing remotely.

Here's one idea for a clean syntax for doing this (also at
http://pastie.caboo.se/55138):

respond_to do |format|
  format.xml { render :xml => User.to_xml }
end

This would mean writing a to_xml function on ActiveRecord::Base as a
class method. The XML this produces could look like the normal AR
to_xml format, just without any content. An example:

http://pastie.caboo.se/55136

I've been working on an ActiveResource client in JavaScript, and it's
impossible to just assign properties to an object and for the object
to know that these are attributes to be submitted along with any POST
or PUT call. ActiveResource has a partial solution to this by taking
advantage of Ruby's method_missing, so any unknown properties that get
assigned can be caught and recognized as formal attributes. Clients
written in strongly typed languages would have an even harder time
than JavaScript. Including an easy way to advertise the schema would
ease the burden of any REST client.

More generally, I think this makes the Rails REST standard more
complete, by affording an ActiveResource client the same level of
schema knowledge that an ActiveRecord server has.

What do you guys think? Would this be a useful extension to the
current REST standard Rails is advocating? I think the most
interesting part of this is the XML format, and I'd really like to
form something the community agrees with as I construct a patch for
this.

-- Eric Mill

Hi Eric,

respond_to do |format|
  format.xml { render :xml => User.to_xml }
end

This would mean writing a to_xml function on ActiveRecord::Base as a
class method. The XML this produces could look like the normal AR
to_xml format, just without any content. An example:

http://pastie.caboo.se/55136

Why not generate an xml schema? or RelaxNG? Inventing a schema
language seems like the wrong direction...

What would a RelaxNG schema look like for the example? Here's one I
found:

http://relaxng.org/tutorial-20011203.html#IDAHDYR

And that looks more complicated than it needs to be.

Ben

And that looks more complicated than it needs to be.

Depends on what your requirements are. If you want a 'schema
language' to help non-rails clients, why not use one that's been
widely tested and has support in many different languages. Making up
something specific for rails would mean it's of questionable value to
non-rails programming environments.

I'm not necessarily sold on the idea of generating a particular schema
language, or even of generating a schema at all, but I do know that
inventing our own is something we should be very wary of.

+1

Michael Koziarski wrote:

Hi Eric,

I propose that Rails and ActiveResource add support for retrieving a
model's schema via REST. This could make it much easier for REST
clients to auto-figure out attributes, the way that ActiveRecord uses
the database to do the same thing.

+1, I think this is a great idea.

Since we're talking about this, I had the idea of using the OPTIONS
method to return the methods allowed on a resource (via the Allow
header), but also returning an XML representation of the routes for
the nested and associated resources.

The OPTIONS response is supposed to return a comma separated list
of methods allowed in the Allow header, but the body can really be
anything we want since the spec doesn't say what it should be. If
people think this is a good fit, we could use it as a way of
"introspecting" a resource.

Combine these two ideas, and we have a way for a remote client to
create new resources AND a way of finding all the associated resources.

Depends on what your requirements are. If you want a 'schema
language' to help non-rails clients, why not use one that's been
widely tested and has support in many different languages. Making up
something specific for rails would mean it's of questionable value to
non-rails programming environments.

Along these lines, I've been investigating generating a W3C XML Schema
from a model and found it delightfully easy with just a bit of
reflection on AR::Base::columns. Throw in a .xsd format and you've
got a great place to serve your schema.

One of the things I noticed was how similar the standard types for an
XML Schema were to the types used by #to_xml. With just a few very
slight changes, Rails can be using a standardized list of types
instead of inventing its own.

I've refactored part of Hash::from_xml and in the process, added
support for the parsing end of things for these types. This also
fixes a few issues as noted in the ticket I submitted it with. It
goes without saying tests are included. If there's interest, I can
fix up the generating part (the various #to_xml's) as well.

http://dev.rubyonrails.org/ticket/8047

I'm not necessarily sold on the idea of generating a particular schema
language, or even of generating a schema at all, but I do know that
inventing our own is something we should be very wary of.

I agree, most of this is destined for plugin land at most. But I
think reusing the types in the core is a Good Thing.

Cheers,
Tim

Hi Tim,

Okay. I see that my first patch was committed this morning (thanks
DHH) so this is the next logical step. Note that by itself, this
change enables nothing without a proper schema definition. Basically
it would be trading a little bit of aesthetics (e.g., datetime for
dateTime) for a little bit of purity (a "standard" type list).

On the subject of XML schemas, the biggest missing piece of the puzzle
is a way to get the appropriate attributes on the root element for
records. I am thinking an interface would look something like

@people.to_xml(
  :root_attributes => {
    'xmlns:xsi' => "http://www.w3.org/2001/XMLSchema-instance",
    'xsi:noNamespaceSchemaLocation' => formatted_people_url(:xsd)
  }
)

The implementation should be simple enough, but before I do it I want
to get some feedback on the interface (or at least, not get negative
feedback).

Cheers,
Tim

Okay. I see that my first patch was committed this morning (thanks
DHH) so this is the next logical step. Note that by itself, this
change enables nothing without a proper schema definition. Basically
it would be trading a little bit of aesthetics (e.g., datetime for
dateTime) for a little bit of purity (a "standard" type list).

Could you elaborate here a bit and list the changes needed to the
current AR::to_xml method to make it W3C compliant? This sounds like
a good idea, though I'm curious as to how heavy the aesthetics-
standards tradeoff will be.

On the subject of XML schemas, the biggest missing piece of the puzzle
is a way to get the appropriate attributes on the root element for
records. I am thinking an interface would look something like

@people.to_xml(
  :root_attributes => {
    'xmlns:xsi' => "http://www.w3.org/2001/XMLSchema-instance",
    'xsi:noNamespaceSchemaLocation' => formatted_people_url(:xsd)
  }
)

Could this be even simpler? And, assuming the schema is at /people/
new.xml:

  @people.to_xml(:format => :w3c, :schema => new_person_url(:format
=> :xml))

And maybe, to address Michael's concern, we could support both a
simple and a RelaxNG format for model schemas:

  Person.to_xml(:format => :rng) # Relax NG format
  Person.to_xml # Simpler, Railsy format

I'm not necessarily sold on the idea of generating a particular schema
language, or even of generating a schema at all, but I do know that
inventing our own is something we should be very wary of.

RelaxNG is a schema for *XML*, not for a *model*. This schema is only
indirectly used to construct XML POSTs/PUTs, its primary use is for
the client to understand the makeup of your data. In fact, "Elements"
and "text nodes" aren't the correct conceptual way to describe a data
model.

In fact, consider that a RelaxNG schema would be outright inaccurate
unless you label every single element in the schema as optional. You
can update as few or as many attributes as you like in a PUT, and you
may never be interested in including data for associations in your
requests, yet you probably want to note all attributes and
associations in a schema for the model. The way you label elements as
optional in RelaxNG seems to be just "zeroOrMore". This terminology
should be used only to describe the nature of associations between
data, not just to make sure the flexible nature of REST is described
technically accurately.

So, I think RelaxNG is inappropriate for a simple, flexible
description of a data model, and that we should keep the format as
close to the current AR#to_xml format as possible.

> Okay. I see that my first patch was committed this morning (thanks
> DHH) so this is the next logical step. Note that by itself, this
> change enables nothing without a proper schema definition. Basically
> it would be trading a little bit of aesthetics (e.g., datetime for
> dateTime) for a little bit of purity (a "standard" type list).

Could you elaborate here a bit and list the changes needed to the
current AR::to_xml method to make it W3C compliant? This sounds like
a good idea, though I'm curious as to how heavy the aesthetics-
standards tradeoff will be.

Well the changes needed to mirror the standard are simply

1. change datetime to dateTime
2. change float to double
3. change encoding="binary" to type="base64Binary"

On top of this, you might consider adding an explicit type="string".
Making these changes would not bring any sort of compliance, though,
as they types are usually indicated in a separte schema file, not
inline.

> On the subject of XML schemas, the biggest missing piece of the puzzle
> is a way to get the appropriate attributes on the root element for
> records. I am thinking an interface would look something like
>
> @people.to_xml(
> :root_attributes => {
> 'xmlns:xsi' => "http://www.w3.org/2001/XMLSchema-instance",
> 'xsi:noNamespaceSchemaLocation' => formatted_people_url(:xsd)
> }
> )

Could this be even simpler? And, assuming the schema is at /people/
new.xml:

  @people.to_xml(:format => :w3c, :schema => new_person_url(:format
=> :xml))

And maybe, to address Michael's concern, we could support both a
simple and a RelaxNG format for model schemas:

Sure, there are many ways it could be done. The interface I discuss
is the bare minimum necessary to play with this sort of thing without
monkey patching several rather large #to_xml methods. Support for
multiple schema formats isn't something that belongs in the core, at
least not without spending a lot of time in plugin land first.

Cheers,
Tim

RelaxNG is a schema for *XML*, not for a *model*. This schema is only
indirectly used to construct XML POSTs/PUTs, its primary use is for
the client to understand the makeup of your data. In fact, "Elements"
and "text nodes" aren't the correct conceptual way to describe a data
model.

Seems a fairly arbitrary distinction, given that we're talking about a
schema for the API which will be returning xml documents...

Automatically generating a schema seems like overkill, given that the
schema for your models changes rarely, what's wrong with just a human
readable API doc page. Like:

http://basecamphq.com/api/

So, I think RelaxNG is inappropriate for a simple, flexible
description of a data model, and that we should keep the format as
close to the current AR#to_xml format as possible.

Your proposed format suffers from the same limitations in that almost
everything provided is optional.

I just don't really see how this enables anything new, surely anything
javascript can do with the schema document, could also be done with
the xml document received in response to /foos/1.xml?

Automatically generating a schema seems like overkill, given that the
schema for your models changes rarely, what's wrong with just a human
readable API doc page.

Because that's a lot of work, and a standard format makes that work
unneeded. The Beast forum doesn't need to publish an API doc, because
it implements ActiveResource and so an ARes client can auto-assume
almost everything it needs.

Highrise hasn't published its API yet, but I bet it's going to be a
lot simpler, since it was the very inspiration for ActiveResource,
which it also implements.

Your proposed format suffers from the same limitations in that almost
everything provided is optional.

How is this a limitation? Everything provided is optional. RelaxNG
just forces you to declare this, which I think is unnecessary.

I just don't really see how this enables anything new, surely anything
javascript can do with the schema document, could also be done with
the xml document received in response to /foos/1.xml?

This only works if there is at least one object saved in the database.

-- Eric

I have implemented this including tests and submitted it as two
tickets, one for ActiveSupport and one for ActiveRecord.

http://dev.rubyonrails.org/ticket/8167
http://dev.rubyonrails.org/ticket/8169

Cheers,
Tim

And if you already know an ID (unless you want go scavenging in
/foos.xml).

It seems to me you're missing a solution that's simple and obvious.
Consider this excerpt from the resource scaffolding:

  # GET /foos/1
  # GET /foos/1.xml
  def show
    @foo = Post.find(params[:id])

    respond_to do |format|
      format.html # show.erb
      format.xml { render :xml => @foo.to_xml }
    end
  end

  # GET /foos/new
  def new
    @foo = Post.new
  end

Now, look what happens when we update the second method to look like
the first.

  # GET /foos/new
  # GET /foos/new.xml
  def new
    @foo = Post.new

    respond_to do |format|
      format.html # new.erb
      format.xml { render :xml => @foo.to_xml }
    end
  end

Voila. Now the HTML response and the XML response refer to the same
object. The XML client gets the field names and types, and defaults
for the field values to boot. And we didn't even have to touch the
Rails core. The only thing left is the route, which I leave as an
exercise to the reader.

The disadvantage to this method is that "new" doesn't fit cleanly into
the four CRUD actions. This was accepted as a necessary compromise
for interactive uses, but now it's creeping into the API as well.

Cheers,
Tim

No compromise needed. Restful controllers manage three broad resources
in a single package: the collection (GET+POST /foos as index and
create), its members (GET+PUT+DELETE /foos/123 as show, update, and
destroy), and the new instance (GET /foo/new as new). So 'new' is a
read-only singleton resource.

I like the feel of returning render :xml => @foo as a 'prototype' for Ares.

jeremy

I like the feel of returning render :xml => @foo as a 'prototype' for Ares.

Same, though I'm not using ARes enough to know if this would solve
any problems we're currently having. However it does solve all the
problems that were mentioned wrt a javascript client, without any
reinvention of a schema language?

Is there anything it can't do relative to the schema / psuedo-schema options?

  # GET /foos/new
  # GET /foos/new.xml
  def new
    @foo = Post.new

    respond_to do |format|
      format.html # new.erb
      format.xml { render :xml => @foo.to_xml }
    end
  end

Voila. Now the HTML response and the XML response refer to the same
object. The XML client gets the field names and types, and defaults
for the field values to boot. And we didn't even have to touch the
Rails core.

I played with exactly this last night, and I think it *is* the better
solution, and of course far less work. The major problem is that this
won't include any associations, even if :include is passed as a param,
because all the associations are either an empty array or nil.
Possible solutions are to add an :include_associations
or :include_empty argument or something to to_xml, or to change
to_xml's default behavior to include associations in the XML
when :include'd, whether they're empty or not.

The only thing left is the route, which I leave as an
exercise to the reader.

Actually, new.xml is already given for free. Sweet!