replace REXML with Hpricot

rubyonrails-talk
1

Hi,

is there any way to replace REXML with Hpricot for rendering XML? I am using Jruby so cant use libxml, and need to significantly increase the speed of XML rendering.

Thanks Adam

2

AD wrote:

is there any way to replace REXML with Hpricot for rendering XML? I am using Jruby so cant use libxml, and need to significantly increase the speed of XML rendering.

What's wrong with Builder::XmlMarkup (iirc)?

Hpricot uses a C module to get fast - like libxml. So if you can't use one in Jruby...

3

I think I actually meant XML::Builder, I was under the initial impression this was actually a REXML Library.

At any rate my performance metrics show that 60-70% is done via “Rendering” b/c of some complicated XML generation so I am looking for some optimization mechanisms here. HPricot is faster than Builder and REXML and is supported in JRuby (running it now).

So basically I am looking to replace XML::Builder with something faster , that is not libXML (like Hpricot)

Adam

4

I think I actually meant XML::Builder, I was under the initial impression this was actually a REXML Library. At any rate my performance metrics show that 60-70% is done via "Rendering" b/c of some complicated XML generation so I am looking for some optimization mechanisms here. HPricot is faster than Builder and REXML and is supported in JRuby (running it now).

So basically I am looking to replace XML::Builder with something faster , that is not libXML (like Hpricot)

There is a jruby version of hpricot. Builder and Rexml are unrelated. The only bit I've ever found to be slow about builder is that it tries to be a bit too clever with its escaping and tries to work out all sorts of stuff that I didn't need. Overriding the _escape method to just replace &, <, > sped things up for me

Fred

5

AD wrote:

So basically I am looking to replace XML::Builder with something faster , that is not libXML (like Hpricot)

Have you time-tested and profiled your code & tests to see what's fast? it's usually a surprise...

6

well I am just noticing when the actions get rendered, its always like (Rendering (78%)) and some of the XML rendering just seems slow in relation to the time for the DB query.

Do you have any profile recommendations besides ruby-prof ?

7

here is some real world output… Any feedback is welcome, notice the very high render time:

Rendering channels/index Completed in 3.38310 (0 reqs/sec) | Rendering: 1.38117 (40%) | DB: 0.48588 (14%) | 200 OK [http://dev1.domain.com/users/1/2008/05/28/index.xml]

[http://dev1.domain.com/users/1/2008/05/28/index.xml]

Thread ID: 3067917240 Total: 2.920000

%self total self wait child calls name

5.82 0.42 0.17 0.00 0.25 18281 Fixnum#xchr 4.45 0.14 0.13 0.00 0.01 79 Array#select 4.11 0.56 0.12 0.00 0.44 2288 Array#map 3.08 0.10 0.09 0.00 0.01 29 ActiveRecord::Associations::ClassMethods#collection_accessor_methods

2.74 0.11 0.08 0.00 0.03 54266 Kernel#=== 1.71 0.16 0.05 0.00 0.11 248 Hash#each 1.71 0.08 0.05 0.00 0.03 54327 Hash# 1.37 0.06 0.04 0.00 0.02 288 Module#define_method

1.37 0.04 0.04 0.00 0.00 15 Mysql::Result#each_hash 1.37 0.04 0.04 0.00 0.00 108 Array#- 1.03 0.03 0.03 0.00 0.00 39128 Hash#default 1.03 0.03 0.03 0.00 0.00 324 Module#constants

1.03 0.03 0.03 0.00 0.00 56670 Fixnum#==

Thread ID: 3067917860 Total: 0.770000

8

here is some real world output.. Any feedback is welcome, notice the
very high render time:

Fixnum.xchr is builder's escaping stuff. See my previous tip for
possible ways out.

Fred

9

I believe you are referencing this method

def _escape(text) text.to_xs end

How do I go about overriding this ?

10

I believe you are referencing this method

def _escape(text)       text.to_xs end

How do I go about overriding this ?

same way as you override any other method

class Builder::XmlMarkup    def _escape(text)      result = text.dup      result.gsub!("&", "&amp;")      result.gsub!("<", "&lt;")      result.gsub!(">", "&gt;")    end end

in my case what i did was

xml = Builder::XmlMarkup.new(:target => $stdout, :indent => 2) xml.instruct!

def xml.output(string)    result = string.dup    result.gsub!("&", "&amp;")    result.gsub!("<", "&lt;")    result.gsub!(">", "&gt;")    self << result end

and then call output rather than text whenever i wanted to output stuff. There was a thread a while back on one of the ruby list about the
fastest way to do this - do you do it in one pass fixing up each of
the 3 possible substitutions in one go, or in 3 passes like i did, and
concluded that the above version was the fastest. It might be
different if you had really long strings.

See also http://bogomips.org/fast_xs/ which is a c implementation of
the escaping thing (haven't tried this, but is probably the fastest of
all).

Fred

11

to expand on that, just gem install fast_xs. It's faster than what I had previously

Fred

12

yea fast_xs looks better, but right now its native C only so no Jruby support.

Is it possible to generated XML without using builder ? If the XML structure is not too complicated, and I am OK sacrificing some of the validation/substitution, could I just @results.each { |r| } and print some XML ?

Thx Adam

13

yea fast_xs looks better, but right now its native C only so no
Jruby support.

Is it possible to generated XML without using builder ? If the XML
structure is not too complicated, and I am OK sacrificing some of
the validation/substitution, could I just @results.each { |r| } and
print some XML ?

Course you can, it's just text after all! I got a significant speed
boost overwriting _escape so that might be enough for you.

Fred

14

Did you just drop this in application.rb ?

class Builder::XmlMarkup def _escape(text) result = text.dup result.gsub!(“&”, “&”) result.gsub!(“<”, “<”) result.gsub!(“>”, “>”) end end

15

Did you just drop this in application.rb ?

class Builder::XmlMarkup   def _escape(text)     result = text.dup     result.gsub!("&", "&amp;")     result.gsub!("<", "&lt;")     result.gsub!(">", "&gt;")   end end

just in a script i was writing. (but it should be ok in
application.rb), with the exception of one line which i left out :slight_smile:

it should be    def _escape(text)      result = text.dup      result.gsub!("&", "&amp;")      result.gsub!("<", "&lt;")      result.gsub!(">", "&gt;")      result    end

jruby is causing you to miss out on the fast_xs fun, but you might be
able to rewrite that method in java, with similar effects.

Fred

16

Frederick,

Thanks for the help but i am getting an error. I dropped this outside the ApplicationController but is throwing an error

class Builder::XmlMarkup def _escape(text) result = text.dup result.gsub!(“&”, “&”) result.gsub!(“<”, “<”) result.gsub!(“>”, “>”) result end end

ActionView::TemplateError (You have a nil object when you didn’t expect it! The error occurred while evaluating nil.gsub) on line #1 of channels/system.xml.builder: 1: xml.instruct! 2: xml.response { 3: xml.result { 4: xml.status “Success”

17

Frederick,

Thanks for the help but i am getting an error. I dropped this
outside the ApplicationController but is throwing an error

class Builder::XmlMarkup   def _escape(text)     result = text.dup     result.gsub!("&", "&amp;")     result.gsub!("<", "&lt;")     result.gsub!(">", "&gt;")     result   end end

Weird, works well enough for me. the error you got looks like what you
would get with the version without the result on the last line.

Fred

18

do you have it in application.rb ?

19

I also noticed that the _escape function is in xmlbase.rb not xmlmarkup.rb. Not sure if that is what is breaking here…

20

I also noticed that the _escape function is in xmlbase.rb not
xmlmarkup.rb. Not sure if that is what is breaking here...

XmlMarkup inherits from XmlBase so it shouldn't make any difference. I
didn't have it in application.rb but I doubt that would make a
difference.

Fred