Railsers:
Seeing as how everyone who works out a Rails technique should blog about it, (and seeing as how I decline to research how to build a gem, plugin, or blog), this post will sketch a system I developed to chat.
If anyone likes this, please put it on your blog, so other blogs can point to it and increase its Google pagerank, etc.
If anyone /doesn't/ like it, speak up now, or forever ... chat about it later!
== Requirements ==
From the user's viewpoint, a chat panel should work like this:
- hit your page - all prior chat utterances appear in a scrolling DIV - user enters text into a long thin field - the text appears at the bottom of the DIV - the DIV scrolls down to display it - anyone else viewing that page sees it, too - you see their chats
That's the core of chat. Here's a naive implementation:
- create a Chat database record and its model - put a DIV on your page - add a JavaScript DOM timer to your page - the timer sends AJAX "refresh" request - the server returns all the chat as HTML - the browser pushes the HTML into that DIV - put a form_remote_for :chat on your page - wire it to call an action, "utter" that adds a chat to the database - when the timer goes off, it "refreshes" this new chat
I hope to improve upon that implementation, but note that my implementation might be naive too. It will religiously observe the guideline "Premature Optimization is the Root of all Hourly Billing Strategies". I hope it is at least more accurate and responsive than some of the chat sample code I downloaded. For example: It won't top-post!
== Analysis ==
So let's guess that sending a long chat history once per timer will be slow. This implies that each batch of chat should append to the end of the history DIV.
Let's also guess we should not set our timer to run faster than every couple seconds. This implies each chat batch could contain more than one line.
We need a system to delimit a batch of chat, so the server can avoid sending the same batch of chat over and over again. We must store a "high water mark" variable, and only send chat that exceeds this value. And we shouldn't store it in a session variable, because they don't scale. Session variables are essentially scaffolding; to use in a pinch, before we figure out a truly stateless solution.
So we add these lines to that algorithm:
- add a hidden last_chat_id input field - the "refresh" timer sends the last_chat_id - the server fetches chat with id > last_chat_id - the "refresh" response contains the highest chat id - the browser pushes this into the hidden field
That system ass-u-me-s that our database will always increase and never roll-over the chat.id variable. We could replace the variable with a timestamp; the rest of the algorithm will work the same. Another solution would be to occassionally "accidentally" erase the entire chat table and reset its index ID. (That solution has the added benefit of insulating us from compliance with court orders requesting proof that someone once chatted something naughty!)
To install a "high water mark", we need one variable's value, last_chat_id, to travel thru these layers:
- a hidden form field - the ActiveForm helper that builds AJAX - the JavaScript inside that helper - the wire - the server's param bundle of joy - the ActiveRecord parameters - the SQL statement - the SQL's data store - the returned recordset (the maximum one in this batch) - the returned HTML inside the wire - the AJAX JavaScript handler - our browser's page's DOM elements - something that pushes last_chat_id back into its hidden field!
Gods bless N-tier architectures, huh?
But focus on "something". If our chat action returned raw HTML (such as a TABLE full of rows, each containing a chat utterance), then our AJAX handler would push this into the DIV tag containing our chat history. Not into the FORM tag containing our hidden field.
This implies our HTML package should return a SCRIPT tag containing JavaScript that updates that field to our hidden value.
That sounds like a plan, but here's another problem.
Suppose our timer goes off, and our browser launches a "refresh" message. Then the server slows down under load, and the timer gets bored and launches another one. Each will have the same last_chat_id variable. Slowing down our timer is not an option, because if everyone starts chatting the server could slow down enough to encounter this problem.
A chat batch should come back to the browser containing enough information to censor duplicate lines. So this implies the chat batch is not HTML, it's all JavaScript. It's a series of addChat() calls, targetting a function in our application.js file with this signature:
addChat(panel_id, chat_id, fighter_name, utterance)
That will first check if our browser already contains a SPAN with an ID set to 'chat_id'. If so, this is a duplicate chat, so just bounce out. (In a stateless, high-latency system, sending a little more data than we need is always more efficient than risking sending a little less!)
If the chat utterance is not in the history DIV yet, push it in. (Code below will show how.)
That JavaScript function has one more variable, and explaining it will finish our requirements.
== Productization ==
The 'panel_id' is a number indicating which chat panel we target. Our application could have more than one chat - sometimes more than one on each page! So each DIV, FORM, and SPAN we have discussed will need an ID with this number inside it, such as <div id='panel_<%= panel_id %>_chat_history'...>, to make it unique. That's so all our excessive JavaScript can find the correct panel when it goes to harvest data out or plug data back in.
So henceforth, if I leave out the excessive <%= panel_id %> baggage, and only write panel_1_, just imagine it's there, because it's easy to code. Relatively.
== The System ==
So here's the complete rig:
* a chat model/table with - id - user_id --> the usual user/login account system - panel_id --> optional panel model/table - uttered_at datetime - utterance
* the chat-endowed page has - 'application.js', containing addChat(panel_id, chat_id, fighter_name, utterance) scrollMe(panel_id) <-- calls div.scrollTop = 0xffff; setLastChatId(panel_id, chat_id) - a DIV called 'panel_1_chat_history' - a FORM to be called 'panel_1_chat_form' with a big edit field and a submit button with a HIDDEN field called panel_1_last_chat_id and an ONSUBMIT with magic AJAXy stuff in it that targets :action => 'utter' and updates this: - a SPAN called 'panel_1_new_content_script' - a periodically_call_remote() that updates 'panel_1_new_content_script' at :frequency => 5 seconds :with => 'panel_1_last_chat_id' targets :action => 'utter'
* the chat-endowed page's controller has (or imports*) - an action 'utter' which finds a chat in the params and saves it returns a SCRIPT containing JavaScript code that updates the 'panel_1_chat_history'
(*Note there's a slight gap between canonical MVC and Rails's convenient implementation. True MVC makes elements in each module pluggable, so elements in Views can plug into elements in Controllers, independent of each other. Rails, by contrast, uses "Controller" to mean "thing that serves logic for one coupled View". [Unless I'm incredibly mistaken.] This is still a very good pattern - it enforces the most important separation; between GUI Layer concerns and Representation Layer concerns. Yet if this were true MVC, then my partial chat View could plug into a truly decoupled Controller. Not a Controller that couples with every page that contains chat panels. But I digress...)
Now notice that the same action, 'utter', serves both saving chat and refreshing the history. This is because we have two AJAX conversations, and we might as well use both to reflect the latest chat back to the user. That allows the user to see their own chat, much sooner than our 5 second timer will allow. This preserves the illusion that other chats arrive promptly, too.
== Implementation ==
The following code snippets illustrate the stations on that cycle that are hard to code. The easy ones (like, uh, lots of RHTML, DDL, etc), are left as an exercise for the reader. The easy tests are left too (and, yes, I have them!).
Create a chat controller/view with one target action, '_index.rhtml'. Go to the your page that will host a chat panel, and add this line, wherever you want chat:
<%= render :partial => 'chat/index', :locals => { :panel_id => 1 } %>
Make the :panel_id unique, somehow. If you only have a few chats, simply hard-code each ones ID. Always remember: Magic Numbers are a perfectly safe and valid coding practice if they are less than 10.
The first part of '_index.rhtml' is the DIV that will contain our scrolling chat history:
<div id='panel_<%= panel_id %>_chat_history' style='height: 30ex; font-size: 90%; border-bottom: #eeeeee solid 1px; overflow: auto; background-color: #ffffee' > </div>
That sets the font size a little smaller than normal (a trick I learned from >cough< Google Chat), and it sets the overflow style to automatic scroll bars.
The <div> goes over the wire empty, and your browser will populate it during the split second before it paints.
Next, '_index.rhtml' starts the FORM that will submit new text:
<% @chat = Chat.new() @chat.user_id = @user.id @chat.panel_id = panel_id
form_remote_for( :chat, :url=>{ :action=>:utter }, :html=>{ :id => 'chat_panel_'+panel_id.to_s, :style => 'display:inline' }, :update=>'panel_'+panel_id.to_s+'_new_content_script', :loading=>load, :complete=>comp, :after=>"chat_utterance.value='';" ) do |f| %>
The form contains TABLE markup to stretch things out, and it finishes with these important fields:
<%= f.text_field :utterance, :style=>'width:100%' %> ... <%= f.hidden_field :user_id %> <%= f.hidden_field :panel_id %>
<input type='hidden' id='panel_<%= panel_id %>_last_chat_id' name='last_chat_id' value='0' />
<%= submit_tag 'Send' if @user %> <span id="loading" style="background-color:white;color:white;"> sending... </span> ... <% end %>
I had to explain why we are doing these things, up in the Analysis section, so you will recognize the variables. Otherwise they would just look like excessive markups. (Right?)
Notice we use the remote form's :loading and :complete parameters to flash the text "sending...". I will eventually upgrade to a system that neither twitches the page's geometry, nor reveals the invisible word "loading..." if you drag-select under the Send button.
Another potential improvement: The controller will soon bounce any new chat with an empty utterance. (We should not reward the user for banging Enter after writing nothing, or spaces!) However, I suspect we could bounce at /this/ layer, if we add more JavaScript into the mix.
When that FORM submits, it triggers the 'utter' action:
def utter return unless request.xhr?
if params[:chat] if validate_utterance(params[:chat]) params[:uttered_at] = Time.now chat = Chat.create!(params[:chat]) else render(:nothing => true) # that's for wasting our time! return end end
last_chat_id = params[:last_chat_id]
tex = Chat.get_chattage( params[:panel_id] || params[:chat][:panel_id], last_chat_id.to_i ) render(:text => tex) end
That (mildly overloaded) action possibly creates a new chat (if the user really had something to say), and it then returns any pending chats, possibly including the recently added chat record.
Without the render(:nothing=>true) line, we would reward any user who banged <Enter> insistently. They would cause more server hits than others, and would fetch chat more often, at the cost of tremendous overhead in their browser and mild overhead in our server. So we don't serve new chat to anyone who did not write an utterance. They can wait for their timer.
(Contrarily, I should have set the limit on our input field. That will also prevent annoying error messages, flooding, etc.)
With only a few more code snippets to go, I will leave out the model stuff that anyone could write. Get your own model! We are nearing the end.
The faux-Controller method Chat.get_chattage() returns a snip of HTML containing JavaScript, like this:
<script> addChat(2, 9, "userA", "chat utterance one"); addChat(2, 10, "userB", "chat utterance two"); addChat(2, 11, "userA", "chat utterance three"); setLastChatId(2, 11); scrollMe(2); </script>
Notice I didn't waste time adding //<![CDATA[ or whatever to mask that script from naive browsers. If the user gets this far, they must have JavaScript turned on! (I have also tested my code in Konqueror, Firefox, and other amateur browsers, and I suspect it's generic...)
AJAX will push that SCRIPT into this SPAN:
<span id='panel_<%= panel_id %>_new_content_script'> <%= Chat.get_chattage(panel_id, 0) %> </span>
The partial '_index.rhtml' also calls get_chattage, so our page loads with a complete chat history. (Note that this call will take care of all the pre-existing <spans>, with their correct IDs, in the history DIV, _and_ it will take care of that sneaky 'last_chat_id' field! Nice, huh?
get_chattage() returns JavaScript that, one way or another, will call these functions, from 'application.js':
function scrollMe(panel_id){ // moves the history DIV to the bottom var chat_history = 'panel_'+panel_id+'_chat_history'; document.getElementById(chat_history).scrollTop = 0xffff; }
var last_user_ids = {}; var last_toggles = {};
function addChat(panel_id, chat_id, fighter_name, utterance) { span_id = 'panel_'+panel_id+'_chat_'+chat_id; if (document.getElementById(span_id)) return; var chat_history = 'panel_'+panel_id+'_chat_history'; div = document.getElementById(chat_history); html = '<span id="'+span_id+'" width="100%" '; var toggle = last_toggles[panel_id]; toggle = !!toggle;
if (last_user_ids[panel_id] != fighter_name) { toggle = !toggle; last_toggles[panel_id] = toggle; }
var bgColor = toggle? 'white': '#eeeeee'; html += 'style="width:100%;color:black;background-color:'+bgColor+';" >'; html += fighter_name; html += ': '; html += utterance; html += '</span><br/>'; div.innerHTML += html; last_user_ids[panel_id] = fighter_name; }
function setLastChatId(panel_id, id) { var last_chat_id = document.getElementById('panel_'+panel_id+'_last_chat_id'); last_chat_id.value = id; }
If you have been following along, you recognize things like 'setLastChatId()'. But the 'bgColor' and 'toggle' stuff is new. Briefly, it assigns the same background color to contiguous utterances by the same user, and it alternates the color when alternate characters speak. So if you send 2 chats, they both get Gray, for example. Then if your opponent chats, they get White. If someone else chats they get Grey, and if you chat again you now get White. So the system works to join utterances into apparent paragraphs, and it always contrasts different speakers.
That elusive support function, Chat.get_chattage(), is straightforward Rails and ActiveRecord code, so I will just touch on its main concepts. Firstly, it needs to sanitize user input, and put it into a "string with quotes". It does this without the help of h(), which I couldn't figure out how to import (or how to Google for!):
def escapeHTML(str) return(str. gsub(/&/, "&"). gsub(/"/, """). gsub(/>/, ">"). gsub(/</, "<") ) end
def enstring(str) # CONSIDER move this to a helper? return '"' + escapeHTML(str) + '"' end
get_chattage also needs to fetch its relevant chat. All the above architecture has _finally_ pulled all these variables together, right where we need them:
def self.get_chattage(panel_id, last_chat_id)
where = ['panel_id = ? and id > ?', panel_id, last_chat_id]
chats = Chat.find(:all, :conditions => where, :order => 'uttered_at')
if != chats tex = '<script>'
chats.each do |chat| tex += chat.to_javascript last_chat_id = chat.id if last_chat_id < chat.id end
tex += 'setLastChatId('+panel_id.to_s+', ' + last_chat_id.to_s + ');' tex += 'scrollMe('+panel_id.to_s+');</script>' return tex end
return nil end
If the system has new chats, we convert each one to a JavaScript statement, and stuff them all together with .to_javascript(). Finally, we detect the new "high water mark" - the new maximum chat_id, and we pass these to the browser. And we trigger the correct history DIV to scroll to the bottom.
Chat::to_javascript() looks a little bit like this:
def to_javascript fighter_name = enstring(user.fighter_name) utterance = enstring(self.utterance) return "addChat(#{self.panel_id}, #{id}, #{fighter_name}, #{utterance});" end
And that's the bottom of the cycle. New chats follow this sequence:
- a user enters them into the big edit field - AJAX sends them to the 'utter' action - 'utter' puts them into the database - the database sets their id to > last_chat_id - our find() detects them - we convert them into JavaScript - we AJAX them back to the browser - the browser evaluates their JavaScript - and sticks them into the history
== Testing ==
Every factoid in this post has a matching unit test. Really.
I will conclude the code snippets with an interesting test. Without introduction, read it and try to see what it does:
require 'rexml/document' include REXML
class ChatTest < Test::Unit::TestCase fixtures :chats, :users
def addChat(*foo) assert_equal 2, foo[0] @utterances << foo end
def setLastChatId(panel_id, id) assert_equal 2, panel_id @last_chat_id = id end
def scrollMe(panel_id) assert_equal 2, panel_id @scrollMe_was_called = true end # TODO better name
def test_get_chattage chattage = Chat.get_chattage(2, 0)
sauce = XPath.first(Document.new(chattage), '/script').text @utterances = @last_chat_id = nil @scrollMe_was_called = false
assert eval(sauce) # no jury would convict me
expect_utterances = [ [2, 9, "userA", "chat utterance one"], [2, 10, "userB" , "chat utterance two"], [2, 11, "userA", "chat utterance three"] ]
assert_equal expect_utterances, @utterances assert_equal 11, @last_chat_id assert @scrollMe_was_called end end
That test case uses a mock pattern called "self shunt", where your own test case object works as your mock object.
We are mocking 'application.js'. Look at our sample SCRIPT again:
<script> addChat(2, 9, "userA", "chat utterance one"); addChat(2, 10, "userB", "chat utterance two"); addChat(2, 11, "userA", "chat utterance three"); setLastChatId(2, 11); scrollMe(2); </script>
The XPath stuff strips out the <script> tags, leaving JavaScript that looks suspiciously similar to Ruby source. Rather than wait for all the web browser venders in the world to get off their butts and add Ruby to their script engines, we simply write only Ruby-style JavaScript.
Then we eval() it. That calls all our mock functions, and they trivially report they were called correctly.
== TODO ==
Right now, nothing retires old chats. If the user returns to a page left on an open server for a couple months, they will (efficiently) see a couple months' worth of chat. Something needs to clean all that up.
The current design is very flexible. (I even flexed it a little as I wrote this post!
We could, for example, decorate each user's name in the list. Decorate means to add user-specific links, icons, colors, etc. Because we have a working cycle, we can extend it by adding variables to the chat records and the user records, then we insert them into addChat(), and render them in the DOM.
I suspect one could use "JSON" to render a record as JavaScript, but I have not researched it yet. It might make mocking our JavaScript hard!
I also suspect that there are libraries available to generate JavaScript from Ruby generators. This implies one could test-first such JavaScript by querying those generators directly.
Lastly, I did not yet re-use this chat into more than one page. This violates the advice I always give, that you should never add extra code that _might_ work when you re-use. My excessive panel_id variable violated that rule! I might report back what happens when I add more chat panels, and I actually see what will happen when I stress that panel_id!
Sorry to hijack your thread, but I
was actually trying to do exactly this as part of my learning process.