Follow-Up: Receiving Inbound Email and Creating Data in Rails App

I just wanted to post back to the group with my potential solution for receiving inbound email and creating data in my Rails application, including attached images. Hope this will be helpful for others. Of course, suggestions for improving my ideas is always welcome.

Thanks to everyone who took the time to post helpful responses to my original (multiple) inquiries.

Nathan

Overall Requirement

I needed a feature in my Rails application that would receive inbound emails (from a Palm Treo 680, in particular) including an attached image. I wanted to use this data to create an entry in a product database for an online shopping web site. This requirement supports a small business owner who travels, finds products to buy and resell, and wanted immediate (easy) posting of available items on the site using a mobile device.

Design:

I decided to use a plain text email, with a defined format…

  • Subject = product title
  • 1st line = product description
  • 2nd line = product price
  • 3rd line = product cost (for accounting application sync)
  • 4th line = product quantity
  • Attachment = product image I created a “mailer” model inheriting from ActionMailer. The mailer model had two important methods (among others). The first was the “check” method which will use Net::IMAP to poll the INBOX folder for new mail. The check method would then call the “receive” method which will take an individual mail, create a product, and create an associated image in the database. Finally, there is a “delete” method which removes the processed mail from the inbox.

PROBLEM: I could not get IMAP to allow deletion of the old messages. I actually had to use POP3 to do it. Something wacky here that may just be my ignorance on this topic.

One of the bigger questions for me was how to periodically process the mail. There were several possibilities here and may well be others that I didn’t know about or consider. Here are a couple of the more popular ones…

  • Use a server script to periodically “push” the mail into the Rails app using cron
  • Use BackgroundRb to create a Rails background process for reading the mail periodically However, I chose something a little simpler and possibly weird. I used the application controller to call the check mail function once per user/browser session. So inside the application.rb I have a before filter that calls a method. The method checks for a session variable that indicates whether mail has been checked yet or not. If not, it calls the model method for doing so. I don’t know the complete implications or trade-offs here, so this is an area where I would be happy to get input from Ruby/Rails experts. Given the low usage of the site and small number of products being entered, I chose to use this given that the only downside seemed to be a small wait (1-2 seconds) on the initial site load for the mail to be checked and the parts/images loaded.

Development/Code:

class Mailer < ActionMailer::Base .

. . def self.check_mail

begin
  imap = Net::IMAP.new('mail.***.com')

  if RAILS_ENV == "production"
    imap.authenticate

(‘LOGIN’, ‘prduser’, ‘prdpasswd’) else

    imap.authenticate('LOGIN', 'devuser', 'devpasswd')
  end

  imap.examine('INBOX')
  imap.search(['FROM', 'validuser@***.com']).each do |message_id|

    msg = imap.fetch(message_id,'RFC822')[0].attr['RFC822']
    envelope = imap.fetch

(message_id, “ENVELOPE”)[0].attr[“ENVELOPE”] Mailer.receive(msg)

  end
  imap.logout
  return true

rescue
  RAILS_DEFAULT_LOGGER.error("Mailer Import Error: " + $!)

  return false
end

end

def receive(email) return unless email.has_attachments?

emailbody = email.body.to_s.gsub(/\r/, ' ')

part = Part.new
part.description = email.subject

part.notes = emailbody.split(/\n/)[0]
part.sellprice = emailbody.split(/\n/)[1].to_f

part.lastcost = emailbody.split(/\n/)[2].to_f
part.onhand = emailbody.split(/\n/)[3].to_i

part.partsgroup_id = 10280
if part.save!

  email.attachments.each do |attachment|
    next unless attachment.original_filename[-4..-1] == '.jpg'

    picture = Picture.new
    [picture.name](http://picture.name) = email.subject

    picture.part_id = [part.id](http://part.id)

    picture.imagedata = attachment.read
    picture.content_type = 'image/jpeg'

    picture.save!     
  end

end

end

def self.delete_mail if RAILS_ENV ==“production”

  Net::POP3.delete_all('mail.***.com', 110, 'prduser', 'prdpasswd')
else

  Net::POP3.delete_all('mail.***.com', 110, 'devuser', 'devpasswd')
end

rescue RAILS_DEFAULT_LOGGER.error("Mailer Import Error: " + $!)

end end

class ApplicationController < ActionController::Base

before_filter :env

def env if !session[:checked_mail] if Mailer.check_mail

    session[:checked_mail] = true
    Mailer.delete_mail

  end
end

end . .

. end

Testing:

So far, this solution seems to work pretty well. I have not tested large numbers of emails or large numbers of concurrent users. This is only a potential solution and not one I would recommend for a larger site. Let’s have it…what do you think? How can this be improved?

Nathan Leach wrote:

I just wanted to post back to the group with my potential solution for receiving inbound email and creating data in my Rails application, including attached images. Hope this will be helpful for others. Of course, suggestions for improving my ideas is always welcome.

Thanks to everyone who took the time to post helpful responses to my original (multiple) inquiries.

Nathan

Overall Requirement

I needed a feature in my Rails application that would receive inbound emails (from a Palm Treo 680, in particular) including an attached image. I wanted to use this data to create an entry in a product database for an online shopping web site. This requirement supports a small business owner who travels, finds products to buy and resell, and wanted immediate (easy) posting of available items on the site using a mobile device.

Design:

I decided to use a plain text email, with a defined format...

    * Subject = product title     * 1st line = product description     * 2nd line = product price     * 3rd line = product cost (for accounting application sync)     * 4th line = product quantity     * Attachment = product image

I created a "mailer" model inheriting from ActionMailer. The mailer model had two important methods (among others). The first was the "check" method which will use Net::IMAP to poll the INBOX folder for new mail. The check method would then call the "receive" method which will take an individual mail, create a product, and create an associated image in the database. Finally, there is a "delete" method which removes the processed mail from the inbox.

PROBLEM: I could not get IMAP to allow deletion of the old messages. I actually had to use POP3 to do it. Something wacky here that may just be my ignorance on this topic.

One of the bigger questions for me was how to periodically process the mail. There were several possibilities here and may well be others that I didn't know about or consider. Here are a couple of the more popular ones...

    * Use a server script to periodically "push" the mail into the       Rails app using cron     * Use BackgroundRb to create a Rails background process for       reading the mail periodically

However, I chose something a little simpler and possibly weird. I used the application controller to call the check mail function once per user/browser session. So inside the application.rb I have a before filter that calls a method. The method checks for a session variable that indicates whether mail has been checked yet or not. If not, it calls the model method for doing so. I don't know the complete implications or trade-offs here, so this is an area where I would be happy to get input from Ruby/Rails experts. Given the low usage of the site and small number of products being entered, I chose to use this given that the only downside seemed to be a small wait (1-2 seconds) on the initial site load for the mail to be checked and the parts/images loaded.

Development/Code:

class Mailer < ActionMailer::Base .   def self.check_mail     begin       imap = Net::IMAP.new('mail.***.com')       if RAILS_ENV == "production"         imap.authenticate ('LOGIN', 'prduser', 'prdpasswd')       else         imap.authenticate('LOGIN', 'devuser', 'devpasswd')       end       imap.examine('INBOX')       imap.search(['FROM', 'validuser@***.com']).each do |message_id|         msg = imap.fetch(message_id,'RFC822')[0].attr['RFC822']         envelope = imap.fetch (message_id, "ENVELOPE")[0].attr["ENVELOPE"]         Mailer.receive(msg)       end       imap.logout       return true     rescue       RAILS_DEFAULT_LOGGER.error("Mailer Import Error: " + $!)       return false     end   end    def receive(email) return unless email.has_attachments?     emailbody = email.body.to_s.gsub(/\r/, ' ')     part = Part.new     part.description = email.subject     part.notes = emailbody.split(/\n/)[0]     part.sellprice = emailbody.split(/\n/)[1].to_f     part.lastcost = emailbody.split(/\n/)[2].to_f     part.onhand = emailbody.split(/\n/)[3].to_i     part.partsgroup_id = 10280     if part.save!       email.attachments.each do |attachment|         next unless attachment.original_filename[-4..-1] == '.jpg'         picture = Picture.new         picture.name <http://picture.name> = email.subject         picture.part_id = part.id <http://part.id>         picture.imagedata = attachment.read         picture.content_type = 'image/jpeg'         picture.save! end     end   end

  def self.delete_mail     if RAILS_ENV =="production"       Net::POP3.delete_all('mail.***.com', 110, 'prduser', 'prdpasswd')     else       Net::POP3.delete_all('mail.***.com', 110, 'devuser', 'devpasswd')     end   rescue     RAILS_DEFAULT_LOGGER.error("Mailer Import Error: " + $!)   end end

class ApplicationController < ActionController::Base   before_filter :env    def env     if !session[:checked_mail]       if Mailer.check_mail         session[:checked_mail] = true         Mailer.delete_mail       end     end   end . end

Testing:

So far, this solution seems to work pretty well. I have not tested large numbers of emails or large numbers of concurrent users. This is only a potential solution and not one I would recommend for a larger site. Let's have it...what do you think? How can this be improved?

hmmm,

The biggest thing is as you said, checking the email on each session will not scale. In fact, the first request for each user will have to wait on the mail to fetched and parsed. cron or backgroundrb would be much better for that.

I think there should be a more rubyesque way for the emailbody.split lines but nothing comes to mind.

You should put the dev and production email passwords into the environment in the config folder along with the database password. Then just use the value set by the environment.

Sincerely, Jason

Jason,

Great points. I will move the settings to the config folder and try to find another way to process the email body.

I am considering the other two background email processing options. I do have another idea to improve the current call to the Mailer model. Not sure how feasible it is for production, but it seems to be working in development ok.

I was wondering about using a process fork to call the email check/receive. Take a look at this code that would reside in the application controller...

    if !session[:checked_mail]       Process.detach fork {         if Mailer.check_mail           session[:checked_mail] = true         end       }     end

Would it alleviate the performance and/or scaling issues? Will this cause problems?

Nathan

Backgroundrbis here :http://backgroundrb.rubyforge.org/

Note that it can integrate with rails easily :wink: as long as you work on a unix machine.

The latest Backgroundrb (0.2.1) runs fine on cygwin (=unix-like env for windows)

/Nils