Heroku, needs constant AppController updates?

Dear All,

Fairly new to rails and Heroku, so could be doing something wacky - do
let me know if you think my code practice is off, even if unrelated to
this error, I'd like to learn!

I'm using Rails 3.0.0, ruby 1.8.7, and 'sqlite3-ruby', '1.2.5', :require
=> 'sqlite3'.

I've got an application that goes off and scrapes websites for event
information, to do this I'm using:

nokogiri - for scraping
date - to help with parsing
chronic - to help recognise dates
open-uri - to open the url

all of these are just gem 'gem-name', so latest unspecified versions.

The code that does this scraping is in the ApplicationController class -
a bad place for it?
I've just got it working as a background process using delayed_job, I
have a class called EventSearcher with the required 'perform' method.

My error occurs when I upload to Heroku, if I make changes to the app,
but NOT to ApplicationController,
do $git add . / git commit .... / git push heroku master / heroku
workers 1 / heroku open
then the process runs, appears successful in the logs, but returns no
events

The error can be solved I've discovered by making small changes to
Application controller, such as adding the line
logger.debug"Changing AppController again"
then doing $git add . / git commit .... / git push heroku master /
heroku workers 1 / heroku open
this time the process runs, appears in logs to take longer, and returns
the scraped info as required.

Any ideas why this should be??

I have tried solving this myself by putting in 'logger.debug" commands,
but these don't show on my heroku logs, any idea how to get them to show
up?

I've put the code below, for ApplicationController, EventSearcher, and
part of SuggestedEvents it's a bit of a mess I know, but runs fine on
local machine.

-------------- Application Controller

class ApplicationController < ActionController::Base

  require 'date'
  require 'nokogiri'
  require 'open-uri'
  require 'chronic'

  protect_from_forgery
  include SessionsHelper

    def authenticate
      deny_access unless signed_in?
    end

  def find_all_events

  @event_searches = EventSearch.all

  @event_searches.each do |es|

  do_search(es)

  end
end

def find_one_band_events(event_id)

  es = EventSearch.where(:id => event_id)[0]
  do_search(es)
end

def do_search(event_search)

  logger.debug"Changing AppController again"
  logger.debug"in do search"
  url = event_search.urlOne

if validate(url)

logger.debug"Passed Validate URL"

  doc = Nokogiri::HTML(open(url))

  #search the doc with the saved css search strings
  if !(event_search.eventDateCSS.empty?)
    startDate = doc.css(event_search.eventDateCSS) else eventDate = 0
end
  if !(event_search.eventNameCSS.empty?)
    eventName = doc.css(event_search.eventNameCSS) else eventName = 0
end
  if !(event_search.eventLocationCSS.empty?)
    location = doc.css(event_search.eventLocationCSS) else location = 0
end
  if !(event_search.eventTimeCSS.empty?)
    time = doc.css(event_search.eventTimeCSS) else time = 0 end
  if !(event_search.priceCSS.empty?)
    price = doc.css(event_search.priceCSS) else price = 0 end
  if !(event_search.descriptionCSS.empty?)
    description = doc.css(event_search.descriptionCSS) else description
= 0 end
  i=0

if eventDate != 0
  while i < startDate.length # find all events on that page going
through each case of start date

    SuggestedEvent.new do |@savedEvent|

      if eventDate != 0
        @savedEvent.eventDate = date_from_string(startDate[i].content)
        logger.debug(startDate[i].content)
        end

      test = SuggestedEvent.where(:bandName => event_search.bandName,
                    :eventDate => @savedEvent.eventDate)

      if !test[0]

      @savedEvent.bandName = event_search.bandName
      if eventName != 0
        @savedEvent.eventName = remove_whitespace(eventName[i].content)
        end
      if time != 0
        @savedEvent.doorsOpen = Chronic.parse(time[i].content) end
      if price != 0
        @savedEvent.price = price[i].content end
      if description !=0
        @savedEvent.description =
remove_whitespace(description[i].content) end

      if location != 0
        savedVenueName = paramMatcher('Venue','venueName',
remove_whitespace(location[i].content))
        if savedVenueName
          @savedEvent.venue = savedVenueName[0]
          @savedEvent.city_location = savedVenueName[1]
          @savedEvent.longitude = savedVenueName[2]
          @savedEvent.latitude = savedVenueName[3]
        else
          @newVenue = Venue.new
          @newVenue.venueName = remove_whitespace(location[i].content)
          @newVenue.old_venue_name = @newVenue.venueName
          @newVenue.save
          @savedEvent.venue = remove_whitespace(location[i].content)
        end #of if
      else
          @savedEvent.venue = 'No saved venue found'
      end #of location !=
      @savedEvent.save
      end # of if !test
    end # of .new do
   i += 1
  end # of while

end #of if eventDate !=0

end # of if url passes

end # of def

# get the date from whatever string gets thrown
def date_from_string(date)

# should remove st rd nd and th

firstParse = Chronic.parse(date)
r1 = /[a-zA-Z]/

#daY Less than 12, assume chronic wrong for Britain, as long as also no
characters such as December,
#where it would be right
if firstParse.day <= 12 and !r1.match(date)

  #swap month with day
  firstParse = firstParse.change(:day => firstParse.month, :month =>
firstParse.day)

  end #of if

  return firstParse

end #of def

# remove whitespace and other characters such as newline

def remove_whitespace(dirty_name)

  return dirty_name.split(' ').join(" ")

end

#find names in strings from the net
def paramMatcher(modelString, param, cssScrapedString)

  case modelString
  when 'Venue'
  venues = Venue.all
  i=0
    venues.each do |v|

      if v.venueName.scan(cssScrapedString)[0] or
cssScrapedString.scan(v.venueName)[0]

      return [v.venueName, v.city_location, v.longitude, v.latitude]
      end

    end # of .each
  end # of case

return nil
end # of def param Matcher

def validate(url)
  begin
    uri = URI.parse(url)
    if uri.class != URI::HTTP
    return false
    end
    rescue URI::InvalidURIError
    return false
    end
  return true
end

end # of class

----------------- Event Searcher

class EventSearcher < ApplicationController

  attr_accessor :all_events
  attr_accessor :one_event_id

  def perform

  logger.debug"in perform"

  if all_events == 1 # searches all band names

  logger.debug"in perform, all_events == 1"

  find_all_events

  end

  if one_event_id != nil #searches just one band name using id

  logger.debug"in perform, one_event != nil"
  logger.debug(one_event_id)
  find_one_band_events(one_event_id)

  end

  end # Of Perform

end #of class

-------------- Suggested Events

class SuggestedEventsController < ApplicationController

#.... other functions removed for brevity....

  def search_all_events

  logger.debug"Sugg E search all events"
  #the instance of the object that gets put on the queue
  @event_searcher = EventSearcher.new
  @event_searcher.one_event_id = nil
  @event_searcher.all_events = 1
  Delayed::Job.enqueue @event_searcher

  flash[:notice] = "Finding All events, this takes a while, refresh
after 10 mins"
  redirect_to suggested_events_path

  end

end

Dear All,

Fairly new to rails and Heroku, so could be doing something wacky - do
let me know if you think my code practice is off, even if unrelated to
this error, I'd like to learn!

I'm using Rails 3.0.0, ruby 1.8.7, and 'sqlite3-ruby', '1.2.5', :require
=> 'sqlite3'.

I've got an application that goes off and scrapes websites for event
information, to do this I'm using:

nokogiri - for scraping
date - to help with parsing
chronic - to help recognise dates
open-uri - to open the url

all of these are just gem 'gem-name', so latest unspecified versions.

The code that does this scraping is in the ApplicationController class -
a bad place for it?

I don't know if it is related to your problem, but it does seem a
bizarre place to put that code - I'd stick it all in the EventSearcher
class and have EventSearcher inherit from object directy)

Fred

Fred

Thanks for the advice Fred,

I've moved the search logic / parsing across to the EventSearcher class
now as you suggested, and it seems to be working. I had it in there to
begin with, and moved it out because it couldn't find other classes it
needed. But I now think this was caused because I didn't restart the
local server before retrying.

When you say "and have EventSearcher inherit from object directy)" could
you explain how to do this, currently I have EventSearcher inherit from
Application controller with

class EventSearcher < ApplicationController

is this what you mean?

thanks,

Mike

Thanks for the advice Fred,

I've moved the search logic / parsing across to the EventSearcher class
now as you suggested, and it seems to be working. I had it in there to
begin with, and moved it out because it couldn't find other classes it
needed. But I now think this was caused because I didn't restart the
local server before retrying.

When you say "and have EventSearcher inherit from object directy)" could
you explain how to do this, currently I have EventSearcher inherit from
Application controller with

I meant just
class EventSearcher
...
end

(which is the same as < Object)

Fred

I tried making EventSearcher just inherit from object, ie:

class EventSearcher

but then I get errors from almost the first line, eg.

failed with NameError: undefined local variable or method `logger'

for my debug statements, and I don't suppose it'll know about my model
Classes either. Having it inherit from ApplicationController solves
these problems, but is there a better way of doing this?

cheers,

Mike

I tried making EventSearcher just inherit from object, ie:

class EventSearcher

but then I get errors from almost the first line, eg.

failed with NameError: undefined local variable or method `logger'

for my debug statements, and I don't suppose it'll know about my model
Classes either. Having it inherit from ApplicationController solves
these problems, but is there a better way of doing this?

Finding your model classes should not be a problem. logger is an instance method on controllers, so you don't get it here. Rails.logger will get you the same logger though (and you could define a logger method yourself to save some typing.

Fred

Cool, learning plenty here, cheers!

I've got a logger method in the class now, and so taken away the
unnecessary inheritance. Cheers!

My heroku app is working for now, although I'm not sure it'll stay
working, it seems to stop fetching events after a day, and needs a new
git push, with changes to specifically the "def do_search(event_search)"
method which is the one that does the work of getting the url, and
parsing the html. A bit weird, as it works every time on local machine.
I've asked them about this on the heroku mailing list as well.

Thanks for your help!

M

Cool, learning plenty here, cheers!

I've got a logger method in the class now, and so taken away the
unnecessary inheritance. Cheers!

My heroku app is working for now, although I'm not sure it'll stay
working, it seems to stop fetching events after a day, and needs a new
git push, with changes to specifically the "def do_search(event_search)"
method which is the one that does the work of getting the url, and
parsing the html. A bit weird, as it works every time on local machine.
I've asked them about this on the heroku mailing list as well.

If your daemon is idle for extended periods of time it might be your
mysql connection timing out. You could try tossing in a call to
ActiveRecord::Base.connection_pool.verify_active_connections! to weed
out dead connections

Fred

Michael Baldock wrote in post #972049:

Cool, learning plenty here, cheers!

I've got a logger method in the class now, and so taken away the
unnecessary inheritance. Cheers!

My heroku app is working for now, although I'm not sure it'll stay
working, it seems to stop fetching events after a day, and needs a new
git push, with changes to specifically the "def do_search(event_search)"
method which is the one that does the work of getting the url, and
parsing the html. A bit weird, as it works every time on local machine.
I've asked them about this on the heroku mailing list as well.

Why don't you set up a cron job or use Resque or delayed_job to run the
job periodically?

Thanks for your help!

M

Best,