Dear All,
Fairly new to rails and Heroku, so could be doing something wacky - do let me know if you think my code practice is off, even if unrelated to this error, I'd like to learn!
I'm using Rails 3.0.0, ruby 1.8.7, and 'sqlite3-ruby', '1.2.5', :require => 'sqlite3'.
I've got an application that goes off and scrapes websites for event information, to do this I'm using:
nokogiri - for scraping date - to help with parsing chronic - to help recognise dates open-uri - to open the url
all of these are just gem 'gem-name', so latest unspecified versions.
The code that does this scraping is in the ApplicationController class - a bad place for it? I've just got it working as a background process using delayed_job, I have a class called EventSearcher with the required 'perform' method.
My error occurs when I upload to Heroku, if I make changes to the app, but NOT to ApplicationController, do $git add . / git commit .... / git push heroku master / heroku workers 1 / heroku open then the process runs, appears successful in the logs, but returns no events
The error can be solved I've discovered by making small changes to Application controller, such as adding the line logger.debug"Changing AppController again" then doing $git add . / git commit .... / git push heroku master / heroku workers 1 / heroku open this time the process runs, appears in logs to take longer, and returns the scraped info as required.
Any ideas why this should be??
I have tried solving this myself by putting in 'logger.debug" commands, but these don't show on my heroku logs, any idea how to get them to show up?
I've put the code below, for ApplicationController, EventSearcher, and part of SuggestedEvents it's a bit of a mess I know, but runs fine on local machine.
-------------- Application Controller
class ApplicationController < ActionController::Base
require 'date' require 'nokogiri' require 'open-uri' require 'chronic'
protect_from_forgery include SessionsHelper
def authenticate deny_access unless signed_in? end
def find_all_events
@event_searches = EventSearch.all
@event_searches.each do |es|
do_search(es)
end end
def find_one_band_events(event_id)
es = EventSearch.where(:id => event_id)[0] do_search(es) end
def do_search(event_search)
logger.debug"Changing AppController again" logger.debug"in do search" url = event_search.urlOne
if validate(url)
logger.debug"Passed Validate URL"
doc = Nokogiri::HTML(open(url))
#search the doc with the saved css search strings if !(event_search.eventDateCSS.empty?) startDate = doc.css(event_search.eventDateCSS) else eventDate = 0 end if !(event_search.eventNameCSS.empty?) eventName = doc.css(event_search.eventNameCSS) else eventName = 0 end if !(event_search.eventLocationCSS.empty?) location = doc.css(event_search.eventLocationCSS) else location = 0 end if !(event_search.eventTimeCSS.empty?) time = doc.css(event_search.eventTimeCSS) else time = 0 end if !(event_search.priceCSS.empty?) price = doc.css(event_search.priceCSS) else price = 0 end if !(event_search.descriptionCSS.empty?) description = doc.css(event_search.descriptionCSS) else description = 0 end i=0
if eventDate != 0 while i < startDate.length # find all events on that page going through each case of start date
SuggestedEvent.new do |@savedEvent|
if eventDate != 0 @savedEvent.eventDate = date_from_string(startDate[i].content) logger.debug(startDate[i].content) end
test = SuggestedEvent.where(:bandName => event_search.bandName, :eventDate => @savedEvent.eventDate)
if !test[0]
@savedEvent.bandName = event_search.bandName if eventName != 0 @savedEvent.eventName = remove_whitespace(eventName[i].content) end if time != 0 @savedEvent.doorsOpen = Chronic.parse(time[i].content) end if price != 0 @savedEvent.price = price[i].content end if description !=0 @savedEvent.description = remove_whitespace(description[i].content) end
if location != 0 savedVenueName = paramMatcher('Venue','venueName', remove_whitespace(location[i].content)) if savedVenueName @savedEvent.venue = savedVenueName[0] @savedEvent.city_location = savedVenueName[1] @savedEvent.longitude = savedVenueName[2] @savedEvent.latitude = savedVenueName[3] else @newVenue = Venue.new @newVenue.venueName = remove_whitespace(location[i].content) @newVenue.old_venue_name = @newVenue.venueName @newVenue.save @savedEvent.venue = remove_whitespace(location[i].content) end #of if else @savedEvent.venue = 'No saved venue found' end #of location != @savedEvent.save end # of if !test end # of .new do i += 1 end # of while
end #of if eventDate !=0
end # of if url passes
end # of def
# get the date from whatever string gets thrown def date_from_string(date)
# should remove st rd nd and th
firstParse = Chronic.parse(date) r1 = /[a-zA-Z]/
#daY Less than 12, assume chronic wrong for Britain, as long as also no characters such as December, #where it would be right if firstParse.day <= 12 and !r1.match(date)
#swap month with day firstParse = firstParse.change(:day => firstParse.month, :month => firstParse.day)
end #of if
return firstParse
end #of def
# remove whitespace and other characters such as newline
def remove_whitespace(dirty_name)
return dirty_name.split(' ').join(" ")
end
#find names in strings from the net def paramMatcher(modelString, param, cssScrapedString)
case modelString when 'Venue' venues = Venue.all i=0 venues.each do |v|
if v.venueName.scan(cssScrapedString)[0] or cssScrapedString.scan(v.venueName)[0]
return [v.venueName, v.city_location, v.longitude, v.latitude] end
end # of .each end # of case
return nil end # of def param Matcher
def validate(url) begin uri = URI.parse(url) if uri.class != URI::HTTP return false end rescue URI::InvalidURIError return false end return true end
end # of class
----------------- Event Searcher
class EventSearcher < ApplicationController
attr_accessor :all_events attr_accessor :one_event_id
def perform
logger.debug"in perform"
if all_events == 1 # searches all band names
logger.debug"in perform, all_events == 1"
find_all_events
end
if one_event_id != nil #searches just one band name using id
logger.debug"in perform, one_event != nil" logger.debug(one_event_id) find_one_band_events(one_event_id)
end
end # Of Perform
end #of class
-------------- Suggested Events
class SuggestedEventsController < ApplicationController
#.... other functions removed for brevity....
def search_all_events
logger.debug"Sugg E search all events" #the instance of the object that gets put on the queue @event_searcher = EventSearcher.new @event_searcher.one_event_id = nil @event_searcher.all_events = 1 Delayed::Job.enqueue @event_searcher
flash[:notice] = "Finding All events, this takes a while, refresh after 10 mins" redirect_to suggested_events_path
end
end
