Handling File on Upload

Hello,

I need some help trying to figure out how to add the ability for a
user to upload a text file, and then have the application parse that
text file, and add each record as instance of a model object. I've
found a lot of helpful articles out there, but I'm stuck trying to
bring it all together since I'm still very new to all of this.

Here's my view code:
# index.html.erb
<% form_for :upload, :url => { :action => :upload }, :html =>
{ :multipart => true } do |f| %>

    <p>Please select a file to upload:</p>
    <%= f.file_field :file %>
    <%= f.submit 'Upload', :disable_with => "Uploading..." %>

<% end %>

In my controller, I've been able to write something like this to work
with the data in my upload view:

# mailing_lists_controller.rb
def upload
    file = params[:upload][:file]
    @data = file.read
  end

so then the view looks like this:
# upload.html.erb
<% @data.split("\r\n").each do |row| %>

    <%= row.split("\t").join(" | ") %>

<% end %>

So, where I'm stuck is translating that into saving each row as a
record in the database. I've tried several different things in the
model (like creating a "process" method or using "before_save"), but
nothing has been successful thus far (probably because I'm not
implementing it correctly).

Any help would be appreciated.

Thanks,
Spencer

Hello,

I need some help trying to figure out how to add the ability for a

user to upload a text file, and then have the application parse that

text file, and add each record as instance of a model object. I’ve

found a lot of helpful articles out there, but I’m stuck trying to

bring it all together since I’m still very new to all of this.

Here’s my view code:

index.html.erb

<% form_for :upload, :url => { :action => :upload }, :html =>

{ :multipart => true } do |f| %>

<p>Please select a file to upload:</p>

<%= f.file_field :file %>

<%= f.submit 'Upload', :disable_with => "Uploading..." %>

<% end %>

In my controller, I’ve been able to write something like this to work

with the data in my upload view:

mailing_lists_controller.rb

def upload

file = params[:upload][:file]

@data = file.read

So here you probably want to read the file line by line and then do your magic on each line to create the model record for the line. You may have to play with it - not sure if you are going to need to save the file in a temp location first to do this. I dealt with a similar issue and file.read on the uploaded file parameter gives you a string, not a file object and you want a file object if you want to read it line by line:

file = File.new(params[:upload][:file] # maybe you can do this… have to try… if not you may just need to save it to a temp location,
# perhaps someone else has a better idea (also check params[:upload][:file] for its class… if the

                                                       # class is File then you are set.

while(line=file.gets)
m = Model.new

do your magic here

m.save
end
file.close

I recommend you to use Paperclip for files uploading.
http://railscasts.com/episodes/134-paperclip

Besides it's other awesome features, you want to use Paperclip
Processors to play with a file after upload.
http://mdeering.com/posts/018-paperclip-processors-doing-so-much-more-with-your-attachment

To read file lines you can go for

File.readlines(@file.path).collect(&:chomp).each do |line|
... (create your models here)
end

to collect all lines.
`chomp` will cut end-of-line symbols, and @file it's an object which
is given by Paperclip.
Or if you have troubles with this one, here's another option

IO.foreach("path/to/file.txt") do |line|
... (create your models here)
end

I recommend you to use Paperclip for files uploading.

http://railscasts.com/episodes/134-paperclip

The only thing if I recall, that kept me from using paperclip for my purposes, is the file essentially attached to the model but does not allow for modification and re-saving the file — am I mistaken? Unless things have changed recently, it would be super-cool if Paperclip would handle file mods just as a mod to any attribute of the model… although I guess this gets away from the basic intent of Paperclip, which for what it does do does it very nicely.

You are mistaken, and although I came late to the Paperclip party, I can't recall a time when it was true. You can edit the model without modifying the image (just don't upload another image) and everything stays the same in the image, or you can upload a new image and it will overwrite the previous version. It's all managed when you save the model that the image is attached to.

Walter

The only thing if I recall, that kept me from using paperclip for my purposes, is the file essentially attached to the model but does not allow for modification and re-saving the file — am I mistaken? Unless things have changed recently, it would be super-cool if Paperclip would handle file mods just as a mod to any attribute of the model… although I guess this gets away from the basic intent of Paperclip, which for what it does do does it very nicely.

You are mistaken, and although I came late to the Paperclip party, I can’t recall a time when it was true. You can edit the model without modifying the image (just don’t upload another image) and everything stays the same in the image, or you can upload a new image and it will overwrite the previous version. It’s all managed when you save the model that the image is attached to.

Right, but what I wanted was to be able to load a model instance, change the file (say I encrypt a portion of the text) and have Paperclip update that file on its own when I call model#save. I am pretty sure Paperclip does not do this. Right, you can re-save the file but it requires manual action beyond calling model#save.

Thanks, Vladimir and David for your replies.

I did experiment with Paperclip without much success. I look at the
post Vladimir linked to and that gave me the idea to create a custom
processor. It didn't work though. No errors thrown, but no records
saved either :-/

I'm sure I did it wrong, but here's the code: http://www.pastie.org/1354053

I also experimented with taking Paperclip out of the picture and put
the code to create the records in the controller. What I get now is
an error that says "invalid attribute". The invalid attribute
provided is the first email address of the first line of the file.

Here's that code: http://www.pastie.org/1354053

Thanks again for any and all help. This has been my largest hurdle to
overcome in Rails thus far.

-S

sorry...that was the wrong URL for the second pastie. Here's the
right one: http://www.pastie.org/1354138

If you code your transformation within a Paperclip Processor, you can have any number of different transformed versions. Here's one that extracts the text from an uploaded PDF:

#lib/paperclip_processors/text.rb
module Paperclip
   # Handles extracting plain text from PDF file attachments
   class Text < Processor

     attr_accessor :whiny

     # Creates a Text extract from PDF
     def make
       src = @file
       dst = Tempfile.new([@basename, 'txt'].compact.join("."))
       command = <<-end_command
         "#{ File.expand_path(src.path) }"
         "#{ File.expand_path(dst.path) }"
       end_command

       begin
         success = Paperclip.run("/usr/bin/pdftotext -nopgbrk", command.gsub(/\s+/, " "))
         Rails.logger.info "Processing #{src.path} to #{dst.path} in the text processor."
       rescue PaperclipCommandLineError
         raise PaperclipError, "There was an error processing the text for #{@basename}" if @whiny
       end
       dst
     end
   end
end

You call it from your model, like this:

#app/models/document.rb
...
   has_attached_file :pdf,:styles => { :text => { :fake => 'variable' } }, :processors => [:text]
...

The :fake => 'variable' part is just in there to get the offset correct for the processors variable. I am not sure if it's still needed, but I have been doing it this way since early this summer.

Later, you can access that version of the file as you would any other paperclip-attached model attribute. In this example, this might look like document.pdf.url(:text). Your untouched original will always be at document.pdf.url(:original).

Walter

I was finally able to get this to work. I ditched Paperclip and went
with putting the code in the controller. Taking it in baby steps, I
was able to work out this code for the controller:

def upload
    ((params[:upload][:file]).read).strip.split("\r\n").each do |line|
      email, account_number, sub_number, eid, premium_code, keycode,
order_date, description = line.split("\t")

      new_record = MailingList.new(:email => email, :account_number =>
account_number, :sub_number => sub_number,
                                    :eid => eid, :premium_code =>
premium_code, :keycode => keycode,
                                    :order_date =>
order_date, :description => description)
      new_record.save
    end

    redirect_to :action => index
  end

Thanks again to everyone. Ultimately it was all of your replies that
helped me piece it together.

Thanks!
Spencer

Hi spncrgr :smiley:

I am trying to do something similar…Is there a possibility that you have this project on Github? (or somewhere, where i can view the code?)

Thank you in advance

George.

Hi spncrgr :smiley:

I am trying to do something similar...Is there a possibility that you have
this project on Github? (or somewhere, where i can view the code?)

I rather doubt whether code from four years ago is likely to still be
the best solution.

Colin