upload very very large file?

Hi, I don't know if there are differing opinions on this, but seems to
me that apps that use such large files will write a file manager
(desktop based app or at least activex) that monitor or possibly
divide such files into smaller pieces. Just seems even with a
webserver that would handle such a direct upload you are going to be
subject to so many extraneous factors which could affect success of
such.

DK wrote:

Hi, I don't know if there are differing opinions on this, but seems to
me that apps that use such large files will write a file manager
(desktop based app or at least activex) that monitor or possibly
divide such files into smaller pieces.

Not necessarily Look at Dropbox (though they do use a Flash upload
tool).

Just seems even with a
webserver that would handle such a direct upload you are going to be
subject to so many extraneous factors which could affect success of
such.

Theoretically, uploading a big file should be just like uploading a
small one. The problem is that there's more time for the connection to
be interrupted.

Hi,

maybe I'm wrong, but I think that you cannot upload files greater than
1Gb with native browser fonctions.

Marnen Laibow-Koser wrote:

DK wrote:

Hi, I don't know if there are differing opinions on this, but seems to
me that apps that use such large files will write a file manager
(desktop based app or at least activex) that monitor or possibly
divide such files into smaller pieces.

Not necessarily Look at Dropbox (though they do use a Flash upload tool).

Dropbox is what he is describing. The start you off by having you install a filesystem extension.

Alan Gutierrez wrote:

Marnen Laibow-Koser wrote:

DK wrote:

Hi, I don't know if there are differing opinions on this, but seems to
me that apps that use such large files will write a file manager
(desktop based app or at least activex) that monitor or possibly
divide such files into smaller pieces.

Not necessarily Look at Dropbox (though they do use a Flash upload
tool).

Dropbox is what he is describing. The start you off by having you
install a filesystem extension.

But you don't have to. I exchange large files (hundreds of MB, not 4
GB) on Dropbox all the time, but I've never installed their desktop
software. It's only for convenience AFAIK.

--
Alan Gutierrez - alan@blogometer.com - http://twitter.com/bigeasy

Best,

Marnen Laibow-Koser wrote:

Alan Gutierrez wrote:

Marnen Laibow-Koser wrote:

DK wrote:

Hi, I don't know if there are differing opinions on this, but seems to
me that apps that use such large files will write a file manager
(desktop based app or at least activex) that monitor or possibly
divide such files into smaller pieces.

Not necessarily Look at Dropbox (though they do use a Flash upload tool).

Dropbox is what he is describing. The start you off by having you
install a filesystem extension.

But you don't have to. I exchange large files (hundreds of MB, not 4 GB) on Dropbox all the time, but I've never installed their desktop software. It's only for convenience AFAIK.

I think it is the other way around. Their getting started link says "Download". They want you to get the download software. The HTTP upload is for convenience. I'm offering this as an opinion.

The OP wants to upload more than 4GB and I wouldn't trust HTTP to transfer 4GB from a client to a server on a regular basis. There is no way to resume a failed upload. If the OP is building a web application, they are going to have to find an alternative. If they are building an application with a web front end, maybe they can use SFTP as a file transfer client for large files, and the web UI to manage the files once they have been uploaded.

Basically, Qin Qin, you are right in noting that HTTP upload is only good for reasonable sized uploads, whatever size reasonable is at the time that you read this. (Images, documents, etc.)

Alan Gutierrez wrote:

Marnen Laibow-Koser wrote:

install a filesystem extension.

But you don't have to. I exchange large files (hundreds of MB, not 4
GB) on Dropbox all the time, but I've never installed their desktop
software. It's only for convenience AFAIK.

I think it is the other way around. Their getting started link says
"Download". They want you to get the download software. The HTTP upload
is for convenience. I'm offering this as an opinion.

However you want to look at it, the point is that the HTTP upload option
(or whatever the Flash side of the tool does) exists.

The OP wants to upload more than 4GB and I wouldn't trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.

I agree with you on that.

If the OP is building a web application,
they are going to have to find an alternative. If they are building an
application with a web front end, maybe they can use SFTP as a file
transfer client for large files, and the web UI to manage the files once
they have been uploaded.

Basically, Qin Qin, you are right in noting that HTTP upload is only
good for reasonable sized uploads, whatever size reasonable is at the
time that you read this. (Images, documents, etc.)

Yup.

Best,

Marnen Laibow-Koser wrote:

Alan Gutierrez wrote:

Marnen Laibow-Koser wrote:

install a filesystem extension.

But you don't have to. I exchange large files (hundreds of MB, not 4
GB) on Dropbox all the time, but I've never installed their desktop
software. It's only for convenience AFAIK.

I think it is the other way around. Their getting started link says
"Download". They want you to get the download software. The HTTP upload
is for convenience. I'm offering this as an opinion.

However you want to look at it, the point is that the HTTP upload option
(or whatever the Flash side of the tool does) exists.

The OP wants to upload more than 4GB and I wouldn't trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.

I agree with you on that.

If the OP is building a web application,
they are going to have to find an alternative. If they are building an
application with a web front end, maybe they can use SFTP as a file
transfer client for large files, and the web UI to manage the files once
they have been uploaded.

Basically, Qin Qin, you are right in noting that HTTP upload is only
good for reasonable sized uploads, whatever size reasonable is at the
time that you read this. (Images, documents, etc.)

Yup.

Best,
--
Marnen Laibow-Koser
http://www.marnen.org
marnen@marnen.org

Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can't browse others
at the same time .
I met with that when I upload a large file the computer system halted
under ubuntu 10.04 LTS, it brings me a lot of trouble.
besides ,when I use mongrel ,the web server will create temp files
,which several times of original file size,in other word ,it needs more
disk or memory to complete it.
maybe I should have a try with nginx firstly ...

Qin Qin wrote:

Marnen Laibow-Koser wrote:

Alan Gutierrez wrote:

Marnen Laibow-Koser wrote:

However you want to look at it, the point is that the HTTP upload option (or whatever the Flash side of the tool does) exists.

The OP wants to upload more than 4GB and I wouldn't trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.

I agree with you on that.

Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can't browse others at the same time .

And then they think it has hung. And then they go away.

I met with that when I upload a large file the computer system halted under ubuntu 10.04 LTS, it brings me a lot of trouble.
besides ,when I use mongrel ,the web server will create temp files ,which several times of original file size,in other word ,it needs more disk or memory to complete it.
maybe I should have a try with nginx firstly ...

This isn't going to get any better for you. Are you expecting that by switching to nginx you're going to be able upload 4GB with nothing more than a screen flicker when the success page is returned?

Why not follow up on by looking at the Flash upload control at Dropbox that Marnen Laibow-Koser describes? See how fast it can upload 250MB, what it does if you interrupt the upload, etc.

FYI

Flash/Flex (AS3 only) now have a File IO Class to handle uploads and
downloads, I use it to upload images to member areas in my Rails App
with no problem, but the file sizes are no bigger than about 250K -
never tried anything larger, although you do get progress information
sent back to the event handler in flash so you can provide a progress
information, and also handle any errors during the upload/download.

paul h wrote:

Alan Gutierrez wrote:

Qin Qin wrote:

Marnen Laibow-Koser wrote:

Alan Gutierrez wrote:

Marnen Laibow-Koser wrote:

However you want to look at it, the point is that the HTTP upload option
(or whatever the Flash side of the tool does) exists.

The OP wants to upload more than 4GB and I wouldn't trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.

I agree with you on that.

Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can't browse others
at the same time .

And then they think it has hung. And then they go away.

I met with that when I upload a large file the computer system halted
under ubuntu 10.04 LTS, it brings me a lot of trouble.
besides ,when I use mongrel ,the web server will create temp files
,which several times of original file size,in other word ,it needs more
disk or memory to complete it.
maybe I should have a try with nginx firstly ...

This isn't going to get any better for you. Are you expecting that by
switching to nginx you're going to be able upload 4GB with nothing more
than a screen flicker when the success page is returned?

Why not follow up on by looking at the Flash upload control at Dropbox
that Marnen Laibow-Koser describes? See how fast it can upload 250MB,
what it does if you interrupt the upload, etc.

--
Alan Gutierrez - alan@blogometer.com - http://twitter.com/bigeasy

I found that the dropbox is just a Cross-platform client,and need to
connect a dropbox account. because of the network environment , we
didn't be allowed to do it. I download the software in other web
site,but I could't connect successfully. And I found that the total
capacity is 2G. So,it is still not a good idea for me.
dropbox is on behalf of network disks. there are so many productions
like this,but the capacity all is small.
So , I doublt that it is possible for browser to upload the very very
large file ? and I whether can to use a web browser to upload files via
FTP ?

Qin Qin wrote:

Alan Gutierrez wrote:

Qin Qin wrote:

Marnen Laibow-Koser wrote:

Alan Gutierrez wrote:

Marnen Laibow-Koser wrote:

However you want to look at it, the point is that the HTTP upload option (or whatever the Flash side of the tool does) exists.

The OP wants to upload more than 4GB and I wouldn't trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.

I agree with you on that.

Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can't browse others at the same time .

And then they think it has hung. And then they go away.

I met with that when I upload a large file the computer system halted under ubuntu 10.04 LTS, it brings me a lot of trouble.
besides ,when I use mongrel ,the web server will create temp files ,which several times of original file size,in other word ,it needs more disk or memory to complete it.
maybe I should have a try with nginx firstly ...

This isn't going to get any better for you. Are you expecting that by
switching to nginx you're going to be able upload 4GB with nothing more
than a screen flicker when the success page is returned?

Why not follow up on by looking at the Flash upload control at Dropbox
that Marnen Laibow-Koser describes? See how fast it can upload 250MB,
what it does if you interrupt the upload, etc.

I found that the dropbox is just a Cross-platform client,and need to connect a dropbox account. because of the network environment , we didn't be allowed to do it. I download the software in other web site,but I could't connect successfully. And I found that the total capacity is 2G. So,it is still not a good idea for me.
dropbox is on behalf of network disks. there are so many productions like this,but the capacity all is small.

So , I doublt that it is possible for browser to upload the very very large file ? and I whether can to use a web browser to upload files via FTP ?

Okay. Seems like a (human) language barrier. I appreciate your taking the time to investigate Dropbox as has people here have suggested. Er, thank you for looking at Dropbox like I asked.

We said look at Dropbox because Dropbox uses a Flash control for upload. You could use a Flash control for upload. Maybe. I'm not sure because I do not know Flash. Another poster said Flash has a File I/O library and a network library. So, maybe there is a Flash based upload control out there, or maybe you could write one.

Maybe this Flash control can upload 4 GB while showing a progress bar and time estimate. Maybe this Flash control can restart an upload that is interrupted.

You would have to write something on the server side that understood how to handle the restart. I believe that restart failed upload is important. Your user will be so unhappy if he accidentally closes a window at 3.9GB and has to start over.

Finally, upload of 4GB just using the HTML file form control is technically possible, and you could iframe it so that page is responsive, but the failed upload thing is going to be a bad user experience.

Qin Qin wrote:

Alan Gutierrez wrote:

Qin Qin wrote:

Marnen Laibow-Koser wrote:

Alan Gutierrez wrote:

Marnen Laibow-Koser wrote:

However you want to look at it, the point is that the HTTP upload option (or whatever the Flash side of the tool does) exists.

The OP wants to upload more than 4GB and I wouldn't trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.

I agree with you on that.

Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can't browse others at the same time .

And then they think it has hung. And then they go away.

I found that the dropbox is just a Cross-platform client,and need to connect a dropbox account. because of the network environment , we didn't be allowed to do it. I download the software in other web site,but I could't connect successfully. And I found that the total capacity is 2G. So,it is still not a good idea for me.
dropbox is on behalf of network disks. there are so many productions like this,but the capacity all is small.
So , I doublt that it is possible for browser to upload the very very large file ? and I whether can to use a web browser to upload files via FTP ?

So, just to be as clear as possible.

I suggested you look at Dropbox to see how they used Flash. Not because they would be a possible 3rd party library or add-on. Just see how they do a big upload. See if they offer restart of a failed upload. Have a look at how they solved the same problem.

Alan Gutierrez wrote:

Qin Qin wrote:

I agree with you on that.

Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can't browse others
at the same time .

And then they think it has hung. And then they go away.

I found that the dropbox is just a Cross-platform client,and need to
connect a dropbox account. because of the network environment , we
didn't be allowed to do it. I download the software in other web
site,but I could't connect successfully. And I found that the total
capacity is 2G. So,it is still not a good idea for me.
dropbox is on behalf of network disks. there are so many productions
like this,but the capacity all is small.
So , I doublt that it is possible for browser to upload the very very
large file ? and I whether can to use a web browser to upload files via
FTP ?

So, just to be as clear as possible.

I suggested you look at Dropbox to see how they used Flash. Not because
they would be a possible 3rd party library or add-on. Just see how they
do a big upload. See if they offer restart of a failed upload. Have a
look at how they solved the same problem.

--
Alan Gutierrez - alan@blogometer.com - http://twitter.com/bigeasy

Thank you very much for your kind help.I see your excellent proposal.
I will learn some knowledge about the realization principles of dropbox
concerning how to upload the large file and how to use Flash to achieve
the goal firstly.
thank you !

If what you mean by restart a failed download is to continue appending
data to the local file, continuing from where the fail happened, then
AFAIK the answer would be no.

But you can handle the errors and provide feedback to the user. It may
also be possible to automatically restart the download after a
failure, but I would have to check that out. In order to begin an
upload or download, the FlashPlayer security requires user activation
by either selecting the file to upload from the local file system or
the location and file name to save a download to on the local file
system - however, once the reference to the files (local or remote)
has been obtained, it may be possible to begin it again after a
failure, and just notify the user of a major error if it fails x
number of times consecutively.

I forgot before that I also use Flash to download .mov files (private
site I set up for my brother to distribute movies of his new baby girl
to the rest of the family) the largest so far is 93MB and it downloads
fine - dunno about upload though, because I wrote an FTP client for
him to do the uploads.

I'll look into the above after the weekend (busy diving), and post the
results to clarify, I'm pretty new to this group, and had a little
help from it so far, so would gladly give something back when I can :slight_smile:

The Rails side is simple, and the Flash side isn't tough - examples
follow:

Rails

Controller

  def upload
    ModelClass.new.save_image( params )
  end

Model

  def save_image params
    data = params[:Filedata]
    name = params[:Filename]
    path = File.join( "public" , "users" , params[:id] ,
"images" , name)

    # params[:Filedata] and params[:Filename] are sent automatically
from Flash
    # params[:id] is appended to the url: www.mysite.com/controller_name/method_name_in_model/id

    File.open(path,'wb') do |file|
      file.puts data.read
    end
  end

Flash/Flex

The place to look for anyone interested would be at the FileReference
class.

Create a FileReference instance and call the browse() method with a
FileFilter if necessary to filter to specific file types.

Once the user has selected a file to upload (or a file location and
name if downloading) the select Event is triggered, and you can then
send the file to the Rails Controller.

Here's some simple AS3 code:

private var _fileReference:FileReference = new FileReference();
private var _imageFilter:FileFilter = new FileFilter("Images (*.jpg,
*.jpeg, *.gif, *.png)", "*.jpg;*.jpeg;*.gif;*.png");
private var _uploadURL:String = "http://www.mysite.com/controller/
model_method/id_if_required";

private function upload():void
{
    _fileReference.addEventListener(Event.SELECT, select); // Add
additional event handlers to handle progress, security errors, IO
errors and of course file complete etc
    _fileReference.browse([_imageFilter]); // You can add as many
different filters as required within the array parameter
}

private function select( event:Event ):void
{
    _fileReference = FileReference(event.target);
   _fileReference.upload(new URLRequest(_uploadURL));
}

HTH

Paul