Transferring files from S3 to Box in chunks without a tmp file

rubyonrails-talk
1

Hi,

I’m using the aws-s3-ruby and boxr gems to transfer large video files from S3 to Box. Both gems allow you to download and upload files in chunks and that’s easy when you’re reading or writing files from disk. However I can’t figure out how to transfer data from S3 to Box in chunks with a StringIO object.

When I call Boxr::Client#upload_file_from_io with a StringIO object from Aws::S3::Client#get_object it results in an exception:

RuntimeError: TypeError no implicit conversion of StringIO into String

filename = "video.mp4"
root_folder_id = "0"

s3_response = s3_client.get_object(bucket: video_bucket, key: filename)

box_client.upload_file_from_io(s3_response.body, root_folder_id, filename)

Do I need to convert the StringIO into a regular IO object in order for it to work with Boxr?

Links:

2

Thanks to https://github.com/cburnette/boxr/pull/106 I saw that using preflight_check with StringIO breaks the boxr io methods. Disabling preflight_check and send_content_md5 made the upload work.

box_file = box_client.upload_file_from_io(                                                        
  s3_response.body,
  root_folder_id,
  name: "#{filename}-#{SecureRandom.hex}.mp4",
  preflight_check: false,
  send_content_md5: false
)
3

Hey! I’m glad you found a solution. I’m the author of the first PR you listed and your post actually made me realize that #chunked_upload_to_session_from_io has the same issue when attempting to upload multiple parts concurrently due to https://github.com/cburnette/boxr/blob/master/lib/boxr/chunked_uploads.rb#L174

I’ll be opening a PR :slight_smile:.

EDIT: https://github.com/cburnette/boxr/pull/110