How to upload gziped file to S3 from URL[Ruby]


This article shows how to upload gziped file to S3 from URL by Ruby.


🍄 Gemfile

Please add RubyGem aws/aws-sdk-ruby and bkeepers/dotenv to Gemfile:

# Loads environment variables from `.env`
gem 'dotenv'

# AWS SDK for Ruby
gem 'aws-sdk'

🐯 Environment Variables

Please set environment variables using .env or etc.

AMAZON_AWS_REGION="AWS Reagion"
AMAZON_AWS_ACCESS_KEY_ID="AWS Access Key Id"
AMAZON_AWS_SECRET_ACCESS_KEY="AWS Secret Access Key"
AMAZON_AWS_S3_BUCKET_NAME="AWS S3 Bucket name"

😀 Code

The following code execute like them:

  • Get a file from argment URL
  • Gzip the file and create a tmp file
  • Upload the gziped file to S3 with public_read access control
module S3Utility
class << self
# @return [String] S3 URL
def upload_by_url(from_url:, object_key:)
content = open(from_url).read
fail "fail to get url: #{from_url}" if content.blank?

temp = Tempfile.new.tap do |t|
t.binmode
t.write(_gzip(content))
t.close
end

obj = _bucket.object(object_key)
result = obj.upload_file(temp.path, acl: 'public-read', content_encoding: 'gzip')
fail "fail to upload url: #{from_url}, object_key: #{object_key}" unless result
_url(object_key)
rescue => e
Airbrake.notify(e)
nil
end

private

def _url(object_key)
"https://#{ENV['AMAZON_AWS_S3_BUCKET_NAME']}.s3-ap-northeast-1.amazonaws.com/#{object_key}"
end

def _bucket
@bucket ||= Aws::S3::Resource.new(
region: ENV['AMAZON_AWS_REGION'],
access_key_id: ENV['AMAZON_AWS_ACCESS_KEY_ID'],
secret_access_key: ENV['AMAZON_AWS_SECRET_ACCESS_KEY'],
).bucket(ENV['AMAZON_AWS_S3_BUCKET_NAME'])
end

def _gzip(data)
sio = StringIO.new
gz = Zlib::GzipWriter.new(sio)
gz.write(data)
gz.close
sio.string
end
end
end

url = 'http://example.com/hoge.jpg'
object_key = '/images/hoge.jpg'
S3Utility.upload_by(from_url: url, object_key: object_key)
#=> https://xxx.s3-ap-northeast-1.amazonaws.com/images/hoge.jpg

Happy Hacking!

🎂 Special Thanks

🖥 Recommended VPS Service

VULTR provides high performance cloud compute environment for you. Vultr has 15 data-centers strategically placed around the globe, you can use a VPS with 512 MB memory for just $ 2.5 / month ($ 0.004 / hour). In addition, Vultr is up to 4 times faster than the competition, so please check it => Check Benchmark Results!!