How to upload gziped file to S3 from URL[Ruby]

This article shows how to upload gziped file to S3 from URL by Ruby.


Gemfile

Please add RubyGem aws/aws-sdk-ruby and bkeepers/dotenv to Gemfile:

1
2
3
4
5
# Loads environment variables from `.env`
gem 'dotenv'
# AWS SDK for Ruby
gem 'aws-sdk'

Environment Variables

Please set environment variables using .env or etc.

1
2
3
4
AMAZON_AWS_REGION="AWS Reagion"
AMAZON_AWS_ACCESS_KEY_ID="AWS Access Key Id"
AMAZON_AWS_SECRET_ACCESS_KEY="AWS Secret Access Key"
AMAZON_AWS_S3_BUCKET_NAME="AWS S3 Bucket name"

Code

The following code execute like them:

  • Get a file from argment URL
  • Gzip the file and create a tmp file
  • Upload the gziped file to S3 with public_read access control
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
module S3Utility
class << self
# @return [String] S3 URL
def upload_by_url(from_url:, object_key:)
content = open(from_url).read
fail "fail to get url: #{from_url}" if content.blank?
temp = Tempfile.new.tap do |t|
t.binmode
t.write(_gzip(content))
t.close
end
obj = _bucket.object(object_key)
result = obj.upload_file(temp.path, acl: 'public-read', content_encoding: 'gzip')
fail "fail to upload url: #{from_url}, object_key: #{object_key}" unless result
_url(object_key)
rescue => e
Airbrake.notify(e)
nil
end
private
def _url(object_key)
"https://#{ENV['AMAZON_AWS_S3_BUCKET_NAME']}.s3-ap-northeast-1.amazonaws.com/#{object_key}"
end
def _bucket
@bucket ||= Aws::S3::Resource.new(
region: ENV['AMAZON_AWS_REGION'],
access_key_id: ENV['AMAZON_AWS_ACCESS_KEY_ID'],
secret_access_key: ENV['AMAZON_AWS_SECRET_ACCESS_KEY'],
).bucket(ENV['AMAZON_AWS_S3_BUCKET_NAME'])
end
def _gzip(data)
sio = StringIO.new
gz = Zlib::GzipWriter.new(sio)
gz.write(data)
gz.close
sio.string
end
end
end
url = 'http://example.com/hoge.jpg'
object_key = '/images/hoge.jpg'
S3Utility.upload_by(from_url: url, object_key: object_key)
#=> https://xxx.s3-ap-northeast-1.amazonaws.com/images/hoge.jpg

Happy Hacking!

Special Thanks