How to upload gziped file to S3 from URL[Ruby]

This article shows how to upload gziped file to S3 from URL by Ruby.

🐞 Gemfile

Please add RubyGem aws/aws-sdk-ruby and bkeepers/dotenv to Gemfile:

# Loads environment variables from `.env`
gem 'dotenv'

# AWS SDK for Ruby
gem 'aws-sdk'

🎳 Environment Variables

Please set environment variables using .env or etc.

AMAZON_AWS_REGION="AWS Reagion"
AMAZON_AWS_ACCESS_KEY_ID="AWS Access Key Id"
AMAZON_AWS_SECRET_ACCESS_KEY="AWS Secret Access Key"
AMAZON_AWS_S3_BUCKET_NAME="AWS S3 Bucket name"

🚕 Code

The following code execute like them:

• Get a file from argment URL
• Gzip the file and create a tmp file
• Upload the gziped file to S3 with public_read access control

module S3Utility
  class << self
    # @return [String] S3 URL
    def upload_by_url(from_url:, object_key:)
      content = open(from_url).read
      fail "fail to get url: #{from_url}" if content.blank?

      temp = Tempfile.new.tap do |t|
        t.binmode
        t.write(_gzip(content))
        t.close
      end

      obj = _bucket.object(object_key)
      result = obj.upload_file(temp.path, acl: 'public-read', content_encoding: 'gzip')
      fail "fail to upload url: #{from_url}, object_key: #{object_key}" unless result
      _url(object_key)
    rescue => e
      Airbrake.notify(e)
      nil
    end

    private

    def _url(object_key)
      "https://#{ENV['AMAZON_AWS_S3_BUCKET_NAME']}.s3-ap-northeast-1.amazonaws.com/#{object_key}"
    end

    def _bucket
      @bucket ||= Aws::S3::Resource.new(
          region:            ENV['AMAZON_AWS_REGION'],
          access_key_id:     ENV['AMAZON_AWS_ACCESS_KEY_ID'],
          secret_access_key: ENV['AMAZON_AWS_SECRET_ACCESS_KEY'],
      ).bucket(ENV['AMAZON_AWS_S3_BUCKET_NAME'])
    end

    def _gzip(data)
      sio = StringIO.new
      gz = Zlib::GzipWriter.new(sio)
      gz.write(data)
      gz.close
      sio.string
    end
  end
end

url = 'http://example.com/hoge.jpg'
object_key = '/images/hoge.jpg'
S3Utility.upload_by(from_url: url, object_key: object_key)
#=> https://xxx.s3-ap-northeast-1.amazonaws.com/images/hoge.jpg

Happy Hacking!

🤔 Special Thanks

• http://akasata.com/articles/292
• http://kinopyo.com/ja/blog/upload-file-to-s3-with-aws-sdk-v2
• http://yamayo.github.io/blog/2013/12/16/aws-sdk-for-ruby-to-gzip/

🖥 Recommended VPS Service

VULTR provides high performance cloud compute environment for you. Vultr has 15 data-centers strategically placed around the globe, you can use a VPS with 512 MB memory for just $ 2.5 / month ($ 0.004 / hour). In addition, Vultr is up to 4 times faster than the competition, so please check it => Check Benchmark Results!!