Serverless framework for Node.js/Python with Docker


Serverless is toolkit for deploying and operating serverless framework due to focus your application.
In this article, I would like to introduce how to build basic lambda API by Node.js/Python with Docker.

🚌 Installation

yarn global add serverless

# Create project folder (e.g. serverless-sample)
mkdir serverless-sample
cd serverless-sample

😀 Docker Config for Python

If you want to develop a function with Python, it is better to use Dockerfile, because it is easy to use pip library in lambda.

Create Dockerfile for development:

FROM python:3.6

# update apt-get
RUN apt-get update -y && apt-get upgrade -y

# Install Nodejs 8
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get install -y nodejs

# install dev tool
RUN apt-get install -y vim

# install aws-cli
RUN pip install awscli

# install serverless framework
RUN npm install -g serverless

# change work directory
RUN mkdir -p /app
WORKDIR /app

Also, create docker-compose.yml:

version: '3.5'
services:
  serverless:
    build: .
    tty: true
    stdin_open: true
    image: serverless
    working_dir: /app
    volumes:
      - .:/app
    container_name: serverless
    env_file:
      - .env.docker
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

After then, create .env.docker:

aws_access_key_id=YOUR_ACCESS_KEY_ID
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

Please add your AWS Access information and do not commit the .env.docker file in git repository.

🏀 Start the image for dev

docker-compose up -d
docker-compose exec serverless /bin/bash

🐠 Set AWS Access key

sls config credentials --provider aws --key $AWS_ACCESS_KEY_ID --secret $AWS_SECRET_ACCESS_KEY

🐡 Create lambda function with Pytnon3 template

Create new service:

# Create service with Python 3.x
sls create -t aws-python3 -p py3test

Modify serverless.yml configuration file:
(diff --no-index --unified=1 serverless.org.yml serverless.yml):

@@ -25,3 +25,3 @@ provider:
 #  stage: dev
-#  region: us-east-1
+  region: ap-northeast-1

@@ -63,6 +63,6 @@ functions:
 #    Check the event documentation for details
-#    events:
-#      - http:
-#          path: users/create
-#          method: get
+    events:
+      - http:
+          path: users/create
+          method: get
 #      - s3: ${env:BUCKET}

🐯 Confirm developing function

Enable to confirm your devloping function on local

# Call the function
sls invoke local -f hello

🗻 Deploy code to AWS

After development, you can deploy your code by one command:

sls deploy -v

🎂 Confirm the deployed function

Enable to confirm your devloping function on local or you can call your API by curl:

# Call the function
sls invoke -f hello

# Call the function's API by curl
curl https://xxx.execute-api.ap-northeast-1.amazonaws.com/dev/users/create

🐞 Add pip libraries for lambda

If you want to use library, please install pip libraries as follows:

# In docker
pip install requests -t /app/lib/requests

🎃 Remove deployed function

sls remove

🍮 Configuration (serverless.yml)

Memory Size, Timeout, etc in AWS lambda

You can configure memory size and timeout for function in AWS lambda:

# https://serverless.com/framework/docs/providers/aws/guide/functions/
provider:
  name: aws
  runtime: python3.6
  memorySize: 128 # optional, in MB, default is 1024
  timeout: 10 # optional, in seconds, default is 6

If you want to know more detail, please see https://serverless.com/framework/docs/providers/aws/guide/functions/ .

Including/excluding file or folder

Please configure including/excluding a file or a folder.

# you can add packaging information here
# https://serverless.com/framework/docs/providers/aws/guide/packaging/
package:
  exclude: # excluding file or folder
    - src/**
  include: # including file or folder
    - src/function/handler.js

If you want to know more detail, please see https://serverless.com/framework/docs/providers/aws/guide/packaging/

Cron(Scheduled job) configuration

Enable to configure cron/scheduled job.

# Cron (scheduled job) information
functions:
  crawl:
    handler: crawl
    events:
      - schedule: rate(2 hours)
      - schedule: cron(0 12 * * ? *)

If you want to know more detail, please see https://serverless.com/framework/docs/providers/aws/events/schedule/

😼 Impression

My impression for serverless is as follows:

  • A deploy command (serverless deploy) is simple and useful to create the required IAM, set up CloudWatch logs, set up S3 for deployment, deploy, etc.
  • Cron (CloudWatch Events) needs just a line to YAML, there are a lot of useful configurations.
  • I thought that it was easy for using Google Cloud Function to manage credential on GCP, I tried a little, but it was buggy. I think that serverless is useful in only AWS lambda.
    • There are some problems in GCF and serverless. Also, GCF is beta yet.
  • I found https://github.com/serverless/serverless-graphql . If it becomes better, it is a good option for GraphQL server.

🐮 [Appendix] Google Cloud function with Node.js

Supported Node.js version

Node version is 6.14.0, so please be careful: https://cloud.google.com/functions/docs/writing/ .

echo '6.14.0' > .nvmrc

Quick Start

Create a project:

serverless create --template google-nodejs --path gcf-nodejs

Install npm libraries:

cd gcf-nodejs
npm install

Create your credential

https://serverless.com/framework/docs/providers/google/guide/credentials

Deploy your code

serverless deploy

Confirmation

# Confirm your code by function
serverless invoke --function first

# Confirm your code by API
curl https://us-central1-price-rank-production.cloudfunctions.net/http

More detail

If you want to know more detail, please see https://serverless.com/framework/docs/providers/google/guide/quick-start/ .

Concerns

  • SLA is not guaranteed because it is in beta test.
  • It is impossible to configure environment variable like AWS.
  • There is no mechanism like cron, scheduled job.
  • It is thought that the test with GCP is not done much.

🎉 Appendix

Manage permission for your Lambda function

If your function read/edit data in DynamoDB table, you can add these additional permission statements directly in your serverless.yml:

provider:
  name: aws
  runtime: nodejs6.10
  iamRoleStatements:
    - Effect: "Allow"
      Action:
       - dynamodb:Query
       - dynamodb:Scan
       - dynamodb:GetItem
       - dynamodb:PutItem
       - dynamodb:UpdateItem
       - dynamodb:DeleteItem
      Resource: "arn:aws:dynamodb:us-west-2:111110002222:table/my-new-table"

If you want to know more details, please see https://serverless.com/blog/abcs-of-iam-permissions/ .

Python / Load and delete a credential

Load and delete a credential data from environment variable in AWS lambda:

def load_credential():
    credential_data = os.environ['CREDENTIAL_DATA']
    if len(credential_data) <= 1:
        raise RuntimeError('No CREDENTIAL_DATA')

    # In AWS lambda, you can write only /tmp folder
    credential_file = os.environ['APPLICATION_CREDENTIALS']
    with open(credential_file, 'w', encoding='utf8') as f:
        print('Credential file {} created.'.format(credential_file))
        f.write(credential_data)


def delete_credential():
    credential_file = os.environ['APPLICATION_CREDENTIALS']
    try:
        os.remove(credential_file)
    except OSError:
        pass

🍄 References

🖥 Recommended VPS Service

VULTR provides high performance cloud compute environment for you. Vultr has 15 data-centers strategically placed around the globe, you can use a VPS with 512 MB memory for just $ 2.5 / month ($ 0.004 / hour). In addition, Vultr is up to 4 times faster than the competition, so please check it => Check Benchmark Results!!