tseki blog

my study room

AWS Lambda のコンテナイメージ対応を試してみた

先日、AWS Lambda でコンテナイメージがサポートされました。

aws.amazon.com

そこで、コンテナイメージを使用した Lambda の作成と実行を試してみます。

実行環境は以下の通りです。

$ aws --version
aws-cli/2.1.28 Python/3.8.8 Linux/4.19.104-microsoft-standard exe/x86_64.ubuntu.20 prompt/off
$ sam --version
SAM CLI, version 1.19.1

作成

まずは、SAM CLI を使ってサンプルアプリケーション(python 3.8)を作成します。

$ sam init
Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice: 1
What package type would you like to use?
        1 - Zip (artifact is a zip uploaded to S3)
        2 - Image (artifact is an image uploaded to an ECR image repository)
Package type: 2

Which base image would you like to use?
        1 - amazon/nodejs14.x-base
        2 - amazon/nodejs12.x-base
        3 - amazon/nodejs10.x-base
        4 - amazon/python3.8-base
        5 - amazon/python3.7-base
        6 - amazon/python3.6-base
        7 - amazon/python2.7-base
        8 - amazon/ruby2.7-base
        9 - amazon/ruby2.5-base
        10 - amazon/go1.x-base
        11 - amazon/java11-base
        12 - amazon/java8.al2-base
        13 - amazon/java8-base
        14 - amazon/dotnet5.0-base
        15 - amazon/dotnetcore3.1-base
        16 - amazon/dotnetcore2.1-base
Base image: 4

Project name [sam-app]: sample-app

Cloning app templates from https://github.com/aws/aws-sam-cli-app-templates

    -----------------------
    Generating application:
    -----------------------
    Name: sample-app
    Base Image: amazon/python3.8-base
    Dependency Manager: pip
    Output Directory: .

    Next steps can be found in the README file at ./sample-app/README.md

生成されたファイルを眺めてみます。

...

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      PackageType: Image
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./hello_world
      DockerTag: python3.8-v1

Outputs:
  ...
FROM public.ecr.aws/lambda/python:3.8

COPY app.py requirements.txt ./

RUN python3.8 -m pip install -r requirements.txt -t .

# Command can be overwritten by providing a different command in the template directly.
CMD ["app.lambda_handler"]

template.yaml には Metadata を使用して Dockerfile をビルドする Context が設定されています。
また、これまで存在した Handler プロパティが無くなっており、
Dockerfile の CMD 命令で指定するようになっています。*1

ビルド

./README.md の手順通りに build します。

$ sam build
Building codeuri: . runtime: None metadata: {'Dockerfile': 'Dockerfile', 'DockerContext': './hello_world', 'DockerTag': 'python3.8-v1'} functions: ['HelloWorldFunction']
Building image for HelloWorldFunction function
Setting DockerBuildArgs: {} for HelloWorldFunction function
Step 1/4 : FROM public.ecr.aws/lambda/python:3.8
 ---> 96d8701bf5d1
Step 2/4 : COPY app.py requirements.txt ./
 ---> 029a59377f95
Step 3/4 : RUN python3.8 -m pip install -r requirements.txt -t .
 ---> Running in fa7c16bc7a59
Collecting requests
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Installing collected packages: idna, chardet, urllib3, certifi, requests
Successfully installed certifi-2020.12.5 chardet-4.0.0 idna-2.10 requests-2.25.1 urllib3-1.26.3
WARNING: You are using pip version 20.2.1; however, version 21.0.1 is available.
You should consider upgrading via the '/var/lang/bin/python3.8 -m pip install --upgrade pip' command.
 ---> d7e0e2543084
Step 4/4 : CMD ["app.lambda_handler"]
 ---> Running in 1c0839958a6d
 ---> 9790df443d64
Successfully built 9790df443d64
Successfully tagged helloworldfunction:python3.8-v1

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

$ docker image ls | grep helloworldfunction
helloworldfunction                                                python3.8-v1                                            9790df443d64   48 seconds ago   599MB

コンテナイメージが出来ました。

.aws-sam/build ディレクトリには以下のような template.yaml と build.toml が作成されました。

...
Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get
      ImageUri: helloworldfunction:python3.8-v1
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./hello_world
      DockerTag: python3.8-v1
...
# This file is auto generated by SAM CLI build command

[function_build_definitions]
[function_build_definitions.c7334742-ebfc-41b8-8278-371e13dc3e38]
packagetype = "Image"
functions = ["HelloWorldFunction"]

[function_build_definitions.c7334742-ebfc-41b8-8278-371e13dc3e38.metadata]
Dockerfile = "Dockerfile"
DockerContext = "./hello_world"
DockerTag = "python3.8-v1"

[layer_build_definitions]

デプロイ

はじめに、コンテナレジストリを作成します。

$ aws ecr create-repository --repository-name sample-app
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:ap-northeast-1:123456789012:repository/sample-app",
        "registryId": "123456789012",
        "repositoryName": "sample-app",
        "repositoryUri": "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/sample-app",
        "createdAt": "2021-02-28T19:27:57+09:00",
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        },
        "encryptionConfiguration": {
            "encryptionType": "AES256"
        }
    }
}

続いて、deploy を行います。

$ sam deploy --guided

Configuring SAM deploy
======================

        Looking for config file [samconfig.toml] :  Not found

        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [sam-app]:
        AWS Region [ap-northeast-1]:
        Image Repository for HelloWorldFunction: 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/sample-app
          helloworldfunction:python3.8-v1 to be pushed to 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/sample-app:helloworldfunction-9790df443d64-python3.8-v1

        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [y/N]:
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]:
        HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
        Save arguments to configuration file [Y/n]:
        SAM configuration file [samconfig.toml]:
        SAM configuration environment [default]:

        Looking for resources needed for deployment: Found!

                Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-31fpkcxpfv0e
                A different default S3 bucket can be set in samconfig.toml

        Saved arguments to config file
        Running 'sam deploy' for future deployments will use the parameters saved above.
        The above parameters can be changed by modifying samconfig.toml
        Learn more about samconfig.toml syntax at
        https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

The push refers to repository [123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/sample-app]
234939dcf430: Pushed
5394ef41e052: Pushed
20b4eff3dd4d: Pushed
11284767d41d: Pushed
d6fa53d6caa6: Pushed
b09e76f63d5d: Pushed
0acabcf564c7: Pushed
f2342b1247df: Pushed
helloworldfunction-9790df443d64-python3.8-v1: digest: sha256:68dc32475a5705c6593497ff35b59a2099b3218956c6307b9123f6373d91fb2b size: 1999


        Deploying with following values
        ===============================
        Stack name                   : sam-app
        Region                       : ap-northeast-1
        Confirm changeset            : False
        Deployment image repository  :
                                       {
                                           "HelloWorldFunction": "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/sample-app"
                                       }
        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-31fpkcxpfv0e
        Capabilities                 : ["CAPABILITY_IAM"]
        Parameter overrides          : {}
        Signing Profiles             : {}

Initiating deployment
=====================
HelloWorldFunction may not have authorization defined.
Uploading to sam-app/0349ea3b59cff300265d848a1ec488a2.template  1169 / 1169  (100.00%)

Waiting for changeset to be created..

CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                          LogicalResourceId                                  ResourceType                                       Replacement
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
* Modify                                           HelloWorldFunction                                 AWS::Lambda::Function                              False
* Modify                                           ServerlessRestApi                                  AWS::ApiGateway::RestApi                           False
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:ap-northeast-1:123456789012:changeSet/samcli-deploy1614508262/9d2ba819-8fbe-4635-94f8-b7c7ed65d116


2021-02-28 19:31:13 - Waiting for stack create/update to complete

CloudFormation events from changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                     ResourceType                                       LogicalResourceId                                  ResourceStatusReason
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UPDATE_IN_PROGRESS                                 AWS::Lambda::Function                              HelloWorldFunction                                 -
UPDATE_COMPLETE                                    AWS::Lambda::Function                              HelloWorldFunction                                 -
UPDATE_COMPLETE_CLEANUP_IN_PROGRESS                AWS::CloudFormation::Stack                         sam-app                                            -
UPDATE_COMPLETE                                    AWS::CloudFormation::Stack                         sam-app                                            -
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 HelloWorldFunctionIamRole
Description         Implicit IAM Role created for Hello World function
Value               arn:aws:iam::123456789012:role/sam-app-HelloWorldFunctionRole-4038SRBBOKKK

Key                 HelloWorldApi
Description         API Gateway endpoint URL for Prod stage for Hello World function
Value               https://fpo1jvgd1d.execute-api.ap-northeast-1.amazonaws.com/Prod/hello/

Key                 HelloWorldFunction
Description         Hello World Lambda Function ARN
Value               arn:aws:lambda:ap-northeast-1:123456789012:function:sam-app-HelloWorldFunction-3WPICAKIXHD7
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - sam-app in ap-northeast-1

出力された samconfig.toml に関数にリポジトリが保存されています。

version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "sam-app"
s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-31fpkcxpfv0e"
s3_prefix = "sam-app"
region = "ap-northeast-1"
capabilities = "CAPABILITY_IAM"
image_repositories = ["HelloWorldFunction=123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/sample-app"]

最終的にデプロイされたCFnテンプレートには、
リポジトリ名を含めた ImageUrl が生成されていました。

...
Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get
      ImageUri: 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/sample-app:helloworldfunction-9790df443d64-python3.8-v1
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./hello_world
      DockerTag: python3.8-v1
...

テスト

$ curl -X GET https://fpo1jvgd1d.execute-api.ap-northeast-1.amazonaws.com/Prod/hello/
{"message": "hello world"}

API Gateway からレスポンスが返ることが確認できました。

ローカルテスト

続いて、ローカルでのテストを確認します。
./README.md に書いてある手順でローカルテストを実行します。

$ sam local invoke HelloWorldFunction --event events/event.json
Invoking Container created from helloworldfunction:python3.8-v1
Image was not found.
Building image..........
Skip pulling image and use local one: helloworldfunction:rapid-1.19.1.

START RequestId: f4b3f8d8-fbda-48d7-8ee8-fba315b874aa Version: $LATEST
END RequestId: f4b3f8d8-fbda-48d7-8ee8-fba315b874aa
REPORT RequestId: f4b3f8d8-fbda-48d7-8ee8-fba315b874aa  Init Duration: 0.10 ms  Duration: 54.73 ms      Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 128 MB
{"statusCode": 200, "body": "{\"message\": \"hello world\"}"}

helloworldfunction:rapid-1.19.1 というイメージが作成され、実行されていました。

$ docker image inspect helloworldfunction:python3.8-v1 > /tmp/python3.8-v1
$ docker image inspect helloworldfunction:rapid-1.19.1 > /tmp/rapid-1.19.1
$ diff /tmp/python3.8-v1 /tmp/rapid-1.19.1
...
5,6c5
<             "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/sample-app:helloworldfunction-9790df443d64-python3.8-v1",
<             "helloworldfunction:python3.8-v1"
---
>             "helloworldfunction:rapid-1.19.1"
...
36,37c33
<                 "#(nop) ",
<                 "CMD [\"app.lambda_handler\"]"
---
>                 "chmod +x /var/rapid/aws-lambda-rie"
...
42,44c38
<             "Entrypoint": [
<                 "/lambda-entrypoint.sh"
<             ],
---
>             "Entrypoint": null,
...
103c97,99
<                 "sha256:234939dcf430ad1d583a8e5fc9a6a8b6e9bd47d7522e68529e7dc6a3a713b08c"
---
>                 "sha256:234939dcf430ad1d583a8e5fc9a6a8b6e9bd47d7522e68529e7dc6a3a713b08c",
>                 "sha256:29131f8b9548941337f7a3f55f91c42c8620f370aa454a3073f08ff1d8320a31",
>                 "sha256:29131f8b9548941337f7a3f55f91c42c8620f370aa454a3073f08ff1d8320a31"
...

rapid-1.19.1 のコンテナイメージは、
python3.8-v1 の EntrypointCMD を変更して作成されているようです。
ローカル実行の仕組みは Runtime Interface Emulator (RIE) として実装されているとのこと。

github.com

使用しているAWS提供のベースイメージには既に RIE が組み込まれているとのことなので、
README の手順でビルドしたイメージを実行してみました。

$ docker run -p 9000:8080 helloworldfunction:python3.8-v1
time="2021-02-28T11:36:25.564" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)"
time="2021-02-28T11:36:36.87" level=info msg="extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory"
time="2021-02-28T11:36:36.87" level=warning msg="Cannot list external agents" error="open /opt/extensions: no such file or directory"
START RequestId: 7ef765f5-d0df-4c12-8235-3f7c70de5056 Version: $LATEST
END RequestId: 7ef765f5-d0df-4c12-8235-3f7c70de5056
REPORT RequestId: 7ef765f5-d0df-4c12-8235-3f7c70de5056  Init Duration: 0.12 ms  Duration: 52.12 ms      Billed Duration: 100 ms Memory Size: 3008 MB    Max Memory Used: 3008 MB
^Ctime="2021-02-28T11:37:51.112" level=info msg="Received signal" signal=interrupt
time="2021-02-28T11:37:51.112" level=info msg="Shutting down..."
time="2021-02-28T11:37:51.112" level=warning msg="Reset initiated: SandboxTerminated"
$ curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
{"statusCode": 200, "body": "{\"message\": \"hello world\"}"}

ローカルで実行したコンテナからレスポンスが返ることが確認できました。
8080 番ポートでListenしてるんですね。

もう一つ、ローカルでの REST API のテストを試します。

$ sam local start-api
Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2021-02-28 20:40:58  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Invoking Container created from helloworldfunction:python3.8-v1
Building image.........
Skip pulling image and use local one: helloworldfunction:rapid-1.19.1.

START RequestId: e67691d8-e55e-408a-96e8-253fdbe760cd Version: $LATEST
END RequestId: e67691d8-e55e-408a-96e8-253fdbe760cd
REPORT RequestId: e67691d8-e55e-408a-96e8-253fdbe760cd  Init Duration: 0.11 ms  Duration: 54.94 ms      Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 128 MB
No Content-Type given. Defaulting to 'application/json'.
2021-02-28 20:41:33 127.0.0.1 - - [28/Feb/2021 20:41:33] "GET /hello/ HTTP/1.1" 200 -
$ curl -XGET "http://localhost:3000/hello/" -d '{}'
{"message": "hello world"}

ローカル実行した REST API からもレスポンスが返ることが確認できました。
API からリクエストがある度にコンテナが起動されています。

sam local start-apiAPIを起動した状態でコードを変更した場合は、
sam build でローカルのイメージを更新することで、
新しいイメージでのリクエストを確認することが出来ます。

まとめ

コンテナ化することで成果物が扱いやすくなり、
バージョン管理もしやすそうに感じました。
ローカルでのテストの手順も簡略化されそうですね。

*1:ImageConfig プロパティで CMD 命令などをオーバーライドできるようにもなっているようです。 AWS::Serverless::Function - AWS Serverless Application Model