Skip to content

Commit

Permalink
Adjust FaaS docs for layout = "zip" (#19180)
Browse files Browse the repository at this point in the history 
This adjusts the AWS Lambda and Google Cloud Function documentation for
the new Zip layout, added in #19076 and targeted for 2.17.

This PR is just what's required for 2.17, ready to cherry-pick. The
"Migrating" section is written with this in mind. It will require
adjustment for 2.18 to reflect the change in defaults, and, hopefully,
support for AWS Lambda Layers (#18880, #19123).

Fixes #19067
  • Loading branch information
@huonw
huonw authored May 30, 2023
1 parent ee3be4d commit 9089aea
Show file tree
Hide file tree
Showing 3 changed files with 152 additions and 25 deletions.
85 changes: 73 additions & 12 deletions docs/markdown/Python/python-integrations/awslambda-python.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ createdAt: "2020-05-05T16:51:03.851Z"
Pants can create a Lambda-compatible zip file from your Python code, allowing you to develop your Lambdas in your repository instead of using the online Cloud9 editor.

> 📘 FYI: how Pants does this
>
> Under-the-hood, Pants uses the [Lambdex](https://github.com/pantsbuild/lambdex) project. First, Pants will convert your code into a [Pex file](doc:pex-files). Then, Pants will use Lambdex to convert the Pex into a zip file understood by AWS.
>
> Under-the-hood, Pants uses the [PEX](https://github.com/pantsbuild/pex) project, to select the appropriate third-party requirements and first-party sources and lay them out in a zip file, in the format recommended by AWS.
Step 1: Activate the Python AWS Lambda backend
----------------------------------------------
Expand All @@ -26,6 +26,17 @@ backend_packages.add = [

This adds the new `python_awslambda` target, which you can confirm by running `pants help python_awslambda`

> 🚧 Set `layout = "zip"` for Pants 2.17
>
> Pants 2.17 is transitioning to a new, better layout, but defaults to the old Lambdex layout for backwards compatibility. To silence the warnings and be ready for Pants 2.18, add the following to the end of your `pants.toml`:
>
> ```toml pants.toml
> [lambdex]
> layout = "zip"
> ```
>
> If you have existing `python_awslambda` targets, this will change the handler from `lambdex_handler.handler` to `lambda_function.handler` (see [below](#migrating-from-pants-216-and-earlier) for more details).
Step 2: Define a `python_awslambda` target
------------------------------------------
Expand Down Expand Up @@ -58,35 +69,34 @@ Pants will use [dependency inference](doc:targets) based on the `handler` field,
You can optionally set the `output_path` field to change the generated zip file's path.

> 🚧 Use `resource` instead of `file`
>
>
> `file` / `files` targets will not be included in the built AWS Lambda because filesystem APIs like `open()` would not load them as expected. Instead, use the `resource` and `resources` target. See [Assets and archives](doc:assets) for further explanation.
Step 3: Run `package`
---------------------

Now run `pants package` on your `python_awslambda` target to create a zipped file.
Now run `pants package` on your `python_awslambda` target to create a zipped file.

For example:

```bash
$ pants package project/awslambda_example.py
Wrote code bundle to dist/project.zip
Runtime: python3.8
Handler: lambdex_handler.handler
$ pants package project/:lambda
Wrote dist/project/lambda.zip
Handler: lambda_function.handler
```

> 🚧 Running from macOS and failing to build?
>
>
> AWS Lambdas must run on Linux, so Pants tells PEX and Pip to build for Linux when resolving your third party dependencies. This means that you can only use pre-built [wheels](https://packaging.python.org/glossary/#term-wheel) (bdists). If your project requires any source distributions ([sdists](https://packaging.python.org/glossary/#term-source-distribution-or-sdist)) that must be built locally, PEX and pip will fail to run.
>
>
> If this happens, you must either change your dependencies to only use dependencies with pre-built [wheels](https://pythonwheels.com) or find a Linux environment to run `pants package`.
Step 4: Upload to AWS
---------------------

You can use any of the various AWS methods to upload your zip file, such as the AWS console or the AWS CLI via `aws lambda create-function` and `aws lambda update-function-code`.

You must specify the AWS lambda handler as `lambdex_handler.handler`.
You can specify the AWS lambda handler as `lambda_function.handler`. This is a re-export of the function referred to by the `handler` field of the target.

Docker Integration
------------------
Expand All @@ -101,7 +111,7 @@ FROM public.ecr.aws/lambda/python:3.8
RUN yum install unzip -y
COPY project/lambda.zip .
RUN unzip lambda.zip -d "${LAMBDA_TASK_ROOT}"
CMD ["lambdex_handler.handler"]
CMD ["lambda_function.handler"]
```
```python project/BUILD
python_sources()
Expand All @@ -119,3 +129,54 @@ docker_image(
```

Then, use `pants package project:my_image`, for example. Pants will first build your AWS Lambda, and then will build the Docker image and copy it into the AWS Lambda.

Advanced: Using PEX directly
----------------------------

In the rare case where you need access to PEX features, such as dynamic selection of dependencies, a PEX file created by `pex_binary` can be used as a Lambda package directly. A PEX file is a carefully constructed zip file, and can be understood natively by AWS. Note: using `pex_binary` results in larger packages and slower cold starts and is likely to be less convenient than using `python_awslambda`.

The handler of a `pex_binary` is not re-exported at the fixed `lambda_function.handler` path, and the Lambda handler must be configured as the `__pex__` pseudo-package followed by the handler's normal module path (for instance, if the handler is called `func` in `some/module/path.py` within [a source root](doc:source-roots), then use `__pex__.some.module.path.func`). The `__pex__` pseudo-package ensures dependencies are initialized before running any of your code.

For example:

```python project/BUILD
python_sources()

pex_binary(
name="lambda",
entry_point="lambda_example.py",
# specify an appropriate platform(s) for the targeted Lambda runtime (complete_platforms works too)
platforms=["linux_x86_64-cp39-cp39"],
)
```
```python project/lambda_example.py
def example_handler(event, context):
print("Hello AWS!")
```

Then, use `pants package project:lambda`, and upload the resulting `project/lambdex.pex` to AWS. The handler will need to be configured in AWS as `__pex__.lambda_example.example_handler` (assuming `project` is a [source root](doc:source-roots)).

Migrating from Pants 2.16 and earlier
-------------------------------------

Pants has implemented a new way to package Lambdas in 2.17, resulting in smaller packages and faster cold starts. This involves some changes:

- In Pants 2.16 and earlier, Pants used the [Lambdex](https://github.com/pantsbuild/lambdex) project. First, Pants would convert your code into a [Pex file](doc:pex-files) and then use Lambdex to adapt this to be better understood by AWS by adding a shim handler at the path `lambdex_handler.handler`. This shim handler first triggers the Pex initialization to choose and unzip dependencies, during the "INIT" phase.
- In Pants 2.17, the use of Lambdex is deprecated, in favour of choosing the appropriate dependencies ahead of time, as described above, without needing to do this on each cold start. This results in a zip file laid out in the format recommended by AWS, and includes a re-export of the handler at the path `lambda_function.handler`.
- In Pants 2.18, the new behaviour will become the default behaviour.
- In Pants 2.19, the old Lambdex behaviour will be entirely removed.

Any existing `python_awslambda` targets will change how they are built. Migrating has three steps:

1. opt-in to the new behaviour in Pants 2.17
2. package the new targets
3. upload those packages to AWS, and update the configured handler from `lambdex_handler.handler` (old) to `lambda_function.handler` (new)

To opt-in to the new behaviour in Pants 2.17, add the following to the end of your `pants.toml`:

``` toml pants.toml
[lambdex]
layout = "zip"
```

To temporarily continue using the old behaviour in Pants 2.17, instead set `layout = "lambdex"`. This will not be supported in Pants 2.19. If you encounter a bug with `layout = "zip"`, [please let us know](https://github.com/pantsbuild/pants/issues/new/choose). If you require advanced PEX features, [switch to using `pex_binary` directly](#advanced-using-pex-directly).
82 changes: 71 additions & 11 deletions docs/markdown/Python/python-integrations/google-cloud-function-python.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@ createdAt: "2021-11-09T20:29:58.330Z"
Pants can create a Google Cloud Function-compatible zip file from your Python code, allowing you to develop your functions in your repository.

> 📘 FYI: how Pants does this
>
> Under-the-hood, Pants uses the [Lambdex](https://github.com/pantsbuild/lambdex) project. First, Pants will convert your code into a [Pex file](doc:pex-files). Then, Pants will use Lambdex to convert the Pex into a zip file understood by Google Cloud Functions.
>
> Under-the-hood, Pants uses the [PEX](https://github.com/pantsbuild/pex) project, to select the appropriate third-party requirements and first-party sources and lay them out in a zip file, in the format recommended by Google Cloud Functions.

Step 1: Activate the Python Google Cloud Function backend
---------------------------------------------------------
Expand All @@ -26,6 +27,15 @@ backend_packages.add = [

This adds the new `python_google_cloud_function` target, which you can confirm by running `pants help python_google_cloud_function `

> 🚧 Set `layout = "zip"` for Pants 2.17
>
> Pants 2.17 is transitioning to a new, better layout, but defaults to the old Lambdex layout for backwards compatibility (see [below](#migrating-from-pants-216-and-earlier) for more details). To silence the warnings and be ready for Pants 2.18, add the following to the end of your `pants.toml`:
>
> ```toml pants.toml
> [lambdex]
> layout = "zip"
> ```
Step 2: Define a `python_google_cloud_function ` target
-------------------------------------------------------
Expand All @@ -44,7 +54,7 @@ python_sources(name="lib")
python_google_cloud_function(
name="cloud_function",
runtime="python38",
# Pants will convert this to `project.lambda_example:example_handler`.
# Pants will convert this to `project.google_cloud_function_example:example_handler`.
handler="google_cloud_function_example.py:example_handler",
type="event",
)
Expand All @@ -59,32 +69,82 @@ Pants will use [dependency inference](doc:targets) based on the `handler` field,
You can optionally set the `output_path` field to change the generated zip file's path.

> 🚧 Use `resource` instead of `file`
>
>
> `file` / `files` targets will not be included in the built Cloud Function because filesystem APIs like `open()` would not load them as expected. Instead, use the `resource` / `resources` target. See [Assets and archives](doc:assets) for further explanation.
Step 3: Run `package`
---------------------

Now run `pants package` on your `python_google_cloud_function` target to create a zipped file.
Now run `pants package` on your `python_google_cloud_function` target to create a zipped file.

For example:

```bash
$ pants package project/google_cloud_function_example.py
Wrote code bundle to dist/project.zip
Runtime: python3.8
$ pants package project/:cloud_function
Wrote dist/project/cloud_function.zip
Handler: handler
```

> 🚧 Running from macOS and failing to build?
>
>
> Cloud Functions must run on Linux, so Pants tells PEX and Pip to build for Linux when resolving your third party dependencies. This means that you can only use pre-built [wheels](https://packaging.python.org/glossary/#term-wheel) (bdists). If your project requires any source distributions ([sdists](https://packaging.python.org/glossary/#term-source-distribution-or-sdist)) that must be built locally, PEX and pip will fail to run.
>
>
> If this happens, you must either change your dependencies to only use dependencies with pre-built [wheels](https://pythonwheels.com) or find a Linux environment to run `pants package`.
Step 4: Upload to Google Cloud
------------------------------

You can use any of the various Google Cloud methods to upload your zip file, such as the Google Cloud console or the [Google Cloud CLI](https://cloud.google.com/functions/docs/deploying/filesystem#deploy_using_the_gcloud_tool).

You must specify the handler as `handler`.
You must specify the handler as `handler`. This is a re-export of the function referred to by the `handler` field of the target.

Advanced: Using PEX directly
----------------------------

In the rare case where you need access to PEX features, such as dynamic selection of dependencies, a PEX file created by `pex_binary` can be used as a Google Cloud Function package directly. A PEX file is a carefully constructed zip file, and can be understood natively by Google Cloud Functions. Note: using `pex_binary` results in larger packages and slower cold starts and is likely to be less convenient than using `python_google_cloud_function`.

The handler of a `pex_binary` is not re-exported at the fixed `main.handler` path, and the Google Cloud Function handler must be configured as the `__pex__` pseudo-package followed by the handler's normal module path (for instance, if the handler is in `some/module/path.py` within [a source root](doc:source-roots), then use `__pex__.some.module.path`). This may require being configured via [`GOOGLE_FUNCTION_SOURCE`](https://cloud.google.com/docs/buildpacks/service-specific-configs#google_function_source). The `__pex__` pseudo-package ensures dependencies are initialized before running any of your code.

For example:

```python project/BUILD
python_sources()

pex_binary(
name="gcf",
entry_point="gcf_example.py",
# specify an appropriate platform(s) for the targeted GCF runtime (complete_platforms works too)
platforms=["linux_x86_64-cp39-cp39"],
)
```
```python project/gcf_example.py
def example_handler(event, context):
print("Hello GCF!")
```

Then, use `pants package project:gcf`, and upload the resulting `project/gcf.pex` to Google Cloud Functions. You will need to specify the handler as `example_handler` and set `GOOGLE_FUNCTION_SOURCE=__pex__.gcf_example` (assuming `project` is a [source root](doc:source-roots)).

Migrating from Pants 2.16 and earlier
-------------------------------------

Pants has implemented a new way to package Google Cloud Functions in 2.17, resulting in smaller packages and faster cold starts. This involves some changes:

- In Pants 2.16 and earlier, Pants used the [Lambdex](https://github.com/pantsbuild/lambdex) project. First, Pants would convert your code into a [Pex file](doc:pex-files) and then use Lambdex to adapt this to be better understood by GCF by adding a shim handler. This shim handler first triggers the Pex initialization to choose and unzip dependencies, during initialization.
- In Pants 2.17, the use of Lambdex is deprecated, in favour of choosing the appropriate dependencies ahead of time, as described above, without needing to do this on each cold start. This results in a zip file laid out in the format recommended by GCF, and includes a re-export of the handler.
- In Pants 2.18, the new behaviour will become the default behaviour.
- In Pants 2.19, the old Lambdex behaviour will be entirely removed.

Any existing `python_google_cloud_function` targets will change how they are built. Migrating has three steps:

1. opt-in to the new behaviour in Pants 2.17
2. package the new targets
3. upload those packages to GCF (the existing handler configuration should still work)

To opt-in to the new behaviour in Pants 2.17, set:

``` toml pants.toml
[lambdex]
layout = "zip"
```

To temporarily continue using the old behaviour in Pants 2.17, instead set `layout = "lambdex"`. This will not be supported in Pants 2.19. If you encounter a bug with `layout = "zip"`, [please let us know](https://github.com/pantsbuild/pants/issues/new/choose). If you require advanced PEX features, [switch to using `pex_binary` directly](#advanced-using-pex-directly).
10 changes: 8 additions & 2 deletions src/python/pants/backend/python/subsystems/lambdex.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from pants.backend.python.target_types import ConsoleScript
from pants.engine.rules import collect_rules
from pants.option.option_types import EnumOption
from pants.util.docutil import doc_url
from pants.util.strutil import softwrap


Expand Down Expand Up @@ -40,8 +41,8 @@ class Lambdex(PythonToolBase):
),
removal_version="2.19.0.dev0",
removal_hint=softwrap(
"""
Remove the whole [lambdex] section, as Lambdex is deprecated and its functionality be
f"""
Remove the whole [lambdex] section, as Lambdex is deprecated and its functionality will be
removed. If you have `layout = "zip"`, no further action is required, as you are already using
the recommended layout.
Expand All @@ -50,6 +51,11 @@ class Lambdex(PythonToolBase):
as recommended by cloud vendors. (If you are using `python_awslambda`, you will need to
also update the handlers configured in the cloud from `lambdex_handler.handler` to
`lambda_function.handler`.)
See the docs for more details:
* {doc_url('awslambda-python#migrating-from-pants-216-and-earlier')}
* {doc_url('google-cloud-function-python#migrating-from-pants-216-and-earlier')}
"""
),
)
Expand Down

0 comments on commit 9089aea

Please sign in to comment.