LambdaSharp "Eurytus" Release (v0.5.0.3) - 2019-03-08
Eurytus was an eminent Pythagorean philosopher. He was a disciple of Philolaus, and Diogenes Laërtius mentions him among the teachers of Plato, though this statement is very doubtful. (Wikipedia)
What's New
The objective of the LambdaSharp 0.5 Eurytus release has been to streamline the iterative development process with the LambdaSharp CLI. The biggest time sink for deploying CloudFormation templates is that errors are only detected during the deploy phase. Consequently, the biggest time optimization is to detect errors as early as possible. This pushes the LambdaSharp CLI into the validating compiler territory, which has always been the objective. With this release, the LambdaSharp CLI validates all properties and attributes for all AWS types. That means, if you forget to set a required property on a resource or you have a typo, the LambdaSharp CLI will detect it during the build phase of the module.
Streamlining error detection would be incomplete if it did not apply to published modules as well. During the build phase, the LambdaSharp CLI downloads the manifests of referenced modules and uses them to validate properties and attributes of custom resource types. Then, during the deploy phase, the CLI checks for the presence of the referenced modules and automatically installs them if needed. This release includes three utility modules: LambdaSharp.S3.IO
provides functionality for writing or deleting files from S3 buckets, LambdaSharp.S3.Subscriber
enables Lambda functions to receive notifications from S3 buckets, and LambdaSharp.Twitter.Query
enables Lambda functions to received notifications from Twitter queries.
In addition to the CLI improvements, LambdaSharp modules have also received a few new features. Amongst them is support for Mapping
and Condition
definitions, which fill the void to provide complete coverage of all CloudFormation constructs. More exciting is built-in support for decrypting parameters and variables directly in CloudFormation. Also, modules now have a constructor/destructor called a Finalizer
allowing for dynamic initialization and clean-up operations.
Finally, as a bonus feature, the LambdaSharp CLI now has the utility command util delete-orphan-lambda-logs
to delete legacy Lambda CloudWatch Logs. This feature is particularly useful for those who have been experimenting with Lambda in the past. By default, CloudWatch Logs are never deleted and keep accumulating over time, only increasing your monthly AWS bill. LambdaSharp modules are quite different. They have default retention policy of 30 days for logs and automatically delete them on tear down.
BREAKING CHANGES
The following change may impact modules created with previous releases.
Module Definition
- The top-level
Module
attribute must now contain at least one period (.
) to denote the module namespace and module name. - The
Inputs
,Outputs
,Variables
, andFunctions
sections of LambdaSharp module have been combined into a singleItems
section to give more organizational freedom. - The
Export
definitions have been removed in favor of a special class forScope
on items that can be exported. - Cross-module references are now grouped into
Using
definitions by module they import from. Package
definitions now compress local files into a deployed zip package only. The zip package can then be used to deploy files to S3--using theLambdaSharp::S3::Unzip
resource--, create Lambda Layers, Alexa skills, or anything else that requires a compressed zip package as its input.- The
Var
definition has been split into two distinct definitions:Variable
for fixed values and expressions, andResource
for existing or new resources. - The
CustomResource
definition has been renamed toResourceType
and enhanced with a specification about its properties and attributes to enable validation at compilation time. Function
definitions lost theirVPC
section andReservedConcurrency
attribute. Instead, likeResource
definitions, they have aProperties
section that can be used to set these, and other settings, directly. This approach is much more future proof.- Nesting items has been limited to
Using
andGroup
definitions. - The default value for the
Version
attribute is now1.0-DEV
.
LambdaSharp Runtime
- The LambdaSharp Runtime has been renamed to LambdaSharp Core. The previous terminology caused too much confusion with the AWS Lambda runtime.
- The LambdaSharp Core is deployed as a single module without using nested modules. This decreases the deployment time.
LambdaSharp CLI
The big change is to run lash
now instead of dotnet lash
. It's shorter and works just as well!
- Build process
- Renamed
--skip-assembly-validation
option to--no-assembly-validation
- Renamed
--cf-output
option to--cfn-output
- Renamed
- Publish process
- Modules are now published to a new location in the S3 deployment bucket:
${Module::Namespace}/Modules/${Module::Name}/Versions/${Module::Version}
- The LambdaSharp manifest is now embedded in the CloudFormation template inside the
Metadata
section underLambdaSharp::Manifest
.
- Modules are now published to a new location in the S3 deployment bucket:
- Deploy Process
- Renamed
--inputs
option to--parameters
for consistency reasons. - Removed
--input
option.
- Renamed
LambdaSharp Assemblies
- Assemblies
- Merged
MindTouch.LambdaSharp.CustomResource
assembly into theMindTouch.LambdaSharp
assembly. - Renamed
MindTouch.LambdaSharp
assembly toLambdaSharp
. - Renamed
MindTouch.LambdaSharp.Slack
assembly toLambdaSharp.Slack
. - Renamed
MindTouch.LambdaSharp.Tool
assembly toLambdaSharp.Tool
.
- Merged
- Namespaces and Classes
- Renamed the
MindTouch.LambdaSharp
namespace toLambdaSharp
. - Renamed
ALambdaEventFunction
toALambdaTopicFunction
.
- Renamed the
New LambdaSharp Module Features
Module Namespace
The Module
attribute now specifies both the module namespace and the module name. The module namespace could be the name of an organization, an individual, or a project the module is part of. The module namespace is used to group related modules in the deployment S3 bucket. Everything after the first period is considered to be part of the module name, including additional periods.
In the following example, Acme
is the module namespace and Accounting.Reports
is the module name:
Module: Acme.Accounting.Reports
Module Dependencies
LambdaSharp modules now support the concept of dependencies. Used modules are listed in the Using
section of the module definition. Listing a module impacts both the build and deploy phases. During the build phase, the module manifests are imported to help with validation. During the deploy phase, used dependencies are checked for and deployed when missing.
In the following example, the module definition indicates that it has a dependency on LambdaSharp.S3.IO
.
Module: Acme.MyModule
Using:
- Module: LambdaSharp.S3.IO
Module Items
All definitions in a LambdaSharp module are now unified in the Items
section. This change allows definitions to be more freely organized. For example, it is now possible to group different definition types (parameters, variables, etc) by their purpose, rather than by their type. Definitions can further be grouped into namespaces to better reflect the organization of the module.
The following example shows how definitions can be placed close to where they are needed or close to other definitions they are related to. This design gives greater flexibility to developers to organize their thoughts, not unlike what is found in most programming languages.
Module: Acme.MyModule
Items:
- Parameter: PersonName
Type: String
- Variable: GreetPerson
Type: String
Scope: Reporting::Calculator
Value: !Sub "Hi ${PersonName}"
- Group: Reporting
Items:
- Resource: Bucket
Scope: Reporting::Calculator
Type: AWS::S3::Bucket
- Function: Calculator
Memory: 128
Timeout: 30
- Variable: AccountingBucket
Scope: public
Value: !GetAtt Reporting::Bucket.Arn
Secret
Type
Parameter
and Variable
definitions with encrypted values (type Secret
) can now be decrypted by accessing a dynamically created nested item called Plaintext
.
The following example defines an encrypted parameter called MySecretParameter
. For illustrative purposes, the variable MyDecryptedValue
shows how easy it is to access the plaintext value. The !Ref
expression could be used in any context where an expression can be used. Note that the module must have been granted access to the encryption key used by the encrypted value.
- Parameter: MySecretParameter
Type: Secret
- Variable:
Value: !Ref MySecretParameter::Plaintext
Module Finalizer
LambdaSharp modules can now have a finalizer function that is automatically run after all resources have been created or before any resources are torn down. There are many use-cases for a module finalizer, including:
- Initialize a DynamoDB table with seed data on initial module deployment.
- Send an module availability notification to other services.
- Migrate existing data after a module has been updated to a newer version.
- Delete dynamically created resources when the module is torn down.
- Delete objects from an S3 bucket so it can be deleted when the module is torn down.
A module finalizer function must be called Finalizer
and must appear in the top-level items of the module. The module finalizer timeout is always set to the maximum duration of 15 minutes. On invocation, the module finalizer receives the module version to allow it to track upgrades or downgrade scenarios. It also receives the module checksum so it track state for each CloudFormation update.
The following example shows how easy it is to define a module finalizer:
- Function: Finalizer
Memory: 128
Function Properties
Section
The Function
definition now exposes its Properties
section to access advanced functionality of Lambda functions, such as VPC configuration and Lambda layers. Similar to Resource
definitions, the function Properties
are validated during compilation.
The following example shows how to configure a Lambda function for VPC:
- Function: MyFunction
Memory: 128
Timeout: 30
Properties:
VpcConfig:
SecurityGroupIds: !Ref SecurityGroupIds
SubnetIds: !Ref SubnetIds
The next example shows how to set Lambda layers for a function:
- Function: MyFunction
Memory: 128
Timeout: 30
Properties:
Layers:
- !Ref MyLambdaLayer
Nested Modules
Nested modules are similar to nested CloudFormation stacks. The module reference is resolved at compile time to a CloudFormation template location. Furthermore, the LambdaSharp CLI seamlessly injects the deployment tier parameters required for deploying modules. See Nested
documentation.
During the build phase, the LambdaSharp CLI validates that all required parameters are supplied and that the supplied parameters exist.
The following example shows how to create a nested module definition and access its output values:
- Nested: MyNestedModule
Module: Acme.MyOtherModule:1.0
Parameters:
Message: !Sub "Hi from ${Module::Name}"
- Variable: NestedOutput
Value: !GetAtt MyNestedModule.Outputs.OutputName
Condition
Item
LambdaSharp modules now support Condition
items and the corresponding !Condition
function. The If
attribute is used on Resource
and Function
items to indicate that they are conditional.
The following example shows a Condition
item keying off a Parameter
and thus controlling the creation of two conditional resources.
- Parameter: EnvType
Description: Environment type.
Default: test
Type: String
AllowedValues:
- prod
- test
ConstraintDescription: must specify prod or test.
- Resource: EC2Instance
Type: "AWS::EC2::Instance"
Properties:
ImageId: ami-0ff8a91507f77f867
- Group: ProductionResources
Items:
- Condition: Create
Value: !Equals [ !Ref EnvType, prod ]
- Resource: MountPoint
Type: AWS::EC2::VolumeAttachment
If: ProductionResources::Create
Properties:
InstanceId: !Ref EC2Instance
VolumeId: !Ref ProductionResources::NewVolume
Device: /dev/sdh
- Resource: NewVolume
Type: AWS::EC2::Volume
If: ProductionResources::Create
Properties:
Size: 100
AvailabilityZone: !GetAtt EC2Instance.AvailabilityZone
The If
attribute can either have the name of a Condition
item or contain the conditional expression directly. The following two examples are identical:
- Condition: Create
Value: !Equals [ !Ref CreateTopic, yes ]
- Resource: ConditionalTopic
Type: AWS::SNS::Topic
If: ProductionResources::Create
-VS-
- Resource: ConditionalTopic
Type: AWS::SNS::Topic
If: !Equals [ !Ref CreateTopic, yes ]
Conditional resources that also have a Scope
attribute are conditionally injected into their specified Lambda functions. Care needs to be taken when reading them from the function configuration to allow for them to be missing.
Mapping
Item
LambdaSharp modules now also support Mapping
items and the corresponding !FindInMap
function.
The next example shows a definition of a Mapping
item and its use:
- Mapping: Greetings
Description: Time of day greeting
Value:
Morning:
Text: Good morning
Day:
Text: Good day
Evening:
Text: Good evening
Night:
Text: Good night
- Parameter: SelectedTime
Description: Parameter for selecting the time of day
AllowedValues:
- Morning
- Day
- Evening
- Night
- Variable: SelectedGreeting
Description: Selected greeting
Value: !FindInMap [ Greetings, !Ref SelectedTime, Text ]
Function Sources
DynamoDB
can now be a!Ref
expression.Kinesis
can now be a!Ref
expression.Queue
can now be a!Ref
expression.S3
can now be a!Ref
expression.Schedule
can now be a!Ref
expression.Topic
can now be a!Ref
expression.Topic
source now supportsFilters
to filter on SNS notifications.
New LambdaSharp CLI Features
Build Command
The LambdaSharp CLI now validates the properties of created resources and access to their attributes. This avoids common errors like missing a required property or having a simple typo in the definition. Similarly, when accessing an attribute with !GetAtt
, the LambdaSharp CLI checks that the attribute exists on the resource. The validation for a resource can be suppressed with the no-type-validation
resource pragma, which is useful when new resource types, properties, or attributes become available that are not yet supported by the LambdaSharp CLI.
The manifests for modules referenced as dependencies or nested modules are now downloaded to validate their usage. Manifests contain information about module parameters, resource type definitions, and output values. This enables the LambdaSharp CLI to validate properties and attributes on custom resource types just like with built-in AWS types. This feature can be disabled with the --no-dependency-validation
CLI option. The list of S3 buckets that are scanned for modules is controlled by a parameter in the AWS Systems Manager parameter store. By default, it is the deployment bucket and the lambdasharp bucket.
The LambdaSharp CLI now also analyzes the usage of all parameters, resources, and variables. If a declared parameter is not used anywhere, the LambdaSharp CLI will issue a warning to draw attention to it. This situation commonly occurs because of a missing Scope
attribute or when a parameter is no longer needed, but its definition lingers. Internally, the analysis is also used to garbage collect optional resource definitions. For example, every module has an embedded IAM Role (i.e. Module::Role
) and API Gateway (i.e. Module::RestApi
). However, unless these are used, the LambdaSharp CLI will automatically remove them to optimize the produced CloudFormation template.
Another benefit of analyzing expressions during the build phase is the ability to inline variables into !Sub
and !Join
expressions. For example, the generated value for Expression
is "My list: First,Second,Third"
:
- Variable: AConstant
Value: First
- Variable: AListOfValues
Value:
- !Ref AConstant
- Second
- Third
- Variable: Expression
Value: !Sub "My list: ${AListOfValues}"
The introduction of Condition
definitions also requires more careful validation of !Ref
and !GetAtt
expressions since these could refer to resources that may not exist during the deploy phase. The LambdaSharp CLI now provides very basic support for validating conditional references. However, because the support is basic, it can lead to false negatives. Consequently, detected violations are shown as warnings instead of errors.
Finally, the entry point for .NET Core Lambda functions is now validated after compilation of the assembly. This avoids deploying a Lambda function that immediately fails, because the Lambda runtime was unable to located the entry point. This error usually occurs after refactoring code and forgetting to update the project file with the new namespace information. Although not a common error, it is extremely time consuming, because it occurs so late in the development process. Entry point validation can be skipped by using the --no-assembly-validation
CLI option or the no-assembly-validation
function pragma.
Publish Command
The LambdaSharp CLI now prevents a module from being published again with the same version number. However, this constraint is lifted for pre-release versions. A pre-release version has a suffix string, such as 1.0-DEV
. If required, the LambdaSharp CLI can be forced to overwrite an existing published module by using the --force-publish
option. This behavior is the building block for caching of module manifests in the future. Republishing a module with breaking changes will cause issues. However, modules with pre-release versions are never cached and are therefore safe to be republished.
Deploy Command
The LambdaSharp CLI now uses CloudFormation Change Sets to deploy modules. CloudFormation change sets are the preferred way to update infrastructure as they provide an additional review mechanism for detecting unwanted changes.
With AWS X-Ray going GA just in time, the LambdaSharp CLI now has a new --xray
option to enable service-call tracing with AWS X-Ray for all functions in the module.
In addition, the LambdaSharp CLI now prompts for missing module parameters before creating or updating the CloudFormation stack. This includes parameters required by modules referenced in the Using
section. If the module was deployed previously, the LambdaSharp CLI will only prompt for parameters that were not set previously. Furthermore, the --parameters
option can be used to supply parameter values for the module being deployed. Parameters with default values are not prompted unless the --prompt-all
option is used.
Having an interactive prompt in a CI/CD setup is about the worst thing that can happen. To avoid this scenario, use the --prompts-as-errors
option to convert prompts into reported errors instead. This is another way to catch early that something is amiss before anything happens.
The following example shows the deployment of the Demos/TwitterNotifier
module:
LambdaSharp CLI (v0.5) - Deploy LambdaSharp module
Readying module for deployment tier 'Sandbox'
...
Configuration for LambdaSharp.Demo.TwitterNotifier (v1.0-DEV)
*** TWITTER SETTINGS ***
|=> TwitterApiKey [Secret]: API Key: AQICAHiCs...wn7WieBEC
|=> TwitterApiSecretKey [Secret]: API Secret Key: AQICAHiCs...BNOJIrKzo
|=> TwitterQuery [String]: Search query: LambdaSharp
Deploying stack: Sandbox-LambdaSharp-Demo-TwitterNotifier [LambdaSharp.Demo.TwitterNotifier:1.0-DEV]
=> Stack create initiated for Sandbox-LambdaSharp-Demo-TwitterNotifier [CAPABILITY_NAMED_IAM, CAPABILITY_AUTO_EXPAND]
REVIEW_IN_PROGRESS AWS::CloudFormation::Stack Sandbox-LambdaSharp-Demo-TwitterNotifier (User Initiated)
CREATE_IN_PROGRESS AWS::CloudFormation::Stack Sandbox-LambdaSharp-Demo-TwitterNotifier (User Initiated)
CREATE_IN_PROGRESS AWS::CloudFormation::Stack TwitterNotify
CREATE_IN_PROGRESS LambdaSharp::Registration::Module Module::Registration
...
CREATE_IN_PROGRESS AWS::Logs::SubscriptionFilter Notify::LogGroupSubscription (Resource creation Initiated)
CREATE_COMPLETE AWS::Logs::SubscriptionFilter Notify::LogGroupSubscription
CREATE_COMPLETE AWS::Lambda::Permission Notify::Source1Permission
CREATE_COMPLETE AWS::CloudFormation::Stack Sandbox-LambdaSharp-Demo-TwitterNotifier
=> Stack create finished
Stack output values:
=> Module: LambdaSharp.Demo.TwitterNotifier:1.0-DEV
Done (finished: 2/1/2019 2:19:28 PM; duration: 00:04:36.2478253)
A small, aesthetic win is that the LambdaSharp CLI uses the manifest to translate the resource names from their logical ID to their module definition name. Similarly, custom resource types are translated to their actual names. For example, update to resources of type Custom::LambdaSharpRegisterFunction
are shown as LambdaSharp::Register::Function
instead.
Last, but not least, all in resources in LambdaSharp module are not automatically tagged, when possible, with the following information:
|Tag Name|Description|Example|
|--------|-----------|-------|
LambdaSharp:DeployedBy
|The IAM identity used to deploy the CloudFormation stack.|user/bob
|
LambdaSharp:Module
|The full module name without version or source bucket specification.|My.Module
|
LambdaSharp:RootStack
|The name of the root CloudFormation stack under which the resource are created. This is the same as aws:cloudformation:stack-name
for non-nested modules. Otherwise, it is the stack name of the topmost, non-nested module.|Sandbox-My-Module-MyResource-1TOI83RQZQZE3
|
LambdaSharp:Tier
|The name of the deployment tier.|Sandbox
|
Config Command
The LambdaSharp CLI now allows to request a specific bucket name during the config
command instead of defaulting to a name created by CloudFormation. The created bucket now also grants permission to the AWS Serverless Repository to access its contents, in preparation to support publishing modules to it in the future.
Similar to the change the deploy
command, the config
command now prompts for new parameters when upgrading an existing configuration.
New Command
The LambdaSharp CLI now allows to add a resource definition to a module, similar to the new function
command. The new new resource
command take a resource name and resource type. It then appends the a skeleont definition to the Module.yml
file where the property values indicate the type of the property and if it is required.
lash new resource MyTopic AWS::SNS::Topic
Appends the following resource definition to the Module.yml
file:
- Resource: MyResource
Description: TO-DO - update resource description
# Scope: List of functions to be given the name of the resource
Type: AWS::SNS::Topic
# Allow: Shorthand or allowed actions
Properties:
DisplayName: String
KmsMasterKeyId: String
Subscription:
- Endpoint: String # Required
Protocol: String # Required
TopicName: String
Util Command
The LambdaSharp CLI now has a new util
command menu with two sub-commands.
The first one is the download-cloudformation-spec
, which is used by LambdaSharp contributors to update the built-in CloudFormation specification. It is not useful for non-contributors.
The second sub-command is delete-orphan-lambda-logs
, which deletes orphaned Lambda CloudWatch log groups. Log groups created by LambdaSharp are automatically deleted when a module is torn down. However, by default, if you have experimented with Lambda functions before, the Lambda function CloudWatch log groups are never deleted. This sub-command scans all log groups and checks if the associated Lambda function still exists. If it does not, it deletes the CloudWatch log group.
Usage is straightforward. There is --dryrun
option to have the sub-command show what it would do without affecting anything.
lash util delete-orphan-lambda-logs
The sub-command shows which CloudWatch log groups were deleted.
LambdaSharp CLI (v0.5) - Delete orphaned Lambda CloudWatch logs
* deleted '/aws/lambda/MyOldFunction'
* deleted '/aws/lambda/LifeBeforeLambdaSharpWasHardFunction'
Found 2 log groups. Deleted 2. Skipped 0.
Done (finished: 2/1/2019 2:57:08 PM; duration: 00:00:00.9142275)
Misc
The LambdaSharp CLI now prints a date-timestamp after each command and how long it took to run. This makes it easy to see if it was run recently and thus avoiding re-running a command in the heat of a frantic development session.
The LambdaSharp CLI now captures information about the git branch during the build phase in addition to the git SHA. If needed, the git branch information can also be supplied with the --git-branch
option.
New LambdaSharp Core Module Features
The LambdaSharp Core module provides the basic capabilities of a deployment tier and is required by every module. The LambdaSharp Core module no longer requires nested modules. Instead, additional functionality is automatically pulled in during the deploy phase by the LambdaSharp CLI.
Module LambdaSharp.S3.IO
This module defines three resource types for interacting with S3 buckets:
LambdaSharp::S3::EmptyBucket
: empty all contents of a bucket on module tear-downLambdaSharp::S3::Unzip
: copy/update a package to an S3 bucketLambdaSharp::S3::WriteJson
: write a JSON file to an S3 bucket
Module LambdaSharp.S3.Subscriber
This module provides the LambdaSharp::S3::Subscription
resource type required for receiving S3 bucket notifications. It is automatically referenced by the LambdaSharp CLI when a Lambda function uses an S3 bucket as its event source.
Module LambdaSharp.Twitter.Query
This module runs a Twitter query on a regular interval and publishes newly found tweets on an SNS topic. The TwitterNotifier
demo module shows how to subscribe to messages and reformat them for delivery.
New LambdaSharp Assembly Features
Class ALambdaFinalizerFunction
This new base class was introduced for creating a module finalizer. This base class defines the proper request and response data structures. See the Finalizer Example for its use.
Class ALambdaCustomResourceFunction
This base class was enhanced to support direct Lambda invocations and indirect SNS topic invocations. The detection of either protocol is done automatically and no additional steps are required.
Internal Changes
There wre some additional internal changes listed here for sake of completeness:
- The default Lambda log group retention period was increased from 7 to 30 days.
- The
ModuleCloudWatchLogsRole
is now defined once in the LambdaSharp Core module and then re-used by all modules. - The
ModuleName
andModuleVersion
module output values have been combined into a singleModule
output value.
Fixes
(v0.5.0.3) - 2019-03-08
- Added support for
!GetConfig
,!GetParam
, and!GetEnv
in module parameters file - Fixed issue where the
DefaultSecretsKey
was not properly set for the functions inLambdaSharp.Core
- Fixed error in
LambdaSharp.Core
log processor - Use ANSI colors for CloudFormation updates
- Apply CFN-spec corrections from
aws-cloudformation/cfn-python-lint
- Added
LambdaSharp.Core
module documentation - Added
LambdaSharp.S3.Subscriber
module documentation
(v0.5.0.2) - 2019-03-01
- Allow
BatchSize
andStartingPosition
attribute for event sources to be expressions - Make
publish
command idempotent - Fix
NullReferenceException
when compiling a module with an invalid CLI profile - Use global AWS Lambda Tools extension for dotnet CLI
- Prevent CLI from exiting with an error due to a network glitch during a CloudFormation stack update