S3 Log Records
All ingested CloudWatch Log events are stored in an S3 logging bucket when Core Services are enabled. During ingestion, the LambdaSharp.Core module parses and converts the CloudWatch Log events into JSON records. This enables Amazon Athena to query the ingested CloudWatch Log events.
Record Schema
The log record schema has the following fields.
Timestamp: Long
ModuleInfo: String
Module: String
ModuleId: String
Function: String
FunctionId: String
Tier: String
RecordType: String
Record: String
Record Properties
Timestamp
-
The
Timestamp
property holds the UNIX epoch timestamp in milliseconds.Type: Long
ModuleInfo
-
The
ModuleInfo
property holds the module name, version, and origin information.Type: String
Module
-
The
Module
property holds the module name.Type: String
ModuleId
-
The
ModuleId
property holds the CloudFormation stack name.Type: String
Function
-
The
Function
property holds the logical Lambda function name.Type: String
FunctionId
-
The
FunctionId
property holds the Lambda function resource name.Type: String
Tier
-
The
Tier
property holds the deployment tier name.Type: String
RecordType
-
The
RecordType
property holds the type of log record. Must be one ofLambdaError
,LambdaEvent
,LambdaMetrics
, orUsageReport
.Type: String
Record
-
The
Record
property holds the contents of the log record as a serialized JSON string.Type: String
Athena Table
The following steps show how to setup an Athena database and table to query ingested CloudWatch Log events.
Step 1 - Create an Athena Database
If you haven't already done so, create an Athena database. Replace MyDatabase
with a database name of your choice.
CREATE DATABASE MyDatabase;
Step 2 - Create an Athena Table
Replace <LOGGING-BUCKET>
with the S3 logging bucket name. The S3 logging bucket name can be found in the output of the lash init
command or, anytime later, in the AWS console. To find the bucket name in the AWS console, got to the CloudFormation service, select the LambdaSharp.Core CloudFormation stack corresponding to the deployment tier, and select Outputs on the tabbed view. The ARN of the S3 logging bucket is under the LoggingBucket
output value. Note that the part after the last colon (:
) is the bucket name.
By default, the log records are stored under the logging-success/
prefix. Adjust the prefix according to your configuration.
Finally, replace MyLogs
with a table name of your choice.
CREATE EXTERNAL TABLE IF NOT EXISTS MyDatabase.MyLogs (
`Timestamp` bigint,
`ModuleInfo` string,
`Module` string,
`ModuleId` string,
`Function` string,
`FunctionId` string,
`Record` string,
`RecordType` string
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1'
) LOCATION 's3://<LOGGING-BUCKET>/logging-success/';
Step 3 - Query Log Records from Athena
The following query fetches all log records for all modules and functions from the S3 logging bucket.
Make sure to replace MyDatabase
and MyLogs
with your database and table names, respectively.
SELECT *
FROM MyDatabase.MyLogs
ORDER BY Timestamp DESC;
Example 1: Show Usage Reports by Used Duration
The following query summarizes all UsageReport
records by count, average used duration in percent, and maximum used duration in percent. This query is useful to identify Lambda functions that may be running too close to their execution limit.
Make sure to replace MyDatabase
and MyLogs
with your database and table names, respectively.
SELECT
Module,
Function,
Count(UsedDurationPercent) AS Count,
Round(Avg(UsedDurationPercent), 2) AS AvgUsedDurationPercent,
Round(Max(UsedDurationPercent), 2) AS MaxUsedDurationPercent
FROM (
SELECT
Module,
Function,
(100.0 * Cast(json_extract_scalar(Record, '$.UsedDurationPercent') AS REAL)) AS UsedDurationPercent
FROM MyDatabase.MyLogs
WHERE RecordType = 'UsageReport'
)
GROUP BY 1, 2
ORDER BY 5 DESC;
Module | Function | Count | AvgUsedDurationPercent | MaxUsedDurationPercent |
---|---|---|---|---|
Sample.Event | SenderFunction | 10 | 6.82 | 16.93 |
Sample.Metric | MyFunction | 4 | 2.02 | 8.07 |
Sample.Event | ReceiverFunction | 10 | 6.08 | 7.84 |
Example 2: Show LambdaError Reports for past 24 Hours
The following query shows all LambdaError reports for the past 24 hours by CloudFormation stack (moduleId
) and Lambda function.
Make sure to replace MyDatabase
and MyLogs
with your database and table names, respectively.
SELECT
from_unixtime(Timestamp / 1000.0) AS "DateTime (UTC)",
ModuleId,
Function,
json_extract_scalar(Record, '$.Level') AS Level,
json_extract_scalar(Record, '$.Message') AS Message
FROM MyDatabase.MyLogs
WHERE Timestamp > (1000.0 * to_unixtime(date_add('day', -1, now()))) AND RecordType='LambdaError'
ORDER BY Timestamp DESC, ModuleId, Function
DateTime | ModuleId | Function | Level | Message |
---|---|---|---|---|
2020-05-13 23:10:08.779 | Sandbox-LambdaSharp-BadModule | FailTimeout | ERROR | Lambda timed out after 15.02 seconds |
2020-05-13 23:10:03.361 | Sandbox-LambdaSharp-BadModule | FailConstructor | ERROR | An exception was thrown when the constructor for type 'BadModule.FailConstructor.Function' was invoked. Check inner exception for more details. |
2020-05-13 23:10:02.816 | Sandbox-LambdaSharp-BadModule | FailConstructor | ERROR | An exception was thrown when the constructor for type 'BadModule.FailConstructor.Function' was invoked. Check inner exception for more details. |
2020-05-13 23:10:02.208 | Sandbox-LambdaSharp-BadModule | FailError | ERROR | this exception was thrown on request |
2020-05-13 23:10:01.582 | Sandbox-LambdaSharp-BadModule | FailBadEntryPoint | ERROR | Unable to load type 'BadModule.FailBadEntryPoint.Function' from assembly 'FailBadEntryPoint'. |
2020-05-13 23:10:01.236 | Sandbox-LambdaSharp-BadModule | FailBadEntryPoint | ERROR | Unable to load type 'BadModule.FailBadEntryPoint.Function' from assembly 'FailBadEntryPoint'. |
2020-05-13 23:10:00.877 | Sandbox-LambdaSharp-BadModule | FailOutOfMemory | ERROR | Exception of type 'System.OutOfMemoryException' was thrown. |
2020-05-13 23:09:59.386 | Sandbox-LambdaSharp-BadModule | FailBadEntryPoint | ERROR | Unable to load type 'BadModule.FailBadEntryPoint.Function' from assembly 'FailBadEntryPoint'. |
2020-05-13 23:09:58.200 | Sandbox-LambdaSharp-BadModule | FailConstructor | ERROR | An exception was thrown when the constructor for type 'BadModule.FailConstructor.Function' was invoked. Check inner exception for more details. |
NOTE: Due to an issue in the Lambda runtime errors that occur either in the constructor or failure to located the constructor result in multiple entries in CloudWatch Logs.