S3 Log Records

All ingested CloudWatch Log events are stored in an S3 logging bucket when Core Services are enabled. During ingestion, the LambdaSharp.Core module parses and converts the CloudWatch Log events into JSON records. This enables Amazon Athena to query the ingested CloudWatch Log events.

Record Schema

The log record schema has the following fields.

Timestamp: Long
ModuleInfo: String
Module: String
ModuleId: String
Function: String
FunctionId: String
Tier: String
RecordType: String
Record: String

Record Properties

Timestamp

The Timestamp property holds the UNIX epoch timestamp in milliseconds.

Type: Long

ModuleInfo

The ModuleInfo property holds the module name, version, and origin information.

Type: String

Module

The Module property holds the module name.

Type: String

ModuleId

The ModuleId property holds the CloudFormation stack name.

Type: String

Function

The Function property holds the logical Lambda function name.

Type: String

FunctionId

The FunctionId property holds the Lambda function resource name.

Type: String

Tier

The Tier property holds the deployment tier name.

Type: String

RecordType

The RecordType property holds the type of log record. Must be one of LambdaError, LambdaEvent, LambdaMetrics, or UsageReport.

Type: String

Record

The Record property holds the contents of the log record as a serialized JSON string.

Type: String

Athena Table

The following steps show how to setup an Athena database and table to query ingested CloudWatch Log events.

Step 1 - Create an Athena Database

If you haven't already done so, create an Athena database. Replace MyDatabase with a database name of your choice.

CREATE DATABASE MyDatabase;

Step 2 - Create an Athena Table

Replace <LOGGING-BUCKET> with the S3 logging bucket name. The S3 logging bucket name can be found in the output of the lash init command or, anytime later, in the AWS console. To find the bucket name in the AWS console, got to the CloudFormation service, select the LambdaSharp.Core CloudFormation stack corresponding to the deployment tier, and select Outputs on the tabbed view. The ARN of the S3 logging bucket is under the LoggingBucket output value. Note that the part after the last colon (:) is the bucket name.

By default, the log records are stored under the logging-success/ prefix. Adjust the prefix according to your configuration.

Finally, replace MyLogs with a table name of your choice.

CREATE EXTERNAL TABLE IF NOT EXISTS MyDatabase.MyLogs (
  `Timestamp` bigint,
  `ModuleInfo` string,
  `Module` string,
  `ModuleId` string,
  `Function` string,
  `FunctionId` string,
  `Record` string,
  `RecordType` string
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
  'serialization.format' = '1'
) LOCATION 's3://<LOGGING-BUCKET>/logging-success/';

Step 3 - Query Log Records from Athena

The following query fetches all log records for all modules and functions from the S3 logging bucket.

Make sure to replace MyDatabase and MyLogs with your database and table names, respectively.

SELECT *
FROM MyDatabase.MyLogs
ORDER BY Timestamp DESC;

Example 1: Show Usage Reports by Used Duration

The following query summarizes all UsageReport records by count, average used duration in percent, and maximum used duration in percent. This query is useful to identify Lambda functions that may be running too close to their execution limit.

Make sure to replace MyDatabase and MyLogs with your database and table names, respectively.

SELECT
  Module,
  Function,
  Count(UsedDurationPercent) AS Count,
  Round(Avg(UsedDurationPercent), 2) AS AvgUsedDurationPercent,
  Round(Max(UsedDurationPercent), 2) AS MaxUsedDurationPercent
FROM (
  SELECT
    Module,
    Function,
    (100.0 * Cast(json_extract_scalar(Record, '$.UsedDurationPercent') AS REAL)) AS UsedDurationPercent
  FROM MyDatabase.MyLogs
  WHERE RecordType = 'UsageReport'
)
GROUP BY 1, 2
ORDER BY 5 DESC;

Module	Function	Count	AvgUsedDurationPercent	MaxUsedDurationPercent
Sample.Event	SenderFunction	10	6.82	16.93
Sample.Metric	MyFunction	4	2.02	8.07
Sample.Event	ReceiverFunction	10	6.08	7.84

Example 2: Show LambdaError Reports for past 24 Hours

The following query shows all LambdaError reports for the past 24 hours by CloudFormation stack (moduleId) and Lambda function.

Make sure to replace MyDatabase and MyLogs with your database and table names, respectively.

SELECT
  from_unixtime(Timestamp / 1000.0) AS "DateTime (UTC)",
  ModuleId,
  Function,
  json_extract_scalar(Record, '$.Level') AS Level,
  json_extract_scalar(Record, '$.Message') AS Message
FROM steveb.stevebv7logs
WHERE Timestamp > (1000.0 * to_unixtime(date_add('day', -1, now()))) AND RecordType='LambdaError'
ORDER BY Timestamp DESC, ModuleId, Function

DateTime	ModuleId	Function	Level	Message
2020-05-13 23:10:08.779	SteveBv7-LambdaSharp-BadModule	FailTimeout	ERROR	Lambda timed out after 15.02 seconds
2020-05-13 23:10:03.361	SteveBv7-LambdaSharp-BadModule	FailConstructor	ERROR	An exception was thrown when the constructor for type 'BadModule.FailConstructor.Function' was invoked. Check inner exception for more details.
2020-05-13 23:10:02.816	SteveBv7-LambdaSharp-BadModule	FailConstructor	ERROR	An exception was thrown when the constructor for type 'BadModule.FailConstructor.Function' was invoked. Check inner exception for more details.
2020-05-13 23:10:02.208	SteveBv7-LambdaSharp-BadModule	FailError	ERROR	this exception was thrown on request
2020-05-13 23:10:01.582	SteveBv7-LambdaSharp-BadModule	FailBadEntryPoint	ERROR	Unable to load type 'BadModule.FailBadEntryPoint.Function' from assembly 'FailBadEntryPoint'.
2020-05-13 23:10:01.236	SteveBv7-LambdaSharp-BadModule	FailBadEntryPoint	ERROR	Unable to load type 'BadModule.FailBadEntryPoint.Function' from assembly 'FailBadEntryPoint'.
2020-05-13 23:10:00.877	SteveBv7-LambdaSharp-BadModule	FailOutOfMemory	ERROR	Exception of type 'System.OutOfMemoryException' was thrown.
2020-05-13 23:09:59.386	SteveBv7-LambdaSharp-BadModule	FailBadEntryPoint	ERROR	Unable to load type 'BadModule.FailBadEntryPoint.Function' from assembly 'FailBadEntryPoint'.
2020-05-13 23:09:58.200	SteveBv7-LambdaSharp-BadModule	FailConstructor	ERROR	An exception was thrown when the constructor for type 'BadModule.FailConstructor.Function' was invoked. Check inner exception for more details.

NOTE: Due to an issue in the Lambda runtime errors that occur either in the constructor or failure to located the constructor result in multiple entries in CloudWatch Logs.