Setting up CloudWatch for on-prem
I’m wanting to kick the tires on CloudWatch as a log collector for my home lab, but I’m finding it difficult to find on-prem instructions. The “To install and configure CloudWatch Logs on an existing Ubuntu Server, CentOS, or Red Hat instance” instructions should work, I hope.
- Created an AWS user with the AmazonDMSCloudWatchLogsRole role. Not sure if that’s the right one. Saved the key and secret securely.
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
sudo python ./awslogs-agent-setup.py --region us-east-1
- Failed. Guessing because I’m using Python 2.7.6.
- Lengthy side-trip
- Tried installing pip (the failing step) with apt
- Failed due to kernel dependencies
- boot partition full
- Tried manually clearing space on /boot
- sudo apt-get autoremove
- Target kernel versions changed at least once during various apt runs
- Copied luckily-backed-up needed files back to /boot
- apt-get <whatevs> finally succeeds
- Note that currently running and latest installed kernels set up for grub, cross fingers and reboot VM. Success!
sudo apt-get install python-pip
wants to install tons of packages including gcc, but I don’t want a dev environment on this particular vm- Update: The installer in the next step silently installed all this dev stuff I intended to skip :P
- Try the aws-provided installer again, and now it’s working
sudo python ./awslogs-agent-setup.py --region us-east-1
- Entered key and secret for logs user
- Log path /var/log/nginx/access-json.log
- Accepted sugested group name /var/log/nginx/access-json.log
- Hostname for log stream name
- Custom timestamp format
%Y-%m-%dT %H:%M:%S
(oops extra space) - From beginning of file
-
Output:
------------------------------------------------------ - Configuration file successfully saved at: /var/awslogs/etc/awslogs.conf - You can begin accessing new log events after a few moments at https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logs: - You can use 'sudo service awslogs start|stop|status|restart' to control the daemon. - To see diagnostic information for the CloudWatch Logs Agent, see /var/log/awslogs.log - You can rerun interactive setup using 'sudo python ./awslogs-agent-setup.py --region us-east-1 --only-generate-config' ------------------------------------------------------
- Don’t see the logs in CloudWatch yet, think I might need to create the group I specified
- Created /var/log/nginx/access-json.log log group in CW console
- Saw in logs that daemon unauthorized to push logs
- Edited config file to remove space from time format, because logs showed that as a problem, too
Found docs for CW logs permissionsOh, that’s for CloudTrail to push to CloudWatch- Found some other CW docs for logs permissions
Created new role in IAMNope…maybe this would work, but took a different tack- Added inline policy
- AWS Service: Amazon CloudWatch Logs
- Actions: PutLogEvents
-
ARN: (arn found in /var/log/awslogs.log)
{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1482266415000", "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:us-east-1:475273263688:log-group:/var/log/nginx/access-json.log:log-stream:router" ] } ] }
- Removed the wrong policy I originally applied
- Restarted awslogs
- Can now view logs in CloudWatch!
- Did I need to manually create the group and stream? Or would it have auto-created if I got the permissions right beforehand?
- I fed it a JSON-document-per-line file; it seems to format it sensible if I expand a line, but how do I search/display/graph fields?
- Poke around search features, very disappointed in lack of formatting/graphing options
- Google
cloudwatch vs elk | splunk
and discover Amazon Elasticsearch Service is a thing
It appears that CloudWatch Logs is a log aggregator that can be mined by other analysis tools according to my brief experience and my reddit question.