I’m wanting to kick the tires on CloudWatch as a log collector for my home lab, but I’m finding it difficult to find on-prem instructions. The “To install and configure CloudWatch Logs on an existing Ubuntu Server, CentOS, or Red Hat instance” instructions should work, I hope.

  • Created an AWS user with the AmazonDMSCloudWatchLogsRole role. Not sure if that’s the right one. Saved the key and secret securely.
  • curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
  • sudo python ./awslogs-agent-setup.py --region us-east-1
  • Failed. Guessing because I’m using Python 2.7.6.
  • Lengthy side-trip
    • Tried installing pip (the failing step) with apt
    • Failed due to kernel dependencies
    • boot partition full
    • Tried manually clearing space on /boot
    • sudo apt-get autoremove
    • Target kernel versions changed at least once during various apt runs
    • Copied luckily-backed-up needed files back to /boot
    • apt-get <whatevs> finally succeeds
    • Note that currently running and latest installed kernels set up for grub, cross fingers and reboot VM. Success!
  • sudo apt-get install python-pip wants to install tons of packages including gcc, but I don’t want a dev environment on this particular vm
    • Update: The installer in the next step silently installed all this dev stuff I intended to skip :P
  • Try the aws-provided installer again, and now it’s working sudo python ./awslogs-agent-setup.py --region us-east-1
    • Entered key and secret for logs user
    • Log path /var/log/nginx/access-json.log
    • Accepted sugested group name /var/log/nginx/access-json.log
    • Hostname for log stream name
    • Custom timestamp format %Y-%m-%dT %H:%M:%S (oops extra space)
    • From beginning of file

    • Output:

        ------------------------------------------------------
  - Configuration file successfully saved at: /var/awslogs/etc/awslogs.conf
  - You can begin accessing new log events after a few moments at https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logs:
  - You can use 'sudo service awslogs start|stop|status|restart' to control the daemon.
  - To see diagnostic information for the CloudWatch Logs Agent, see /var/log/awslogs.log
  - You can rerun interactive setup using 'sudo python ./awslogs-agent-setup.py --region us-east-1 --only-generate-config'
  ------------------------------------------------------
  • Don’t see the logs in CloudWatch yet, think I might need to create the group I specified
  • Created /var/log/nginx/access-json.log log group in CW console
  • Saw in logs that daemon unauthorized to push logs
  • Edited config file to remove space from time format, because logs showed that as a problem, too
  • Found docs for CW logs permissions Oh, that’s for CloudTrail to push to CloudWatch
  • Found some other CW docs for logs permissions
  • Created new role in IAM Nope…maybe this would work, but took a different tack
  • Added inline policy
    • AWS Service: Amazon CloudWatch Logs
    • Actions: PutLogEvents

    • ARN: (arn found in /var/log/awslogs.log)

        {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Sid": "Stmt1482266415000",
              "Effect": "Allow",
              "Action": [
                  "logs:PutLogEvents"
              ],
              "Resource": [
                  "arn:aws:logs:us-east-1:475273263688:log-group:/var/log/nginx/access-json.log:log-stream:router"
              ]
          }
      ]
  }
  • Removed the wrong policy I originally applied
  • Restarted awslogs
  • Can now view logs in CloudWatch!
    • Did I need to manually create the group and stream? Or would it have auto-created if I got the permissions right beforehand?
    • I fed it a JSON-document-per-line file; it seems to format it sensible if I expand a line, but how do I search/display/graph fields?
  • Poke around search features, very disappointed in lack of formatting/graphing options
  • Google cloudwatch vs elk | splunk and discover Amazon Elasticsearch Service is a thing

It appears that CloudWatch Logs is a log aggregator that can be mined by other analysis tools according to my brief experience and my reddit question.