Consolidated logging

rubyonrails-talk
1

I'm looking for recommendations on consolidating Rails logs from multiple servers[*] into one.

[*] AWS EC2 instances running Ubuntu 16.04 + Ruby 2.3.x/Rails 5

Bonus points for workable integration with Docker instances.

Extra bonus points if the end point is a third-party service AND it offers a HIPAA-compliant BA environment :slight_smile:

TIA!

2

try this?

3

try this? https://newrelic.com/ruby

Thanks, we already use NewRelic - even the free tier is super useful - but that has nothing to do with exporting/consolidating logs.

4

probably not getting any Bonus points for pointing out that there are existing ami-images for setting up an Elastic-Stack(https://www.elastic.co), consisting of logstash (for sending out logfiles), elasticsearch (for collecting them centrally and making them searchable), and kibana (for visualising log-events). Integrating this into a ruby/rails-app is pretty straightforward, for rails there already exist log-adapters to properly json-format log-entries. We’re also using this setup for sending out application-events we want to monitor (user logged in, purchase completed, item viewed etc.) - and then let logstash sort out where to put them. So apart from consolidating, this also comes in handy for splitting up logs into different buckets. Logstash had had its issues in the beginning, but has come a long way since Elastic has put it under its wings and now basically "just works"™.

There are also existing docker-images for the whole elastic-stack, right now we just throw logstash in with our application-container and have the elasticsearch-server run in another container.

Works quite straightforward for us, reasonably easy to set-up, covers a wide range of scenarios, works for us,

ok, now for half a bonus-point: There is also a hosted version called ElasticCloud (Sign up for the Elasticsearch Service with a free 14-day trial | Elastic), but it isn’t exactly cheap and we haven’t tested it…

5

If you’re on aws, then aws cloudwatch has a logs feature these days. It can do some things like creating metrics / alarms from logs, but it’s not anything as full featured as logstash / elasticsearch/ kibana. If you just want to store logs for possible future analysis, it’s fine though.

Fred

6

On Thu, Nov 24, 2016 at 2:17 AM, 'Stefan Frank' via Ruby on Rails:

setting up an Elastic-Stack(https://www.elastic.co),

ELK is an option since we're already using Elasticsearch for text search but I'd rather have log files stay in plain-text format.

TBD, might have to give on that... :-/

application-events we want to monitor

That sounds useful, but in this case it's already handled by another 3rd-party service because events are also generated in iOS/Android mobile clients.

Thanks for the perspective!

7

This is an option, but the AWS web UI is painful at best, and there doesn't appear to be (or I haven't found) any CloudWatch CLI that provides access to the saved logs.

The main use case is troubleshooting errors where Honeybadger doesn't provide enough context, if that helps clarify :slight_smile:

Thanks for the thoughts!

8

If you’re on aws, then aws cloudwatch has a logs feature these days. It can

do some things like creating metrics / alarms from logs, but it’s not

anything as full featured as logstash / elasticsearch/ kibana. If you just

want to store logs for possible future analysis, it’s fine though.

This is an option, but the AWS web UI is painful at best, and there

doesn’t appear to be (or I haven’t found) any CloudWatch CLI that

provides access to the saved logs.

You can export the logs to s3 ( Export log data to Amazon S3 using the AWS CLI - Amazon CloudWatch Logs )

They’ve updated the cloud watch logs UI recently (within the past month or two I think). Still a little clunky, but better than it was

Fred