Stream Rails Logs to the Google Cloud

Part 1: Replacing Rails Logger with Lograge

The stackdriver gem contains a bunch of monitoring tools:

google-cloud-debugger
google-cloud-error_reporting
google-cloud-logging
google-cloud-trace

Only google-cloud-logging is needed for logs though.

Configuring GCP

Navigate to the Google Cloud Console and create an IAM & admin → Service Account with a Logging Admin rights. Then navigate to IAM & admin → IAM and make sure the service account has a role Logging Admin, click Add otherwise.

How to stream Rails logs to the GCP

Binding a default Rails logger to the GCP is quite easy, add the following code to the config/application.rb:

require "google/cloud/logging"

credentials = Google::Cloud::Logging::Credentials.new(JSON.parse(*service_account_key*))

config.google_cloud.project_id  = *project_id*
config.google_cloud.keyfile     = credentials # or *path_to_keyfile*
config.google_cloud.use_logging = true # if you want to send non-production logs

The google-cloud-logging gem is well-documented, you can find the examples for how to:

One of the reasons logs don't appear in the Google Cloud is incorrect rights, to check what went wrong you can debug the logger writer:

$ bundle exec rails c
> Rails.logger # Must be a Google Cloud instance
> Rails.logger.info "test" # Try to create a log
...
> Rails.logger.writer.last_exception # See what went wrong

How to stream Lograge logs to the GCP

The set of existing formatters for this gem is located here: lib/lograge/formatters. We'd like to stream jsonPayload to the cloud, however if you select Lograge::Formatters::Json.new as a formatter — lograge will send a textPayload. That's because this formatter does JSON.dump(message) which transforms a Hash to a String with JSON inside(which is good if you want to send it to the STDOUT) and that String is being sent to the cloud. To have a jsonPayload in the cloud you need to send a Hash without any transformations. So let's write our own formatter, lib/lograge/formatters/json_custom.rb:

# frozen_string_literal: true

module Lograge
  module Formatters
    class JsonCustom
      # @param data [Hash] Contains the log message as key-values.
      #   In development-like environments we'd like to convert
      #   this hash to a JSON string to display in STDOUT/file/...
      #   In production-like environments we'd like to send it raw
      #   to the Google Cloud, which interprets it as JSON payload.
      def call(data)
        if Log::GCLOUD_ENVIRONMENTS.include?(Rails.env.to_sym)
          data
        else
          JSON.dump(data)
        end
      end
    end
  end
end

And configure the lograge custom events class to send JSON to STDOUT and Hash to the GCP:

# frozen_string_literal: true

# Adds logger Log.*level* methods
module Log
  extend self

  GCLOUD_ENVIRONMENTS = %i[production staging].freeze

  %i[debug info warn error fatal unknown].each do |severity|
    define_method severity do |message, params = {}|
      raise ArgumentError, "Hash is expected as 'params'" unless params.is_a?(Hash)
      payload = {
        m: message
      }.merge(params)
      unless GCLOUD_ENVIRONMENTS.include?(Rails.env.to_sym)
        payload = payload.to_json
      end
      logger.public_send(severity, payload)
    end
  end

  private

  def logger
    Rails.application.config.lograge.logger
  end
end

Setting up Google Cloud auth and lograge

The config/initializers/lograge.rb will do the GCP authentication for Log::GCLOUD_ENVIRONMENTS, otherwise set STDOUT as a logging stream. :credentials is a GCP JSON key.

# frozen_string_literal: true

if Log::GCLOUD_ENVIRONMENTS.include?(Rails.env.to_sym)
  require "google/cloud/logging"

  google_cloud_config = Rails.application.credentials.dig(:google_cloud, Rails.env.to_sym)

  credentials = Google::Cloud::Logging::Credentials.new(
    JSON.parse(google_cloud_config[:credentials])
  )

  logging = Google::Cloud::Logging.new(
    project_id: google_cloud_config[:project_id],
    credentials: credentials
  )

  resource = logging.resource("gae_app", module_id: "1")
  logger = logging.logger(Rails.env, resource, env: Rails.env.to_sym)
else
  logger = ActiveSupport::Logger.new(STDOUT)
end

Rails.application.configure do
  config.lograge.enabled = true
  config.lograge.keep_original_rails_log = true
  config.lograge.formatter = Lograge::Formatters::JsonCustom.new
  config.lograge.logger = logger
  config.lograge.ignore_actions = [
    "ActiveStorage::DiskController#show",
    "ActiveStorage::BlobsController#show",
    "ActiveStorage::RepresentationsController#show"
  ]
  config.lograge.custom_options = lambda do |event|
    {
      user_id: event.payload[:user_id]
    }
  end
end

To add user_id to the payload add the following method to the application_controller.rb:

def append_info_to_payload(payload)
  super
  payload[:user_id] = current_user&.id
end