Unified Rails Logging with Syslog-ng

Posted 8 months ago

Unified rails logging is an incredibly helpful tool, and isn’t that difficult to accomplish. Unfortunately, searches for the term come up mostly blank. Tim Lucas wrote a great article on using Eric Hodel’s SyslogLogger, but what’s missing are instructions on how to unify those logs on a central logging server. Don’t fear, this is a rather simple process with syslog-ng!

Syslog-ng is the “next generation” syslog drop in. It provides a much more advanced feature set than the default system syslog, and allows you to unify multiple log sources into single file with simple to manage directories. The configuration for syslog-ng can be a bit confusing at first, so we thought we would take the opportunity to document this setup from one of our customers installations. Hopefully this will help others create unified rails logs.

This article is written with a few assumptions. First, I’m assuming you’re running a rails application that has multiple application servers. This could be both a “staging” and “production” server, or multiple production servers. Second, I’m assuming you’ve got a dedicated host you want to use as the receiving server. We normally recommend this to be a specialty slice and often, a customer will put this on their repository server.

The final setup will have your Rails application logging to the local syslog-ng setup on the same server as the rails application. That local syslog-ng will then replicate the logs out to the remote syslog-ng server which will collect and sort them into a shared log file.

These steps should be fairly straight forward, but if you have any questions, please don’t hesitate to contact us at support@blueboxgrp.com!

If you do your rails hosting with Blue Box Group, then you’re most likely running CentOS. If that’s the case, you can install syslog-ng with a simple yum command. You’ll then want to set it to start on boot!

[root@ staging-app01 /]# yum install syslog-ng 
<snip>
Installed: syslog-ng.x86_64 0:2.1.4-1.el5 
Complete!

[root@staging-app01 /]# /sbin/chkconfig syslog-ng on 

This process should be completed on all your application servers, as well as on the server you’re going to use as the central log box.

If you’re using a different distribution, you’ll want to use the appropriate tools (apt-get, etc) to install the syslog-ng package.

The next step is to configure syslog-ng on each machine. As stated in the assumptions, there are two different types of configurations you need to worry about. The first configuration runs on your application servers and is what your Rails application talks to. The second runs on the collection server. I’ve laid out the configuration for both below.

Note: On CentOS, these configs are located in /etc/syslog-ng/syslog-ng.conf.

Our example application server configuration is below. The important part you need to modify for your own purposes is contained at the bottom of the file and is annotated with comments. The top section tells syslog-ng to route the various system log messages to the appropriate files (much like the system syslog daemon would do). The bottom section then defines a rails-* facility and tells syslog-ng to send those logs to the IP address you specify.

options { 
    chain_hostnames(off); 
    long_hostnames (on); 
    use_fqdn (on); 
    sync (0); 

    stats(43200); 
    log_msg_size(1048576); 
}; 

source s_sys { 
    file ("/proc/kmsg" log_prefix("kernel: ")); 
    unix-stream ("/dev/log"); 
    internal(); 
}; 

destination d_cons { file("/dev/console"); }; 
destination d_mesg { file("/var/log/messages"); }; 
destination d_auth { file("/var/log/secure"); }; 
destination d_mail { file("/var/log/maillog" sync(10)); }; 
destination d_spol { file("/var/log/spooler"); }; 
destination d_boot { file("/var/log/boot.log"); }; 
destination d_cron { file("/var/log/cron"); }; 
destination d_kern { file("/var/log/kern"); }; 
destination d_mlal { usertty("*"); }; 

filter f_kernel     { facility(kern); }; 
filter f_default    { level(info..emerg) and 
                        not (facility(mail) 
                        or facility(authpriv) 
                        or facility(cron)); }; 
filter f_auth       { facility(authpriv); }; 
filter f_mail       { facility(mail); }; 
filter f_emergency  { level(emerg); }; 
filter f_news       { facility(uucp) or 
                        (facility(news) 
                        and level(crit..emerg)); }; 
filter f_boot   { facility(local7); }; 
filter f_cron   { facility(cron); }; 

log { source(s_sys); filter(f_kernel); destination(d_kern); }; 
log { source(s_sys); filter(f_default); destination(d_mesg); }; 
log { source(s_sys); filter(f_auth); destination(d_auth); }; 
log { source(s_sys); filter(f_mail); destination(d_mail); }; 
log { source(s_sys); filter(f_emergency); destination(d_mlal); }; 
log { source(s_sys); filter(f_news); destination(d_spol); }; 
log { source(s_sys); filter(f_boot); destination(d_boot); }; 
log { source(s_sys); filter(f_cron); destination(d_cron); }; 

# BBG - Configure the rails portion of our syslog-ng setup! 
# This section can be left as is. 
log {    source(s_sys); 
    filter(f_rails_apps); 
    destination(log_server); 
    flags(final); }; 

# You should create a filter here that will catch your facility 
# you will define within your rails application. 
filter f_rails_apps { program("rails-*"); }; 

# You should set your IP address / port of the remote logging server. 
destination log_server { 
    udp("10.1.1.2" port(515)); 
}; 

The logging server is the machine that will collect the log files from the network and combine them into a unified logging server. We define four custom items within this configuration: a source, a destination, a filter, and a log entry. The source tells syslog-ng to listen on an accessible IP (this should either be an internal IP that’s non routable, or should be protected with firewall rules). The destination tells syslog-ng where it should write the combined log files to. The filter is what we use to parse out our rails specific data, and the log line takes those above elements and combines them so syslog-ng knows what to do.

options { 
    chain_hostnames(off); 
    sync (0); 
    dir_owner(root); 
    dir_group(logs); 
    stats(43200); 
    use_dns(yes); 
    dns_cache(yes); 
    dns_cache_size(100); 
    dns_cache_expire(3600); 
    dns_cache_expire_failed(600); 
    keep_hostname(yes); 
    long_hostnames(on); 
    use_fqdn(no); 
    log_msg_size(1048576); 
    log_fifo_size (1000); 
}; 

source s_sys { 
    file ("/proc/kmsg" log_prefix("kernel: ")); 
    unix-stream ("/dev/log"); 
    internal(); 
}; 

destination d_cons { file("/dev/console"); }; 
destination d_mesg { file("/var/log/messages"); }; 
destination d_auth { file("/var/log/secure"); }; 
destination d_mail { file("/var/log/maillog" sync(10)); }; 
destination d_spol { file("/var/log/spooler"); }; 
destination d_boot { file("/var/log/boot.log"); }; 
destination d_cron { file("/var/log/cron"); }; 
destination d_kern { file("/var/log/kern"); }; 
destination d_mlal { usertty("***"); }; 

filter f_kernel     { facility(kern); }; 
filter f_default    { level(info..emerg) and 
                        not (facility(mail) 
                        or facility(authpriv) 
                        or facility(cron)); }; 
filter f_auth       { facility(authpriv); }; 
filter f_mail       { facility(mail); }; 
filter f_emergency  { level(emerg); }; 
filter f_news       { facility(uucp) or 
                        (facility(news) 
                        and level(crit..emerg)); }; 
filter f_boot   { facility(local7); }; 
filter f_cron   { facility(cron); }; 

log { source(s_sys); filter(f_kernel); destination(d_kern); }; 
log { source(s_sys); filter(f_default); destination(d_mesg); }; 
log { source(s_sys); filter(f_auth); destination(d_auth); }; 
log { source(s_sys); filter(f_mail); destination(d_mail); }; 
log { source(s_sys); filter(f_emergency); destination(d_mlal); }; 
log { source(s_sys); filter(f_news); destination(d_spol); }; 
log { source(s_sys); filter(f_boot); destination(d_boot); }; 
log { source(s_sys); filter(f_cron); destination(d_cron); }; 

# BBG - Define a source for log files.  This causes syslog-ng to
# listen on an IP that then allows your application server
# syslog-ng servers to talk to. 
source app_cluster { 
    udp( ip(10.1.1.2)); 
}; 

# BBG - Define a destination.  Our filter will then map anything
# from our above source to this destination. 
destination d_rails_apps { 
    file("/var/log/rails/prod/$PROGRAM/$YEAR-$MONTH/$DAY/$PROGRAM-$YEAR$MONTH$DAY" 
        owner("deploy") 
        group("logs") 
        perm(0660) 
        dir_perm(0770) 
        create_dirs(yes) 
    ); 
}; 

# BBG - Define our filter that we'll use to map our logs. 
filter f_rails_apps { program("rails-***"); }; 

# BBG - Now use the above source and destination to route our
# log file to the appropriate place. 
log { source(app_cluster); 
    filter (f_rails_apps); 
    destination(d_rails_apps); 
    flags(final); };

On each one of your servers, start syslog-ng. It should start cleanly:

[root@log rails]# /etc/init.d/syslog-ng start 
Starting syslog-ng:                                        [  OK  ] 

If you get errors, it probably means you typo’d something in your configuration. Take a look at the output, fix the error, and try restarting syslog-ng.

On your application servers, install the syslog_logger gem:

[root@staging-app01 environments]# gem install SyslogLogger 
Successfully installed SyslogLogger-1.4.0 
1 gem installed 
Installing ri documentation for SyslogLogger-1.4.0... 
Installing RDoc documentation for SyslogLogger-1.4.0... 

Within your application, you’ll want to add the following to your config/environments/production.rb (or config/environments/staging.rb). You can pass an argument to SyslogLogger.new to define the “facility” to use if you want to separate staging logs from production logs. Make sure to start your facility with rails- so our filters above work.

It should be noted that our syslog-ng configuration will separate apps based on this token, so if you’re running a staging and production app, you’d want to name them uniquely: rails-my_app-staging / rails-my_app-prod.

# Use a different logger for distributed setups 
config.gem 'SyslogLogger', :lib => 'syslog_logger' 
require 'syslog_logger' 
config.logger = SyslogLogger.new ('rails-my_app-staging')

With a simple application restart, you should now have unified logs! Take a peak in /var/log/rails on your logging server and you should have as much data as you care to digest.

That’s it! You now should have a unified production log. This should simplify debugging for production errors and generally make your life much happier.

Additionally, you can now start to play with cool features such as the Production Log Analyzer

- Jesse Proudman
Blue Box Group, LLC
Rails Hosting without Headaches

Ruby Big Decimal Vulnerability

Posted 9 months ago

Yesterday, both the Rails core team and the Ruby team announce a vulnerability in the Big Decimal libraries of Ruby. This vulnerability does not allow for remote access to your data, however it can allow an attacker to create a Denial of Service attack on your application, essentially rendering it down. To quote the Ruby Team…

We’ve promptly patched our Ruby RPMs and posted information on how to update in our System Status blog. For users on Debian or Ubuntu, updates should be hitting their repos over the next few days. If you can’t upgrade, we recommend you implement the work arounds provided by the Rails core team. Information on those can be found here:

http://weblog.rubyonrails.org/2009/6/10/dos-vulnerability-in-ruby

Our patching instructions for our RPM versions are available here:

http://www.blueboxgrp.com/system_status/2009/06/ruby_security_bug

We strongly recommend for all of our customers to follow those instructions as soon as possible. For assistance, please don’t hesitate to contact us.

Thanks!

- Jesse Proudman
Blue Box Group

5 Things You Didn't Know You Could Do With Box Panel

Posted 9 months ago

Probably my favorite thing about my life at Blue Box is getting to work alongside a team that’s really dedicated to improving the quality of the services we provide. Scarcely a day goes by where we don’t have some kind of update to our systems that makes life a little easier for both our customers and our technicians. That said, one of the tools that’s changed the most since I started working here has been the Box Panel.

Our long term goal is to make the Box Panel your one-stop resource for everything related to account management, and it’s become that in more ways than I could have initially imagined. The only problem is, with things progressing at such a fast pace, it’s easy to overlook new features as they are added. With that in mind, I’ve decided to put together a little list of useful tools you might not have even known you had.

1. E-Mail Administration

• Vacation Messages – Once turned on you can set an automated response to be sent to your contacts when you’re unavailable, and you can set a date to have your message automatically disable itself.
• Automatic CC – If you want your e-mail to also go to another address, simply define it here, and you’re good to go
• Simple Spam Filter – If you have spam problems, but don’t want to pay extra for postini, the simple spam filter may be your next best bet. It uses a filtering tool called SpamAssassin to automatically move spam messages to a “Junk Mail” folder. It’s not as robust or flexible as postini, but it has the upshot of being a more than adequate free alternative.

2. Bandwidth Data/New Relic

As your site grows, it’s going to become more and more important to have good data on how it’s performing. Under the “Monitoring and Reporting” group (all the way on the bottom of your Box Panel page) are bandwidth data and graphs, which will tell you how much data your servers push to and receive from the internet, and New Relic RPM which is an invaluable tool for easily finding bottlenecks in your applications.

We announced our partnership with New Relic a few months back and have had a great response from developers who’ve used these tools to help tune their applications. The best part is, not only can you associate your New Relic account with Box Panel (whether it’s pre-existing or newly created) but you can also get a free month trial of their gold membership plan for hosting with Blue Box. Be sure and let us know if you’re interested!

3. Registering Domains

In the domain names section of Box Panel, you can Update your DNS and renew your domains at the click of the button. It’s also easier than ever to register domains! While you could always check domain name availability and add domains from the Box Panel, until recently if you wanted to add more than one domain at a time, you’d have to order each individually. With that in mind we’ve implemented a cart system—when you search for a domain and want to purchase it, it will add it to an online shopping cart and will allow you to continue adding domains until you’re ready to check out.

4. Online Chat

One of our most exciting new Box Panel features is the live support chat. Since our initial announcement we’ve seen a lot of positive feedback from customers who have appreciated having another way to get in touch with us, and the chat interface is fantastic for quickly copying and pasting quick links to customers’ sites or error messages they might need help troubleshooting. Keep in mind that the service is still in beta, and we’re still ironing out the bugs, but so far it’s been working like a charm. What are you waiting for—check it out!

5. Quick help links

Probably the best kept secret of the Box Panel is the handy online support button in the upper right-hand corner. Take a look up there the next time you login—you should see something like this:

Our contact number, of course, needs no explanation, but what you might not know about that support link is that it’s context specific to each page of the Box Panel. If you see any features that you’re not entirely sure how to use, click that button and you’ll be taken to our public knowledge base article for that specific service. For example, if you were to navigate to the DNS administration page, but aren’t entirely sure how to add entries for your domain, when you click that button, you’ll be instantly taken here! It can really be a life saver when you just have a quick question, and don’t want to search through the knowledge base to find an answer.

The Box Panel is constantly being updated and improved, so hopefully this has been a helpful quick list of features, but as ever if you find anything you have questions about, and you can’t seem to find the answers online, feel free to let us know and maybe it’ll be about time for us to throw together the top 10 things you didn’t know you could do with Box Panel!

-Nick Rycar
Blue Box Group Systems Administrator