I recently installed Mailman with Exim4, which was a challenge considering all the manual configuration you have to do. I found a few guides on the install, but they didn’t seem to be “complete” enough to get the system working. Turns out, the biggest challenge was Exim4. The configuration files are confusing, especially since there are two sets of configuration files.

I used the following guide from http://www.debian-administration.org/article/Mailman_and_Exim4 as a baseline. I’ve added my own notes to make the install go through smoother. This is probably the best guide I’ve found so far on this topic. Debian allows for easy installation of the software packages, however the configuration is all manual work.

Installing and Configuring Mailman

To install mailman, simply run the following command:

apt-get install mailman

During the install, you will be prompted to choose which languages you want mailman to support.

After the install is complete, follow the instructions given during the install and setup the Mailman-specific mailing list.

newlist mailman

There are just a few changes that must be made to the basic configuration. Open /etc/mailman/mm_cfg.py and edit the following items:

# Default domain for email addresses of newly created mailing lists
DEFAULT_EMAIL_HOST = 'list.example.org'

# Default host for the web interface of newly created mailing lists
DEFAULT_URL_HOST   = ’list.example.org’

# Uncomment this. In this setup, the alias file won’t need to be changed.
MTA=None   # Misnomer, suppresses alias output on newlist

Restart mailman so that the configuration changes take effect:

/etc/init.d/mailman restart

Now would be a good time to set up any other mailing lists you will need using the same “newlist” command. If your list will be using anything other than the DEFAULT_URL_HOST we set up earlier as its web interface hostname, make sure to pass that to newlist with the -u flag.

Exim Configuration

Previously, you had to update the /etc/alias for each list you add on Mailman. This is no longer necessary with Exim. I strongly suggest using the split config since its much easier to locate the right section in the configuration file to modify.  By default, its setup as a single file so you will need to update the setting by running “dpkg-reconfigure exim4-config“. On the next to last screen, set the configuration on multiple files vs. a single file.

Create the files listed below.

/etc/exim4/conf.d/main/04_mailman_options:
# Mailman macro definitions

# Home dir for the Mailman installation
MM_HOME=/var/lib/mailman

# User and group for Mailman
MM_UID=list
MM_GID=list

#
# Domains that your lists are in - colon separated list
# you may wish to add these into local_domains as well
domainlist mm_domains=list.example.org

# The path of the Mailman mail wrapper script
MM_WRAP=MM_HOME/mail/mailman
#
# The path of the list config file (used as a required file when
# verifying list addresses)
MM_LISTCHK=MM_HOME/lists/${lc::$local_part}/config.pck

/etc/exim4/conf.d/router/450_mailman_aliases:
mailman_router:
  driver = accept
  domains = +mm_domains
  require_files = MM_LISTCHK
  local_part_suffix_optional
  local_part_suffix = -admin : \
    -bounces   : -bounces+*  : \
    -confirm   : -confirm+*  : \
    -join      : -leave      : \
    -owner     : -request    : \
    -subscribe : -unsubscribe
  transport = mailman_transport


/etc/exim4/conf.d/transport/40_mailman_pipe:
mailman_transport:
  driver = pipe
  command = MM_WRAP \
    '${if def:local_part_suffix \
    {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
    {post}}' \
    $local_part
  current_directory = MM_HOME
  home_directory = MM_HOME
  user = MM_UID
  group = MM_GID


After you finish creating the various configuration files, run the following commands to build the updated configuration file and restart exim:
update-exim4.conf
/etc/init.d/exim4 restart

Apache Configuration

mailman uses CGI to create a web interface for its mailing lists. We need to configure Apache in order to get this piece working. First create a file to store some new aliases for the web server.

/etc/apache2/conf.d/mailman:
Alias /pipermail /var/lib/mailman/archives/public
Alias /images/mailman /usr/share/images/mailman
<directory /var/lib/mailman/archives/public>
    DirectoryIndex index.html
</directory>

Then create (or edit) a VirtualHost entry to allow the scripts to run.

/etc/apache2/sites-available/list.example.org:
<virtualhost *:80>
        ServerName list.example.org
        ServerAdmin webmaster@list.example.org
        DocumentRoot /var/www/
        <directory /var/www/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
                # This directive allows us to have apache2's default start page
                # in /apache2-default/, but still have / go to the right place
                RedirectMatch ^/$ /cgi-bin/mailman/listinfo
        </directory>
If this is a new file, remember to symlink it to the sites-enabled directory.

        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        <directory ”/usr/lib/cgi-bin”>
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        </directory>
</virtualhost>

Finally, restart Apache so that the changes take effect.
/etc/init.d/apache2 restart

Mailman troubleshooting:

If your lists are not showing up on the website, you will need to change
/var/lib/mailman/Mailman/Defaults.py

In the section “VIRTUAL_HOST_OVERVIEW“, set it to “No“.
Restart Mailman /etc/init.d/mailman restart

Locking down the Create Lists “feature”
e.g. If you have the proper authority, you can also create a new mailing list
By default, Mailman leaves the Create Lists feature wide open so anyone can create new lists. For most places, this is a bad thing.  To lock it down, go to your Apache2 configuration to block access to the folder.  Go to /etc/apache2/site-enabled/000-default (if you only have 1 web host on the server) and find the section before

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/

Add the following above the ScriptAlias line:

<Location /cgi-bin/mailman/create>
Allow from [YOUR IP]
Deny from all
</Location>

Restart Apache2 and the creation of lists should now be limited to your computer only.

Exim4 troubleshooting:

ERROR: Low level smtp error: (111,’Connection refused’)

This basically means a few things.

First, is your hostname a FQDN? Fully qualified domain name. Basically, it needs to be “mysite.com” as opposed to just “mysite”.  Go to /etc/hosts file. Is 127.0.0.1 setup as localhost and 127.0.1.1 setup as mydomain.com?

Second, your firewall could be interferring. If your iptables rules are too strict, then you won’t be able to connect to your own smtp port.  To test, run “telnet localhost 25″
If you get a connection refused or it hangs, then this means a few things.

Third, Exim4 is not configured properly. Run “dpkg-reconfigure exim4-config”
Make sure your sendmail is setup to be an “internet site” so that other computers can send email to it. Set relay domains to blank. And make sure the configration is setup using the multiple configuration files.

ERROR: 550 relay not permitted
This has to do with Mailman unable to send email to the clients due to the relay setting in Exim not being configured properly. If you get this far, Exim is able to take the e-mail posts and route it to the Mailman. In Exim, make sure localhost has permission to post messages. Run “dpkg-reconfigure exim4-config“  In the relay network option, set them to: “127.0.1.1;127.0.0.1″. This will allow your computer to send email out. By default, Mailman sends everything through localhost, so it needs to match up with Exim. To verify whether you can connect, make sure you can “telnet localhost 25″.

One thing I’ve encountered with Awstats is adding in old apache log files. This is useful if you are migrating data from servers. By default, Awstats will ignore ALL past dates in the log that occur before the most recent log entry date. In order to avoid Awstats from finding the “recent log entry date”, you will need move all Awstats cached files into a separate folder. I’ve verified that this works, so if you have any questions feel free to comment.

1) Locate your awstats data directory. Most commonly found in /var/lib/awstats

2) Create a new folder inside the directory. Move all awstats* files to the new folder.

3) Run  awstats update process on all log files in chronological order. AWStats will not complain about the “too old record” because there is no history files in DirData directory that contains compiled data more recent than record.

Edit the awstats.conf inside /etc/awstats
Locate the Apache logfile you need. It must be the oldest one you want to import. Awstats will chronologically add in the files, so make sure you add them in the right order. Edit the line starting with LogFile=”[log file location]”

Run the Awstats command (through the web browser or through shell via the perl command line)

4) Move the history files back. Once you process through all the logfiles, move the Awstats history files inside the /var/lib/awstats/newfolder back into /var/lib/awstats

Here’s a quick tip to make cleaner looking URLs in CodeIgniter PHP framework. By default, if you have an application, the path of the application usually ends after index.php.  Using Apache RewriteEngine, you can make cleaner looking URLs. You can have something similar to mysite.com/cigniter/MyApplication instead of mysite.com/cigniter/index.php/MyApplication

• Create a file named “.htaccess” inside your root CodeIgniter directory. It should look like the following:RewriteEngine on
  RewriteCond $1 !^(index\.php|images|css|robots\.txt)
  RewriteRule ^(.*)$ /[CODE IGNITER BASE]/index.php/$1 [L]where [CODE IGNITER BASE]  is the directory of your Code Igniter install. For instance, if you named your base install folder “cigniter”, the Rewrite Rule would look like:RewriteRule ^(.*)$ /cigniter/index.php/$1 [L]
• If you get an error, make sure your Apache install allows for RewriteEngine in that particular folder. This is usually located in /etc/apache2/sites-enabled/YOURSITE The folder needs to have permissions of:AllowOverride AuthConfig
  Options +FollowSymlinksSee: http://www.whoopis.com/howtos/apache-rewrite.html
• More info on CodeIgniter framework:
  http://codeigniter.com/

I recently implemented a way to limit access by IP range on specific blogs on WordPress MU. As you know, WordPress MU uses Apache Rewrite engine to rewrite URLs. For instance, you have a blog on WordPress MU called “intranet”. Apache Rewrite takes the “intranet” string in the URL and automatically rewrites it as a value in the PHP script. A side effect to Apache Rewrite is that “Directory” .htaccess parameters don’t work. So, if you wanted only your company IPs to access an internal blog, you will need to use Apache Rewrite parameters instead.

Here’s how to limit access to an IP or subnet on a particular blog on your WordPress MU install:

DISCLAIMER: Modifying .htaccess files can break your WordPress MU install. ALWAYS backup your .htaccess file. Simply, copy .htaccess file and rename it to .htaccess-backup. (cp .htaccess .htaccess-backup)

Step 1:

Add a section after the “RewriteEngine On
RewriteBase /…” section

RewriteCond %{REMOTE_ADDR} !^1\.2\.3\.4
RewriteCond %{REMOTE_ADDR} !^1\.2\.3\.5
RewriteCond %{REQUEST_URI} ^/BLOG1
RewriteRule .* - [F]


DO NOT SAVE YET. You will need edit the IP address info and blog info first:

Step 2: Replace the 1.2.3.4 number with your company IPs (its easier if you have an entire subnet or you can use internal IPs)

For a class C, the part after {REMOTE ADDR} would be
!^1\.2\3\.

Step 3: Replace BLOG1 with the blog you want to limit access to those IPs. So http://www.mysite.com/myfirstblog would be “^/blogs/

Optional: If you have WordPressMU installed in a folder (e.g. not your root folder), you will need to append the directory in front of the blog name.

For instance, if you have http://mysite.com/blog (as your WordPress MU root folder) the ^/ BLOG1 would be

^/blog/BLOG1

Optional: Multiple blogs with same access restrictions
By default, the Apache Rewrite treats every line as an AND statement. If you have multiple blogs, you will need to have an [OR] at the end of the line.

RewriteCond %{REQUEST_URI} ^/BLOG1 [OR]
RewriteCond %{REQUEST_URI} ^/BLOG2

Errors?

• Make sure you have ^/ marks in front of the blog names
• IP addresses must have a backslash before each dot. Regular expression for dot is concatenate by default, so it needs to be escaped
• Make sure you don’t forget the !^ sign before the IP, otherwise you will be forbidden.

If all else fails, if can’t fix the error, just copy back the .htaccess-backup to the .htaccess file.

Sitemaps are useful if you want search engines to look in specific directories of your website. The standard robots.txt notation only has the exclusion list; where not to look and the search frequency.

For instance, a really basic robots.txt file looks like this:

User-agent: *
Crawl-delay: 3
Disallow:/cgi-bin/


For me, I set the Crawl-delay to 3 as a general rule to prevent crawlers from consuming all the web server bandwidth. Generally, Yahoo crawlers are the most aggressive on your site, Google averages about ~13 seconds per request. Anyway, a sitemap gives the crawler a better idea of where to search, rather than trying to discover on its own by looking at the root file.

Here’s a great resource for generating sitemaps:
http://code.google.com/p/sitemap-generators/wiki/SitemapGenerators

About Google Sitemap Generator

Our new open-source Google Sitemap Generator finds new and modified URLs based on your webserver’s traffic, its log files, or the files found on the server. By combining these methods, Google Sitemap Generator can be very fast in finding these URLs and calculating relevant metadata, thereby making your Sitemap files as effective as possible. Once Google Sitemap Generator has collected the URLs, it can create the following Sitemap files for you:

The fastest way to trim down large web log files is through UNIX/Linux shell. Large files exceeding 1 GB (millions of lines of logs) are not easily editable using a GUI interface, so the fastest way is to parse them is via command line. You can trim them down according to a time range, remove internal requests from within the company, and remove bots/crawlers data from the log files.

For instance, I have a 4GB log file with about two years worth of info (2007-2009) in there. What if I just wanted the logs from 2008? First, run the “head -10″ command on the log file to see what the general format of the log is.

Preferably, it is in the YYYY-MM-DD hour:min:second format. If you are building a custom log file from scratch, make sure you have the dash delimiter between the dates to make it easier to work with after. For me, the log file that I was given was in the YYYYMMDDHHMMSS (e.g. timestamp format) Luckily, with regular expressions, you can still parse the file accordingly.

20070101000013 4ms 0ms 8ms
20070101000019 4ms 0ms 8ms

Make sure you have enough disk space before proceeding, at least enough to handle twice the size of the logfile. Run a “df” to check available space and “ls -lah” to see the size of the files in your log directory.
So, if you only want the logs from 2008 (using the example above) you can run the following command.

cat [INPUTLOGFILE] | grep '^2008' > [OUTPUTLOGFILE]

This will look through each line, run a regular expression match with 2008 at the beginning of the line. If the logfile does not start with the date, you can still run a match if the date appears anywhere in the line (if in the YYYY-DD-MM format)

cat [INPUTLOGFILE] | grep '2008-' > [OUTPUTLOGFILE]

What if you want to omit googlebots and crawlers directly from the log file? Most log analysis programs have a filter that looks in the user agent string to check if the source is from a crawler. But, not all crawlers use the user agent string. Some set the user agent string to a common browser. If you have dns lookups on, you can try filtering logs that match the domain googlebot.com . The “grep -v ” command is handy since it excludes all lines that match that particular value.

cat [INPUTLOGFILE] | grep -v 'googlebot.com' > [OUTPUTLOGFILE]

Another use is to filter out all local requests coming from your department/building/company. This would be good to see where your external users are coming from. If you have DNS turned on:

cat [INPUTLOGFILE] | grep -v 'mycompany.com' > [OUTPUTLOGFILE]

If you don’t have DNS turned on, you can filter by IP range

cat [INPUTLOGFILE] | grep -v '123.456.789.' > [OUTPUTLOGFILE]

Check the number of lines in the logfile (before and after comparison) with:
wc -l [LOGFILE]

For more information, checkout these resources:
http://www.robelle.com/smugbook/regexpr.html

What if you don’t have access to UNIX/Linux? There’s Cygwin for Windows, although I don’t recommend running it with huge log files. I’ll eventually try it at some point to see if it’ll work
http://www.cygwin.com/

If you want to remove image files from your awstats reports, modify the SkipFiles variable in the /etc/awstats/awstats.conf (or awstats.YOURHOST.conf)

Do a search for “SkipFiles” in the file using nano/vi/emacs and find the section that talks about “Use SkipFiles to ignore access to URLs that match one of the following entries…” The SkipFiles line should look similar to the following:

SkipFiles="REGEX[.jpg$] REGEX[.gif$] REGEX[.png$]"


Installing awstats in Debian with GeoIP caching

First, use apt-get to get the software:

apt-get install awstats

Configure awstats:
Instead of using the awstats configure tool (written in Perl), manually add in the awstats Apache configuration. Using the tutorial from debuntu, make a file inside your apache config folder called awstats.conf with the following lines.

Alias /awstatsclasses "/usr/share/awstats/lib/"
Alias /awstats-icon/ "/usr/share/awstats/icon/"
Alias /awstatscss "/usr/share/doc/awstats/examples/css"
Options ExecCGI -MultiViews +SymLinksIfOwnerMatch

Inside the Apache2.conf file (or virtual host conf file inside the sites-available folder) add this to the end of the file:

Include /etc/apache2/awstats.conf

Next, copy the /usr/lib/cgi-bin/awstats.pl file to the apache cgi-bin folder. If you have virtual hosts enabled, copy it to the cgi-bin located in the designated virtual hosts folder.

Configure the awstats.conf sample file located in /etc/awstats/awstats.conf
1) Set the LogFile to the server’s logfile directory (usually /var/log/apache2/access.log)
2) Set LogFormat to 1.
3) Set DNSLookup=0 (No DNS Lookup is necessary when you use GeoIP DB, instructions are below)
4) Uncomment this line (remove the # from the beginning)
LoadPlugin=”geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat”

Install GeoIP to speed up the hostname lookups. This will significanly improve the performance since DNS lookups will generally take a long time. Using this tutorial, install the GeoIP library:

wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.5.tar.gz

Download and extract the file.  Inside the GeoIP 1.4.5 folder, run:

./configure
make
make check
make install


Next, download the latest GeoIP perl module -and- latest GeoLite country database.
http://www.maxmind.com/download/geoip/api/perl/ I usually run wget to download the file from shell. wget http://geolite.maxmind.com/download/geoip/api/perl/Geo-IP-1.36.tar.gz

Extract the tar file and inside the Geo-IP folder, run:

perl Makefile.PL
make
make test
make install

GeoIP should now be installed. You can now generate the reports:
/usr/lib/cgi-bin/awstats.pl -config=[NAME OF YOUR INSTALL] -update
For instance, if you called the install “sample”, you should have a file called /etc/awstats/awstats.sample.conf    The command for sample would be:
/usr/lib/cgi-bin/awstats.pl -config=sample -update

To run awstats:
http://[SERVER]/cgi-bin/awstats.pl?config=[NAME OF YOUR INSTALL]