IMF Tune - Bringing Back the Exchange Connection Filter
Configuring IMF Tune v5.5 Moderator/Reporting
The IMF Tune v5.5 server configuration includes the necessary interface for administering the Web Moderator/Reporting component. This document goes through the configuration elements relevant to this component.
In Installing IMF Tune v5.5 Moderator/Reporting we walked through the setup of the new Moderation and Reporting functionality. The setup discussed is depicted below:
Once the setup is completed, we can proceed with configuring the IMF Tune server(s) for them to start publishing emails at the database server.
The interaction between an IMF Tune server and the database starts from the registration process. This is done from the Configuration |Quarantine | Quarantine Database category and establishes a connection between an IMF Tune server and a database server.
We already discussed the Registration process in Installing IMF Tune v5.5 Moderator/Reporting. So we won't discuss this process any further here.
The web component does not include its own configuration interface. Instead, all settings are available from the IMF Tune server configuration. Thus, even if the IMF Tune server and Web interface are installed on different machines, we are still able to manage everything from a single interface.
Before delving into the specific configuration options, let's have a look at how configuration settings are being stored.
Each IMF Tune server has its own local database where most settings are stored. At the database server we store the list of Users that are allowed to access the web interface. Being centralized, all IMF Tune servers share the same User configuration.
As already discussed the Moderator/Reporting is controlled from the IMF Tune server configuration interface. We will now go through the various setting relevant to email quarantining and reporting. The following image highlights the configuration categories relevant to our discussion.
The categories Archiving and Disk Maintenance | Archives have been renamed to Archiving/Quarantine and Disk Maintenance | Archives/Quarantine. The new names highlight the changes made to accommodate additional options for email Quarantining. Furthermore we also have two new categories:
One of the most significant design changes introduced in IMF Tune v5.5 is the shift from email disk archiving to SQL database quarantining. Disk archiving is still 100% backwards compatible. We can easily implement archiving exactly in the same manner as we did in previous releases. However we now have the option to do more by publishing archived emails to a central database server.
The updated Archiving category, now renamed to Archiving/Quarantining, is the place where we identify which emails are to be published to the database server.
Select the Archiving/Quarantining category and click Add to open a new Archiving Profile.
This dialog is nearly identical to what we used to have in earlier versions with a single addition, the checkbox 'Also add archived email to quarantine database'
If we do not set this checkbox, we have the traditional disk archiving where emails are saved to the specified HDD location. On selecting this option, we instruct IMF Tune to also publish emails to the database server.
IMF Tune has always been able to archive all emails, both those blocked as spam and those accepted as legitimate. This feature is now inherited by the database server. At the Archive profile select the SCL configuration page:
Here we select the SCL ratings to which this Archiving profile applies. SCLs 7, 8, 9 and blacklisted are being blocked as spam. Thus we select their checkboxes so that emails with these ratings are published to the database for review.
Similarly, selecting the Whitelisted checkbox, instructs IMF Tune to also publish whitelisted emails. It really is totally up to us. We can choose to only publish emails blocked at the server, but we can also choose to publish emails that were delivered to the user mailboxes.
Note: The web interface includes a very simple switch that allows us to quickly hide accepted emails.
One reason why we might want to publish accepted emails to the database is to analyze the type of emails being assigned midrange SCL ratings, for example SCL4. It is not unusual for us to receive support questions on how to best handle emails assigned such ratings. By publishing these emails, an administrator may gain better understanding of the type of emails falling in this category and thus choose the most appropriate email handling action.
The Archiving interface should immediately highlight the tight coupling that exists between disk archiving and the database server. Indeed for an email to be published to the database a copy must also be archived to disk.
The database server is not fed with a full copy of the email. Most notably the database is only supplied with up to 32Kb of body text, no html body content and no file attachments. These potentially large pieces of data are kept at the disk archive.
Let's see what happens when a user, through the IMF Tune Web interface, releases a quarantined email for delivery. In this case, the IMF Tune server fulfills the request by fetching the email from the disk archive and submits the email for delivery.
Furthering the twinning between disk archiving and the database server we now look at the changes available under the Maintenance | Archives/Quarantine category.
At the top of Maintenance | Archives/Quarantine we find a Warning saying:
If the Archive Maintenance option
At the bottom of Maintenance | Archives/Quarantine category we have a new checkbox:
This option determines whether archived emails should be immediately deleted from disk as soon as these are moderated from the Web interface. Consider a user resubmitting and deleting emails at the moderator. IMF Tune on fulfilling these operations removes the emails from the database. In addition it checks this setting to determine whether or not a copy of the email should be retained on disk.
Retaining emails on disk is useful in a multi-layer backup system where a predictable audit trail is required. You can choose to keep all emails on disk until the archives are backed up and later purged by disk maintenance. This renders the availability of emails archived to disk predictable. If we need to go back and dig some old email from the disk archive, it will still be available.
The last option at the bottom of Maintenance | Archives/Quarantine category is:
This is the age limit for email retention at the database server. Emails reaching this limit are purged completely. Note how there is no way to disable this purging and the maximum value here is 999 days.
Some might be tempted to employ the IMF Tune database as some kind of permanent email archive. However the Quarantine functionality has not been designed for this purpose. Thus we discourage employing the system in this manner.
The option reads 'Retain quarantine information for reporting purposes' because this age limit determines the number of days reporting information is retained within the database. In other words the number of days configured here will determine the span of time covered by all reports. As an example let's say we set this to 30 days. The bar chart showing the number of accepted, rerouted, rejected and deleted emails will show the totals for the last 30 days.
We now move to the new Quarantine configuration nodes starting from the Quarantine Database category.
The top most configuration elements are all related to the IMF Tune server registration/un-registration process. This is the process that connects/disconnects an IMF Tune server to/from the database.
At the top we have two read-only fields, SQL Server and Database. These identify the database against which IMF Tune is currently registered or was last registered with. On the right we have a button that will read 'Register' or 'Unregister' depending on the current IMF Tune server state. If the server is currently not registered the button will read 'Register' and vice versa. Here is the dialog that pops on clicking the button.
The registration process is discussed in detail in Installing IMF Tune v5.5 Moderator/Reporting. The un-registration dialog is identical, but performs a clean disconnection from the database server.
In Exchange 2003 the default path is that of the SMTP Virtual Server pickup directory:
In Exchange 2007 and Exchange 2010, IMF Tune is installed on the server running the Hub or Edge Transport role. In this case we set the path to the transport replay directory:
In case we are only interested in using the Web interface as an email moderation tool, it is a good idea to disable this functionality by clearing the checkbox. IMF Tune will still generate reports however these will only take into consideration the set of emails published to the database through the Archiving profiles configured under the Archiving/Quarantining category. Thus the reports won't include the complete set of processed emails. Disabling this functionality will reduce the amount of data we load to the database server, hence reducing resource consumption.
Emails loaded to the database exclusively for reporting, will not be visible at the Web Moderator interface. This information is retained at the database until it reaches the age specified at:
On reaching the age limit the information is purged from the database.
The Moderator Web interface allows users to Approve (resubmit) and Delete quarantined emails permanently. These tasks are fulfilled by the IMF Tune server that was responsible for publishing the email at the database. For the best performance, this processing is done in batches.
The publishing of new email information at the database server is also done in batches. The volume of information uploaded, depends on the Exchange server email load. If we enable
The settings discussed so far control the interaction between an IMF Tune server and the database server. These are stored locally at the IMF Tune server machine. Installations running multiple IMF Tune servers will thus have a set of settings for each server.
We now take a look at the User configuration, a list of users stored at the database server.
Being stored at the database, this list only becomes accessible once IMF Tune is registered with the database server as discussed in Registering IMF Tune Servers with the Database Server.
Here users are granted access rights over the Web interface. To create a new user, click on Add.
This opens the Add/Edit User wizard. In the first step we specify the login credentials and the initial enablement status for the new user. If enabled, the user is allowed to login to the Web interface. Otherwise, it is blocked as if the user did not exist.
Clicking Next we move to the Rights assignment step.
IMF Tune includes a set of predefined Roles available from the top most combo box list. A role is simply a set of rights that you will most often want to assign:
Role | User
Role | User Read Only
Role | Admin (full access rights)
Role | Admin Read Only
Apart for the fixed Roles, it is also possible to assign a custom set of rights. To do this, from the roles list select Custom. This will enable all the other UI elements in this step, giving us full control on the rights assignments. Custom rights are discussed in detail in the section that follows.
Depending on the assigned rights, the Wizard may require us to go through another step. For example if we allow a user to only see emails addressed to him, then we need to identify the user's email addresses. On the other hand this is not necessary for an Administrator who is allowed to see all emails.
So in the former case the Next button will be enabled, leading us to the Email Addresses step:
Here we click Add for us to manually enter the user's addresses. We can enter multiple addresses separated by carriage returns. Otherwise, we can click on Import and select the user Active Directory object to import the address from there.
It is good to appreciate that by adding addresses we are granting the user rights over those emails where the address appears in the recipient list. If we wanted, we could allow a user to manage emails for other users. All we have to do is to include the addresses for the other users here as well.
This completes the wizard. Just click Finish for the new user to be added to the list.
If we need to change any settings, including resetting the password, click edit to launch the wizard again.
Once ready, click Apply at the top of the Configuration for all settings to be persisted. At this point the IMF Tune configuration will connect to the database server and save all changes in the user list.
As discussed in the Quarantine Users section of this document, on configuring users, it is possible to assign a custom set of rights instead of the predefined roles. We now take a closer look at the options we have when assigning custom rights.
The above image shows side-by-side the user rights selection and the web interface. The top most set of rights, controls access to the Web Moderator email list. The last two checkboxes control access to the General Spam Detection Report and the Detailed Spam Detection Report.
Email Viewing | View all emails - The user is granted the right to view all emails published at the database. We have to be careful with this right, especially when publishing emails having low SCL ratings, Whitelisted or Unprocessed. Quite obviously this right can easily grant access to confidential emails.
Email Viewing | View own emails only - The user is only allowed to see emails addressed to him. This right is combined with the list of email addresses configured on creating the User. The addresses will identify the emails owned by the user.
Message processing details viewing | View message processing details - For each published email, IMF Tune includes a lot of additional information. As an example here is an email as displayed at the moderator:
The email information is organized into a number of pages. The
Email delete rights - This set of rights controls the ability to delete emails from the Moderator. Here we have four options:
Note how it is also possible to grant a user the right to view all emails (with Email Viewing | View all emails) but only allow him to delete his own emails.
Email re-submit rights - the set of rights available to control resubmitting emails, is identical to that discussed in
Access to reporting is controlled by granting/denying the rights:
The general report includes a set of graphs showing the current filtering performance. Being generic, this information is usually not considered to be very sensitive. However this type of reporting is not very relevant to the casual user.
The detailed report includes lists of email addresses, IPs and other more specific information on the email traffic flowing through the server. The information here is more sensitive.