IMF Tune Quarantine Performance Considerations

The IMF Tune Moderator/Reporting interface very often provides access to thousands of emails. At the same time new emails are being added to the backend database. This requires the system to be very responsive. Here are some recommendations on how to install and configure this component with performance in mind.

1. Background Information

The IMF Tune Moderator/Reporting interface is made up of:

1. An IIS ASP.NET application providing the Moderator/Reporting interface.

2. An MS SQL database backend storing the email data.

The Quarantine system responsiveness is very dependent on the backend database performance. So here we focus on the MS SQL database. We start with some basic installation time considerations. Next we look at configuration settings that impact the volume of data the system has to handle.

 

2. Processing Power and Server Resources

Whenever discussing application performance, the server processing power and available resources are a good starting point.

The most common performance related problems our support investigates, very often concern relatively small organizations that try to run all their server applications on a single machine.

WinDeveloper recommends running the SQL database on a different server, other than the MS Exchange server where the IMF Tune filtering component is installed.

 

3. Moving the Quarantine Database to new Hardware

The long term solution to performance related problems is to make sure the Database has enough resources to handle the email load. This could mean that we have to move the Quarantine Database to a different server. For details on how to do this check the article Moving the Quarantine Database to New Hardware.

 

4. Reduce the Data Retention Time

IMF Tune provides the functionality to automatically purge old Quarantine and Reporting data. These settings have a significant impact on the amount of data the Quarantine system handles. For example, reducing the data retention from 30 days to 20 days we cut the data volume by a third.

This functionality is controlled through two edit boxes under Disk Maintenance | Archives/Quarantine.

'Compress and backup files after (days)' is the number of days emails are available for Moderation at the Quarantine interface. This is the first age check IMF Tune performs. Once an email ages beyond this number of days it will no longer be visible through the Moderator interface. Although not visible, some of the email data is still retained for reporting purposes.

Reducing this value reduces the number of days an email is available at the Moderator interface. So the administrator has fewer days for reviewing blocked emails.

'Retain quarantine information for reporting purposes for (days)' is the number of days emails are retained for reporting purposes. This is the second and final "age check" IMF Tune applies. Once an email ages beyond this number of days, all the information is deleted from the SQL Database.

Basically this value determines the number of days reporting information will cover. For example a chart showing the total number of blacklisted emails effectively shows the sums of all blacklist matches for the period of time configured here.

Considering the two settings together we can easily see why 'Retain quarantine...' has to be greater or equal to 'Compress and backup...'.

Secondly we can see that 'Retain quarantine...' is the value that ultimately controls the data retention length in days.

These two edit boxes give us the opportunity to control the data volume at the Database. We can reduce these until we reach a level that our current hardware is able to handle.

 

5. Reduce Storage of Reporting Data

Performance can be improved by reducing the amount of data collected. In this manner we reduce the SQL database load.

The IMF Tune web interface covers two functionality areas, Quarantine Moderation and Reporting.

By default IMF Tune is configured to store information on all processed emails for reporting purposes. This includes both spam and legitimate emails. Whereas information on spam emails is used for both the Quarantine and Reporting functionality, legitimate emails are mostly useful for Reporting purposes.

This functionality is controlled by the checkbox under Quarantine | Quarantine Database:
'Store information on all processed emails for reporting purposes'

Clearing this checkbox, IMF Tune stops recording information on each email for reporting purposes. Thus less data is written to the SQL database.

Note: When the checkbox 'Store information on all processed emails...' is cleared, the IMF Tune Web interface will still display all the usual reports. However these won't take into account emails with low SCL ratings. When looking at the reports, you will need to keep in mind that these are only covering blocked emails.

Note: When the checkbox 'Store information on all processed emails...' is cleared, IMF Tune will not immediately delete email information already accumulated at the database. This data will be purged gradually as the information ages based on the setting:
Disk Maintenance | Archives/Quarantine | 'Retain quarantine information for reporting purposes for (days)'

 

6. Only Store Blocked Emails

The load on the SQL database can be reduced further by making sure that only blocked emails are recorded. The logic is similar to that discussed in the previous point but the settings involved are different.

IMF Tune allows us to select which emails to Quarantine based on the SCL rating. This is configured at the Archiving/Quarantine Category.

Here we create Archiving/Quarantine profiles. If we open an existing profile or create a new one we get the dialog that follows.

Here we find the checkbox:
'Also add archived email to quarantine database'

When this checkbox is set, emails whose SCL match the profile are uploaded to the SQL database.

IMF Tune leaves us completely free when configuring the profile SCL set. Thus we can easily configure the system to upload legitimate emails to the database. This can be useful when fine tuning SCL thresholds for example. However, if in need to relieve the database from some load, it is good to keep the database for blocked emails exclusively.

Do this as follows:

1. Open the IMF Tune Email Handling category and determine which thresholds are configured to block email delivery:

   

2. In this example emails with SCL 6 or greater are not reaching user mailboxes.

3. Now move back to the Archiving/Quarantine category.

   Here make sure that only emails with SCL 6 or greater are loaded to the database. In the process you may need to create new Quarantine profiles such that:

   SCL 6 and higher are included in a profile having 'Also add archived email to quarantine database' selected.

   SCL 5 and lower (if archived) are included in a profile where the checkbox is cleared.

Note: When the IMF Tune configuration is changed, any emails already uploaded to the database won't be immediately deleted. This data will be purged gradually as the information ages based on the setting:
Disk Maintenance | Archives/Quarantine | 'Retain quarantine information for reporting purposes for (days)'

 

7. IMF Tune Quarantine Database Purging

In this document we described configuration changes that allow us to reduce the amount of data uploaded to the Quarantine database. However as already highlighted IMF Tune won't immediately delete email information already accumulated at the database. To force an immediate database purge check IMF Tune Quarantine Database Purging.