CTO Tech Blog

Wouldn’t it be great if all unsolicited commercial email, or spam as it’s more commonly known, came with a tag in the subject line that identified it as spam?  Not surprisingly, the people who send spam don’t see the value in that idea.  The job of defending networks and users against spam is on the shoulders of network administrators and users.  Understanding what defences are available and how they work will help users and network administrators choose the best solution for their environment.

Spam filtering can be done in any of four points in the path of an email; as it is sent, before it reaches the destination server, on the destination server, or at the end client.  Checking for spam as it is sent seems, on the outside, to be futile since someone intending to broadcast spam would certainly not do it.  In fact, a large portion of spam is sent using malware that is unknowingly installed on end user hosts.  Checking for spam leaving a host would identify that there is a problem which could then be corrected using malware tools.  Companies that host their own mail servers, as well as many hosted servers check email for spam as it arrives or while it is being processed by the mail server.  Filtering spam before it arrives at the server reduces network traffic and isolates the server from malware that may be contained in spam, but often limits the ability for users to allow email that may appear to be spam from people or businesses that are known.  This is called white listing a sender or email domain.  Moving the spam filter to the mail server often adds simpler configuration tools to allow administrators or users to adjust the white list (desired) and blacklist (malicious) settings either on a global or per user basis.  Client based filtering allows users to individually decide how to identify spam and how to act on each message based on the rules that they configure.  Any or all of these types of filtering can be used depending on how much control a network administrator wants to retain or offload to users and most solutions are a hybrid mix.

Regardless of what stage on the email flow a filter is testing for spam, the same criteria can be used.  Most spam filters use a number of tests in a certain order with specific settings for action and logging for each test.  The best filter will allow all desirable mail through and block all spam, but since the spammers are constantly fighting the filters, new rules must constantly be implemented.  Having a subscription with the spam filter vendor allows the filter to be updated with defences against new spamming methods very soon after they are discovered.  Here are some of the more common methods included in spam filters to detect and protect against spam.

Heuristics/Bayesian analysis – These intelligent filters learn and use statistics to determine if a message is spam.

Reputation – Most anti-spam vendors keep databases of known spammers and tag any mail from their domain or IP address as spam.

Phishing – The scan engine looks for links in emails to known phishing sites and tags them.

RBL – Realtime or Relay blacklist – These are third party lists of reported spammers on the Internet.  These blacklists often list IPs and domains that have temporary issues with malware, or are part of a large range of addresses.

Header checking – The scan engine compares the SMTP and MIME email addresses in the message header to make sure they match as well as other header anomalies.  Email clients set the MIME address, while the server sets the SMTP address.  Spammers use this to mask the origin of an email.

Directory harvesting – Emails sent to multiple non-existent accounts in an email domain are marked as spam.  The term directory harvesting is used because by sending large lists of names to a mail server, most will bounce back as failed, but some may not.  Eventually the sender can, by process of elimination, build a list of valid email addresses on the domain.

SPF – Sender Policy Framework – A relatively new DNS record type that defines domain names and hosts that are authorized to send email on a given domain.  Although not widely used initially this method of protecting a domain has become more common.

rDNS – Reverse Domain Name Service – Where DNS takes a domain name and translates it to an IP address, rDNS looks at the IP address and ensures that the domain name in the email header matches the domain name that points to the IP.  This identifies servers that are relaying mail for other, typically unauthorized, domains.

Blacklist – A list on the anti-spam server that is generated automatically or by user intervention that identifies domains or IP addresses that users have reported as spam.

Keyword checking – A list generated automatically or by user intervention or words that are in undesirable emails.  These words could be profanity, pornographic, pharmaceutical, or any other s.

Regardless of which type of spam filtering is used and which tests are implemented, all but the most expensive appliances require configuration and some amount of learning.  Don’t expect a product to stop all spam right out of the box.  With some adjustments and ongoing updates they are great tools to keep users productive.

Like many concepts in computer networking, DNS is a simple function that is difficult to grasp initially.  DNS is an acronym for Domain Name System and the role it plays is very similar to that of a telephone book.  If we look at how computers communicate over networks or over the Internet in a very basic form, it is all based on numbers.  Like telephones, every device or host has a unique number assigned to it called an Internet Protocol (IP) address.  This can be a little confusing because your computer and mine may currently have the same IP address, but chances are that neither is connected directly to the Internet.  My computer is on a network that has a firewall connecting to the Internet.  My firewall has a unique IP address on the Internet even though my computer may not.  You could look at this like peoples’ names in that there are a number of entries for Siverns in the phone book, but if you call my number there is only one Glenn in the house.  It is not typical for someone to remember someone by their telephone number or their street address.  The designers of the Internet understood this and created DNS as a system that is similar to a telephone book for people and other computers to use as a directory to locate hosts on the Internet.

When configuring the IP settings on a computer a DNS server address is defined.  This may be done manually or, for a computer receiving its IP from a Dynamic Host Configuration Protocol (DHCP) server, automatically.  A host is also typically given a name to identify itself.  DNS servers keep records of host names and the IP addresses associated with them.  If an IP address is changed, users and computers can still find the host by name using a DNS server.  DNS can also work closely with DHCP servers to keep records up to date without the need for a system administrator to manually update them.

On the Internet there are a number of “root” servers that are used to direct traffic to other servers which hold the DNS records of hosts.  When attempting to access a host such as www.basicbusiness.com, DNS will first look for the record in the DNS server that is configured on the host.  If the record is not found the DNS server will forward the request to the root server that holds records for the root level domain .com.  This server will have a record for the domain registry server which knows where the basicbusiness.com records are stored.  The request will then be forwarded to that DNS server which will reply with the IP address of the host “www” in the basicbusiness.com domain.  You could think of this process as being similar to looking for a phone number for Glenn Siverns in Vancouver, BC, Canada.  You might go to a library to find a shelf with Canadian phone books and select the Vancouver book which would contain the number.

Many problems related to accessing web sites on the Internet can be attributed to DNS problems.  When an Internet browser reports that if is unable to find a web site, yet the computer can still receive email, the problem is very likely a name resolution issue attributed to DNS.  Here are some terms you may hear when someone is helping diagnose a DNS issue:

 IPCONFIG /all– This command will display the IP configuration of a host.  When you type IPCONFIG at a command prompt, the computer will display the IP address of the host, the subnet mask, and the gateway address.  IF the command “IPCONFIG /all” is entered, the computer also reports the DNS server addresses among other things.

PING – Similar to the use of sonic pings in submarines, ping is used to see if something responds.  Ping can be used to identify or verify a host by name or by IP address.  For example, if you type “PING www.basicbusiness.com” at a command prompt the command should list the IP address assigned to www.basicbusiness.com and send a number of packets to the IP address to see if it responds.  Note that many hosts will not respond because of security configurations, but if the ping command is able to determine the IP address then DNS is likely working.  Another useful test may be to try to ping a host by IP address if it’s known.  If the host responds to a ping by IP address, but not by name then DNS is not resolving names properly.  If neither test gets a response then there may be other network issues.

I hope this takes some of the mystery out of DNS and helps point you in the right direction next time you have a problem with accessing the Internet.

I recently received some digital photos from a friend and noticed that the files were named differently from the ones from my own camera.  I was wondering why different camera manufacturers would use different file formats, so I did a little digging.

Camera manufacturers are always trying to get an edge by providing better features and image quality.  Much of the image quality is determined by the lenses, but once the picture is snapped, the image data needs to be stored.  I had a look at the specs on a high end camera, the Canon EOS 1DS Mark III digital SLR and they listed the image format as:

DCF 2.0 (Exif 2.21): JPEG, RAW and RAW+JPEG simultaneous recording possible. Multiple options for recording images on two memory cards, and onto compatible external USB hard drives (via optional Wireless File Transmitter WFT-E2A)

Since this is a high end camera, I thought that someone considering this purchase would be well up on the terminology and acronyms, so I looked at one of the consumer rated models instead.  The Canon PowerShot SX 120 IS Digital Camera had two separate specs for file format and compression:

Design rule for camera file system, DPOF (Version 1.1) compliant

Still Image: Exif 2.2 (JPEG)
Movie: AVI (Image data: Motion JPEG; Audio data: WAVE (Monaural))

Thinking that maybe it was just common for Canon to use these acronyms, I checked other manufacturers and was greeted with terms like CCD-RAW, TIFF, DPOF, and BMP.  There are some important things to look for in a digital camera, but choosing the right specs to compare may depend on your personal needs. First some basics about image files.  Image data can be stored in two very basic formats; raster and vector.  Vector graphics are more suited to line and shape drawings, including 3D images.  The basic premise of vector graphics is to define a starting point for a line, then a direction and distance.  The direction can be on two planes (2D) or three (3D).  Other parameters such as colour, fill, and weight are also defined.  Vector graphics are commonly used in computer aided design (CAD) drawings.  Raster graphics use a method similar to that of a television screen where the image starts forming by defining a single dot or pixel, then moving to the adjacent pixel and defining it.  This would continue across one dimension of the image, then move to the next line or row until the entire image is formed.  The definition would include colour and intensity for each dot.  As you can guess, vector graphics take relatively less space to store because they can define large areas with just a few parameter definitions.  Raster graphics by contrast, must define each pixel in the image.  To combat this space requirement, a number of different image formats and compression schemes have been developed.  The best quality images have no compression, but take the most room.  Because of the detail that can be stored is raster files they are used as the storage format for cameras.  Here are some descriptions that may help:

CCD - Charge Coupled Device.  This is the special device in digital cameras that convert light to electronic data.  They are an array of microscopic devices on one silicon chip and usually number in the millions.  The term megapixel refers to the number of pixels in a CCD.  More pixels allow the CCD to detect more detail in an image.

RAW - This is not an acronym.  Image data is taken directly from the CCD and stored on the storage media.  RAW images are created very quickly, but take a lot of space.  If space is not an issue, then this is the best format for a camera.

DCF - Design rule for Camera File system.  This is a standard adopted by a number of camera manufacturers to define a standard format for camera images file systems.  The standard includes folder structure and EXIF formatted files and has been adopted by most camera manufacturers.

EXIF - Exchangeable Image file Format.  EXIF adds structure and photographic information to existing image formats to standardize the file structure and help compatibility.  Information such as shutter speed, camera manufacturer and orientation are among the parameters defined.  Various versions of EXIF are in use by most camera manufacturers today.

JPEG - Joint Photographic Experts Group defined this standard for image compression.  There are a number of levels of compression available which decrease the storage footprint at the expense of image quality.  If you're looking for a camera that can store many pictures, then look for JPEG formats.  A low level of compression will retain most of the image quality with a good saving of storage space.

TIFF - Tagged Image Format was developed by Adobe Systems in an attempt to create a common image standard for scanners.  TIFF acts as a container which defined the geometry and other parameters of an image and may contain images in other formats such as JPEG.  TIFF formats with embedded JPEG give the same storage saving as JPEG, but don't add much additional value.

DPOF - Digital Print Order Format adds definitions for output to images.  These definitions may include number of copies, output image size, orientation, and title text.  This standard was developed by a consortium of camera manufacturers to simplify printing of digital images.  DPOF was not widely adopted and has become unneccessary for digital printing.

BMP - Bitmap images were developed by Microsoft as a high quality uncompressed image format.  The large size of BMP images makes them unsuitable for cameras or as images on web pages.

PNG - Portable Network Graphics files are based on the BMP with the addition of compression for use on the Internet.  The compression of PNG results in too much image quality loss for most camera use.

While there are many other formats defined and in use today, I hope this sheds some light on what to look for when choosing an image format or a camera that uses them.