At the beginning of 2003, spam – bulk, unsolicited email – was around ten percent of all email traffic, according to Mark Sunner, CTO of U.K. email security outsourcer MessageLabs. "I did a simple trend analysis and came up with a figure of 50 percent by year end. In the event, I was wrong – the figure was actually 62 percent. I've done the same exercise again and I'm expecting a figure of around 80 percent by April 2004."
Who are the spammers and why are they doing it? Recent numbers from anti-virus vendor Sophos show that more than 50 percent of spam originates from the U.S. Spammers from Canada, China and South Korea also contribute but in much smaller numbers. Roger Dean, one founder of EEMA, the European forum for e-business, points out that 160 spammers produce 80 percent of all spam worldwide.
According to George Webb, of Microsoft's anti-spam group, a spammer's motivation is nearly always financial. He quotes U.S. research from 2003 that claimed that 33 percent of the recipients said that they had responded to a spam message and a staggering seven percent had gone on to buy goods or services.
Sunner notes that the spammers' business model depends on a conversion rate of 0.01 percent from mail shots that routinely consist of millions of messages.
The problem does not stand still. Actions taken to date have begun to deal with some problems but the spammers continue to adapt.
Industry-wide action has closed many of the "open relays" that allowed mail to be passed through SMTP servers, concealing its origin. Effective blocking of IP addresses used to originate spam has closed routes used by spammers too.
Now a new approach is evident notes Sunner. More than two-thirds of spam intercepted by MessageLabs is sent via "open proxies" – ordinary office and (mostly) home computers that have been "prepared" by spammers to allow bulk mail to be sent without the user's knowledge.
Virus writers appear to be working with spammers to develop "blended threats," as Mikko Hypponen, director of anti-virus research at Finnish anti-virus specialist F-Secure, explains.
"Ten days after MyDoomA was first identified, we saw the first conscious attempt to scan the internet to use the 'backdoors' that it had opened," he says. "Someone, quite possibly the original authors of the virus, then began installing Trojan.Mitglieder on compromised machines, at the same time removing the virus (presumably to remove evidence of a link). Within 12 to 20 hours, spam began flowing through these zombie computers."
In Germany, c't magazine recently investigated open proxies and was able to purchase a batch of them for spam use. In the case it investigated, a Trojan was installed on thousands of computers with the help of the "Randex" virus.
"This small program contacted its 'master' through the chat protocol IRC," says Hypponen. "From its master it received commands to, for example, look for CD keys of games, launch syn flood attacks from the infected machine, or secretly load additional software. This way, the Trojan was also able to install a SOCKS proxy server, which can be used to relay spam through infected PCs. The virus also infects local subnets using Windows directory services."
In an interview with c't, an investigating officer commented: "We fear that this is just the beginning. In the case in question, the authors and distributors of the viruses no longer do their work for fun or ego. The scene is becoming more professional and has recognized how much money can easily be gained illicitly this way."
The specter of organized crime, even if it is only one part of the problem, highlights the need for solutions. Some commentators even see the end of email as a usable service unless emerging blended threats can be eliminated.
Hypponen points to the approach used by MyDoom as a step change. "MyDoom doesn't stop once it has sent messages to the infected machine's address book," he says. "It has the potential to keep issuing messages in a loop, and to create huge volumes of genuine-looking spam." Such volumes may cause email users to simply give up on using email altogether.
Microsoft's Webb sees the need for activity to be coordinated using legal measures, improved user education, advances in technology and alliances between like-minded organizations. Legislation has been produced in the E.U. and in the U.S. and while many commentators view both the CAN-SPAM Act and the E.U. directive on Privacy and Electronic Communications as inadequate, there is no doubt that the legislators are moving in the right direction.
Court actions, such as that taken by Earthlink, an ISP, against 16 people known collectively as the "Alabama Spammers" are also a potentially productive way forward.
As for user education, Kelly Martin of Security Focus wrote recently: "There's a myth that should be dispelled in the computer security world and that is the belief that the growing and pervasive use of computers in the past 20 years has enabled the average person to become functionally computer literate." Their limited knowledge and concern for security is "a major reason why so many hundreds of thousands of drones and bots exist and are under the control of black-hat folks."
As larger businesses become more security conscious, the battleground is likely to continue shifting to small businesses and home users.
No single technical solution will completely deal with spam now or in the future. Steve Kille, CEO of messaging company Isode, contends that spam solutions work in two ways. One solution uses the anti-virus model of creating spam "signatures." Each signature takes time to produce and, as spam is more volatile than viruses, it requires frequent updates of signature files.
The other solution uses a number of techniques to identify messages that "look like spam." Isode's approach, similar to many other proprietary solutions, is to deploy a variety of techniques.
Bayesian filters use methods first deployed by the Reverend Thomas Bayes (1702-1761). The filters examine a wide range of message characteristics derived from these techniques, creating a mathematical score that can be used to indicate the probability that the message analyzed is spam.
User-controlled parameters then control what action is taken with "definite spam" and "probable spam." The aim is to reach a balance between spam that reaches the mailbox (false negatives) and real mail that is removed by the filtering process (false positives).
Kille suggests that "the rate of false positives needs to be 0.001 percent before spam can be automatically deleted with confidence."
Isode has recently implemented a technique called "greylisting" in its anti-spam product – one developed by the Anti-Spam Group of the Internet Engineering Task Force .
"Most spam sent by scripts is generated on the fly," says Kille. "So the message is not stored. We look at the sender and recipient addresses and if we haven't previously seen the combination, the message is temporarily failed for a short period. For legitimate mail, the sending process in SMTP will simply queue and re-send, at which time we pass it through. Spam cannot retry and so does not re-present the message."
A small percentage of genuine mail is script-generated, but this can be allowed by using a "whitelist."
Boots Software, an offshoot of a European ISP, got into spam filtering out of necessity. The company found it was having to increase its capacity as message volumes grew out of control.
Clive Homewood, CEO, explains: "As we examined the problem it seemed to us that many spam messages used some form of falsification of the HTML, which is detectable. We could think of no legitimate user that would do this, so we decided to stop falsified messages before they hit our systems."
It implemented front-end SMTP processors to filter out those messages and to suspend the connection with the originator's IP address for up to an hour. "In effect, it looks like the connection is active but nothing is passing across it."
In the meantime, a second level processor notifies all other front-end processors to suspend the same IP address. "Before we switched Spam Exile on, we were processing 1.8 million messages per day, with 60 percent spam. Once switched on, we came down to 600,000 messages and only 15 percent spam and we were able to switch one of our systems off. Now, we are detecting 98 percent of all spam," he claims.
While managing spam at the desktop is fine for home users and smaller businesses, the costs of additional network bandwidth and employee time are causing company CSOs to find solutions at a corporate level. These solutions will be implemented at mail server or firewall level or may be outsourced completely.
Vendors of corporate solutions, such as Isode, point to the importance of maintaining control over email, especially where critical communications might be subject to deletion as spam.
Outsourcing supporters, like MessageLabs, point to the dramatic reduction in bandwidth and hardware costs as well as the improvements in people productivity that follow when the problem is managed externally.
In February 2004, at the World Economic Forum, Bill Gates, chairman and chief software architect of Microsoft, apparently upped the stakes by declaring that spam would be defeated within two years. This was at odds with the message being put out two months earlier by his own anti-spam team and, indeed, with that presented by Gates himself to the RSA Conference in San Francisco later the same month.
Microsoft's Webb, addressing the EEMA Conference in a speech titled "Spam – The Death of Email?" in Dublin in December 2003, predicted that some 18 to 24 months from now "containment might be possible." From that point it would be "measure and countermeasure, as it is now in the anti-virus world," a point reiterated by Gates at the RSA Conference.
Gates' announcement at the RSA Conference (see panel, page 30), included a number of measures aimed at "dramatically reducing spam." Commenting on this apparent difference in message, Tony Lock, chief analyst at Bloor Research, seems unconcerned noting that "Gates was speaking to two very different audiences."
The emphasis in Microsoft's proposals on corporate policy will have been music to the ears of Dean and the members of EEMA who have been busy developing a white paper on spam and email abuse management since their December conference. Contributors include U.K. postal service Royal Mail, Siemens Business Services, Volvo Information Technology, the U.K. National Health Service Information Authority and Microsoft.
The white paper, available to members of the EEMA, outlines the problems and potential solutions for corporations. Dean believes having effective policies on the use of email and internet access is essential.
Microsoft, like other major players AOL and Yahoo, has been concerned by the growth in "spoofing," or the imitation of sender addresses in SMTP. SMTP has no mechanism for validating sender addresses and there have been suggestions that SMTP should be completely rewritten. For now though, that does not appear to be on the agenda.
The three different approaches (see panel, above) known as Caller ID for Email (Microsoft), the Sender Policy Framework (AOL/IETF) and DomainKeys (Yahoo) all endeavor to use the internet's Domain Name System to validate the sending domain.
All three companies are founder members of the Anti-Spam Technology Alliance and the hope is that they will work with each other and with the industry as a whole so that a clear, collaborative road map emerges.
These proposals are only "the first step on a perhaps never-ending road," according to Bloor Research's Lock. He continues, "At the heart of the spam issue is really identity management."
Lock suggests that it will not be long before "Governments or pan-national entities begin issuing electronic identities. It is already being done in some countries – Belgium, Sweden and Malaysia, for example – and it is technically a matter of scalability. The big issue will be political and may be driven by national security issues – like the recent U.S. passport changes – by the bigger western economies such as the U.S. and U.K."
Moving forward and planning for the long term, defeating spam will involve dealing with the motive of, and eliminating opportunities for, the spammers.
Their motive is to make money by exploiting an essentially free medium that makes it easy to reach millions of people. This motive is being attacked by proposals to add cost to the process as a tax on every message, by imposing additional machine cycles, by using encryption to validate senders or simply by leaving millions of messages undelivered on the spammers' servers.
The greater likelihood of being caught and prosecuted or sued will help deter spammers but opportunity is a bigger problem.
Currently, too many internet users fail to understand, or fail to act on, the imperative to protect themselves for the greater good. Access to email addresses seems to be unlimited and users help spammers by responding to spam, thus confirming their addresses.
Spammers are also finding it too easy to infect and control vast numbers of machines. Here the anti-virus and anti-spam lobbies have a common cause.
The pessimism of F-Secure's Hypponen may yet prove to be unfounded, but can we risk the continuing viability of email when so many businesses now depend upon it?
Some of the proposals now on the table may lead to a two-speed internet, with those capable of proving that their security measures are adequate paying more to secure a spam-free service.
If users securing a spam-free service represent a large portion of the target audience of the spammers, that may reduce the creation of spam.
The worst case scenario is that we fail to make progress before nine out of every ten (or worse, 99 out of every 100) messages are an unsolicited, unwanted intrusion and people choose to desert email and return to more traditional methods of communication.
Andy Coote is a freelance writer
