M3AAWG has presented the 2023 Mary Litynski Award to Chris Lewis, one of the most prolific and expert spam fighters in the anti-abuse ecosystem. The award recognizes the lifetime achievements of an individual who has significantly contributed to making the Internet safer for all, reflecting that same work ethic and dedication.

Members can view a video discussion of Lewis’ work here on the members-only site. More here, https://www.m3aawg.org/members/home.

Lewis has worked deep inside anti-spam technology for several decades. He is the former chief scientist at SpamhausTechnology, one of the most well-respected sources of anti-spam and malware filtering information in the world. Before that, he was senior security architect at Bell Northern Research, later Nortel, from 1991 through to 2012.

In 2003, Lewis developed a new technology that greatly increased the effectiveness of message filtering. This required vast amounts of sensor data from all over the Internet, analysis of the data, and by way of return, he published anti-abuse intelligence for free for all to use. He is a founding member and active in a number of industry, legislative, policy and law enforcement efforts to prevent and reduce spam.

Note: this Q&A has been lightly edited from a video interview of Lewis shown at M3AAWG’s 57th members-only general meeting February 21, 2023.

Q. Chris, how did you get interested in spam and botnets initially and how did your early efforts work out?

I started my career as a specialist in compilers and operating systems and evolved into developing/supporting system security monitoring applications.

I was heavily involved with Usenet and infrastructure tools in the days of dialup network, when the Internet was a handful of sites. I believed in it, and started making efforts to improve it for everyone, contributing to software, interfaces, access methods, basic security/authentication and associated RFC standards.

Then Usenet spam started to happen; it was trivial to send just a few hundred or thousand messages that everyone would get.  And thus, the writing was on the wall about impending destruction when every advertiser would want to take advantage.

A set of rules were developed by consensus that would determine when a set of Usenet postings stepped over the bounds, and protocols were developed to seek out and remove the postings.  I was one of the first to step into those shoes, both deleting spam and other measures making it possible to discuss these issues with other operators and interested people.

I believe we succeeded in keeping Usenet working usefully for a few years past the time spam would have killed it. But Internet connectivity started to explode, making email and the web the preferred ways to communicate. The spammers we frustrated in Usenet started to shift to email.  Email was where the important fight was going to take place, and it was a war we couldn’t afford to lose.

Q. Can you offer us a bit of inside intel on how the CBL (Chris’s Black List) was developed and why?

It became apparent through both the usenet->email spam shift and the advent of email-born viruses (mostly hackers looking for the bragging rights of taking down networks) that I was going to have to come up with a way of protecting my current employer’s (Nortel’s) business email.  Otherwise, email would not just be useless, but provide an avenue for malware to disrupt our sales and operations.

I initiated a skunkworks (meaning: a personal project done on Nortel’s time while not being an official project) effort to build inbound email filtering for when we needed it.  The Nortel email group was mystified as to why anyone would want to block email, but I knew what was coming, and persisted anyway.

First, I needed a platform to build spam filters into.  The main criteria was that it was going to be fast enough to handle our email, and be sufficiently flexible to use many different kinds of filters, different responses, implement a policy by which EVERY email that we blocked would be blocked with a message that would allow the sender to contact us to correct the problem, and allowed us to resend email that we had held in quarantine.  I asked and looked around for such a platform.  At the time email spam filtering tools were toys, inadequate for even a hundredth of the volume of email we had.  Everything, even including well-known, high-volume email platforms could not handle filtering at our scale, and their flexibility was practically non-existent.

Out of the blue a stranger asked me “we’re thinking about building something like this, could you give us a functional specification?”  I jumped at the opportunity, and the result became eventually known as Lyris Mailshield.

At the time, the Mailshield project had fairly limited filtering methods.  But it was the platform I needed to start developing filters. When I took this system to the email group for possible full deployment, they still had their previous inclination that nobody would want to block any email, and so it languished as a small-scale trial with me still doing filter development in my spare time.

Then Mellisa hit and took just about all medium-large companies down hard.  Email and a lot of computers crashed.  On a Saturday I was called by my director, and asked “how fast could you deploy this for all of our email?”. I boldly I said “I’ll have it up in four hours”.  We were back in operation by Sunday, beating many very large companies by several days.

For a while, it was a matter of stabilizing Mailshield at such a load, and building support procedures around it.  But I was still doing a lot of research and tweaking my simple filtering.

At about this time the people I knew were getting interested in better filters, and early efforts such as MAPS (Mail Abuse Prevention System) were created, along with various other methods that did not prove to work well. I wanted to do something like MAPS.  But how could I do something better?

By June 2003, I had been staring at email spam for years and had started to look at aspects of email that people don’t normally see.  One day I noticed something common to particular email campaigns running on things we knew to be email-spewing botnets.  Then I thought that instead of filtering on that characteristic, I would set up a local blacklist listing the IP addresses showing that fingerprint.  Over the next several days I noticed several other peculiarities of other spambots, and blacklisted them too.

Thus what I called “Chris’s Black List” (later known as CBL) was born, and it was successfully blocking more than 85 percent of all the spam hitting us with no known false positives out of millions of blocked emails - nobody was surprised by those numbers more than me!

I reported my results on a mailing list, and over the next few weeks I started getting many  requests to try it out.  Then someone offered to host it for publication, others offered to feed us the tiny bit of sensor data for each email from their own mail servers to broaden its coverage and applicability.

At that time, the CBL was being offered for free to the Internet, and I continued to tune the results, and answer complaints.  After a few months, I got contacted by Spamhaus, which asked if they could redistribute it too.

Spamhaus then republished it (with a small piece from elsewhere) as the XBL.  Coupled with Spamhaus helped obtain many new spam sources for me to analyze and hardware to do some of the work.  But the primary analysis and list production remained at Nortel.

Thus it remained this way from 2003 to 2012. During this whole time, I was giving it away to everyone for free, only requesting large organizations to get paid subscriptions to Spamhaus to help them pay for the infrastructure, and in turn get far bigger resources serving it up.  Along the way I developed new techniques for detecting compromised IPs that were not necessarily spam-emitting, but still clearly compromised (emitting malware, or facilitating it) and the CBL started publishing those too.

After Nortel declared bankruptcy in 2009 (they kept me until 2012) I simply transferred to Spamhaus and got paid for doing “spam and botnets” for the first time in my career.  It remained this way until about 2020, when the CBL was finally fully absorbed into the Spamhaus core and essentially became real-time listing for the first time.

Q. What were the initial results of deploying CBL and industry response? 

My intent to make the CBL the best I possibly could, and was alert to anything I could find that would help me compare the CBL to other solutions so I could improve it.  I relied on colleagues reporting how well the CBL was performing, comments from industry and academia on how well it worked, and examining how much spam the CBL blocked at Nortel and all my traps. 

Most of the time, the actual bot detections was in excess of 95 percent of all inbound email on most of my traps.  When factoring in botspam that missed the detectors, of the same IP (which’d get blocked by the blacklist), the overall filtering rate was even higher.  If it dropped below 95 percent, it was my signal to go in and tune up some more botspam rules.

Part of my effort was providing as much information as possible about each individual listing, and writing guides on how to find/fix them.  Many complemented us on the detail and quality of the diagnostics we published. I invariably found that the CBL almost always performed better in all respects to every other blacklist that used single listing criteria (eg: listing bots, or those listing dynamic IP ranges etc). 

Q. Given how much effort has been put into fighting spam and botnets by yourself and others, what do you think about the future of these issues?

After almost 30 years fighting spam I’ve learned:

Most spammers are small and new amateurs, often tempted by bad guys offering them email services.  They rarely last long because they’ve been dissuaded by complaints, ISPs disconnecting them, blacklists and other filters stopping them and killing any profit they can hope to make.  These are greatly helped by such things as M3AAWG’s outreach, laws and regulations, and those ESPs (email service providers) that have learned how to do things properly and provide tools and guidance on how to do it right (including consent).  This is why I spent a lot of time with M3AAWG’s sender community to help educate ESPs so they could teach their customers to do it right.

The spam volumes are EXTREMELY variable.  Usually they’re highly dominated by one or perhaps two botnet networks.  Examples include Bagle, Rustock, Szrizbi, Cutwail (at times), Kelihos and the spambot now spewing bitcoin extortion attacks (likely Tofsee).  These heavily dominate all botnet spam for weeks, months and sometimes years.  Some of them were capable of sending 60+ billion spams per day, and simply dominate all traffic.  My traps peaked out at 1.5 billion spams per day (but we were getting intel for further spams from sources in the billions of emails per day).

When the volume of email into a spamtrap drops, it’s NOT likely to be that the bot owners have learned how to avoid your traps.  It’s almost always that one of the high-volume botnets has been destructively taken down by law enforcement, or the botnet owner has simply given up.  In fact, the general botspam volume actually HAS gone way down.  The peak was back in the days of Rustock, and it’s been declining ever since, with huge bumps due to the advent of newer high volume spambots, who in turn eventually die too.

The only significant volume spambot currently in operation is <the bitcoin extortion bot>, and its volumes are a tiny shadow of the earlier giants.

The threat landscape for email spam has simply shifted a lot since 2003.  Gone are the days of massive blasts of dubious/fraudulent products via 10’s of thousands of infected IPs. Almost everybody got multiple copies per day. At best these campaigns would have a return of pennies per 100,000 spams.  Email spam has gone to narrow targeting with potentially very high returns per victim.  Such as the gangs sending encryption ransomware, and corporate phishing.  The volumes are very low, and filter organizations almost never see them.  Furthermore, even if they do, the volumes are so low that the spammers can afford to purchase/hack legitimate email servers or even send most of them by hand, and they don’t expect their hosting to last long.  The filter organizations can’t see them to block them, and even if they do, the spammers have skipped elsewhere by then.

This indicates to me that email-based filters have to get MUCH MUCH smarter in examining content.  While some blacklists will remain effective (even with IPv6) many will decline in effectiveness, because they aren’t targeting the new threat landscape.  However, it still behooves us to ensure that even the now less effective methods remain around, because spammers DO reinvent the wheel, and such solutions will probably again become much more useful.

Q. You have been quite involved in policy and legislation, law enforcement efforts and industry-wide efforts in the spam fight. Give us some color around those in terms of the battles, successes and what you see for the future. 

I was a founding member of The Coalition Against Unsolicited Commercial Spam (CAUCE) - a grassroots effort against spam, and later was a director up to the late 2010s and a founding member of NCFTA/slamspam initiative via FBI. I participated in an early meeting with the Direct Marketing Association to get them to encourage consent-based emailings, which they reneged on less than 6 hours later. I’ve presented to the U.S. Federal Trade Commission.  I think my most important was the work towards the Canadian Anti-spam Law (CSL).

This was a multi-year consultative/advisory/stakeholder process. I was involved with review of final drafts, last consulted to the Canadian Parliamentary committee for the law’s required review. I was invited to M3AAWG as a senior technical advisor and have led training, intel and consultation in many parts of industry, law enforcement, and regulatory bodies about specific spam/malware cases, and intel/investigation techniques.

M3AAWG thanks Chris for his many contributions. You can read more about this topic by going to our website, www.M3AAWG.org, and we welcome industry participation in our efforts. Learn more about us here, https://www.m3aawg.org/about-m3aawg. Best practices are here, https://www.m3aawg.org/published-documents.