Sherpa Filter Test Results Show 4% Mail Filtered, Possibly More

"Welcome to the Wickes Lumber email filtering system. SPAM is NOT welcome here! Have a nice day!" - bounce message today

At 9 A.M. Eastern Time this morning MarketingSherpa sent out three test messages to approximately 10,000 MarketingSherpa double-opt-in readers to learn how much filtering is affecting permission emailers, and if typos make a difference.

Within just a few minutes we were deluged with responses, more than 2,000 readers took the survey online and more than 300 emailed separate personal notes which we are still replying to. Here is what we learned, and how it might affect your own email broadcasts.

Only 4% of messages were filtered no matter what

Completely unexpected results. Turns out the same percent of survey respondents got, or did not get, each of the three test messages. We do not think we have ever seen such an even group of responses from any survey!

Test #1 Contained 10 potential filter words
Test #2 Contained the 10 words only with typos-on-purpose
Test #3 Did not contain the list of 10 words

97% of respondents received each message in their in-box. 3% of respondents received each message in their bulk mail folder. A tiny 1% of respondents did not receive one of each of the messages at all. Every individual did not report the exact same results, but added together every test letter got roughly the same overall results.

Naturally people who did not get any of the three messages did not respond at all, because they did not get anything to respond to. The 1% number could be low.

Bounces were roughly the same for all three messages, at 1% 'transient problems' and 1% 'undeliverables' (which include the few filters that proactively tell us they are filtering) after four delivery attempts.

Of the bulk mail folder people, several noted they used Cloudmark and others mentioned they used Outlook 2000 filters. Lots of readers using Hotmail and Yahoo wrote in to say even with filters on, they got all three emails.

However, this is worth noting: About 30% of the personal notes we received from respondents said they are not using personally activated filters currently, but they (or their company IT departments) are considering adding filters soon.

The percent filtered is still a moving target.

Last but not least, two readers wrote in to say they got the tests, but they had not got any regular issues of MarketingSherpa for a few weeks. Our newsletter was filtered, but not the test. Which leads us to:

Our test may have been flawed: Too few words

Our test was probably flawed because we only wrote each of the 10 words once. (Plus the word "unsubscribe" was added a second time in the automatic message at the end of each note.)

Our Tech Editor said, "Filters that weigh message body are looking at all the words in your message. You may be able to get away with using a word once, but not multiple times because you'll get scored for each use."

If you write a very short note, as we did, chances are you will not be filtered. If you write a long note, you increase your chances of being filtered for the exact same words just because you used them in more places.

Filters are evolving quickly so this will not always be the case.

However in the meantime, you can avoid some by keeping content short.

Other mistakes we made that you can avoid

Several other email list owners are conducting tests like the one we conducted today, including Debbie Weil of WordBiz. (Her results are in Section 5 below.)

If you are considering it, here are some mistakes we made that you will certainly want to avoid. Our faults for having a brainwave late at night and implementing by seat-of-the-pants.

a. Ask what email system or service respondents use so you can sort results in a meaningful way.

b. Give respondents a comment field in your survey form in case they have other questions, ideas, etc.

c. Be prepared to forget your day's scheduled work in order to read and reply to reader notes. (It has been harried, but absolutely wonderful and very insightful.)

d. If you have a big enough list that a segment would provide statistically reliable results (you need at least 100-150 results per question for any answer to be reliable), then consider setting up your own Test Forum of readers or customers instead of sending everyone tests.

Some people strongly dislike being asked to test things, or receiving any email whatsoever aside from their regular issue. We completely respect that.

Test results from another publisher: WordBiz

Debbie Weil, Publisher of WordBiz Report newsletter, emailed us, "My open rate for Nov 6 issue was down a bit. I decided to ask my readers if they received the issue. I sent out a little text message politely asking."

Within 24 hours, 1.36% of Weil's list responded saying they had not gotten the issue. (An additional 1.32% of her list emailed in that their copies had not been stopped by filters.)

On one hand some may have received it, but forgotten they did. On the other hand, many who did not get it may not have responded to this note either.

The two sides probably balance each other out.

Weil does not know how many of the 1.36% did not look in their bulk folders, so some mail may have been stuck there. She added two thoughts though:

a. Weil's issue did not have lots of obvious filterable words. In fact she put a typo-on-purpose into it for the word "free."

b. Weil got messages, just like we did, from readers who had not gotten issues in weeks and assumed she was no longer publishing!