Website Testing: Research and testing leads to new IBM Software Group homepage, 23% increase in demos

by David Kirkpatrick, Reporter

CHALLENGE

Last week's B2B newsletter article covered website testing at IBM from an overall corporate marketing perspective. This week we revisit Big Blue’s methodology for homepage redesign, with a case study that examines how IBM implemented a digital brand evolution one step deeper than its primary brand homepage.

Both case studies involve the same basic testing process and protocol at IBM, and both tested areas have a global reach with homepages for many international markets. But, as you may experience in any company that has differences between corporate marketing and marketing within a group or division, the similarities stop there.

Last week’s case study looked at testing on the homepage for IBM's "Smarter Planet" initiative. The goal of that online property was deeper engagement with Web visitors, by increasing clickthroughs to in-depth content on industry-specific topics, and the overall business goals were messaging and branding.

This case study involves IBM's Software Group. The goal for this test was lead generation and engagement. Because of the lead gen aspect, instead of looking to create strong top-down branding to deploy across 60 international, country-specific websites, the idea was test for, and create, a new homepage template that would have enough flexibility to handle lead generation at each unique local market.

Marjorie Madfis, Digital Marketing Strategy and Worldwide Editor, IBM Software homepage, explained that IBM Software Group was looking for a website design that could provide a template every country site could use for top-of-the-funnel marketing activities and could accommodate different updating frequencies and content offerings based on what worked best in each local market.

Read on to learn more about the preparatory steps of collecting internal and external information that helped inform the new page design, the actual tests IBM's Software Group ran, and details on where IBM is going after finding a new homepage for the division. Madfis will present an in-depth look at this test at the upcoming MarketingSherpa B2B Summit 2011.


CAMPAIGN

IBM's Software Group has homepages in 60 countries around the world, and about 40 of those Web properties are in languages other than English.

Because the Software Group homepages are lead generation sites, an overarching goal of this effort was to find a winning design that was based, in part, on input from the local market homepage owners to ensure the new basic site template had the flexibility to address the unique lead gen needs in every country.


Step #1. Define business objectives

The Software Group had traditionally gone to market with its five main brand organizations -- Lotus, Tivoli, Rational, Websphere, and Information Management -- some homegrown and some acquired. But, Madfis said, market research found that, "the IBM Software name was much more recognized and had a greater impact in the marketplace, particularly in growth market areas."

With this research information in hand, the group interviewed eight IBM executives in the Software Group and in Corporate Digital Marketing Strategy to better understand the business priorities.

Instead of marketing the Software Group through individual brands, the high-level internal interviews resulted in an objective to market the division’s overall capabilities under the IBM brand.

"We undertook a brand evolution," said Madfis, "An approach to positioning IBM around our capabilities and not by any traditional (brand) names or the collections of products in our portfolio."

With the new objective defined, the group began looking at the customer Web experience to make sure IBM's capabilities were being expressed through

o Information architecture
o Messaging
o Design
o Content
o Call-to-action

After determining the best way to show the group's capabilities was to share IBM's expertise with prospects and customers, and to enable those groups to connect with each other, the group decided redesigning and testing the homepage would be the best way fully meet the business objective and drive an increase in lead generation.


Step #2. Understand internal user requirements and external user goals

Because each of the Software Group teams in 60 global countries don't have the same resources the worldwide team can draw on, it was important to learn

o Their top priorities
o Their requirements
o Their challenges

To gain this information the team conducted online surveys with IBM Software homepage owners across the division's geographic reach.

- External visitor goals and customer requirements

The business objective of this effort was to change to presentation of the IBM Software Group from five different brand offerings to a simplified single brand of "IBM Software" with an emphasis on the different capabilities of the division, with the measurable goal being an increase in generated leads.

To accomplish this, Madfis stated the team wanted to understand their visitors and customers better.

In order to identify country-specific IBM Software homepage visitor motivation, an intercept pop question link was added to eight countries' Software homepages that polled visitors about their current goals for the page visit.

Learning more about Web prospect and customer requirements and priorities, Madfis said, "We did an online survey with 291 prospects or customers."

These surveys assessed site use and visitor requirements, and included feedback on the current IBM Software homepage.


Step #3. Evaluate results to design and refine the new website

Taking into account the newly gathered internal (country Software homepage owners) and external (site visitors, prospects and customers) user information, website designs were created based on survey input.

Before fully testing the new designs against the existing division homepage, IBM conducted 13 one-to-one 90 minute usability study phone sessions, with customers in the United States and the United Kingdom.

One major result of these one-on-one surveys was that an area of the page featuring IBM "experts" website users could connect with was a "huge winner" with survey participants. The expert area

o Instilled feelings of confidence, trust, empowerment and partnership with IBM

o Was considered a competitive advantage and differentiator for IBM

"We then made adjustments to those [new] designs based on that input," said Madfis.


Step #4. Test the new website designs against the existing site

The design process ended up with two new designs to test against the current homepage, which served as the control. The two new designs were variations on a visually similar theme, and were radical departures from the control page.

The metrics Madfis' team looked for in the test included improvements on clickthrough rates and increased downloads and offer registrations.
Because of a high level of traffic to the tested pages, statistically significant results were quickly generated to determine the winning page.

- Control

The key feature of the control was a centralized box featuring three tabs:

o Find Products
o Highlights
o Fast ROI

The "Find Products" tab served as the default for the page.

Other page features included a left column with navigation links, and content boxes of various sizes surrounding the tabs box with content and links to more information or calls-to-action, such as "Follow IBM Software news on Twitter."

The sub-head box included a graphical element

- Treatment #1

Treatment #1 featured four tabs:

o Overview (note: also served as the default tab)
o Find Products
o Solutions by Industry
o Solutions by IT Issue

The tabs were located below a horizontal bar with links to resources, news and information; and a horizontal headline bar with very sparse headline content and an image of a man with his arms crossed behind his head.

Below the tab bar was the "meet the experts" section. The entire layout of this treatment was a one page-wide single column with each new information section found directly below the previous.

- Treatment #2

Treatment #2 featured a similar look and feel to treatment #1 with the page layout featuring the same page-wide single column.

This treatment opened with the headline bar with more content than Treatment A and the same image of the man.

Below the headline bar was a "Find Products" bar featuring four tabs:

o by Need
o by Category
o by Product Line
o A-Z Index

The next horizontal bar was "Featured content" with "Meet our featured experts" below the content bar. In this treatment the bar featuring links to products, news, downloads, services and support was found at the bottom of the page.


Step #5. Deploy winning homepage and continue the effort

Treatment #1 ended up the winning design version, but it was not the actual new IBM Software homepage that was deployed Because IBM included individual winning elements from the non-winning treatment in the new homepage.

"We were looking for many different interactions on the page," explained Madfis. "There was no single factor, but (actually) a combination of interactions that produced a winner. The winning design wasn't better in every factor, but it was in most."

With a settled design for the division's homepage, there was a process of "globalization" so all 60 countries could use a version of the new design based on the content that local market develops and the frequency they refresh that content.

Madfis said she refreshes the worldwide (also the U.S. version) Software Group homepage content weekly. Other countries refresh every other week, monthly, or even on an ad hoc basis for a particular event or a critical announcement.

"We developed a template that was flexible enough for our countries, depending on the resources and scales that they have available, to still be able to benefit from the design strategy that we created," stated Madfis.

This flexibility was important because each local market needed the ability to deploy types of content that would locally drive lead generation within the design of the new homepage template. Something that might work in the United States or United Kingdom markets might not work in Croatia or Poland.

- Remarket winning ideas

This is one area where IBM flexed its marketing budget muscles. Effective testing can be conducted with a relatively low level of expenditure, but huge television ad spends are a different matter altogether. The takeaway is when you gain a learning from testing a marketing effort, use that new knowledge in other areas.

When IBM found that the "expert area" of the homepage design created such a positive reaction, it began a television campaign called, "I am an IBMer," to promote its experts, and even tied the entire effort to its "Smarter Planet" initiative.

- Continue testing

Just because you find a winning webpage doesn’t mean it's time to stop the testing and optimization cycle. For this effort, because the Software Group did so much internal and external surveying to solidify business objectives, find out more about the needs of its country homepage owners, and learn what resonates with site visitors and customers, it can now continue testing to refine the new design template without going back through the first three steps of this case study.
RESULTS

Treatment #1 was the overall winner of the three-way test.

- Product page views (a key lead generation metric for this test):

o The control had a clickthrough rate of 4.38%
o Treatment #1 had a clickthrough rate of 4.98%
o Treatment #2 had a clickthrough rate of 4.29%

Treatment #1 beat both the control and Treatment #2, and produced a 16.12% lift over Treatment #2.

- Clicks on "expert" modules

Treatment #1 had a 95.6% lift over Treatment #2

- Clicks on interactive "community" channels

Treatment #1 beat Treatment #2 by 89.4%

- Downloads of trials and demos (an indicator of product interest and lead qualifier)

Treatment #1 beat the control by 23.99%

One interesting learning from the test was on the physical layout of tabs on the page. Treatment #1 featured tabs that were separated by a small amount of space. The control featured tabs that touched each other.

Treatment #1 had a 468% lift in clicks on content in one of the tabs over the control. This result has actually influenced overall corporate design standards at IBM.

Finding out the "expert" area resonated so positively with website visitors was another key result.

"I think it was really a great moment where what IBM wanted to do, and what our audience wanted from us, actually came together," said Madfis.
She added the big takeaway from this homepage redesign process is "data speaks."

Having quantifiable results also helps in convincing the 60 international Software Group homepage owners to accept the changes, and provides confidence the effort involved will yield results.

"I think the takeaway is that the data speaks first," said Madfis. "Now that we have methodologies to get everybody's input (internal and external) and then do testing, we can ultimately better meet our business objectives."

Marjorie Madfis will present an in-depth look at this test at the upcoming MarketingSherpa B2B Summit in Boston, Sept. 26-27.


Useful links related to this article

CREATIVE SAMPLES:
1. User survey highlights
2. Test control
3. Test treatment one
4. Test treatment two

IBM Software Group

Website Testing: IBM's navigation elements test leads to 128% increase in clickthroughs

Members Library -- Website Redesign: Customer surveys and testing help increase conversion by 21.6%

How to Plan Landing Page Tests: 6 Steps to Guide Your Process

Quick Lift Ideas: How switching a headline with a sub-headline could increase conversion (+6 more ideas)

Website Redesign: Wondering what to test? Just ask your customers

Homepage Optimization: No single metric will do